WEBVTT 00:00.260 --> 00:02.600 - This is gonna be like a WeWork-style operation 00:02.600 --> 00:04.480 with all sorts of really interesting 00:04.480 --> 00:06.480 technology groups here in this building. 00:07.780 --> 00:09.403 So as a bit of background, 00:10.250 --> 00:13.570 we collectively are a subset of 00:13.570 --> 00:15.650 the Defense Innovation Board members 00:15.650 --> 00:20.420 and we work for free as special government employees 00:20.420 --> 00:24.170 for the DoD to try to help drive innovation, 00:24.170 --> 00:25.820 drive change, so forth and so on. 00:26.840 --> 00:29.490 The way our structure works is we have, 00:29.490 --> 00:31.310 as part of being a FACA Committee, 00:31.310 --> 00:32.143 we have public meetings. 00:32.143 --> 00:36.080 This is a public meeting, which we are very happy to do, 00:36.080 --> 00:39.010 and it's literally an official meeting of our group 00:39.010 --> 00:39.843 to talk about things, 00:39.843 --> 00:43.040 so you can watch the way we make our recommendations 00:43.040 --> 00:45.293 and help us shape what we wanna do. 00:48.660 --> 00:50.850 Austin has a fantastic innovation community, 00:50.850 --> 00:52.630 that's not a new fact for you all. 00:52.630 --> 00:54.980 And I wanna make sure I thank the hosts 00:54.980 --> 00:57.253 at the Capital Factory for this thing. 00:58.460 --> 00:59.440 You know who I am. 00:59.440 --> 01:01.110 Kurt, where's Kurt? 01:01.110 --> 01:03.730 Kurt is nowadays the Executive Vice President 01:03.730 --> 01:05.940 of Corporate Strategy and has run many 01:05.940 --> 01:09.570 of the products and services that you use from Microsoft. 01:09.570 --> 01:13.020 Michael McQuade was a big time aerospace executive 01:13.020 --> 01:14.787 and is now Vice President for Research 01:14.787 --> 01:17.410 at Carnegie Mellon University. 01:17.410 --> 01:20.870 Richard Murray is a Professor at Caltech of Control 01:20.870 --> 01:22.983 and Dynamical Systems and Bioengineering. 01:22.983 --> 01:27.190 and Mark Sirangelo is now an entrepreneur-in-residence 01:27.190 --> 01:28.023 at the University of Colorado, 01:28.023 --> 01:32.600 and was head of Space Systems at Sierra Nevada Corporation. 01:32.600 --> 01:34.470 So he's the person who understands the space 01:34.470 --> 01:36.573 and space activities best among our group. 01:37.540 --> 01:39.560 On our line, we have a number of other people. 01:39.560 --> 01:41.610 We have Eric Lander, 01:41.610 --> 01:43.863 who's the President of the Broad Institute, 01:44.777 --> 01:48.230 and one of the foremost biologists in the country. 01:48.230 --> 01:52.670 Jen Pahlka, who is the Founder and Executive Director 01:52.670 --> 01:54.400 of Code for America and the former 01:54.400 --> 01:57.180 Deputy CTO of the US Government. 01:57.180 --> 02:00.274 Dr. Daniela Rus, who is the Director of Computer Science 02:00.274 --> 02:05.274 and AI and a Professor at MIT and a foremost AI expert. 02:06.968 --> 02:10.570 Neil Degrasse Tyson, he's well-known to this audience. 02:10.570 --> 02:14.130 An Astrophysicist at the American Museum of Natural History, 02:14.130 --> 02:18.080 in New York City and an astrophysicist by training as well. 02:18.080 --> 02:21.080 Missy Cummings is the Director of Humans and Autonomy Lab, 02:21.080 --> 02:24.360 and a Professor of Engineering at Duke University, 02:24.360 --> 02:28.040 and was also a fighter pilot in the Navy 02:28.040 --> 02:30.663 before she turned to university work. 02:32.220 --> 02:34.450 Marnie Levine is the VP of Global Partnerships 02:34.450 --> 02:35.900 and business development at Facebook, 02:35.900 --> 02:38.830 and a former Chief Operating Officer of Instagram. 02:38.830 --> 02:40.990 And Danny Hillis, who's a Computer Theorist 02:40.990 --> 02:42.660 and co-founder of Applied Inventions 02:42.660 --> 02:45.780 and inventor of many of the things that we use today. 02:45.780 --> 02:49.560 So the way this meeting works 02:49.560 --> 02:52.903 is we're gonna have a set of subcommittee discussions, 02:52.903 --> 02:56.140 where we wanna talk about two recommendations. 02:56.140 --> 02:59.550 A report on something called biology and data digitization 02:59.550 --> 03:01.520 with something called the Joint Pathology Center, 03:01.520 --> 03:03.740 it's something you've been working on for a long time. 03:03.740 --> 03:07.410 And then the Congress has mandated an assessment 03:07.410 --> 03:09.680 of software-related talent programs 03:09.680 --> 03:13.260 and a response to congressional testing through the NDAA. 03:13.260 --> 03:16.510 When we started, our objective was to have no reports 03:16.510 --> 03:18.880 at all 'cause we figured nobody would read them. 03:18.880 --> 03:20.860 And the government has recently been mandating 03:20.860 --> 03:23.580 that we actually produce reports so that they can read them, 03:23.580 --> 03:25.250 which I guess is a good sign. 03:25.250 --> 03:27.833 So we're gonna go through those with you. 03:29.680 --> 03:34.410 We're going to talk about workforce behaviors and culture. 03:34.410 --> 03:36.180 This has been an area that's been a high priority 03:36.180 --> 03:37.027 for us for a while. 03:37.027 --> 03:40.014 And we're going to discuss a recommendation 03:40.014 --> 03:42.170 of the appointment of a new digital engineering 03:42.170 --> 03:44.180 recruitment and management officer, 03:44.180 --> 03:46.150 that's which is again part of the NDAA. 03:46.150 --> 03:47.940 And if you're not familiar with the NDAA, 03:47.940 --> 03:50.830 most of you are, it's the legislation that's passed 03:50.830 --> 03:53.757 every year that authorizes the Defense Department 03:53.757 --> 03:55.420 and the defense activities of the nation 03:55.420 --> 03:57.963 and we are subject to that law obviously. 03:59.770 --> 04:03.040 We'll give a report on how we're doing 04:03.040 --> 04:04.710 implementing some of the DIB. 04:04.710 --> 04:07.030 And then we're going to go to public comment. 04:07.030 --> 04:07.890 Colleen. 04:07.890 --> 04:08.723 - That sounds great. 04:08.723 --> 04:10.144 Thank you, Eric. 04:10.144 --> 04:11.730 I think first we'll start off, 04:11.730 --> 04:14.380 we have a representative from Army Futures Command here, 04:14.380 --> 04:16.430 who's just gonna give some brief remarks. 04:19.210 --> 04:20.350 - Good afternoon, everybody. 04:20.350 --> 04:22.380 Whoa, tall person (laughing). 04:22.380 --> 04:25.110 I'm good, I could stand on my toes (laughing). 04:25.110 --> 04:25.970 Good afternoon, everybody. 04:25.970 --> 04:26.803 I'm Sue Goodyear. 04:26.803 --> 04:28.800 I'm General Murray's Executive Deputy, 04:28.800 --> 04:31.240 and so on behalf of General Murray 04:31.240 --> 04:33.510 and behalf of Army Futures Command, 04:33.510 --> 04:36.150 and on behalf of the City of Austin, we wanna welcome you. 04:36.150 --> 04:40.720 This is a really good innovation ecosystem here. 04:40.720 --> 04:43.730 That's one of the reasons why Army Futures Command is here. 04:43.730 --> 04:46.140 And I think this audience and everyone online 04:46.140 --> 04:48.770 and everybody out there would like to welcome you 04:48.770 --> 04:50.950 and wanna look forward to taking advantage 04:50.950 --> 04:52.630 of what you're gonna talk about 04:52.630 --> 04:54.840 and then take advantage of the public forum. 04:54.840 --> 04:55.940 So on behalf of General Murray, 04:55.940 --> 04:58.303 I'd like to welcome you all here, and thank you. 05:01.650 --> 05:04.310 - Great, I'm gonna do some FACA throat-clearing language 05:04.310 --> 05:07.430 up front and then we'll get on with the show here. 05:07.430 --> 05:09.880 So, welcome to everyone here in Austin, 05:09.880 --> 05:12.090 the Capital Factory for hosting us. 05:12.090 --> 05:13.230 My name is Colleen Laughlin 05:13.230 --> 05:15.070 and I am the Executive Director 05:15.070 --> 05:16.861 and designated federal official 05:16.861 --> 05:19.250 for the Defense Innovation Board. 05:19.250 --> 05:20.420 If you have not done so already, 05:20.420 --> 05:23.740 please go ahead and silence your electronic devices. 05:23.740 --> 05:27.020 Today's meeting is being recorded and livestreamed 05:27.020 --> 05:28.880 to allow members of the public to attend 05:28.880 --> 05:30.950 today's meeting virtually. 05:30.950 --> 05:34.130 I wanna do a quick shout out to the Defense Media Activity 05:34.130 --> 05:37.190 for providing your expert services and support today, 05:37.190 --> 05:38.400 so thank you. 05:38.400 --> 05:41.990 And welcome to all of our in-person and virtual attendees. 05:41.990 --> 05:44.910 The board will now convene its public session. 05:44.910 --> 05:46.030 As we begin this meeting, 05:46.030 --> 05:48.790 allow me to share a few procedural remarks. 05:48.790 --> 05:51.540 As Eric noted, the board is a discretionary 05:51.540 --> 05:53.150 independent advisory board, 05:53.150 --> 05:56.010 operated under the Federal Advisory Committee Act 05:56.010 --> 05:58.120 and The Government in Sunshine Act. 05:58.120 --> 06:01.400 Today's meeting was announced in the Federal Register Notice 06:01.400 --> 06:04.850 posted on February 17, 2020. 06:04.850 --> 06:07.140 The public was invited to submit written comments 06:07.140 --> 06:09.240 for the board members to consider. 06:09.240 --> 06:12.530 14 comments were received in advance of today's meeting, 06:12.530 --> 06:15.870 and will be posted online with the minutes of this meeting. 06:15.870 --> 06:17.270 There have been no significant changes 06:17.270 --> 06:18.580 to the meeting's agenda as posted 06:18.580 --> 06:20.720 in the Federal Register notice. 06:20.720 --> 06:23.240 Depending on time constraints, there may be an opportunity 06:23.240 --> 06:25.850 for members of the public attending here today 06:25.850 --> 06:27.970 to provide verbal comments to the board 06:27.970 --> 06:29.350 at the end of the meeting. 06:29.350 --> 06:31.950 Just a quick reminder, when you do your public comments, 06:31.950 --> 06:33.770 the board is not able to respond, 06:33.770 --> 06:35.720 but we will, we of course welcome 06:35.720 --> 06:39.220 and would love to hear all your remarks. 06:39.220 --> 06:42.460 All of our papers for today's meeting can be found 06:42.460 --> 06:46.980 on the DIB's website, which is innovation.defense.gov. 06:46.980 --> 06:49.450 We encourage you to follow along. 06:49.450 --> 06:51.387 You can follow us on our Twitter handle 06:51.387 --> 06:55.477 @innovationboard as another way to follow the conversation. 06:55.477 --> 06:59.013 And with that, we will go ahead and kick off the meeting. 06:59.954 --> 07:01.339 - Do you want me to say a few things about 07:01.339 --> 07:03.070 where I think we are? 07:03.070 --> 07:04.920 - Yes I think that would be wonderful. 07:04.920 --> 07:07.170 - We've had a series of meetings with Secretary Esper, 07:07.170 --> 07:10.090 who is a very impressive public servant. 07:10.090 --> 07:13.301 And he continues to emphasize AI talent management, 07:13.301 --> 07:17.300 and then reducing the foreign influence on the US 07:17.300 --> 07:19.933 tech sector as themes that we should be focused on. 07:20.900 --> 07:22.822 We work very closely with Secretary Griffin, 07:22.822 --> 07:27.800 and he's focused us on biotechnology, commercial space, 07:27.800 --> 07:29.640 and trying to strengthen our ties 07:29.640 --> 07:32.540 to the National Security Innovation Base. 07:32.540 --> 07:36.470 Secretary Lord is very, very focused with us 07:36.470 --> 07:39.414 on the software study and the swap study. 07:39.414 --> 07:42.340 This is a good forum to talk about that. 07:42.340 --> 07:45.820 And she's pushing for variable report reforms 07:45.820 --> 07:47.268 on acquisition. 07:47.268 --> 07:49.360 She's also doing a very interesting initiative 07:49.360 --> 07:52.970 around trusted capital, to try to sort of streamline 07:52.970 --> 07:55.908 the relationship between the DoD and startups 07:55.908 --> 07:58.410 and more mature companies that are building 07:58.410 --> 08:00.453 technologies that DoD needs. 08:02.600 --> 08:05.220 We're very, very proud to be a small component 08:05.220 --> 08:09.673 of the joint AI center that's run through the OSD CIO. 08:10.520 --> 08:15.460 And we're beyond excited that the DoD announced 08:15.460 --> 08:18.220 AI ethical principles which we heavily participated 08:18.220 --> 08:19.610 in a week ago. 08:19.610 --> 08:22.593 If you haven't seen them, please read them. 08:22.593 --> 08:24.658 I think they're going to be a... 08:24.658 --> 08:26.502 My opinion, you guys can say, 08:26.502 --> 08:28.720 I think they're going to be the index that all 08:28.720 --> 08:30.140 the other countries are now gonna judge. 08:30.140 --> 08:35.140 I think we really, as a nation, made a strong component. 08:35.800 --> 08:39.330 So in any case, General Janet Shanahan will be here 08:39.330 --> 08:41.180 next week as part of that conference. 08:42.300 --> 08:45.540 So there's other things going on, 08:45.540 --> 08:49.230 the FY21 budget increases federal research by 6%, 08:49.230 --> 08:52.083 including increases specifically for AI. 08:54.790 --> 08:56.110 I'm the chairman of something called 08:56.110 --> 08:57.766 the National Security Commission on AI. 08:57.766 --> 09:00.599 We released an interim report in November. 09:00.599 --> 09:02.850 These folks work very closely with me 09:02.850 --> 09:04.670 to help make that a success. 09:04.670 --> 09:06.410 I won't summarize it, but you can read it. 09:06.410 --> 09:10.094 It's been very well-received on a bipartisan basis. 09:10.094 --> 09:13.740 We were going to hear some recommendations 09:13.740 --> 09:15.780 from some new topics that you'll hear 09:15.780 --> 09:17.730 that were initially introduced at Georgetown 09:17.730 --> 09:19.100 in our last meeting. 09:19.100 --> 09:21.280 And I guess I'd say to my colleagues, 09:21.280 --> 09:22.806 thank you guys for coming. 09:22.806 --> 09:25.250 This is just fantastic to be here. 09:25.250 --> 09:26.671 So Colleen, how do you wanna do this? 09:26.671 --> 09:27.592 You want to-- 09:27.592 --> 09:28.793 - [Colleen] So I think right now we'll jump right 09:28.793 --> 09:31.566 into the S & T update, starting off with Dr. McQuade there. 09:31.566 --> 09:32.399 - Okay. 09:33.953 --> 09:37.630 - Thank you and thanks again for everybody for being here. 09:37.630 --> 09:39.900 One of our committees in the Defense Innovation Board 09:39.900 --> 09:41.850 is the S & T subcommittee. 09:41.850 --> 09:43.840 My colleague Richard Murray and I co-chair that, 09:43.840 --> 09:45.381 and we're actually gonna split the duties here 09:45.381 --> 09:47.180 for two subjects. 09:47.180 --> 09:48.934 I'm gonna lead a discussion with the board 09:48.934 --> 09:51.860 on some recommendations we're making on 09:51.860 --> 09:54.293 the Joint Pathology Center Repository, 09:54.293 --> 09:56.960 enhancements to the digitization of that, 09:56.960 --> 10:01.110 and then Richard's gonna talk about the DIB's 10:01.110 --> 10:03.390 assessment on the progress that the department 10:03.390 --> 10:06.027 is making relative to software development 10:06.027 --> 10:08.115 and software acquisition, testing and management plans. 10:08.115 --> 10:09.581 So both of those on the agenda for today. 10:09.581 --> 10:14.340 So let me jump in on the Joint Pathology Center. 10:14.340 --> 10:15.880 And before I do that, I wanna recognize 10:15.880 --> 10:17.470 two people in the audience. 10:17.470 --> 10:20.540 It's always good to have the people who have the task 10:20.540 --> 10:22.620 of implementing whatever it is that you recommend. 10:22.620 --> 10:26.220 So Colonel Joe Monsour is here with us, 10:26.220 --> 10:28.210 who's the Director of the Joint Pathology Center, 10:28.210 --> 10:29.328 and Dr. Frank Rendus, who's the Director 10:29.328 --> 10:31.800 of R & D JPC. Oaxaca 10:31.800 --> 10:34.120 The gentlemen are both right here, you can wave your hands. 10:34.120 --> 10:35.183 Thank you very much. 10:36.313 --> 10:39.780 On the phone is Eric Lander and I am counting 10:39.780 --> 10:43.210 on Eric jumping in at any moment when I either (laughing) 10:43.210 --> 10:44.580 misspeak or don't speak enough. 10:44.580 --> 10:46.573 So Eric, thanks in advance for that. 10:47.550 --> 10:48.383 So our first recommendation today-- 10:48.383 --> 10:49.440 - [Eric] Nice to be here. 10:49.440 --> 10:50.368 - Thank you, 10:50.368 --> 10:53.920 is a project that has a significant impact 10:53.920 --> 10:55.360 for the global community, 10:55.360 --> 10:57.090 as well as for the Department of Defense. 10:57.090 --> 11:00.185 It's focused on a very large volume of medical data 11:00.185 --> 11:03.080 that we have and a way to better understand 11:03.080 --> 11:06.122 and treat infectious diseases, cancers and other 11:06.122 --> 11:09.470 places where large scale data analysis 11:09.470 --> 11:12.583 can provide value to the public health community. 11:12.583 --> 11:15.430 I guess I don't wanna be the first person 11:15.430 --> 11:18.327 but I'll simply say the word coronavirus and then move on. 11:18.327 --> 11:22.010 So next slide please. (attendees chuckling) 11:22.010 --> 11:24.980 So we jump into specific recommendations. 11:24.980 --> 11:27.090 There's some things that's important to keep in context. 11:27.090 --> 11:28.930 So when we say medical data, 11:28.930 --> 11:31.350 what are we talking about at JPC? 11:31.350 --> 11:33.730 We're talking about biopsies, so tissue samples 11:33.730 --> 11:37.820 taken from patients with certain types of cancer or disease. 11:37.820 --> 11:42.070 Those samples have to be, want to be collected 11:42.070 --> 11:45.160 and associated paired with medical records 11:45.160 --> 11:47.284 of the patients who gave those tissue samples 11:47.284 --> 11:50.130 for the medical data to be useful. 11:50.130 --> 11:52.380 Large volumes of data from discrete disease 11:52.380 --> 11:55.700 and cancers are sought but rarely available. 11:55.700 --> 11:57.300 And I'll show you some numbers in a moment 11:57.300 --> 12:00.600 about the amazing value of this repository. 12:00.600 --> 12:03.270 There are lots of barriers to accumulating 12:03.270 --> 12:06.490 and accessing a meaningful amount of data. 12:06.490 --> 12:08.910 Generally, there are few centralized resources 12:08.910 --> 12:10.450 of this kind of data. 12:10.450 --> 12:13.910 And major organizations typically dispose of the samples 12:13.910 --> 12:17.190 of these kinda data after a time that's either required 12:17.190 --> 12:21.170 by volume limitations, by legal or statutory 12:21.170 --> 12:22.980 limitations, et cetera. 12:22.980 --> 12:25.590 If we have access to that data, the medical community 12:25.590 --> 12:28.770 is better able to identify and understand current 12:28.770 --> 12:30.450 and future disease. 12:30.450 --> 12:32.970 It enables the global community to better respond 12:32.970 --> 12:36.210 to future global health crises. 12:36.210 --> 12:38.400 And it promotes a better sense of health and well being, 12:38.400 --> 12:40.800 both in the DoD and in the general population. 12:40.800 --> 12:44.020 So we see this as an amazingly valuable resource. 12:44.020 --> 12:47.210 And then coupled with artificial intelligence, 12:47.210 --> 12:49.770 machine learning, things that the DIB has talked about 12:49.770 --> 12:52.110 for a very long time, only serves 12:52.110 --> 12:55.180 to amplify the value of that repository. 12:55.180 --> 12:58.180 So there are specific benefits for DoD, 12:58.180 --> 13:00.770 not just for the global health community. 13:00.770 --> 13:03.410 Number one, improving the service member's health 13:03.410 --> 13:07.180 and readiness, providing a resource to establish 13:07.180 --> 13:09.510 and deepen partnerships with the private sector, 13:09.510 --> 13:11.930 the academic sector, or quite honestly, 13:11.930 --> 13:14.050 with pure adversaries. 13:14.050 --> 13:17.210 Provide the massive data sets that allow us to explore 13:17.210 --> 13:21.710 how AI and ML can really provide value in healthcare. 13:21.710 --> 13:25.640 It's a subject that is obviously catalytic for all of us. 13:25.640 --> 13:28.467 And ultimately lowering the cost of health care for the DoD, 13:28.467 --> 13:31.023 which is billions of dollars in the budget. 13:33.540 --> 13:35.051 Next slide. 13:35.051 --> 13:38.740 So there's the challenge, how do we improve healthcare? 13:38.740 --> 13:40.280 And the main topic for the discussion 13:40.280 --> 13:43.620 then is that the repository at the DoD's 13:43.620 --> 13:45.700 Joint Pathology Center. 13:45.700 --> 13:48.960 JPC is organized as part of the defense health agency, 13:48.960 --> 13:51.040 it's overseen by the Office of the Under Secretary 13:51.040 --> 13:54.298 for Personnel and Readiness, and it holds the largest 13:54.298 --> 13:58.570 repository of disease and cancer-related data in the world. 13:58.570 --> 14:02.400 So just to give you scale, there are 55 million plus 14:02.400 --> 14:04.780 glass pathology slides. 14:04.780 --> 14:08.888 There are 31 million paraffin embedded tissue blocks. 14:08.888 --> 14:12.127 There are over 500,000 wet tissue samples, 14:12.127 --> 14:15.500 data that stretches back over 100 years. 14:15.500 --> 14:18.710 So if you think about the value of the data 14:18.710 --> 14:22.170 that we ingest into data analytic studies 14:22.170 --> 14:23.760 and into machine learning studies, 14:23.760 --> 14:27.320 this is an amazing resource that's available to us. 14:27.320 --> 14:29.650 It also, as I said, holds tremendous value 14:29.650 --> 14:32.230 for the department's digital modernization. 14:32.230 --> 14:34.770 So just the pure action of how do we deal 14:34.770 --> 14:36.200 with data sets this large? 14:36.200 --> 14:39.923 How do we extract value from our algorithmic capabilities? 14:41.110 --> 14:42.670 But of course, it's not hard. 14:42.670 --> 14:44.220 I'm sorry, it's not easy, sorry. 14:44.220 --> 14:45.490 It is hard. (audience laughing) 14:45.490 --> 14:47.290 It's not hard, it's not hard at all. 14:48.170 --> 14:52.080 Glass slides, paraffin blocks, 55 million. 14:52.080 --> 14:54.950 They're hard to share, they're hard to analyze, 14:54.950 --> 14:58.730 and most importantly of all, they are degrading over time. 14:58.730 --> 15:00.899 So for this reason, we are recommending 15:00.899 --> 15:03.700 a set of recommendations for the JPC 15:03.700 --> 15:08.330 to rapidly scan the full repository into digital fashion. 15:08.330 --> 15:10.010 And we have a series of recommendations 15:10.010 --> 15:12.040 on how we proceed to do that. 15:12.040 --> 15:14.560 I'm gonna pause and just give Eric a chance to jump in. 15:14.560 --> 15:17.490 Eric, you wanna provide any more background here? 15:17.490 --> 15:20.260 - [Eric] Well, I'll just say a moment or two here. 15:20.260 --> 15:22.700 I had the pleasure and the honor 15:22.700 --> 15:27.560 to tour the facility that the JPC maintains, 15:27.560 --> 15:29.820 and it is a marvel. 15:29.820 --> 15:34.820 It is a vast warehouse with just stacks and stacks, 15:34.860 --> 15:39.180 and stacks of shelves with both slides 15:39.180 --> 15:43.310 and pathology blocks that can be cut. 15:43.310 --> 15:47.460 I had a chance to look at how this operation is organized, 15:47.460 --> 15:50.610 and it is just a national treasure. 15:50.610 --> 15:55.240 Who thought to save all of this for more than a century 15:56.180 --> 15:58.370 should get a really big metal 15:58.370 --> 16:02.500 because it's not something you could ever create today, 16:02.500 --> 16:06.213 it required a lot of foresight to do that. 16:07.590 --> 16:09.850 Michael has already described many of the reasons 16:09.850 --> 16:13.328 why this is important and I just wanna add one more. 16:13.328 --> 16:15.490 This Defense Innovation Board has talked about 16:15.490 --> 16:20.490 the importance of ML and AI as a strategic capability 16:22.710 --> 16:24.400 for the DoD. 16:24.400 --> 16:26.710 Many of the problems to which that's gonna 16:26.710 --> 16:29.850 be applied are highly classified problems 16:29.850 --> 16:32.390 with highly classified data. 16:32.390 --> 16:36.330 I'm excited about the opportunity that there are a set 16:36.330 --> 16:40.220 of data, which while they involve patient confidentiality, 16:40.220 --> 16:45.210 properly done can be a magnet for the DoD to work together 16:45.210 --> 16:49.610 with the tech companies, the startup companies, 16:49.610 --> 16:53.110 the academic sector, because I think it's a great way 16:53.110 --> 16:55.780 to improve your game to have the best data set 16:55.780 --> 16:58.280 in town of defense agencies in the country 16:59.320 --> 17:02.300 that will truly be a magnet for the development 17:02.300 --> 17:05.170 of new approaches in machine learning. 17:05.170 --> 17:10.050 So both for all of the health reasons and cost reasons, 17:10.050 --> 17:13.310 and it's an amazing asset, and it's advancing ML, 17:13.310 --> 17:16.100 I also think it'll build a special kind of relationship 17:16.100 --> 17:21.100 and position the DoD in a very constructive way relative 17:22.830 --> 17:26.710 to this whole ML community, and that's a value too. 17:26.710 --> 17:29.280 So carry on, but anybody who gets a chance 17:29.280 --> 17:32.763 to take a tour of it, you'll see why I was so impressed. 17:34.050 --> 17:37.600 - Can I try to understand, who is opposed 17:37.600 --> 17:39.950 to this brilliant idea of Michael, and Eric? 17:39.950 --> 17:42.860 I mean, like this benefits humanity, 17:42.860 --> 17:44.995 it benefits the military. 17:44.995 --> 17:47.930 What are the issues? 17:47.930 --> 17:50.800 - The issues relate to do we have the technology? 17:50.800 --> 17:53.050 Do we have the capability to do this 17:53.050 --> 17:55.180 at scale and at volume? 17:55.180 --> 17:57.840 There's a minor issue of money, but it's a minor issue. 17:57.840 --> 18:01.200 So I would just say, Eric Lander made this comment before, 18:01.200 --> 18:02.880 but I'll get to repeat it. 18:02.880 --> 18:06.650 Think about this repository and think about Indiana Jones. 18:06.650 --> 18:08.650 So that's the mental picture you should have 18:08.650 --> 18:11.690 of this vast warehouse full of samples. 18:11.690 --> 18:14.930 So the first three recommendations relate to the pilots. 18:14.930 --> 18:17.120 I keep referencing back to the standard way 18:17.120 --> 18:20.620 the Defensive Innovation Board operates. 18:20.620 --> 18:23.280 Get out there, try something and improve it as you go. 18:23.280 --> 18:25.940 So our first recommendation is to execute a pilot 18:25.940 --> 18:28.400 on the slide scanning itself. 18:28.400 --> 18:30.400 So the Secretary of Defense should direct the launching 18:30.400 --> 18:32.780 of a pilot to scan an initial batch 18:32.780 --> 18:34.800 of slides within 12 months. 18:34.800 --> 18:37.230 And the goal here is to establish the tools, 18:37.230 --> 18:40.770 the methods, the processes, the analyses, 18:40.770 --> 18:44.400 to determine how to scale up beyond the first set of slides. 18:44.400 --> 18:47.140 So the goal here that we have laid out 18:47.140 --> 18:49.300 in our recommendation is to scan something 18:49.300 --> 18:51.780 between a million and two million slides, 18:51.780 --> 18:54.002 to do that over the first 12 months 18:54.002 --> 18:56.750 at an estimated cost of something 18:56.750 --> 18:59.240 into the 10 to $15 million range. 18:59.240 --> 19:01.380 And at the end of the 12 months be in a position 19:01.380 --> 19:05.400 to double the volume in a second 12-month period 19:06.580 --> 19:09.240 at reduced costs with an upgraded set 19:09.240 --> 19:10.450 of tools and capabilities. 19:10.450 --> 19:12.957 So that's recommendation number one. 19:12.957 --> 19:15.560 - Michael, I think it's also that we're gonna do 19:15.560 --> 19:17.740 a sampling from the different periods, 19:17.740 --> 19:19.370 and a sampling of the different types 19:19.370 --> 19:22.130 so that this batch is gonna represent. 19:22.130 --> 19:23.050 Does it make sense? 19:23.050 --> 19:24.820 How far back does it make sense to go, 19:24.820 --> 19:26.770 and what's the quality of the slides 19:26.770 --> 19:28.410 in those early periods as well? 19:28.410 --> 19:29.250 - Right, thank you. 19:29.250 --> 19:31.960 And again, the whole idea here is you do a pilot 19:31.960 --> 19:36.130 to find out what it is that is gonna deliver maximum value, 19:36.130 --> 19:38.140 and can allow you to set the parameters 19:38.140 --> 19:39.410 for the ongoing work. 19:39.410 --> 19:43.120 - Usually in these data extraction strategies, 19:43.120 --> 19:46.010 the more recent data is much more useful 19:46.010 --> 19:47.840 than the hundred years old data. 19:47.840 --> 19:48.990 Do you have any reason to believe 19:48.990 --> 19:50.420 that that's not true? 19:50.420 --> 19:51.253 - Eric? 19:52.780 --> 19:56.860 - [Eric] Well, one reason why the old data is useful 19:56.860 --> 19:59.200 is there are conditions and diseases 19:59.200 --> 20:00.970 we don't have today. 20:00.970 --> 20:04.320 So as a general rule, what Eric Schmidt 20:04.320 --> 20:05.820 said is exactly right. 20:05.820 --> 20:09.430 But there's the 1918 flu, there's many things. 20:09.430 --> 20:12.990 So I think there will be a set of medical problems 20:12.990 --> 20:15.770 that can only be addressed through this archive. 20:15.770 --> 20:19.943 And it's worth exploring what issues we'll encounter there. 20:20.830 --> 20:21.663 - The other thing I would say, 20:21.663 --> 20:25.370 and go back to Mark's comment is part of this first pilot 20:25.370 --> 20:28.670 needs to be to determine which slides are both valuable 20:28.670 --> 20:31.020 and deteriorating most rapidly, that may allow you 20:31.020 --> 20:32.687 to prioritize how you'd move forward. 20:32.687 --> 20:35.020 - And I would also add that there are some diseases 20:35.020 --> 20:38.760 that make a comeback very late after they were thought 20:38.760 --> 20:39.610 to have gone away. 20:39.610 --> 20:41.570 We're seeing some of that these days. 20:41.570 --> 20:44.370 So having some of that older data even if it's not perfect, 20:44.370 --> 20:46.740 it might help us if we combat certain areas 20:46.740 --> 20:48.470 like the plague and other things, 20:48.470 --> 20:50.473 which have reappeared in the world. 20:51.450 --> 20:54.360 - So we have two then ancillary recommendations 20:54.360 --> 20:56.840 to go along with this, also to be accomplished 20:56.840 --> 20:59.540 during this first 12-month pilot. 20:59.540 --> 21:02.240 One is to launch a parallel effort to build on 21:02.240 --> 21:05.930 and improve the linkage of slides to medical records. 21:05.930 --> 21:08.740 There's a quite a wide array of the quality 21:08.740 --> 21:11.010 of the medical records, as Eric mentioned, 21:11.010 --> 21:13.720 the newer ones better than the older ones. 21:13.720 --> 21:16.220 How you access the connection between slides 21:16.220 --> 21:17.380 and medical records? 21:17.380 --> 21:19.270 That all needs to be piloted over 21:19.270 --> 21:22.000 the first 12 months to connect the sample 21:22.000 --> 21:23.523 specifically to the records, 21:24.640 --> 21:25.850 and specifically to things 21:25.850 --> 21:29.020 like additional medical visits, et cetera. 21:29.020 --> 21:30.570 The additional information that 21:30.570 --> 21:31.800 can go along with the slides. 21:31.800 --> 21:33.460 So that's the second recommendation. 21:33.460 --> 21:36.980 And the third is to initiate a pilot to allow 21:36.980 --> 21:41.277 some enhancements, molecular annotations to the slides 21:41.277 --> 21:43.500 for the physical slides and tissue blocks 21:43.500 --> 21:45.790 to see if there is even more value 21:45.790 --> 21:47.260 that can be obtained from the slides. 21:47.260 --> 21:51.295 So these are things like immunohistochemistry 21:51.295 --> 21:54.260 and insitu hybridization of the slides 21:54.260 --> 21:55.730 ahead of the scanning process. 21:55.730 --> 21:58.320 And so given that I just said those words Eric 21:58.320 --> 22:00.029 can explain what they mean. 22:00.029 --> 22:00.862 - [Eric] (laughing) Sure, so the idea is 22:00.862 --> 22:05.050 when you make a slide, 22:05.050 --> 22:06.790 you've just stained it with some kind 22:06.790 --> 22:08.840 of a dye and you see a picture. 22:08.840 --> 22:11.560 But underlying that picture is biological material 22:11.560 --> 22:15.020 like 20,000 different proteins distributed 22:15.020 --> 22:16.180 in a different cells, 22:16.180 --> 22:21.180 and 20,000 different types of RNA that are distributed. 22:22.510 --> 22:24.870 And it's possible now to reveal those 22:24.870 --> 22:27.330 by different kinds of hybridization methods. 22:27.330 --> 22:30.940 And even a few years ago, you'd do one at a time. 22:30.940 --> 22:34.230 But now there are ways to do 100 at a time, 22:34.230 --> 22:37.400 and for reading out gene expression, 22:37.400 --> 22:40.150 yeah, even thousands at a time. 22:40.150 --> 22:42.370 And we're beginning to see machine learning 22:42.370 --> 22:45.483 make it possible to take a subset of images 22:45.483 --> 22:49.130 that have been enriched and annotated in this way, 22:49.130 --> 22:52.030 and build tools that'll let you carry that over to many, 22:52.030 --> 22:53.670 many more slides. 22:53.670 --> 22:58.660 So I think this recommendation, while it's modest in scale, 22:58.660 --> 23:01.340 is to make sure that the JPC is connected 23:01.340 --> 23:03.480 to the cutting edge of this work, 23:03.480 --> 23:06.190 because I think it will add great value. 23:06.190 --> 23:10.070 So that's what's going on as the picture has lots more, 23:10.070 --> 23:11.458 the slide and lots more information 23:11.458 --> 23:14.967 than we've been able to see until the past couple years. 23:14.967 --> 23:16.548 - And Eric, this is Mark. 23:16.548 --> 23:17.381 Isn't there also the opportunity 23:17.381 --> 23:21.760 to help inform the collection of new data, 23:21.760 --> 23:23.097 new slides by trying to determine 23:23.097 --> 23:25.772 - Oh, absolutely. - how best to do that 23:25.772 --> 23:28.703 from this process of doing the sampling? 23:30.160 --> 23:33.390 - [Eric] Yes because, of course the military is continuing 23:33.390 --> 23:35.780 to collect slides and blocks. 23:35.780 --> 23:37.870 And I think engaging in a project 23:37.870 --> 23:40.370 like this will very much inform it, 23:40.370 --> 23:41.930 the whole scanning process 23:41.930 --> 23:45.150 but also the molecular adaptation process, 23:45.150 --> 23:48.040 I think will inform how this gets done in the years ahead. 23:48.040 --> 23:49.660 - Right, so slight modification, 23:49.660 --> 23:52.845 it might produce much better data as we go forward. 23:52.845 --> 23:54.243 Danny go ahead. 23:56.090 --> 23:58.010 - [Danny] Yeah, I just wanted to add one thing 23:58.010 --> 23:59.760 to what Eric said. 23:59.760 --> 24:02.240 Which is this is a pilot, I think this should be recorded 24:02.240 --> 24:05.230 as a pilot in two different senses. 24:05.230 --> 24:08.960 It's both a pilot for doing the entire repository. 24:08.960 --> 24:13.830 But also this pattern of the DoD having had the foresight 24:13.830 --> 24:17.400 to collect sample sets and associated records 24:17.400 --> 24:21.400 has actually been repeated in other places within the DoD. 24:21.400 --> 24:23.970 So there are other repositories too. 24:23.970 --> 24:28.727 And so if we can successfully show 24:29.820 --> 24:34.820 the advantages of the results of this pilot, 24:35.560 --> 24:38.820 there's other places within DoD 24:38.820 --> 24:41.600 that these same kinds of things that could be applied 24:41.600 --> 24:45.183 to other kinds of samples of other kinds of datasets. 24:45.183 --> 24:46.533 - Very good, thanks, Danny. 24:47.420 --> 24:50.440 Okay, so those are the three pilots 24:50.440 --> 24:53.140 or three elements of the first year's pilot, 24:53.140 --> 24:54.630 and then the second set 24:54.630 --> 24:56.925 of recommendations relate to what happens next. 24:56.925 --> 24:59.960 So we are clearly recommending 24:59.960 --> 25:02.770 that this entire repository be scanned. 25:02.770 --> 25:05.410 Presumably after the first set of pilots, 25:05.410 --> 25:07.520 we will have determined whether there is the value 25:07.520 --> 25:09.584 that we think is there, et cetera. 25:09.584 --> 25:11.990 But then what are the additional recommendations? 25:11.990 --> 25:15.450 One is that you have to work a process improvement plan 25:15.450 --> 25:19.110 to be able to digitize the entire repository 25:19.110 --> 25:21.010 at the most cost effective scale. 25:21.010 --> 25:23.390 So that means probably something, 25:23.390 --> 25:25.820 maybe the target is below $2 a slide, 25:25.820 --> 25:27.330 maybe it's a whole lot smaller than that 25:27.330 --> 25:28.570 to be able to get to the scope 25:28.570 --> 25:30.110 that we're talking about here. 25:30.110 --> 25:33.340 It's to fully developed the clear medical records 25:33.340 --> 25:35.860 linkage strategy, and the data strategy 25:35.860 --> 25:36.870 that goes along with it. 25:36.870 --> 25:38.920 So we were talking about exabytes of data. 25:38.920 --> 25:41.530 And you can't do this for the full repository 25:41.530 --> 25:43.500 without understanding where, when, 25:43.500 --> 25:45.600 how and what you're gonna do with the data. 25:45.600 --> 25:47.600 So improving the slide scanning 25:47.600 --> 25:49.820 by targeting specific components, 25:49.820 --> 25:52.570 by understanding if there are new scanning technologies 25:52.570 --> 25:53.900 that are coming along, 25:53.900 --> 25:56.530 and maybe even re-imagining the whole process, 25:56.530 --> 25:58.330 that is in the pilot program to something 25:58.330 --> 26:00.457 that develops over time. 26:00.457 --> 26:03.560 We need a clear data strategy, as I mentioned before, 26:03.560 --> 26:08.500 to make it easy and productive to use the data, 26:08.500 --> 26:12.190 and to make it accessible for not only AI and ML, 26:12.190 --> 26:14.300 but other forms of data analysis. 26:14.300 --> 26:17.060 And we are suggesting that the JPC needs 26:17.060 --> 26:20.790 to work directly with the Defense Digital Service on that. 26:20.790 --> 26:23.530 We need to worry about de-identifying patient data 26:23.530 --> 26:26.430 while still maintaining medical records linkages. 26:26.430 --> 26:28.100 We need to continue to look at this issue 26:28.100 --> 26:29.060 we've just been talking about, 26:29.060 --> 26:30.416 about enhancing the tissue slides 26:30.416 --> 26:32.031 and the tissue blocks. 26:32.031 --> 26:35.470 And there's an issue here that a holy grail, 26:35.470 --> 26:36.780 amongst a whole bunch of other ones, 26:36.780 --> 26:39.010 is that what we learn in the pilot 26:39.010 --> 26:41.790 from this additional molecular annotation 26:41.790 --> 26:44.330 becomes a training mechanism 26:44.330 --> 26:45.890 for the machine learning system, 26:45.890 --> 26:46.950 so that we don't actually have to do 26:46.950 --> 26:48.460 that on all the subsequent slides. 26:48.460 --> 26:50.920 So to be determined whether that can work or not. 26:50.920 --> 26:55.223 So that's this next recommendation. 26:56.390 --> 26:57.223 Sorry. 26:59.430 --> 27:01.460 The fifth recommendation then relates 27:01.460 --> 27:03.490 to developing a partnership plan. 27:03.490 --> 27:04.930 So we're gonna have all this data, 27:04.930 --> 27:06.933 how do we make maximum use of it? 27:07.920 --> 27:10.350 Either partners who assist in the process, 27:10.350 --> 27:13.020 so that's robotic experts, experts in pathology, 27:13.020 --> 27:15.950 experts in annotation, data storage, et cetera. 27:15.950 --> 27:18.550 So that needs to be completed soon. 27:18.550 --> 27:21.860 That needs to be completed within the first sort of 90 days 27:21.860 --> 27:24.080 after launching the process. 27:24.080 --> 27:25.790 We need to build the partnerships across 27:25.790 --> 27:28.570 the research environment, the commercial environment, 27:28.570 --> 27:29.800 across government sectors. 27:29.800 --> 27:32.590 To Danny's point, understanding how what we learn here 27:32.590 --> 27:35.350 may be more relevant for other repositories 27:35.350 --> 27:36.820 in the government. 27:36.820 --> 27:39.000 The JPC should reach out to organizations 27:39.000 --> 27:41.760 like Memorial Sloan Kettering Cancer Center, 27:41.760 --> 27:44.350 who has experience in conducting similar exercises 27:44.350 --> 27:45.770 on other samples. 27:45.770 --> 27:49.180 They have a repository, which is roughly half this size, 27:49.180 --> 27:51.610 and they're scanning at roughly a million slides per year. 27:51.610 --> 27:54.340 So there's some learning that can happen there. 27:54.340 --> 27:57.140 And we need to seek information and background 27:57.140 --> 27:58.300 from people who have been involved 27:58.300 --> 28:01.100 in very large digitization efforts in the past. 28:01.100 --> 28:03.810 So things like the Human Genome Project, 28:03.810 --> 28:06.170 and other forms of data collection efforts 28:06.170 --> 28:08.810 that have been underway for large scale data. 28:08.810 --> 28:10.800 And then the ultimate goal is to make 28:10.800 --> 28:14.401 a fully digitized accessible repository available 28:14.401 --> 28:16.288 within the next five to 10 years 28:16.288 --> 28:19.000 after completion of the pilot project. 28:19.000 --> 28:22.100 All physical glass slides and tissue sample blocks 28:22.100 --> 28:24.010 should then be in digital format, 28:24.010 --> 28:25.080 they should be associated 28:25.080 --> 28:27.382 with secure personal health information, 28:27.382 --> 28:31.217 and exist in a data lake that is owned by the DoD, 28:32.100 --> 28:35.360 is protected as this kind of data should be protected, 28:35.360 --> 28:39.390 but is available to people who can do the analysis 28:39.390 --> 28:43.070 and derive both DoD and community value. 28:43.070 --> 28:45.980 So the scale as we said before, at the very beginning, 28:45.980 --> 28:47.793 is 55 million slides. 28:49.350 --> 28:50.590 And time is going by. 28:50.590 --> 28:52.680 Time is going by as these slides deteriorate, 28:52.680 --> 28:55.180 and time is going by as we are not accessing 28:55.180 --> 28:56.900 the value from this repository. 28:56.900 --> 28:59.220 So those are the six recommendations. 28:59.220 --> 29:00.870 I'm gonna turn the floor back over to Eric 29:00.870 --> 29:02.277 for additional comments, 29:02.277 --> 29:03.500 and then I'll turn it over 29:03.500 --> 29:04.880 to the group for more discussion. 29:04.880 --> 29:06.500 So Eric. 29:06.500 --> 29:08.510 - [Eric] No, you did a great job on that. 29:08.510 --> 29:10.860 And so I think I have nothing more to add to that. 29:10.860 --> 29:12.380 Thank you. 29:12.380 --> 29:13.213 - Comments? 29:15.007 --> 29:17.300 - I think it'd be useful to talk about how this relates 29:17.300 --> 29:20.600 to the VA and the VA administration. 29:20.600 --> 29:22.500 Eric, do you have any comment to that? 29:24.090 --> 29:26.160 - [Eric] Sorry, I can't hear the question. 29:26.160 --> 29:28.880 - How does this project if it goes 29:28.880 --> 29:31.990 to its ultimate conclusion and is adopted fully, 29:31.990 --> 29:33.730 how does it relate to the work being done 29:33.730 --> 29:35.843 at the VA, the Veterans Administration? 29:37.600 --> 29:39.668 - [Eric] Well, I think the methodologies here 29:39.668 --> 29:41.290 are very general. 29:41.290 --> 29:45.127 I think the specific repository 29:45.127 --> 29:48.200 will have a lot of very specific problems, 29:48.200 --> 29:51.840 it's gonna have to solve for this amazing resource. 29:51.840 --> 29:56.313 So I wanna separate out the value of the DoD and VA 29:57.230 --> 29:59.590 being involved in this kind of research 29:59.590 --> 30:01.470 and this kind of medical improvement, 30:01.470 --> 30:05.270 which I think will be general across all healthcare 30:05.270 --> 30:08.320 for veterans and for active duty, 30:08.320 --> 30:12.280 and other Defense Department personnel. 30:12.280 --> 30:15.530 And then I think this resource, quite honestly, 30:15.530 --> 30:19.820 is gonna need a lot of detailed focus to make sure 30:19.820 --> 30:22.620 that these fragile things are dealt with. 30:22.620 --> 30:24.523 So I'll put them in those two buckets. 30:26.330 --> 30:27.365 - Other questions? 30:27.365 --> 30:28.447 - Yeah, I had a couple of questions, 30:28.447 --> 30:30.750 more comments than questions. 30:30.750 --> 30:33.340 The first I mean, we're in this hub of innovation, 30:33.340 --> 30:35.970 I think that we should really stress the opportunity 30:35.970 --> 30:38.000 for the commercial sector to play a role here. 30:38.000 --> 30:41.010 I know we talked about it for the sake of consultation. 30:41.010 --> 30:42.810 I think the paper does a great job, 30:42.810 --> 30:45.820 and actually I support the recommendations wholeheartedly. 30:45.820 --> 30:47.860 It does a great job of talking about some of the pilots 30:47.860 --> 30:49.691 that are being done in Sloan Kettering you mentioned. 30:49.691 --> 30:51.810 I think we can even think about whether there's a way 30:51.810 --> 30:56.700 to outsource, under the right guidance, a ton of that. 30:56.700 --> 30:59.360 The other thing is there was I think the discussion 30:59.360 --> 31:01.630 around having all the data open, 31:01.630 --> 31:04.310 again getting to the innovation community the ability to, 31:04.310 --> 31:06.730 as long as you protect the privacy, 31:06.730 --> 31:08.550 the ability to make this a data resource 31:08.550 --> 31:11.290 that others can do research on, 31:11.290 --> 31:12.700 can build AI models on. 31:12.700 --> 31:15.700 I think that requires us to think very deeply 31:15.700 --> 31:18.700 about how much of this is open to the public for innovation. 31:20.640 --> 31:25.520 - So I'm concerned that when we get into this, 31:25.520 --> 31:27.257 we'll decide that we're not gonna do 31:27.257 --> 31:30.063 all 55 million slides, we're gonna do a subset. 31:31.640 --> 31:34.590 For all sorts of costs and the usual problems, 31:34.590 --> 31:36.663 as you described, it's really hard. 31:38.770 --> 31:41.720 Can you imagine in the first phase, 31:41.720 --> 31:46.720 the highest value targets and pick those as a priority? 31:47.710 --> 31:51.193 I mean you mentioned the 1918 flu as an example. 31:52.248 --> 31:55.270 In the notes, there are other examples of strange diseases 31:55.270 --> 31:57.350 that I have never heard of. 31:57.350 --> 31:59.430 Can we get a team to say, 31:59.430 --> 32:01.760 this is the most fruitful area to start, 32:01.760 --> 32:03.290 or is that part of the plan already? 32:03.290 --> 32:04.123 - I think that's part of the plan, 32:04.123 --> 32:05.650 I'll have Eric comment in just a moment. 32:05.650 --> 32:07.630 But that's clearly part of the plan, 32:07.630 --> 32:09.543 is the definition of the pilot. 32:10.860 --> 32:13.810 Not just the pilot, but what comes after the pilot 32:13.810 --> 32:15.700 has to include what's the prioritization 32:15.700 --> 32:16.770 of how you would go after them. 32:16.770 --> 32:19.810 So, Eric I know you've spent time thinking about this 32:19.810 --> 32:20.960 from a prioritization point of view. 32:20.960 --> 32:23.240 I know there's one issue which is sort of 32:23.240 --> 32:25.670 which ones might be most valuable and are most 32:25.670 --> 32:28.300 in danger of sort of falling apart, 32:28.300 --> 32:29.740 but there are other disease-related 32:29.740 --> 32:31.030 or clinical-related reasons. 32:31.030 --> 32:32.762 So Eric, any other comments? 32:32.762 --> 32:34.670 - [Eric] There are definitely reasons 32:34.670 --> 32:35.960 why you would prioritize. 32:35.960 --> 32:38.870 But let me also speak to the value 32:38.870 --> 32:41.890 of taking a chunk of this and trying to work through. 32:41.890 --> 32:44.900 The slides reside in sets of boxes, 32:44.900 --> 32:45.733 and much like if you were trying 32:45.733 --> 32:48.963 to scan a library at a university, 32:48.963 --> 32:51.630 one could make the case that perhaps 32:51.630 --> 32:53.879 prioritizing which books to scan, 32:53.879 --> 32:55.770 it takes a lot of work 32:55.770 --> 32:58.700 to find each of those things and cherry pick it, 32:58.700 --> 33:00.720 might not actually be the best solution 33:00.720 --> 33:02.600 if you can drive down the costs. 33:02.600 --> 33:05.090 So I think there are two things to go in parallel. 33:05.090 --> 33:09.263 One, prioritize by medical importance and representation. 33:10.130 --> 33:13.220 And the other is when I walked around the facility, 33:13.220 --> 33:16.230 I didn't see anything that 33:16.230 --> 33:17.560 there was a law of physics 33:17.560 --> 33:19.960 said couldn't be made pretty cheap. 33:19.960 --> 33:22.915 So I think one of our recommendations here is to ask, 33:22.915 --> 33:27.915 could this be done well under sub dollar costs? 33:28.180 --> 33:31.670 In which case, maybe the scanning the whole thing works. 33:31.670 --> 33:34.573 So I wanna see both tracks go forward. 33:36.267 --> 33:38.470 - The amount of time and energy and cost it would take 33:38.470 --> 33:40.580 to try to put some qualitative methods 33:40.580 --> 33:45.580 on this entire 55 million would probably be better used 33:45.720 --> 33:47.837 just scanning the entire set. 33:47.837 --> 33:49.080 - I see your point. 33:49.080 --> 33:49.983 Yeah, understood. 33:51.060 --> 33:51.893 - Other comments? 33:51.893 --> 33:53.703 - [Eric] I think that horse might win. 33:55.290 --> 33:57.473 - Anybody else on the phone with comments? 33:59.430 --> 34:00.570 - [Daniela] So this is Daniela. 34:00.570 --> 34:03.340 I would like to make just a few comments 34:03.340 --> 34:06.599 about how the latest advancements 34:06.599 --> 34:09.690 in dealing with data at scale 34:09.690 --> 34:12.180 and considering data governance 34:12.180 --> 34:16.190 and data sharing strategies can contribute 34:16.190 --> 34:18.107 and help this project along. 34:18.107 --> 34:22.430 So first, we have significant advancements 34:22.430 --> 34:27.430 in database systems that are coupled with policy awareness. 34:28.730 --> 34:31.960 And this is leading to new models for systems 34:31.960 --> 34:35.000 that can help govern data at scale, 34:35.000 --> 34:38.210 and enable engagements with policymakers 34:38.210 --> 34:40.730 about what are the most effective, 34:40.730 --> 34:42.543 most appropriate approaches. 34:43.550 --> 34:45.860 Second, we have new advances 34:45.860 --> 34:49.290 on privacy preserving trustworthy 34:49.290 --> 34:52.970 machine learning that can potentially 34:52.970 --> 34:55.900 meet global legal requirements, 34:55.900 --> 34:59.580 and that can provide explanation 34:59.580 --> 35:03.490 and bias assessment in how they're trained, 35:03.490 --> 35:08.490 and this can be utilized to great benefit for this project. 35:10.200 --> 35:12.320 And certainly I think it's important 35:12.320 --> 35:16.750 to consider a data flow across different systems, 35:16.750 --> 35:20.885 because data portability has clear benefits, 35:20.885 --> 35:24.173 but also significant impact on privacy. 35:27.690 --> 35:29.900 - I think the other thing related to that last question, 35:29.900 --> 35:31.880 I think there's a lot of data architecture 35:31.880 --> 35:34.000 and software architecture issues here. 35:34.000 --> 35:36.390 The whole notion of the amount of data that exists 35:36.390 --> 35:39.350 in these 80 million plus slides means you're not actually 35:39.350 --> 35:41.110 gonna be able to do a lot of transfer of the data 35:41.110 --> 35:42.470 from one place to the other. 35:42.470 --> 35:44.640 So you've gotta think about what's your platform look like 35:44.640 --> 35:47.880 so that you can bring compute to the actual database 35:47.880 --> 35:49.410 as opposed to your, 35:49.410 --> 35:50.430 privacy reasons too. 35:50.430 --> 35:53.000 You're not gonna wanna download a bunch of this data. 35:53.000 --> 35:54.160 - Yeah, I agree. 35:54.160 --> 35:56.230 And I have to say given all the work we've done 35:56.230 --> 35:59.510 on AI and machine learning, this to me, 35:59.510 --> 36:01.670 if I can step aside from the clinical value, 36:01.670 --> 36:04.190 just the sheer value of the Department of Defense 36:04.190 --> 36:06.540 having this data source and being able 36:06.540 --> 36:10.130 to learn what it means to deal with data of this scale is, 36:10.130 --> 36:12.463 that in and of itself is extremely valuable. 36:15.060 --> 36:17.060 - We've previously had made the point 36:17.060 --> 36:19.310 that the DoD generates an enormous amount 36:19.310 --> 36:22.830 of data that it never actually uses, right? 36:22.830 --> 36:25.460 I mean, like there's enormous amount of data 36:25.460 --> 36:27.565 in every operation that they do. 36:27.565 --> 36:28.850 And this is just one example. 36:28.850 --> 36:30.220 - [Kurt] So let's use one. 36:30.220 --> 36:32.047 - Yeah, but it sounds like a lot of data. 36:32.047 --> 36:34.240 And Eric you know this better than anyone, 36:34.240 --> 36:38.300 55 million images on the scale of what is created 36:38.300 --> 36:41.590 every day in terms of images is not as significant 36:41.590 --> 36:42.680 as it sounds. 36:42.680 --> 36:43.810 - [Eric] But for the DoD it is. 36:43.810 --> 36:44.693 - Yeah. - Yeah. 36:46.240 --> 36:47.300 - [Kurt] Okay. 36:47.300 --> 36:48.450 - [Colleen] Mr. Chair, if you're ready. 36:48.450 --> 36:49.283 - Yeah. - Can we go ahead 36:49.283 --> 36:50.520 and do a vote? - Yes, okay. 36:50.520 --> 36:52.140 - Great, and I think what I'll do 36:52.140 --> 36:53.750 is I'll go through the dial-ins first, 36:53.750 --> 36:55.840 and then let you guys raise hands. 36:55.840 --> 36:57.833 So Missy? 37:00.870 --> 37:01.830 - [Missy] Yes, I'm here. 37:01.830 --> 37:04.763 - Great, do you concur with the recommendations? 37:05.700 --> 37:06.620 - [Missy] I concur. 37:06.620 --> 37:08.170 - Great. 37:08.170 --> 37:09.003 Danny? 37:10.510 --> 37:11.443 - [Danny] I concur. 37:12.344 --> 37:13.663 - Dr. Lander? 37:15.520 --> 37:17.392 - [Eric] As a matter of fact, I concur too. 37:17.392 --> 37:18.225 (board members laughing) 37:18.225 --> 37:19.165 - [Eric] Yeah, you better. 37:19.165 --> 37:20.800 (board members laughing) 37:20.800 --> 37:21.633 - Marnie? 37:23.550 --> 37:24.650 - [Marnie] I concur. 37:24.650 --> 37:25.943 - Great, Jen? 37:27.580 --> 37:29.083 - [Jen] Strongly concur. 37:30.260 --> 37:31.093 - Daniela? 37:33.090 --> 37:34.170 - [Daniela] I concur. 37:34.170 --> 37:35.003 - Fantastic. 37:35.003 --> 37:36.610 And do we have Neil on the line also? 37:37.800 --> 37:38.900 - [Neil] Yes, I am. 37:38.900 --> 37:39.733 Can you hear me? 37:39.733 --> 37:40.566 - Yes, we can. 37:41.840 --> 37:43.870 - [Neil] Yes, I double concur. 37:43.870 --> 37:46.140 Does that count as two votes (laughing)? 37:46.140 --> 37:48.016 - I can't do that for you, Neil. 37:48.016 --> 37:49.420 Excellent. - [Neil] Okay. 37:49.420 --> 37:51.880 - Show of hands here on the stage. 37:51.880 --> 37:52.800 Great. 37:52.800 --> 37:54.680 So we'll go ahead and approve that recommendation. 37:54.680 --> 37:56.446 - So it appears to be universal-- 37:56.446 --> 37:58.210 - [Eric] Before we move on, 37:58.210 --> 38:00.980 can I just express my enormous admiration 38:00.980 --> 38:03.033 to the JPC staff? 38:04.210 --> 38:07.710 Since I got to tour on behalf of the DIB, 38:07.710 --> 38:12.710 and then host a meeting with the JPC and others, 38:13.420 --> 38:16.010 I just think they're doing an amazing job. 38:16.010 --> 38:17.900 And in all of these recommendations 38:17.900 --> 38:19.930 of saying they should be doing more, 38:19.930 --> 38:21.390 this is very supportive and none 38:21.390 --> 38:23.040 of this would be possible without 38:23.890 --> 38:25.410 all of what they're doing already. 38:25.410 --> 38:26.900 So thank you. - Thanks. 38:26.900 --> 38:29.750 - And I'll just close this by also recognizing Skyler Moore, 38:29.750 --> 38:31.940 who's part of the Defense Innovation Board staff, 38:31.940 --> 38:34.310 who was the lead on the activity here. 38:34.310 --> 38:36.170 So thank you, Skyler. 38:36.170 --> 38:37.386 - Okay, thank you. 38:37.386 --> 38:38.750 Very well done. 38:38.750 --> 38:40.290 Shall we move to say Richard, 38:40.290 --> 38:42.355 you wanna talk about software again? 38:42.355 --> 38:46.580 - (laughing) So thank you, Eric. 38:46.580 --> 38:49.290 So the second recommendation that we have coming out 38:49.290 --> 38:50.470 of the S & T subcommittee is a little bit different 38:50.470 --> 38:53.170 than the first. 38:53.170 --> 38:56.760 As Eric mentioned, the NDAA that comes out 38:56.760 --> 38:59.610 each year authorizing all of our activities, 38:59.610 --> 39:04.577 and this year's NDAA, the 2020 NDAA, has in it section 862, 39:05.610 --> 39:08.399 in which Congress asked the Defense Innovation Board 39:08.399 --> 39:10.130 to create an independent assessment 39:10.130 --> 39:11.060 of the department's progress 39:11.060 --> 39:12.390 on implementing software development, 39:12.390 --> 39:14.840 software management and software training programs. 39:14.840 --> 39:17.380 And this is directly related to the swap report 39:17.380 --> 39:20.210 that Michael and I co-chaired, 39:20.210 --> 39:24.823 along with Jen and Milo, and Gilman and Trey, 39:25.840 --> 39:27.810 and it came out last year. 39:27.810 --> 39:30.660 And so just the way that's being done is that, 39:30.660 --> 39:33.230 in Section 862, Congress has asked 39:33.230 --> 39:35.720 the Department of Defense, the Secretary, 39:35.720 --> 39:38.550 to report back an initial report 39:38.550 --> 39:42.360 on what's being done to implement various activities 39:42.360 --> 39:43.960 in terms of software development, software management, 39:43.960 --> 39:45.079 and software training programs. 39:45.079 --> 39:49.250 An initial report around now and then a final report 39:49.250 --> 39:52.660 a year after the NDAA was passed in 2020, 39:52.660 --> 39:54.680 or the '20 NDAA was passed. 39:54.680 --> 39:58.040 And so as part of that initial report and final report, 39:58.040 --> 39:59.580 the Defense Innovation Board has been asked 39:59.580 --> 40:01.200 to give an independent assessment 40:01.200 --> 40:02.720 of the department's activities, 40:02.720 --> 40:05.140 and so that will be included in those reports. 40:05.140 --> 40:07.630 So as it happens, the department is going 40:07.630 --> 40:10.307 to request an extension of the initial report, 40:10.307 --> 40:12.533 and ask for a little bit more time. 40:12.533 --> 40:16.660 And so that probably won't come out for a couple of months. 40:16.660 --> 40:19.680 But because of the Federal Advisories Committee Act, 40:19.680 --> 40:22.460 we have to publicly deliberate on our comments 40:22.460 --> 40:26.752 that will go into that future report now, 40:26.752 --> 40:28.840 because this is what our public meeting is. 40:28.840 --> 40:33.390 And so to do that, we have collected some data 40:33.390 --> 40:35.390 from the people who are working on that initial report 40:35.390 --> 40:39.400 in terms of their initial thoughts on where that was going, 40:39.400 --> 40:41.911 in addition to some of the materials around 40:41.911 --> 40:45.720 they're basing their kind of implementation activities on. 40:45.720 --> 40:49.040 And so this is something where we're seeing 40:49.040 --> 40:50.880 the beginnings of the activity. 40:50.880 --> 40:52.460 They will report in another few months 40:52.460 --> 40:54.197 on what the actual implementation is. 40:54.197 --> 40:57.650 But we're deliberating now on kind of our sense 40:57.650 --> 41:00.100 of all of that and an assessment. 41:00.100 --> 41:02.850 And then we'll have a short report 41:02.850 --> 41:05.480 that we'll need to approve in order for that to go in. 41:05.480 --> 41:08.040 So overall, our assessment 41:08.040 --> 41:10.689 of the initial activities is positive. 41:10.689 --> 41:13.480 The directions are right, it's just starting. 41:13.480 --> 41:15.580 And so I think people are pointed in the right direction. 41:15.580 --> 41:18.080 As is often I think the case 41:18.080 --> 41:19.460 with the Defense Innovation Board, 41:19.460 --> 41:21.190 we'd love to see things move 41:21.190 --> 41:23.360 more quickly toward actual implementation. 41:23.360 --> 41:25.990 And so we almost uniformly give the advice 41:25.990 --> 41:27.830 of let's get going on this. 41:27.830 --> 41:31.300 And in particular, to do something that is very consistent 41:31.300 --> 41:33.910 with the swap report, which is to start small 41:33.910 --> 41:37.390 and iterate quickly, and then identify those ways 41:37.390 --> 41:40.130 to expand that and scale it and optimize it. 41:40.130 --> 41:43.310 And so that's also the case here. 41:43.310 --> 41:45.900 We also think that there are opportunities in the context 41:45.900 --> 41:48.170 of these initial activities that we've seen started, 41:48.170 --> 41:50.570 and I'm being vague about it because they're not 41:50.570 --> 41:52.280 publicly available. 41:52.280 --> 41:55.250 So I can't say what they are, but we've assessed them. 41:55.250 --> 41:58.500 But there are some opportunities in there for the department 41:58.500 --> 42:02.130 to partner with Congress towards seeking legislative support 42:02.130 --> 42:03.810 for some of the things that they would like to do, 42:03.810 --> 42:07.220 in particular in the areas around software expertise 42:07.220 --> 42:10.400 within major development of acquisition programs and apps, 42:10.400 --> 42:12.920 as well as major information systems. 42:12.920 --> 42:17.600 So that's sort of where we're at, the directions are good. 42:17.600 --> 42:21.040 We believe that the department should look 42:21.040 --> 42:23.980 toward implementing things as quickly as it can. 42:23.980 --> 42:26.610 Start small, iterate quickly, we hope to see that 42:26.610 --> 42:29.660 when the actual initial report comes out, 42:29.660 --> 42:33.210 and that's what we will be looking for when the final report 42:33.210 --> 42:35.470 comes out about those implementation activities 42:35.470 --> 42:39.610 in terms of really providing the types of training 42:39.610 --> 42:41.725 and career paths and other things that are required 42:41.725 --> 42:44.460 and that are part of our stock swap report. 42:44.460 --> 42:49.460 So with that, I will stop and take any questions, 42:49.610 --> 42:51.940 comments from the board, regarding things. 42:51.940 --> 42:53.720 And Michael, you've been involved with this as well, 42:53.720 --> 42:55.820 you may wanna throw in your own two cents. 42:56.820 --> 42:58.970 - Just two very quick comments, and maybe a question 42:58.970 --> 43:00.440 that Richard, you can sort of opine back on. 43:00.440 --> 43:03.530 So I would just reiterate what Richard said, 43:03.530 --> 43:06.740 we've seen very good support in the department 43:07.850 --> 43:10.423 for going after the recommendations. 43:11.850 --> 43:14.800 And simply repeat that in all cases, 43:14.800 --> 43:18.060 the mantra is move quickly, do something. 43:18.060 --> 43:21.460 If it doesn't work fix it, move again and move quickly. 43:21.460 --> 43:23.400 So general reinforcing comment. 43:23.400 --> 43:25.240 Richard I guess the one question is, 43:25.240 --> 43:27.260 do we wanna make any comments on sort of 43:27.260 --> 43:29.040 where we see the issue around 43:29.040 --> 43:31.910 sort of tools and capabilities? 43:31.910 --> 43:33.390 I mean, this has been an ongoing refrain 43:33.390 --> 43:35.610 that we've heard from a long time. 43:35.610 --> 43:38.630 - Yeah, I mean generally looking at how 43:38.630 --> 43:42.340 to increase the ability of DoD 43:42.340 --> 43:46.410 through both the services and military 43:47.440 --> 43:48.760 as well as the civilian workforce, 43:48.760 --> 43:51.390 as well as contractors to just have more capability 43:51.390 --> 43:54.260 in software available to the Department of Defense, 43:54.260 --> 43:56.250 both within actually doing a software, 43:56.250 --> 43:57.870 implementing a software, getting it out in the field, 43:57.870 --> 43:58.890 doing things quickly. 43:58.890 --> 44:00.760 But also within acquisition programs. 44:00.760 --> 44:03.470 The acquisition side of that and understanding 44:03.470 --> 44:05.040 how does modern software work, 44:05.040 --> 44:07.470 and figuring out how to get modern software 44:07.470 --> 44:08.820 through that contracting process 44:08.820 --> 44:09.880 in a way that makes sense. 44:09.880 --> 44:12.760 And again, going back to our swap report, 44:12.760 --> 44:15.267 software is never done, that was the title of the report. 44:15.267 --> 44:16.100 And so you can't sort of think 44:16.100 --> 44:19.460 about software as something where you somehow 44:19.460 --> 44:21.570 you write a bunch of recommendations, 44:21.570 --> 44:23.410 you spend a bunch of time 44:23.410 --> 44:25.490 sort of putting out a competition and deciding, 44:25.490 --> 44:28.080 and then you acquire that software, and then you sustain. 44:28.080 --> 44:29.200 That's just not the way it works. 44:29.200 --> 44:31.600 We all know this just from our daily lives. 44:31.600 --> 44:32.810 You buy a piece of software 44:32.810 --> 44:34.060 and it's constantly getting updated, 44:34.060 --> 44:35.460 and it needs to be constantly updated. 44:35.460 --> 44:37.730 Software is never done, it's an ongoing process. 44:37.730 --> 44:39.770 Getting people within the Department of Defense 44:39.770 --> 44:40.830 who understand that, 44:40.830 --> 44:43.202 getting the capabilities within the Department of Defense 44:43.202 --> 44:46.270 so that we can see that type of operation going on 44:46.270 --> 44:48.340 that when a bug or a flaw comes up, 44:48.340 --> 44:50.480 a security flaw in particular, that it's not, 44:50.480 --> 44:52.940 oh well, we need another release of the software. 44:52.940 --> 44:54.700 Let's start writing what the requirements are 44:54.700 --> 44:55.900 and 10 years from now we'll have 44:55.900 --> 44:57.100 that release that protects against it. 44:57.100 --> 44:58.450 It's way too late. 44:58.450 --> 45:01.080 How do we say this is an ongoing capability, 45:01.080 --> 45:03.420 this is something that needs to be solved soon? 45:03.420 --> 45:05.174 How do we get something out on Friday, 45:05.174 --> 45:06.820 because it needs to go out that quickly. 45:06.820 --> 45:08.820 And again, that requires a different way 45:08.820 --> 45:09.820 of thinking about software. 45:09.820 --> 45:12.000 We need that capability and expertise 45:12.000 --> 45:13.910 within the department in a variety of ways. 45:13.910 --> 45:16.020 And so I think, again on the workforce side, 45:16.020 --> 45:19.650 and we had sort of four lines of effort, 45:19.650 --> 45:22.300 and the third line of effort was really increasing 45:22.300 --> 45:25.900 the capabilities within the Department of Defense 45:25.900 --> 45:28.870 on the people side, because software is 45:28.870 --> 45:30.870 by people and for people. 45:30.870 --> 45:33.413 And so we just really need to recognize that. 45:33.413 --> 45:37.430 - To pick up on that, Richard, it's not just the number 45:37.430 --> 45:40.000 of people attached to this, but I think certainly, 45:40.000 --> 45:42.880 I would recommend that we strongly emphasize 45:42.880 --> 45:45.700 that those people need to be given a plan 45:45.700 --> 45:47.661 to make sure that they stay. 45:47.661 --> 45:50.470 That there is a qualitative path 45:50.470 --> 45:53.128 for them to move forward, because they're gonna amass 45:53.128 --> 45:53.961 a large amount of information even though 45:53.961 --> 45:57.160 they're gonna be a relatively small group 45:57.160 --> 46:00.320 in the beginning and we don't want any, 46:00.320 --> 46:02.640 as much as possible, we don't want any 46:02.640 --> 46:04.451 significant turnover of that team 46:04.451 --> 46:06.770 because it could set this project back a lot. 46:06.770 --> 46:09.288 So I would just recommend that we strongly emphasize 46:09.288 --> 46:12.100 that as we're doing all this work for the software side, 46:12.100 --> 46:15.760 we make sure that we are pathing and making sure 46:15.760 --> 46:18.892 that the support for the people are there, 46:18.892 --> 46:22.350 that those people will stay for the length 46:22.350 --> 46:24.970 of the duration of the project and have an ability 46:24.970 --> 46:26.790 to train new people that come in after. 46:26.790 --> 46:27.740 - Yeah, absolutely. 46:27.740 --> 46:29.980 And again, I'd say that providing people 46:29.980 --> 46:30.813 with the right environment 46:30.813 --> 46:33.550 so that they can do their job effectively. 46:33.550 --> 46:36.327 And then having them say, "Boy, this is the best place 46:36.327 --> 46:38.347 "I could imagine being for having the type of impact 46:38.347 --> 46:40.040 "that I can have as a software developer" 46:40.040 --> 46:42.070 I think is something that we absolutely have to do. 46:42.070 --> 46:44.530 And I would say we want sort of both, right? 46:44.530 --> 46:47.470 That is that we want people who stay in those positions 46:47.470 --> 46:49.780 rather than hopping out and going into-- 46:49.780 --> 46:51.067 - So it's essentially making it a career track. 46:51.067 --> 46:52.320 - Making it career track. 46:52.320 --> 46:54.340 - And making them understand it's rare to be able 46:54.340 --> 46:56.510 to be in a DoD where you can do a job that affects 46:56.510 --> 46:58.270 the entire DoD like this could. 46:58.270 --> 47:00.860 - Right, and at the same time, allowing people who are 47:00.860 --> 47:03.060 in the private sector to come in and spend some time 47:03.060 --> 47:05.900 in DoD and help and contribute and then go back, 47:05.900 --> 47:07.710 and they'll still be part of that DoD family 47:07.710 --> 47:09.307 but now back in their commercial companies and other things. 47:09.307 --> 47:10.600 And so, how do we do both? 47:10.600 --> 47:12.050 How do we allow people to have a career path 47:12.050 --> 47:14.130 but also allow people to kind of hop in and out 47:14.130 --> 47:16.905 a little bit within the context of the Department of Defense 47:16.905 --> 47:18.200 in an effective way? 47:18.200 --> 47:19.480 - Very good, thank you. 47:19.480 --> 47:22.510 So we've had enormously strong support 47:22.510 --> 47:24.160 from Secretary Lord, as I said earlier, 47:24.160 --> 47:27.174 and the Chief Information Officer and his teams, 47:27.174 --> 47:29.223 so that's in pretty good shape. 47:30.240 --> 47:33.600 There were a set of changes in procurement 47:33.600 --> 47:35.340 that were in flight. 47:35.340 --> 47:37.070 The most important one was addressing 47:37.070 --> 47:39.413 this question of color of money. 47:39.413 --> 47:41.960 Are you reasonably confident that those changes 47:41.960 --> 47:43.569 will go through? 47:43.569 --> 47:45.610 - (laughing) I don't know that 47:45.610 --> 47:47.420 I'm reasonably confident that anything complicated 47:47.420 --> 47:49.380 will go through, but I'm very confident 47:49.380 --> 47:51.940 that that is something that Secretary Lord has put 47:51.940 --> 47:53.900 very high up on her list of priorities 47:53.900 --> 47:56.330 in terms of really thinking how do we change 47:56.330 --> 47:59.580 that acquisition process and have a single color of money? 47:59.580 --> 48:01.680 It's a hard lift to get all of the pieces 48:01.680 --> 48:02.703 to sort of do that. 48:04.408 --> 48:06.250 - This is the bureaucracy fighting back 48:06.250 --> 48:08.000 against sort of nationally. - Absolutely. 48:08.000 --> 48:09.260 So we want the leadership to win 48:09.260 --> 48:10.270 not the bureaucracy. - Absolutely. 48:10.270 --> 48:11.960 And and we are strong advocates of that. 48:11.960 --> 48:14.070 Again, why is that important? 48:14.070 --> 48:17.540 Because when you split things into multiple pots, 48:17.540 --> 48:19.530 this is not the way software works. 48:19.530 --> 48:20.930 And to sort of say, oh, no, no, no, 48:20.930 --> 48:22.690 we can't add, that's a new feature. 48:22.690 --> 48:25.200 We have to go out and write the requirements 48:25.200 --> 48:27.400 for that new feature and rebid. 48:27.400 --> 48:29.200 Sorry, that is not software. 48:29.200 --> 48:31.480 And so again, that single color of money that says, 48:31.480 --> 48:32.870 software is never done. 48:32.870 --> 48:35.450 We need to say, we're gonna spend $50 million a year, 48:35.450 --> 48:37.220 or $100 million on this capability 48:37.220 --> 48:39.250 that is software-enabled forever. 48:39.250 --> 48:41.100 And we want improvements all the time. 48:41.100 --> 48:43.380 We wanna be rolling out improvements every month, 48:43.380 --> 48:45.360 every quarter, every year, whatever it is, 48:45.360 --> 48:46.350 depending on the size of that. 48:46.350 --> 48:47.980 So I think getting there is absolutely 48:47.980 --> 48:48.813 the right way to go. 48:48.813 --> 48:50.367 I have confidence that the leadership 48:50.367 --> 48:52.930 within the DoD understands that 48:52.930 --> 48:53.763 and would like to move that forward. 48:53.763 --> 48:55.890 But and there are lots of pieces that need to agree, 48:55.890 --> 48:57.803 and I think that that conversation is happening. 49:00.670 --> 49:02.180 - I wanna pick up on a theme. 49:02.180 --> 49:03.160 One of the things we've had a bunch 49:03.160 --> 49:04.840 of discussion about is you've got the DoD, 49:04.840 --> 49:08.450 which is not a very software centric organization today, 49:08.450 --> 49:09.560 one of the experiences we had 49:09.560 --> 49:12.140 when we kind of recovered healthcare.gov, 49:12.140 --> 49:13.470 and you kinda touched on it, 49:13.470 --> 49:14.960 the notion of rotation. 49:14.960 --> 49:16.451 I think the thing we need to target 49:16.451 --> 49:19.430 as a first thing, is to get the right kernel 49:19.430 --> 49:21.960 of software development in all the right places. 49:21.960 --> 49:23.740 Because one of the things we noticed is 49:23.740 --> 49:25.510 if you can get trusted advisors 49:25.510 --> 49:27.620 that do the design correctly, 49:27.620 --> 49:29.870 that know when, if a lot of it's outsource, 49:29.870 --> 49:32.380 can know what's good and what's bad, 49:32.380 --> 49:33.870 it's not as daunting as you think. 49:33.870 --> 49:36.050 You'd get up to this level of software capability 49:36.050 --> 49:37.910 in the organization internally, 49:37.910 --> 49:39.470 and then you can build from there, 49:39.470 --> 49:42.320 and your outcomes become better just from the beginning. 49:42.320 --> 49:44.429 - Which kind of colonel were you referring to? 49:44.429 --> 49:46.740 (audience laughing) - Not that kind of colonel. 49:46.740 --> 49:47.706 Both perhaps. 49:47.706 --> 49:48.539 - Both perhaps. 49:48.539 --> 49:49.372 - A good Colonel would be good too. 49:49.372 --> 49:51.857 - Can we get some comments from folks on the phone? 49:57.206 --> 49:58.039 - Maybe not. 49:58.039 --> 49:59.700 - Or not. 49:59.700 --> 50:01.250 All right, maybe I'll just pull it back and say 50:01.250 --> 50:02.940 so we have prepared a page and a half, 50:02.940 --> 50:05.463 which is again on our website, and you can read that, 50:05.463 --> 50:08.360 that we are asking for the board approval to put 50:08.360 --> 50:11.220 into the initial response that goes back. 50:11.220 --> 50:14.750 I'll also say we're looking for ways to also comment 50:14.750 --> 50:17.293 on the final version that comes out, 50:18.210 --> 50:20.700 when it comes out, and to augment what we've already done, 50:20.700 --> 50:22.110 but we're asking for official approval 50:22.110 --> 50:24.474 now on this particular page and a half assessment. 50:24.474 --> 50:26.360 - So is there agreement we should, 50:26.360 --> 50:27.193 so Colleen, do you wanna-- 50:27.193 --> 50:28.640 - [Colleen] We'll go ahead and do the vote, yeah? 50:28.640 --> 50:31.810 So I'll do the online dial-ins first and then on stage. 50:31.810 --> 50:32.713 So Missy? 50:35.153 --> 50:36.827 - [Missy] Yes, are you looking? 50:36.827 --> 50:38.156 - [Kurt] Missy, do you approve or not? 50:38.156 --> 50:39.873 - Yes, do you concur? 50:41.770 --> 50:42.860 - [Missy] Do I have any comments, 50:42.860 --> 50:43.777 is that what you're asking? 50:43.777 --> 50:46.805 - No, no, we're doing the vote right now. Do you concur? 50:46.805 --> 50:48.540 - [Missy] The vote, okay yes, I concur. 50:48.540 --> 50:50.613 - Okay, Danny? 50:58.310 --> 50:59.333 - [Kurt] Are you there Danny? 51:00.460 --> 51:01.740 - [Danny] Yeah, I concur. 51:01.740 --> 51:03.640 - Great, thanks. - Sorry, on mute. 51:03.640 --> 51:04.703 - Dr. Lander? 51:06.090 --> 51:07.120 - [Eric] Concur. 51:07.120 --> 51:07.953 - Marnie? 51:10.083 --> 51:11.215 - [Marnie] Concur. 51:11.215 --> 51:12.298 - Great, Jen? 51:13.658 --> 51:15.040 - [Jen] Absolutely, concur. 51:15.040 --> 51:16.423 - Great, Daniela? 51:18.160 --> 51:18.993 - [Daniela] I concur. 51:18.993 --> 51:19.826 - Thank you. 51:19.826 --> 51:20.833 And then Neil? 51:22.140 --> 51:22.973 - [Neil] I concur. 51:22.973 --> 51:25.830 - Great, and then those on stage, by show of hands. 51:25.830 --> 51:28.409 - [Kurt] Neil, you only concurred once. 51:28.409 --> 51:31.076 (Neil laughing) 51:32.000 --> 51:36.250 - So that appears that we have approved 51:36.250 --> 51:38.193 that number two unanimously. 51:39.970 --> 51:42.730 So we've been interested in this workforce 51:42.730 --> 51:44.600 and behavior of stuff for a long time. 51:44.600 --> 51:47.229 And now there's a team, that you're part 51:47.229 --> 51:48.890 of this really pushing hard on it. 51:48.890 --> 51:52.570 So take us through what you wanna get done. 51:52.570 --> 51:53.403 - Okay. 51:53.403 --> 51:54.236 - Give us an update. 51:54.236 --> 51:58.760 - So, thanks a lot for the intro there. 51:58.760 --> 52:01.240 We obviously had talked a lot about workforce issues 52:01.240 --> 52:04.060 as part of the transformation of the DoD, 52:04.060 --> 52:05.180 both in the software study. 52:05.180 --> 52:06.900 There was also a Workforce Now or set 52:06.900 --> 52:07.960 of recommendations that we looked at 52:07.960 --> 52:12.960 in the October meeting, and approved at that meeting. 52:12.960 --> 52:15.590 But there's another piece that comes from the NDAA again, 52:15.590 --> 52:17.490 the Defense Authorization Act. 52:17.490 --> 52:19.360 This section is 230. 52:19.360 --> 52:21.970 And 230 directs the DoD to promote 52:21.970 --> 52:23.530 and maintain digital expertise 52:23.530 --> 52:26.330 in software development, as a core competency, 52:26.330 --> 52:29.750 in the both the civilian and the military workforce. 52:29.750 --> 52:31.890 And the first thing I'd say is 52:31.890 --> 52:34.610 that we completely support this notion overall. 52:34.610 --> 52:36.680 It's consistent with what the Workforce Now recommendations 52:36.680 --> 52:39.058 were that were in October. 52:39.058 --> 52:40.853 There's an interesting provision in it that we wanted 52:40.853 --> 52:44.680 to talk about here and it's an optional provision 52:44.680 --> 52:48.010 that the Secretary can appoint a civilian official 52:48.010 --> 52:48.900 to a new position. 52:48.900 --> 52:51.620 The new position is called the Chief Digital Engineering 52:51.620 --> 52:53.630 Recruitment and Management Officer. 52:53.630 --> 52:56.226 A bit of a mouthful of a position, 52:56.226 --> 52:58.550 but it's a really cool idea. 52:58.550 --> 53:02.740 The DIB supports the notion of the Secretary appointing 53:02.740 --> 53:04.170 this position. 53:04.170 --> 53:06.867 And the idea is if I'll introduce it, 53:06.867 --> 53:09.830 and we'll get a sense of what that person might do. 53:09.830 --> 53:13.050 But if you think about the workforce personnel divisions, 53:13.050 --> 53:15.880 it's very disparate across the organization. 53:15.880 --> 53:19.130 It is kind of mired in the past in some cases. 53:19.130 --> 53:20.130 We actually had a briefing 53:20.130 --> 53:23.150 from the Army's Future Command today, 53:23.150 --> 53:26.330 and the Chief Human Capital Officer, 53:26.330 --> 53:27.780 her name is Katie Kelly, 53:27.780 --> 53:28.954 and she talked about the different parts 53:28.954 --> 53:32.930 of the organization having very different policies 53:32.930 --> 53:35.520 in terms of how they recruit people. 53:35.520 --> 53:36.790 It's very complex. 53:36.790 --> 53:39.760 It's not coordinated across all those divisions. 53:39.760 --> 53:43.640 And so at the same time, digital talent is hugely important. 53:43.640 --> 53:44.800 We just talked about it here, 53:44.800 --> 53:47.940 the importance of having great software expertise. 53:47.940 --> 53:49.560 And so we think that actually appointing 53:49.560 --> 53:51.650 this position could be game changing. 53:51.650 --> 53:53.500 It's an opportunity that we actually think 53:53.500 --> 53:55.380 that they should take advantage of. 53:55.380 --> 53:56.540 We actually like a better name 53:56.540 --> 53:58.030 the Digital People Officer, 53:58.030 --> 54:01.010 it's less of a mouthful, so I think that's a good thing. 54:01.010 --> 54:03.590 And we think actually, in addition to kind of setting 54:03.590 --> 54:06.380 on a better course, we think it has an opportunity 54:06.380 --> 54:08.990 to kind of bridge the gap of how do you get started again. 54:08.990 --> 54:10.120 How do you get people starting 54:10.120 --> 54:12.623 to bring the right talent into the organization? 54:13.570 --> 54:17.090 So we think they should really go at this boldly 54:17.090 --> 54:18.750 and put this position in place. 54:18.750 --> 54:21.450 There's a few things, we actually did a job description, 54:21.450 --> 54:23.780 and you can see that in the report. 54:23.780 --> 54:26.800 But there's a few things we think you could do. 54:26.800 --> 54:28.430 One, they could actually just get started 54:28.430 --> 54:31.210 and start figuring out how do we effectively recruit people, 54:31.210 --> 54:33.100 and what are the right techniques for doing that. 54:33.100 --> 54:35.000 Actually just start recruiting people, 54:35.961 --> 54:38.890 core software people, digital people. 54:38.890 --> 54:41.440 We think it's the most effective way overall to do that. 54:41.440 --> 54:43.170 We also think a piece of this is gonna be 54:43.170 --> 54:46.410 about best practices, and how do you kind of bring together 54:46.410 --> 54:48.970 a community of the HR officials, 54:48.970 --> 54:50.883 kind of all over the organization? 54:52.850 --> 54:57.060 And develop what's the best approach to do this overall. 54:57.060 --> 54:59.670 So we think that's a good opportunity as well. 54:59.670 --> 55:01.480 And then the other thing is when you look across 55:01.480 --> 55:04.420 the organization to figure out, how is it working today? 55:04.420 --> 55:05.940 How do you pull it together into a structure 55:05.940 --> 55:07.930 that makes sense for the future? 55:07.930 --> 55:11.010 So as I said, we pulled together a job description. 55:11.010 --> 55:12.590 We also put thought, think a bit 55:12.590 --> 55:14.280 about where we should put this. 55:14.280 --> 55:17.610 And we think putting it in the DDS makes the most sense, 55:17.610 --> 55:20.460 that's Defense Digital Service 55:20.460 --> 55:22.333 that actually got created as an offshoot 55:22.333 --> 55:25.220 of when we had the issues with healthcare.gov. 55:25.220 --> 55:27.550 How do you create an organization that's gonna kickstart 55:27.550 --> 55:29.920 great software development in the different organizations, 55:29.920 --> 55:32.740 one got created for the Department of Defense. 55:32.740 --> 55:34.960 And so we think that that actually is already started 55:34.960 --> 55:37.980 by creating the infrastructure by which you can bootstrap. 55:37.980 --> 55:40.020 They'll have all their offices together, 55:40.020 --> 55:42.270 they know how to work with the DoD overall. 55:42.270 --> 55:43.610 So we think it's probably the best way 55:43.610 --> 55:45.310 to get kickstarted really quickly. 55:46.590 --> 55:47.423 The other thing I think is important 55:47.423 --> 55:51.070 to recognize is that it should use what's going 55:51.070 --> 55:53.090 on in the private sector as an example 55:53.090 --> 55:55.420 for what we can do in the DoD as well. 55:55.420 --> 55:58.530 Every corporation is going through transformation 55:58.530 --> 56:00.580 to being a digital organization. 56:00.580 --> 56:03.240 And so we've already learned a bunch about what this means, 56:03.240 --> 56:05.140 what it means in terms of having the right people 56:05.140 --> 56:06.370 to do the job. 56:06.370 --> 56:08.700 And so one of the organizations that I lead in Microsoft 56:08.700 --> 56:11.380 is called Core Services, Engineering and Operations. 56:11.380 --> 56:13.720 And we've actually transformed Microsoft 56:13.720 --> 56:15.480 from thinking about our internal systems 56:15.480 --> 56:17.150 as things that you run, 56:17.150 --> 56:19.330 to actually thinking of them as things you build 56:19.330 --> 56:21.850 using great software development techniques. 56:21.850 --> 56:23.560 It means you have to change who you have 56:23.560 --> 56:26.370 in the organization, what the career ladders look like. 56:26.370 --> 56:28.530 And we can actually, we wrote it up in the paper, 56:28.530 --> 56:31.110 but we'd be happy to talk to the DoD about what that means. 56:31.110 --> 56:34.710 We also cited a couple other places like Novartis and PwC, 56:34.710 --> 56:36.680 who've done a similar transformation. 56:36.680 --> 56:38.440 So lots of examples in industry 56:38.440 --> 56:40.400 that we can leverage there as well. 56:40.400 --> 56:43.360 So overall, we all kinda as part of the workforce, 56:43.360 --> 56:44.907 we strongly recommend that they adopt this, 56:44.907 --> 56:46.930 and I wanted to toss it over to Jen. 56:46.930 --> 56:48.880 Jen, do you have some comments as well? 56:50.773 --> 56:52.160 - [Jen] Okay, thanks for that setup Kurt. 56:52.160 --> 56:54.500 This is a really important part of the meeting today 56:54.500 --> 56:55.630 and an important part of the work 56:55.630 --> 56:57.573 that we're trying to do here. 56:59.060 --> 57:01.110 I guess I'll start just by commending Congress 57:01.110 --> 57:03.043 for the great idea of this position. 57:03.930 --> 57:06.560 And also echoing Kurt's comments 57:06.560 --> 57:10.740 that the Digital People Officer is a little bit 57:10.740 --> 57:12.940 of a catchier title and would probably 57:14.160 --> 57:17.180 serve the position well in its simplicity. 57:17.180 --> 57:19.970 So recommending that the department adopt 57:19.970 --> 57:21.820 that name rather than the longer one. 57:24.060 --> 57:26.539 I think the rest of the board's with me on that. 57:26.539 --> 57:27.540 - [Kurt] Absolutely. 57:27.540 --> 57:31.680 - [Jen] I also just wanted to recognize and assuring 57:31.680 --> 57:35.820 that all of you and on the DIB understand this, 57:35.820 --> 57:38.523 all of us this year, that while this is something 57:38.523 --> 57:40.262 that is coming from Congress, 57:40.262 --> 57:43.500 you can't simply command the system 57:43.500 --> 57:46.220 to rapidly fill this digital talent gap. 57:46.220 --> 57:48.603 It is obviously more complicated than that. 57:48.603 --> 57:52.590 The work that we did in Workforce Now that Kurt referenced, 57:52.590 --> 57:54.910 revealed that however well-intended the department 57:54.910 --> 57:57.910 might be on these issues of filling this gap, 57:57.910 --> 58:01.820 there are really a lot of regulations and policies. 58:01.820 --> 58:04.410 They're deeply complex, deeply interconnected, 58:04.410 --> 58:05.970 and you get kind of mired down, 58:05.970 --> 58:10.320 and it's very easy for the work that we recommend 58:10.320 --> 58:13.433 to get limited to making change on the margins. 58:14.800 --> 58:16.780 This limited incremental change, 58:16.780 --> 58:19.320 as we're all I think in agreement here, 58:19.320 --> 58:20.670 isn't enough right now 58:20.670 --> 58:22.170 at this particular moment in time, 58:22.170 --> 58:24.670 and the issues that face our national defense. 58:24.670 --> 58:27.230 It's not fast enough, and it's not bold enough. 58:27.230 --> 58:29.380 And so what we're recommending here 58:29.380 --> 58:31.553 I think sort of takes it up a notch. 58:33.520 --> 58:38.280 I think Kurt reference to cyber workforce example, 58:38.280 --> 58:40.400 at least it's in the report. 58:40.400 --> 58:41.790 That one was driven in large part 58:41.790 --> 58:44.220 by federal requirements to stand up a cyber workforce, 58:44.220 --> 58:48.298 which has been slower than we'd hoped to come. 58:48.298 --> 58:50.930 And as with anything in the Federal Government, 58:50.930 --> 58:54.010 we really appreciate that where this individual sits, 58:54.010 --> 58:57.690 that hopefully the Digital People Officer, 58:57.690 --> 59:00.180 can really make or break their success, 59:00.180 --> 59:03.700 how close they are to the power. 59:03.700 --> 59:05.630 So we're obviously really looking out 59:05.630 --> 59:07.920 for where this position will end up being. 59:07.920 --> 59:11.390 And as Kurt said, we think that the digital I'm sorry, 59:11.390 --> 59:14.273 the Defense Digital Service is the right place to put it. 59:16.960 --> 59:20.060 I guess I would also just echo Kurt's comments 59:20.060 --> 59:22.780 on the important role of hiring processes 59:22.780 --> 59:24.883 in what we're recommending here. 59:25.960 --> 59:28.700 This DPO would be sort of a repository of knowledge 59:28.700 --> 59:31.420 and best practices of hiring around the department, 59:31.420 --> 59:34.320 at least hiring for these digital positions. 59:34.320 --> 59:37.940 And there has been a lot of study and introspection, 59:37.940 --> 59:41.630 of course we're looking at what the private sector has done. 59:41.630 --> 59:43.860 I'll also add that I think that's happening 59:43.860 --> 59:46.433 in lots of places around the public sector. 59:48.060 --> 59:50.070 You might note that Google for instance, 59:50.070 --> 59:52.650 has started including peer in their hiring processes, 59:52.650 --> 59:56.590 so it's not solely managers making hiring decisions. 59:56.590 --> 59:59.160 Why did they do something like that? 59:59.160 --> 01:00:01.240 Because they measured it and saw that 01:00:01.240 --> 01:00:03.060 that got better results. 01:00:03.060 --> 01:00:05.640 You have other practices coming from the public sector, 01:00:05.640 --> 01:00:08.900 many people have looked at how places like Symphonis 01:00:08.900 --> 01:00:12.330 it started following blind hiring auditions 01:00:12.330 --> 01:00:14.170 as part of their hiring practice. 01:00:14.170 --> 01:00:15.610 And again, it's because the data shows 01:00:15.610 --> 01:00:18.500 you get a better workforce because of that. 01:00:18.500 --> 01:00:20.850 So we really wanna emphasize how much 01:00:20.850 --> 01:00:22.680 the department should be looking at practices 01:00:22.680 --> 01:00:26.750 from the outside, not just technology considerations. 01:00:26.750 --> 01:00:28.300 And what we haven't mentioned yet 01:00:28.300 --> 01:00:30.370 is just the speed of hiring. 01:00:30.370 --> 01:00:32.390 We are absolutely in an incredibly 01:00:32.390 --> 01:00:35.570 competitive environment here and speed will matter. 01:00:35.570 --> 01:00:39.140 In fact, Colleen was kind enough to forward a comment 01:00:39.140 --> 01:00:42.060 from the Army Chief of Staff General McConville, 01:00:42.060 --> 01:00:44.050 I think this was just yesterday at an event. 01:00:44.050 --> 01:00:45.567 He was quoted as saying, 01:00:45.567 --> 01:00:48.137 "We are in a war for talent." 01:00:49.020 --> 01:00:50.257 That's true, and we should act that way. 01:00:50.257 --> 01:00:52.170 And we should be as aggressive 01:00:52.170 --> 01:00:54.520 as that language suggests we should be. 01:00:54.520 --> 01:00:58.530 So just excited to see this brought forward. 01:00:58.530 --> 01:00:59.840 I really believe this is critical 01:00:59.840 --> 01:01:02.940 and a game changing opportunity for the department. 01:01:02.940 --> 01:01:05.300 We can rapidly scale recruitment, 01:01:05.300 --> 01:01:08.310 development and retention of digital talents 01:01:08.310 --> 01:01:09.930 with this particular proposal. 01:01:09.930 --> 01:01:13.000 So thanks for setting it up. 01:01:13.000 --> 01:01:14.660 And I'll turn it back over to you, Kurt. 01:01:14.660 --> 01:01:15.760 - Great, thanks Jen. 01:01:15.760 --> 01:01:17.770 So I'm gonna toss it over to questions. 01:01:17.770 --> 01:01:19.530 There are a few we might wanna think about. 01:01:19.530 --> 01:01:23.040 One thing is the section talks about digital engineering. 01:01:23.040 --> 01:01:25.280 Do you wanna define what that means is more specifically? 01:01:25.280 --> 01:01:28.360 That's a question we should probably resolve at some point. 01:01:28.360 --> 01:01:30.160 There's the whole notion that was talked about 01:01:30.160 --> 01:01:32.530 as cyber talent acquisition 01:01:32.530 --> 01:01:34.900 or development versus digital engineering. 01:01:34.900 --> 01:01:37.340 How do those relate to each other? 01:01:37.340 --> 01:01:39.110 There's digital transformation overall. 01:01:39.110 --> 01:01:41.380 How do we think about building the workforce 01:01:41.380 --> 01:01:45.190 versus the digital transformation effort underway overall? 01:01:45.190 --> 01:01:46.418 Also we had some discussion earlier 01:01:46.418 --> 01:01:50.620 and it was mentioned here around housing the DDS. 01:01:50.620 --> 01:01:53.560 The provisions talk about this as a formative stage 01:01:53.560 --> 01:01:55.860 of I think it's like up to two years. 01:01:55.860 --> 01:01:57.270 But we can have a discussion around 01:01:57.270 --> 01:01:59.564 is that like the right place to put it? 01:01:59.564 --> 01:02:01.830 What's the value there? 01:02:01.830 --> 01:02:04.640 Would we see a transitioning over a period of time? 01:02:04.640 --> 01:02:06.090 And any other thoughts people have 01:02:06.090 --> 01:02:08.710 about additional things we might put in the capabilities. 01:02:08.710 --> 01:02:11.173 But let me toss it open for discussion. 01:02:14.040 --> 01:02:15.570 - Maybe I'll just make two comments, 01:02:15.570 --> 01:02:16.595 they're just in reverse order. 01:02:16.595 --> 01:02:19.940 So I very much liked this position 01:02:19.940 --> 01:02:23.130 and I think it's good to see the impetus 01:02:23.130 --> 01:02:24.996 from Congress to make this happen. 01:02:24.996 --> 01:02:28.670 I think the attachment to DDS makes a lot of sense, 01:02:28.670 --> 01:02:30.730 and don't take my comment as negative any way, 01:02:30.730 --> 01:02:32.980 'cause I think there's a lot of good reason for that. 01:02:32.980 --> 01:02:33.961 I mean, we just feel go one step further, 01:02:33.961 --> 01:02:38.961 then at some point, having even more maximum visibility 01:02:39.710 --> 01:02:42.010 into the Secretary's Office. 01:02:42.010 --> 01:02:44.170 That can be while the person is in DDS or not, 01:02:44.170 --> 01:02:47.740 but given that this was conceived as a sort of a short term, 01:02:47.740 --> 01:02:50.920 a year or two before it transitions into something more, 01:02:50.920 --> 01:02:52.870 and given everything the Defense Innovation Board 01:02:52.870 --> 01:02:55.030 has been saying for a long time, 01:02:55.030 --> 01:02:58.010 that there is no more important role that has to get done. 01:02:58.010 --> 01:03:01.870 So we should think a little bit about how do we make sure 01:03:01.870 --> 01:03:04.900 that this is maximally visible across the organization? 01:03:04.900 --> 01:03:05.733 - I think it's good point. 01:03:05.733 --> 01:03:06.659 I think the other thing to think about 01:03:06.659 --> 01:03:09.540 in exactly the same regard is there are lots 01:03:09.540 --> 01:03:11.530 of personnel divisions already in the DoD 01:03:11.530 --> 01:03:13.020 that have that function. 01:03:13.020 --> 01:03:14.540 And I think part of this formative period 01:03:14.540 --> 01:03:15.960 also has to be how does it relate 01:03:15.960 --> 01:03:16.793 to all those different organizations 01:03:16.793 --> 01:03:19.180 and what does that transformation look like as well? 01:03:19.180 --> 01:03:20.560 - Without getting buried in those organizations. 01:03:20.560 --> 01:03:21.948 - Yeah precisely. 01:03:21.948 --> 01:03:23.077 - I mean and the only other comment I would make, 01:03:23.077 --> 01:03:24.580 and just follow on to what you said before, 01:03:24.580 --> 01:03:27.600 I think we can't look at this as just 01:03:28.590 --> 01:03:30.140 how do we hire the right people? 01:03:30.140 --> 01:03:32.230 Because as we have all said all along, 01:03:32.230 --> 01:03:34.370 hiring the right people without having the right tools, 01:03:34.370 --> 01:03:35.810 without having the right environment. 01:03:35.810 --> 01:03:39.790 So if you're going to make a major step move 01:03:39.790 --> 01:03:41.170 in the digital personnel, 01:03:41.170 --> 01:03:42.430 you have digital engineering personnel, 01:03:42.430 --> 01:03:43.770 whatever we want to call them, 01:03:43.770 --> 01:03:44.650 you have to have an environment 01:03:44.650 --> 01:03:46.640 where people can actually do the work. 01:03:46.640 --> 01:03:49.480 And so that means this is highly coupled to the systems, 01:03:49.480 --> 01:03:51.290 and the tools, and the capabilities that people have. 01:03:51.290 --> 01:03:55.320 And we need to be sure that the DPO has as much play 01:03:55.320 --> 01:03:57.250 in that part of the conversation as anything else. 01:03:57.250 --> 01:03:59.237 - Yeah, totally, totally agree. 01:03:59.237 --> 01:04:00.580 - The other thing Kurt, is that I understand 01:04:00.580 --> 01:04:04.080 that there may be some similar efforts being undertaken 01:04:04.080 --> 01:04:05.810 in other agencies, 01:04:05.810 --> 01:04:09.130 other US government agencies and departments. 01:04:09.130 --> 01:04:11.330 Once this is established, we should encourage 01:04:11.330 --> 01:04:14.990 that outreach to see what type of connection 01:04:14.990 --> 01:04:17.670 because there is a world beyond the DoD 01:04:17.670 --> 01:04:20.320 that we should also be understanding and connecting 01:04:20.320 --> 01:04:21.590 into if we could. 01:04:21.590 --> 01:04:24.300 - Yes, and other issues in some of the departments 01:04:24.300 --> 01:04:25.640 within the DoD around, 01:04:25.640 --> 01:04:27.910 we had the discussion earlier today around, 01:04:27.910 --> 01:04:30.650 how do you hire for particular positions? 01:04:30.650 --> 01:04:31.800 What authorities do you have 01:04:31.800 --> 01:04:33.240 that you might be able to scale? 01:04:33.240 --> 01:04:35.440 - Yeah, to some extent that this job is gonna have 01:04:35.440 --> 01:04:38.490 similar impact in other agencies as well, 01:04:38.490 --> 01:04:41.160 it's gonna look and look like the same kind of thing. 01:04:41.160 --> 01:04:43.330 Maybe not to the extent that it is in a DoD 01:04:43.330 --> 01:04:46.047 in terms of size, but it's gonna have similar impact 01:04:46.047 --> 01:04:48.160 and some may be further ahead. 01:04:48.160 --> 01:04:50.540 So we might wanna see what mistakes might have been made 01:04:50.540 --> 01:04:53.250 in organizing that might be avoided 01:04:53.250 --> 01:04:54.673 and get this going quicker. 01:04:57.850 --> 01:04:59.047 - So it seems to me that, 01:04:59.047 --> 01:05:01.146 in the first place, I support this proposal 01:05:01.146 --> 01:05:03.540 and your analysis of it. 01:05:03.540 --> 01:05:07.590 It seems to me that we are still willing around the edges 01:05:07.590 --> 01:05:09.540 of the bureaucracy. 01:05:09.540 --> 01:05:11.230 So we have the swab study, 01:05:11.230 --> 01:05:12.740 which we're busy getting implemented, 01:05:12.740 --> 01:05:15.420 we have this proposal, we're trying to get 01:05:15.420 --> 01:05:17.520 more software talent into the DoD, 01:05:18.430 --> 01:05:21.809 in one form or another through various mechanisms. 01:05:21.809 --> 01:05:25.876 I don't think it's going to happen fast enough. 01:05:25.876 --> 01:05:27.580 At the end of the day, people will say, 01:05:27.580 --> 01:05:31.800 this was a good try, but it wasn't aggressive enough. 01:05:31.800 --> 01:05:33.550 And I've been trying to think 01:05:33.550 --> 01:05:34.870 about what is the most aggressive 01:05:34.870 --> 01:05:36.047 thing that could realistically be done. 01:05:36.047 --> 01:05:37.950 And I'll give you a crazy idea, 01:05:37.950 --> 01:05:39.480 which is not under the current discussion, 01:05:39.480 --> 01:05:42.140 but maybe we should discuss at some future point, 01:05:42.140 --> 01:05:45.793 which is we have roughly five service academies. 01:05:46.870 --> 01:05:50.060 Create a sixth one and call it a Cyber Academy. 01:05:50.060 --> 01:05:52.290 When we visited the Air Force Academy, the way it worked, 01:05:52.290 --> 01:05:53.800 is you were appointed. 01:05:53.800 --> 01:05:55.550 You could go there for the first two years, 01:05:55.550 --> 01:05:56.890 and if you didn't want your commission, 01:05:56.890 --> 01:05:58.590 you could go somewhere else. 01:05:58.590 --> 01:06:00.960 But if you chose to finish your degree 01:06:00.960 --> 01:06:01.890 at the Air Force Academy, 01:06:01.890 --> 01:06:06.400 you were then signed up for four years and the usual rules. 01:06:06.400 --> 01:06:08.710 And all of a sudden, you'd have a guaranteed pipeline 01:06:08.710 --> 01:06:11.460 of people who were officers, 01:06:11.460 --> 01:06:14.640 officer level people who would be flowing 01:06:14.640 --> 01:06:18.090 through the intelligence community, the CIA, 01:06:18.090 --> 01:06:19.620 all the various hiring groups that need 01:06:19.620 --> 01:06:21.490 these kinds of talents. 01:06:21.490 --> 01:06:23.400 And if you don't like that idea, 01:06:23.400 --> 01:06:24.530 'cause we don't want more than, 01:06:24.530 --> 01:06:26.450 we've only had five and we don't to add a sixth 01:06:26.450 --> 01:06:27.800 for the last hundred years, 01:06:28.830 --> 01:06:31.440 you could create a virtual equivalent of that. 01:06:31.440 --> 01:06:33.750 That's a sourcing mechanism. 01:06:33.750 --> 01:06:36.290 But I just don't think that we're going 01:06:36.290 --> 01:06:38.550 to get enough software people 01:06:38.550 --> 01:06:42.173 without having them have some more traditional pipeline. 01:06:43.930 --> 01:06:45.920 So for example, the military does in fact, 01:06:45.920 --> 01:06:47.660 produce excellent software engineers, 01:06:47.660 --> 01:06:50.683 but it gets confused and it then assigns them to guard duty. 01:06:53.350 --> 01:06:54.970 Right? 01:06:54.970 --> 01:06:56.100 Everyone's who's a software engineer, 01:06:56.100 --> 01:06:57.500 you've done your guard duty too. 01:06:57.500 --> 01:06:58.440 I'm sure the software, 01:06:58.440 --> 01:07:01.240 you wrote software while you were doing guard duty. 01:07:01.240 --> 01:07:03.670 But there's some sort of confusion 01:07:03.670 --> 01:07:06.390 around what digital means. 01:07:06.390 --> 01:07:08.360 And so if we could give these digital people 01:07:08.360 --> 01:07:13.360 some kind of status that had meaning inside the empire, 01:07:14.200 --> 01:07:18.390 and also conferred a commitment from them to the DoD. 01:07:18.390 --> 01:07:20.470 So in other words, they can't just quit, right? 01:07:20.470 --> 01:07:23.910 So we grant your education, you have to work here, 01:07:23.910 --> 01:07:24.770 you have to work hard. 01:07:24.770 --> 01:07:27.510 And then after three or four years, you can go off. 01:07:27.510 --> 01:07:28.450 - I think it's interesting idea. 01:07:28.450 --> 01:07:30.320 I think it's some of the stuff 01:07:30.320 --> 01:07:32.580 that this Digital People Officer could kind of think 01:07:32.580 --> 01:07:34.670 through what might be a good idea. 01:07:34.670 --> 01:07:35.730 I do have reason to hope. 01:07:35.730 --> 01:07:36.835 I mean, there were things like the Jake, 01:07:36.835 --> 01:07:39.505 things stood up that are really super focused 01:07:39.505 --> 01:07:41.820 on software expertise. 01:07:41.820 --> 01:07:44.300 I think the DDS is another place. 01:07:44.300 --> 01:07:47.210 I think we're also seeing particular projects 01:07:47.210 --> 01:07:49.833 within the DoD that are very software intensive. 01:07:50.730 --> 01:07:51.700 - Just to be obnoxious. 01:07:51.700 --> 01:07:53.360 So the problem with this is we're talking 01:07:53.360 --> 01:07:55.882 about hundreds of incredibly talented people 01:07:55.882 --> 01:07:59.390 in 3 million person organization, right? 01:07:59.390 --> 01:08:01.783 I'm just looking for a scale strategy. 01:08:02.770 --> 01:08:04.790 I'm looking for some way to have this be 01:08:04.790 --> 01:08:06.620 not the exceptional conversation, 01:08:06.620 --> 01:08:07.453 but the normal conversation. 01:08:07.453 --> 01:08:08.390 Oh, yeah, they're always here. 01:08:08.390 --> 01:08:09.980 Yeah, yeah, we have to deal with them. 01:08:09.980 --> 01:08:10.813 They're always there. 01:08:10.813 --> 01:08:12.550 They're always around, there's lots of them. 01:08:12.550 --> 01:08:14.580 They have incredible value. 01:08:14.580 --> 01:08:15.870 - And I think we should take it away and think it through. 01:08:15.870 --> 01:08:16.703 - Okay. 01:08:16.703 --> 01:08:19.850 So think of ways in which you could scale literally 01:08:19.850 --> 01:08:23.440 the number of such people through traditional mechanisms, 01:08:23.440 --> 01:08:26.070 hiring processes, promotion processes, 01:08:26.070 --> 01:08:27.950 changes and more in the various strange 01:08:27.950 --> 01:08:29.660 rules that the government has. 01:08:29.660 --> 01:08:32.090 - The other thing I think, is this whole notion of I mean, 01:08:32.090 --> 01:08:33.610 in Microsoft after healthcare, 01:08:33.610 --> 01:08:34.980 I came back from healthcare.gov, 01:08:34.980 --> 01:08:37.270 we started a civic leave policy. 01:08:37.270 --> 01:08:39.220 And that basically said you can leave Microsoft 01:08:39.220 --> 01:08:40.769 for a period of time and then you can come back, 01:08:40.769 --> 01:08:42.990 and having more and more of that being something 01:08:42.990 --> 01:08:45.220 that can be done in agencies I think helps as well 01:08:45.220 --> 01:08:48.203 'cause there are a lot of people wanna help the country. 01:08:49.098 --> 01:08:50.720 - And Eric, I really like your idea. 01:08:50.720 --> 01:08:54.700 I think one of the big concerns here is continuity. 01:08:54.700 --> 01:08:57.230 And this type of program, this type of idea, 01:08:57.230 --> 01:08:58.740 whether or not it's a formal Academy 01:08:58.740 --> 01:09:01.120 or virtually provides that continuity. 01:09:01.120 --> 01:09:04.340 But even more so, the type of person we want here 01:09:04.340 --> 01:09:06.680 is we want an intellectual refresh 01:09:06.680 --> 01:09:09.750 with the age group that comes in 01:09:09.750 --> 01:09:11.500 because there is a difference here. 01:09:11.500 --> 01:09:12.830 Maybe there isn't a difference 01:09:12.830 --> 01:09:15.570 in some of the other disciplines that the academies do, 01:09:15.570 --> 01:09:17.600 but in this area, it does matter. 01:09:17.600 --> 01:09:20.057 You want to stay as current, and you want to bring along 01:09:20.057 --> 01:09:24.740 as many people in a continuous form as you can, 01:09:24.740 --> 01:09:27.450 because that's what this is gonna need. 01:09:27.450 --> 01:09:28.930 - I know it's hard online. 01:09:28.930 --> 01:09:30.115 Jen, did you have any comments you wanted 01:09:30.115 --> 01:09:34.087 to add, as an author of this paper? 01:09:34.087 --> 01:09:35.753 Anything to add? 01:09:35.753 --> 01:09:38.580 - [Jen] I just I think that Eric is exactly right, 01:09:38.580 --> 01:09:41.670 that the scale isn't quite ambitious enough 01:09:41.670 --> 01:09:44.340 for the needs of the department. 01:09:44.340 --> 01:09:47.720 I also, just sort of maybe reframe a little something 01:09:47.720 --> 01:09:50.340 that Mike I think it was, Michael said, 01:09:50.340 --> 01:09:51.700 as we started the discussion here, 01:09:51.700 --> 01:09:54.910 which is that it's not just about the recruitment. 01:09:54.910 --> 01:09:59.220 To get hundreds of thousands of great digital people 01:09:59.220 --> 01:10:01.080 into the department, we're also gonna have 01:10:01.080 --> 01:10:02.450 to have them land in positions 01:10:02.450 --> 01:10:05.450 where they can do the job they know they came here to do. 01:10:05.450 --> 01:10:08.350 And that every time we talked about 01:10:08.350 --> 01:10:12.610 every single one of these recommendations or proposals, 01:10:12.610 --> 01:10:13.580 when we get together, 01:10:13.580 --> 01:10:16.890 we talk about the need for greater speed and greater scale. 01:10:16.890 --> 01:10:19.560 And I think that it's really 01:10:19.560 --> 01:10:21.790 about those things coming together. 01:10:21.790 --> 01:10:24.400 To have people coming in at a time, 01:10:24.400 --> 01:10:26.990 if they're coming in in larger numbers, 01:10:26.990 --> 01:10:30.090 when the procurement rules are changing, 01:10:30.090 --> 01:10:32.350 the tools are changing, the ability for them 01:10:32.350 --> 01:10:34.933 to succeed when they come in is also changing, 01:10:35.930 --> 01:10:38.750 it's so important to getting them to be effective, 01:10:38.750 --> 01:10:41.241 to stick, and then to go out and say to other people 01:10:41.241 --> 01:10:43.970 in their field, the Department of Defense 01:10:43.970 --> 01:10:46.930 is the biggest place where you can have enormous impact 01:10:46.930 --> 01:10:48.380 and change the world. 01:10:48.380 --> 01:10:50.530 So I completely concur. 01:10:50.530 --> 01:10:54.060 And I want to sort of tie these threads together 01:10:54.060 --> 01:10:55.690 so that they all add up to greater 01:10:55.690 --> 01:10:57.850 than the sum of their parts. 01:11:00.300 --> 01:11:01.413 - Anybody else online? 01:11:04.290 --> 01:11:06.414 - Great, Mr. Chair. I'll go ahead and start the votes. 01:11:06.414 --> 01:11:08.933 - Yes, can we start? - Great, Missy. 01:11:09.950 --> 01:11:10.883 Do you concur? 01:11:11.870 --> 01:11:13.180 - [Missy] I concur. 01:11:13.180 --> 01:11:14.340 - Thank you. 01:11:14.340 --> 01:11:15.173 Marnie? 01:11:18.200 --> 01:11:19.040 - [Marnie] I concur. 01:11:19.040 --> 01:11:19.873 - Thank you. 01:11:19.873 --> 01:11:20.823 Jen? 01:11:22.080 --> 01:11:23.280 - [Jen] I concur. 01:11:23.280 --> 01:11:24.113 - Thank you. 01:11:24.113 --> 01:11:25.363 And then Neil? 01:11:26.660 --> 01:11:27.920 - [Neil] I concur. 01:11:27.920 --> 01:11:29.363 - Excellent, thank you. 01:11:31.632 --> 01:11:34.230 I think yes, yep. 01:11:34.230 --> 01:11:35.702 Fantastic, okay. 01:11:35.702 --> 01:11:38.760 Oh and then (laughing) board members by show of hands here. 01:11:38.760 --> 01:11:41.283 - We all concur. - Great, thank you. 01:11:44.750 --> 01:11:46.920 Fantastic, and then with that, Dr. Schmidt 01:11:46.920 --> 01:11:49.365 I'm gonna go ahead and move. - I think it's your turn. 01:11:49.365 --> 01:11:51.460 - It absolutely is, thank you. 01:11:51.460 --> 01:11:53.610 So I'm gonna go ahead and give a update 01:11:53.610 --> 01:11:56.200 to the board on progress implementing 01:11:56.200 --> 01:11:58.953 some of your recommendations within DoD. 01:12:02.290 --> 01:12:06.810 So just to give everyone a little background on this, 01:12:06.810 --> 01:12:09.310 because they are a Federal Advisory Committee, 01:12:09.310 --> 01:12:10.950 providing advice to senior leaders, 01:12:10.950 --> 01:12:13.760 they actually have no authority to implement. 01:12:13.760 --> 01:12:15.660 And so the information is provided here 01:12:15.660 --> 01:12:17.730 to inform them of actions undertaken 01:12:17.730 --> 01:12:20.340 by the department relevant to their recommendations. 01:12:20.340 --> 01:12:23.280 And updates on recommendations are included 01:12:23.280 --> 01:12:24.733 in all of our meetings. 01:12:25.830 --> 01:12:28.380 So we will move through these. 01:12:28.380 --> 01:12:30.420 So recommendation number five we've had 01:12:30.420 --> 01:12:32.970 catalyzing innovation in artificial intelligence 01:12:32.970 --> 01:12:34.113 and machine learning. 01:12:35.510 --> 01:12:38.960 As you discussed at the top of the meeting, Dr. Schmidt, 01:12:38.960 --> 01:12:41.150 the Secretary of Defense approved a set 01:12:41.150 --> 01:12:43.310 of AI Ethical Principles based 01:12:43.310 --> 01:12:45.240 on the DIB's AI Ethics Principles 01:12:45.240 --> 01:12:47.810 that will guide the department's adoption of AI. 01:12:47.810 --> 01:12:49.980 They aim to shape global norms on the development 01:12:49.980 --> 01:12:52.740 and deployment of the military AI. 01:12:52.740 --> 01:12:55.860 And the CIO through the Jake will lead implementation 01:12:55.860 --> 01:12:59.260 of these principles to ensure that DoD AI capabilities 01:12:59.260 --> 01:13:01.563 are both effective and responsible. 01:13:03.130 --> 01:13:04.563 If we could do the next slide. 01:13:05.900 --> 01:13:08.640 Recommendation number 10, making computing 01:13:08.640 --> 01:13:10.770 and bandwidth abundant. 01:13:10.770 --> 01:13:13.570 In November 2019, the Air Force announced their intention 01:13:13.570 --> 01:13:17.410 to invest 9 billion in digital and cloud architecture, 01:13:17.410 --> 01:13:20.060 dependent on funding and FY21. 01:13:20.060 --> 01:13:21.530 This will be part of an effort 01:13:21.530 --> 01:13:23.962 to create a digital Air Force. 01:13:23.962 --> 01:13:26.430 The investment is part of a new digital strategy 01:13:26.430 --> 01:13:28.080 for the Air Force that aims to field 01:13:28.080 --> 01:13:30.770 a 21st century IT infrastructure. 01:13:30.770 --> 01:13:35.000 Use data for AI to help airman make better decisions faster, 01:13:35.000 --> 01:13:36.630 and reform business practices 01:13:36.630 --> 01:13:39.103 to improve effectiveness and efficiency. 01:13:41.690 --> 01:13:44.640 Recommendation number 12, forge new approaches 01:13:44.640 --> 01:13:48.210 to data collection, sharing and analysis. 01:13:48.210 --> 01:13:50.600 The army has taken steps to increase the amount 01:13:50.600 --> 01:13:53.560 of data it collects, and to make better use of that data, 01:13:53.560 --> 01:13:55.840 both at home and in the field. 01:13:55.840 --> 01:13:58.940 For example, the Army's AI Task Force has pushed 01:13:58.940 --> 01:14:01.730 to process larger amounts of existing data, 01:14:01.730 --> 01:14:03.960 namely in the areas of talent management 01:14:03.960 --> 01:14:06.100 and predictive maintenance. 01:14:06.100 --> 01:14:08.220 The army also plans to outfit a large number 01:14:08.220 --> 01:14:10.030 of infantry soldiers with integrated 01:14:10.030 --> 01:14:12.650 visual augmentation system goggles. 01:14:12.650 --> 01:14:15.760 This will allow soldiers in the field to act as sensors, 01:14:15.760 --> 01:14:18.410 building a database of what sensor-readable events 01:14:18.410 --> 01:14:19.710 happen on the battlefield. 01:14:22.320 --> 01:14:25.250 Recommendations on refactoring the acquisition code 01:14:25.250 --> 01:14:28.180 for competitive advantage via the DIB's software 01:14:28.180 --> 01:14:30.603 acquisition and practices study. 01:14:31.440 --> 01:14:34.504 General Hyden, Vice Chairman of the Joint Chiefs 01:14:34.504 --> 01:14:36.590 of Staff has stated that one of his top priorities 01:14:36.590 --> 01:14:40.520 will be to re-inject speed into DoD's requirements 01:14:40.520 --> 01:14:42.370 and acquisition process, 01:14:42.370 --> 01:14:44.320 especially when it comes to software. 01:14:44.320 --> 01:14:46.930 His desire to have acquisition specialists 01:14:46.930 --> 01:14:49.770 write software acquisition requirements at a speed 01:14:49.770 --> 01:14:52.890 to match evolving cyber threats falls in line 01:14:52.890 --> 01:14:55.743 with the recommendations in the DIBs swap study. 01:14:58.640 --> 01:15:02.670 The next one, implementing a zero trust architecture 01:15:02.670 --> 01:15:03.613 across DoD. 01:15:04.682 --> 01:15:08.010 DoD established a Zero Trust Action Group 01:15:08.010 --> 01:15:11.580 comprised of representatives from the services DISA, 01:15:11.580 --> 01:15:16.580 NSA, US Cybercom, DoD, CIO and others 01:15:17.310 --> 01:15:19.670 to share best practices across ongoing 01:15:19.670 --> 01:15:22.750 zero trust architecture efforts in the department. 01:15:22.750 --> 01:15:25.120 The group is also meeting with the commercial sector 01:15:25.120 --> 01:15:28.143 to better understand commercial best practices. 01:15:30.320 --> 01:15:33.010 Recommendations on 5G. 01:15:33.010 --> 01:15:36.610 Last fall DoD named four military bases 01:15:36.610 --> 01:15:39.350 that would host 5G testing. 01:15:39.350 --> 01:15:41.860 Naval base San Diego in California, 01:15:41.860 --> 01:15:44.710 Joint Base Lewis McChord in Washington, 01:15:44.710 --> 01:15:46.640 Hill Air Force Base in Utah, 01:15:46.640 --> 01:15:50.110 and the Marine Corps Logistics Base in Albany, Georgia. 01:15:50.110 --> 01:15:53.900 DoD is also involved in ongoing discussions with NTIA 01:15:53.900 --> 01:15:58.100 and Congress to share hundreds of megahertz of spectrum. 01:15:58.100 --> 01:16:00.230 It has subsequently taken steps 01:16:00.230 --> 01:16:02.163 to work with industry in that regard. 01:16:04.750 --> 01:16:06.610 And then last, as we're going forward 01:16:06.610 --> 01:16:08.880 in our implementation updates, 01:16:08.880 --> 01:16:10.800 we would like to welcome the public 01:16:10.800 --> 01:16:12.670 and those in the Department of Defense 01:16:12.670 --> 01:16:15.033 to tell us your innovation stories. 01:16:16.100 --> 01:16:17.190 We can go to the next slide, 01:16:17.190 --> 01:16:19.060 and we'll post this online for you. 01:16:19.060 --> 01:16:21.390 But we'd like you to send us your stories. 01:16:21.390 --> 01:16:23.610 And then going forward, we will use those 01:16:23.610 --> 01:16:27.363 to present at our upcoming public meetings. 01:16:30.190 --> 01:16:34.840 And with that, that completes our implementation update. 01:16:34.840 --> 01:16:37.373 So I will now move into public comments. 01:16:38.310 --> 01:16:39.730 For those who are interested, 01:16:39.730 --> 01:16:42.030 we have two microphones set up here. 01:16:42.030 --> 01:16:44.200 If you have I think you filled out public comment cards 01:16:44.200 --> 01:16:45.201 when you came in. 01:16:45.201 --> 01:16:49.010 If you could hand those off to some of our staff here. 01:16:49.010 --> 01:16:50.800 I think they'll raise their hand and sort 01:16:50.800 --> 01:16:51.780 of pre-position themselves. 01:16:51.780 --> 01:16:53.520 But we welcome folks to come up 01:16:53.520 --> 01:16:56.130 and go ahead and make any public comments 01:16:56.130 --> 01:16:58.730 or remarks you would like for the board's awareness. 01:17:02.130 --> 01:17:03.013 Don't be shy. 01:17:12.530 --> 01:17:14.060 - [Eric] Hi, my name is Eric Gupta. 01:17:14.060 --> 01:17:15.760 I'm at Carnegie Mellon University. 01:17:17.000 --> 01:17:19.576 So I really, really appreciate the fact that we're creating 01:17:19.576 --> 01:17:22.990 a position to do digital recruitment. 01:17:22.990 --> 01:17:24.770 I want to emphasize that we really need 01:17:24.770 --> 01:17:25.923 to focus on retainment. 01:17:26.810 --> 01:17:28.120 I've worked a long time with the DoD now, 01:17:28.120 --> 01:17:30.770 two and a half years, which is not as long as you guys, 01:17:30.770 --> 01:17:32.213 but been around. 01:17:33.210 --> 01:17:35.410 I'd like to just kind of share a case study. 01:17:36.280 --> 01:17:39.770 One of my interns last summer, USAFA grad, 01:17:39.770 --> 01:17:42.670 masters at CMU, amazing, amazing practitioner 01:17:42.670 --> 01:17:44.570 of cybersecurity and machine learning, 01:17:46.965 --> 01:17:50.340 worked with us, won some awards, great guy. 01:17:50.340 --> 01:17:54.050 Went off to cyber school at Biloxi for six months, 01:17:54.050 --> 01:17:57.630 had a ehh time and now he's promoted 01:17:57.630 --> 01:18:00.319 to First Lieutenant and is currently 01:18:00.319 --> 01:18:03.576 stationed at his first position as an officer, 01:18:03.576 --> 01:18:07.470 and his technical talents are not being used at all. 01:18:07.470 --> 01:18:11.070 He's just a manager of enlisted men and women. 01:18:11.070 --> 01:18:13.180 He is gonna be sitting there, 01:18:13.180 --> 01:18:15.090 he's gonna be trying to figure what to do with his life. 01:18:15.090 --> 01:18:16.170 And he's gonna go through this, 01:18:16.170 --> 01:18:17.690 he's gonna go through all these different steps 01:18:17.690 --> 01:18:20.170 to yeah, you know, he can leave the military, 01:18:20.170 --> 01:18:23.090 and then go to industry and do stuff with his talents. 01:18:23.090 --> 01:18:24.350 I would really emphasize that we need 01:18:24.350 --> 01:18:26.580 to focus on retainment. 01:18:26.580 --> 01:18:28.410 So that's the officer side. 01:18:28.410 --> 01:18:31.690 Amongst the GS side, we have a lot of amazing people 01:18:31.690 --> 01:18:33.280 that I've worked with. 01:18:33.280 --> 01:18:35.570 And one of the issues is that they come to the government 01:18:35.570 --> 01:18:37.270 because they really wanna make a difference. 01:18:37.270 --> 01:18:39.780 They face issues against bureaucracy, 01:18:39.780 --> 01:18:43.290 against a lot of just a lot of mismanagement. 01:18:43.290 --> 01:18:44.830 And then just frankly, they're not paid well enough 01:18:44.830 --> 01:18:45.770 as guys compared to industry. 01:18:45.770 --> 01:18:47.930 So once they get the experience they leave. 01:18:47.930 --> 01:18:49.320 So we do have a war for talent. 01:18:49.320 --> 01:18:51.260 It's not like necessarily against foreign adversaries, 01:18:51.260 --> 01:18:53.310 it's against our own industry base. 01:18:53.310 --> 01:18:54.980 So how do we kind of balance that out? 01:18:54.980 --> 01:18:56.620 How do we not only encourage industry 01:18:56.620 --> 01:18:57.750 to work with the government, 01:18:57.750 --> 01:19:00.010 but how do we keep our talented people 01:19:00.010 --> 01:19:02.243 within the DoD, DoE, wherever? 01:19:03.200 --> 01:19:04.783 - Great, thank you. 01:19:06.071 --> 01:19:08.240 - Thank you very much. 01:19:08.240 --> 01:19:09.073 More comments? 01:19:09.073 --> 01:19:13.251 - [Colleen] Do we have anyone else? (panelists whispering) 01:19:16.210 --> 01:19:18.860 - [David] Good afternoon, I'm commander David Saw, 01:19:18.860 --> 01:19:21.990 I worked down the road with 01:19:21.990 --> 01:19:24.600 the Defense Health Agency as well. 01:19:24.600 --> 01:19:26.380 My disclaimer is, of course I don't represent 01:19:26.380 --> 01:19:28.843 my department or the Navy when I speak. 01:19:29.770 --> 01:19:33.350 But I would like to highlight just the opportunity 01:19:33.350 --> 01:19:35.350 that the board has in front of them 01:19:35.350 --> 01:19:38.010 to encourage recruitment and retention 01:19:38.010 --> 01:19:39.940 through just your presence. 01:19:39.940 --> 01:19:44.010 Up until today, I wasn't aware of the scope 01:19:44.010 --> 01:19:47.550 of this board, and what kind of problems you're solving. 01:19:47.550 --> 01:19:50.210 So when I hear you speak, you're speaking to the problems 01:19:50.210 --> 01:19:54.230 that I get angry with during my job. 01:19:54.230 --> 01:19:56.287 And so to hear you saying, 01:19:56.287 --> 01:19:58.060 "Hey, here's a possible solution" 01:19:58.060 --> 01:20:00.190 is very inspiring. 01:20:00.190 --> 01:20:02.280 Not to mention that you're all leaders in your field 01:20:02.280 --> 01:20:04.960 and people know who you are, they look up to you. 01:20:04.960 --> 01:20:07.150 So just knowing you're involved, and you want them 01:20:07.150 --> 01:20:10.030 to go do that work is a recruitment tool. 01:20:10.030 --> 01:20:13.440 So increasing your exposure in the community, 01:20:13.440 --> 01:20:16.770 and answer the active duty community as well, 01:20:16.770 --> 01:20:21.350 really helps with keeping us inspired and motivated. 01:20:21.350 --> 01:20:22.670 So thank you. 01:20:22.670 --> 01:20:23.963 - Thank you very much. 01:20:23.963 --> 01:20:26.173 More comments or questions? 01:20:28.010 --> 01:20:29.260 Yes, sir. 01:20:29.260 --> 01:20:30.760 - [Richard] Yes, my name is Richard Rodriguez. 01:20:30.760 --> 01:20:32.810 I'm the Director of Partnership Development 01:20:32.810 --> 01:20:36.100 at the National Security Collaboration Center at UTSA. 01:20:36.100 --> 01:20:36.933 I'm not sure if you've heard 01:20:36.933 --> 01:20:38.720 about this center where we're collaborating 01:20:38.720 --> 01:20:40.960 with many researchers. 01:20:40.960 --> 01:20:43.083 And recently UTSA, in the fall, 01:20:43.940 --> 01:20:45.960 did a cluster hire by artificial intelligence. 01:20:45.960 --> 01:20:47.670 And so we had faculty research, 01:20:47.670 --> 01:20:49.160 looking for research projects and maybe 01:20:49.160 --> 01:20:53.200 to aid in a timely result of the scanning. 01:20:53.200 --> 01:20:54.960 Maybe there might be some opportunity there, 01:20:54.960 --> 01:20:56.180 but I'd certainly like to speak to any one 01:20:56.180 --> 01:20:58.780 of you after this, about that. 01:20:58.780 --> 01:20:59.783 - Okay, thank you. 01:21:06.710 --> 01:21:08.110 - [Colleen] Okay, last call. 01:21:09.050 --> 01:21:09.900 - Anybody online? 01:21:11.567 --> 01:21:12.400 (panelists mumbling) 01:21:12.400 --> 01:21:13.910 - Then thank you all for your time. 01:21:13.910 --> 01:21:16.540 - Great, okay with that will close 01:21:16.540 --> 01:21:18.443 the public comment period. 01:21:18.443 --> 01:21:21.890 Dr. Schmidt, board members, I think this is a really 01:21:21.890 --> 01:21:24.300 exciting meeting with a lot of actionable items 01:21:24.300 --> 01:21:27.190 that we presented here today, 01:21:27.190 --> 01:21:29.470 that we will take back, and hopefully the department 01:21:29.470 --> 01:21:32.010 hears us and helps move some of this forward. 01:21:32.010 --> 01:21:34.420 And I'll sort of turn it over to you. 01:21:34.420 --> 01:21:35.290 - Thank you. 01:21:35.290 --> 01:21:36.920 There's a set of things that I'm really proud of. 01:21:36.920 --> 01:21:40.000 First, I'm obviously, at least speaking for myself, 01:21:40.000 --> 01:21:42.570 I've had the best time working with my colleagues 01:21:42.570 --> 01:21:45.770 on the phone and here working on these problems. 01:21:45.770 --> 01:21:47.510 We have a sense of purpose. 01:21:47.510 --> 01:21:49.100 We have a sense that it matters, 01:21:49.100 --> 01:21:51.420 and we have an audience that's listening to us. 01:21:51.420 --> 01:21:53.180 It's now up to us whether we can actually 01:21:53.180 --> 01:21:54.470 get these things to happen or not. 01:21:54.470 --> 01:21:55.370 That's the charge. 01:21:56.610 --> 01:21:59.220 Particularly, as I mentioned in my opening statement, 01:21:59.220 --> 01:22:02.070 I'm particularly proud of the ethics work, 01:22:02.070 --> 01:22:05.240 which we approved I think last cycle, 01:22:05.240 --> 01:22:09.287 and the staff working with the OSD, 01:22:09.287 --> 01:22:12.150 and so forth in a very expeditious way. 01:22:12.150 --> 01:22:14.550 Got it supported and in fact issued 01:22:14.550 --> 01:22:16.360 by the Defense Department, 01:22:16.360 --> 01:22:18.593 which sets it's a sort of historical point, 01:22:18.593 --> 01:22:22.380 it said something about morality and ethics and AI 01:22:22.380 --> 01:22:24.130 that is something I want to be associated with 01:22:24.130 --> 01:22:25.150 and I know the rest of us do 01:22:25.150 --> 01:22:27.480 because we all discussed it at great length. 01:22:27.480 --> 01:22:30.290 It's a good example of this process working. 01:22:30.290 --> 01:22:33.650 Now, for me this meeting is a bittersweet meeting, 01:22:33.650 --> 01:22:37.010 because we are in the process of losing 01:22:37.010 --> 01:22:40.610 our historic first founder Executive Director 01:22:40.610 --> 01:22:42.667 who's been replaced by an even better version 01:22:42.667 --> 01:22:43.508 (Michael laughing) 01:22:43.508 --> 01:22:44.341 but we will avoid that - That you recruited. 01:22:44.341 --> 01:22:45.174 - for the moment. 01:22:45.174 --> 01:22:50.174 That Josh recruited but Josh Marcuse, when I started, 01:22:50.710 --> 01:22:52.407 I was introduced to him and he said aww he'll be good. 01:22:52.407 --> 01:22:53.707 And I said, "Okay, well whatever. 01:22:53.707 --> 01:22:55.166 "How good can he be?" 01:22:55.166 --> 01:22:55.999 (audience laughing) 01:22:55.999 --> 01:22:58.800 The answer is he was perfect for the job. 01:22:58.800 --> 01:23:02.570 His background is in workforce development in fact, 01:23:02.570 --> 01:23:05.981 and so he had an unusual approach to trying to get people 01:23:05.981 --> 01:23:10.981 and his talent turned out to be building political support 01:23:12.770 --> 01:23:14.330 within the bureaucracy. 01:23:14.330 --> 01:23:16.150 So I can rail against the bureaucracy, 01:23:16.150 --> 01:23:19.230 while he can go build political support inside of it. 01:23:19.230 --> 01:23:21.100 Which means, as Josh will tell you, 01:23:21.100 --> 01:23:22.902 an innumerable number of meetings, 01:23:22.902 --> 01:23:27.082 because the DoD is very, very big. 01:23:27.082 --> 01:23:29.420 In addition, he used the same techniques 01:23:29.420 --> 01:23:33.500 that he developed over many years to work with Congress. 01:23:33.500 --> 01:23:36.040 So one of the reasons why our trajectory 01:23:36.040 --> 01:23:38.560 over the last five years has been so effective, 01:23:38.560 --> 01:23:41.950 is because Josh, personally knew not only 01:23:41.950 --> 01:23:44.440 how to recruit the team that has now brought 01:23:44.440 --> 01:23:46.780 what you saw today and what we had before, 01:23:46.780 --> 01:23:49.500 but also understood the politics of this. 01:23:49.500 --> 01:23:50.920 And in an organization when you're trying 01:23:50.920 --> 01:23:52.400 to change the politics, 01:23:52.400 --> 01:23:55.100 you have to have some inner supporters, 01:23:55.100 --> 01:23:57.500 you have to have a strategy of internal change 01:23:57.500 --> 01:23:58.850 and he developed that. 01:23:58.850 --> 01:24:00.600 And so if you go back to, for example, 01:24:00.600 --> 01:24:02.550 the Jake right when that was created, 01:24:02.550 --> 01:24:04.670 it didn't just happen because a general who happens 01:24:04.670 --> 01:24:06.760 to be incredibly talented decided. 01:24:06.760 --> 01:24:09.170 It's because there was a great deal of preparatory work. 01:24:09.170 --> 01:24:10.003 So what I've learned about the DoD 01:24:10.003 --> 01:24:13.260 is that the DoD works through consensus 01:24:13.260 --> 01:24:14.880 building among the staffs. 01:24:14.880 --> 01:24:16.458 And there is no better person than I've worked with 01:24:16.458 --> 01:24:18.020 than Josh Mark Hughes. 01:24:18.020 --> 01:24:20.549 Thank you Josh, for your five years of service. 01:24:20.549 --> 01:24:23.716 (audience applauding) 01:24:25.961 --> 01:24:26.837 - And since this was the public commentary... 01:24:26.837 --> 01:24:27.790 (phone attendee clapping loudly) 01:24:27.790 --> 01:24:29.170 You see, even the people (board members laughing) 01:24:29.170 --> 01:24:30.337 on the phone. 01:24:30.337 --> 01:24:31.930 And since this is a public comment comment, 01:24:31.930 --> 01:24:34.590 you're not allowed to respond to my praise. 01:24:34.590 --> 01:24:37.433 Just take it as a genuine gift from us. 01:24:38.488 --> 01:24:39.321 Colleen? 01:24:39.321 --> 01:24:40.370 - Excellent. 01:24:40.370 --> 01:24:44.770 And so I think with that, this meeting is adjourned. 01:24:44.770 --> 01:24:48.220 I think we have some event over here for folks after, 01:24:48.220 --> 01:24:50.410 but thank you very much, board members. 01:24:50.410 --> 01:24:52.420 Thank you audience members for participating, 01:24:52.420 --> 01:24:55.083 and those who were able to watch us on livestream.