The best DARPA program managers are the ones who can look at an entire literature in an area and notice a systemic bias

Exploratory Program Organization

DARPA Program managers pull control and risk away from both researchers and directors

DARPA PMs need to think for themselves, be curious, and have low ego

It is hard for a private ARPA to capture value without hamstringing impact

By definition outlier analysis can’t rely on data as comparison to other organizations

Where do ideas come from?

Who is the ARPA Model for and what do I hope to achieve

DeepMind combines bottom-up research work with top-down strike teams

Lessons learned from publishing ARPA Model

ARPA Model Conclusions

Startups and Academia are bad at addressing structural constraints

Why try to emulate DARPA and not Bell Labs

Why doesn’t ARPA use prizes instead of grants?

ARPA-E faces different industry incentives than DARPA

DARPA works on a mix of Big H and Little H hard problems

# §ARPA model
This is not meant to be a primer on *how* the ARPA model works or a history of DARPA. See references for plenty of good writing on that. It is *not* meant to be an unbiased presentation of the facts or argue for a general thesis about the world (though it hints at some.) The explicit goal of this analysis is to acknowledge that DARPA is a massive outlier and to figure out which of the attributes of DARPA contributed to its outlier results with the explicit intention of creating more organizations that can enable more outlier results.  [[§How do we get more awesome sci-fi shit?]]

The nature of outliers means that you can’t do a data-based analysis. Perhaps controversially, I think it’s foolish to try to find patterns among outliers because outliers are outliers in *different* ways except in the most broad strokes. Instead, I dig into the things that seem distinctive and ask why they might lead to outlier results.  At the end of the day, this is storytelling. But stories are powerful.  Of course, it’s easy to take this approach too far and create a story about how  *anything* weird about an outlier contributed to their success. “Steve Jobs only wore one outfit which is important because it allowed him to have EXTREME FOCUS.”  I try to avoid this by focusing on why the pieces lead to outlier performance *in combination* - many people who try to replicate outliers fail by picking and choosing the pieces they want to replicate.

There is some relevant history to deciding what to pay attention to without [[Cargo cult]]ing.

- [[The definition of the ARPA model has changed over time]]
- [[ARPA became DARPA in 1972 because of the increased scrutiny on military spending both in the government and outside of it]]
- [[Most of DARPA’s outlier output happened before 1972 when it was ARPA]]
- [[Was the shift from ARPA to DARPA a focus change or a process change?]]
Upshot:
- [[Pay attention to DARPA’s informal process and ignore formal process]]

### Program Managers
Anything written about DARPA agrees that it’s all about the program managers.
* [[DARPA Program managers pull control and risk away from both researchers and directors]]
* [[The best DARPA program managers are the ones who can look at an entire literature in an area and notice a systemic bias]]
*  [[DARPA is incredibly flexible with who it hires to be program managers]]
* [[DARPA Program managers have a tenure of four to five years]] 
* [[Why do people become DARPA Program managers?]]
* [[A large part of a DARPA program manager’s job is focused network building]] 
	* [[Maverick work is lonely and DARPA provides peers and community]] ::new::
* [[DARPA PMs need to think for themselves, be curious, and have low ego]]
*  [[The dependence of DARPA on high quality program managers mirrors the obsession with “talent” in other disciplines]] ::Updated::

### Incentives and Structure
* [[The government’s influence on DARPA is buffered by opacity and the director]]
* [[Opacity is important to DARPA’s outlier success]]
* [[DARPAs director is important]] ::needs work::
* [[DARPA does multiple levels of top-down problem generation and bottom-up solution generation]]
* [[The transient nature of most people at DARPA enables ideas to be revisited]]
* [[DARPA is relatively tiny and flat]] 
* [[DARPA Employees aren’t paid very much compared to what they could be]]
* [[DARPA’s aversion to people with a web presence may be how they avoid asymmetric career risk]]
* [[DARPA doesn’t do any research in house]] ::Updated::
* [[DARPA works on programs that go against the established paradigm]] ::new:: ::needs work::
* [[The ARPA Model is an unbuffered system]]::new::
* [[DARPA Staff are Important]] ::new:: ::needs work::
* [[Negative results are important to DARPA]]

### Funding
* [[DARPA is ~0.5% of the Department of Defense Budget and ~12% of that is basic research]]
* [[DARPA is more ideas limited than money limited]] 
* [[DARPA funds wacky things that go nowhere]]
* [[It is relatively easy for DARPA PMs to re-deploy funding]]
*  [[Program Managers have the ability to deploy money without much overhead]] 

### Process 
* [[The DARPA execution framework boils down to showing that thing is not impossible, showing that thing is possible, and then making that thing possible]] 
* [[There are many tight feedback loops built into the ARPA model]] 
*  [[The initial exploratory tranche of a DARPA program is approximately $1.5m]]
*  [[DARPA PMs use seedling programs to ‘acid test’ the riskiest pieces of a program idea]]
* [[The ‘framework’ DARPA PMs use to create a program is by modeling the tech council]]
* [[Every program at DARPA is intensely technically scrutinized by the tech council]] 
* [[DARPA facilitates idea sex both between PMs and Performers]]
* [[DARPA provides a derisking role for people in other organizations]]
* [[Decoupling from market discipline is like cave diving]] ::new:: ::Needs work::
* [[DARPA works where markets dont]] ::New::

### Riffing on ARPA
- [[A handful of organizations have riffed on the ARPA Model with mixed success]]::New::
	- Answers:  [[How does IARPA work? What are its successes?]]
* [[It’s easy to set out to build a DARPA but end up building a Skunkworks]]
* [[Most DARPA clones start too big or with heavy process]]
* [[Can the ARPA model exist outside of the government?]]

### Closed Questions
-  [[How successful did DARPA look ten years in?]]
	- Pretty successful

## Open Questions
Because a lot of this is just storytelling there are a lot of open questions about different knobs, what happens when you turn them and what they mean in different domains.  

* [[When is externalized research better than internalized research and vice versa?]]
*  [[Is it essential that a research org is a small fraction of the money factory’s budget?]]
*  [[Why doesn’t ARPA use prizes instead of grants?]]
* [[Is there a minimum effective budget to generate outlier results?]]
- [[How could the ARPA Model take advantage of the internet?]]

## Conclusions
* [[ARPA Model Conclusions]]
* [[Things about ARPA that can be improved]]

## Related

## References
* [[Funding Breakthrough Research]]
* [[“Special Forces” Innovation: How DARPA Attacks Problems]]
* [[Sam Moleneux convo 16/1/20]]
* [[Sam Moleneux Conversation 31/10/2019]]
* [[Know When to Fold ‘Em: An Empirical Description of Risk Management in Public Research Funding]]
* [[What makes DARPA tick]]
* [[What else makes DARPA tick]]
* [[Cloning DARPA Successfully - Article]]
* [[fuchsRethinkingRoleState2010]]
* [[Mark Micire Podcast]]

 
          
[Web URL for this note](http://notes.benjaminreinhardt.com/§ARPA_model)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/§ARPA_model)




§PARPA

ARPA Does Windows - The Defense Underpinning of the PC Revolution

What are the specific tactics for PARPA fundraising?

Ways an ARPA-riff could improve on Things about DARPA that can be improved

weaverNSNotesOfficer1946

A handful of organizations have riffed on the ARPA Model with mixed success

ICT Distillation

An Institute for Creative Technology - Riffing on ARPA

ICT Outline

# Anybody attempting to emulate DARPA needs to be willing to sit with a lot of discomfort for a long time
Heuretics enabled by DARPA take a long time to get out into the world and the attribution is usually mixed because of DARPA’s role as a creator of fields and not as a commercializer of technology. 

While there were intermediate products, the [[Mother of All Demos]] happened in 1968 - six years after [[JCR Licklider]] joined DARPA and it took another five years after that to make its way into a commercial product in the Xerox Alto and arguably another ten years after that to actually have a massive impact in the Macintosh. 

The [[DARPA grand challenge]] that arguably kicked off autonomous driving was in 2006 and of this writing in 2020, autonomous vehicles are still not mainstream. 

Adding to the uncertainty is the fact that a big part of DARPA’s role is to get the ball rolling, not get it to the goal. This role is important but muddles attribution. You see this historically in how it seeded some of the best CS departments in the country and how often its money is used to derisk research enough that the NSF is willing to fund it.  ([[DARPA provides a derisking role for people in other organizations]].)  Almost by design there will be arguments about who deserves credit for things DARPA was involved in. Any organization following that path will suffer the same problem.

On top of uncertainty, there is going to be discomfort because trying to emulate ARPA will entail working on *weird shit*. [[DARPA funds wacky things that go nowhere]]. Over time, DARPA directors have needed to fight to keep DARPA from being `toned down` and turned into a normal R&D org. 

It is very easy to say “we accept failure” in theory but in practice it’s really uncomfortable. 

### Related
- [[How successful did DARPA look ten years in?]]
- [[How do you motivate people via Kleos if they can’t take credit for their work?]]
- [[Credit actually matters a lot because its the difference between someone writing a blog on the side and getting to do it as their job]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Anybody_attempting_to_emulate_DARPA_needs_to_be_willing_to_sit_with_a_lot_of_discomfort_for_a_long_time)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Anybody_attempting_to_emulate_DARPA_needs_to_be_willing_to_sit_with_a_lot_of_discomfort_for_a_long_time)




§ARPA model

Anybody attempting to emulate DARPA needs to be willing to sit with a lot of discomfort for a long time

Visions of ARPA

# How successful did DARPA look ten years in?
# How successful did DARPA look ten years in
Pretty fucking successful - started in 1958 and you had The mother of all Demos in 1968 with the first mouse, shakey the robot, the first space-based navigation satellite, and first weather satellite. 



[Web URL for this note](http://notes.benjaminreinhardt.com/How_successful_did_DARPA_look_ten_years_in)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/How_successful_did_DARPA_look_ten_years_in)




Government procurement and prizes are the same thing

# Visions of ARPA
- Hoover I
- States that ARPA became DARPA in 1996 
`Pushing the Frontiers of Basic Science to solve a well defined use-inspired need`

### Policy Exchange Recommendations
1. Government should move rapidly to establish UK ARPA
2. Government should clearly set out role, objectives, and mission of ARPA
3. Government should get bipartisan support for ARPA
4. ARPA should not execute research or have its own labs
5. ARPA should focus on developing technologies on a 10-15 year horizon
6. ARPA should be judged on successes not rate of failure
	- Ruthlessly cut off funds for failing projects
7. Small number of missions
	- 2-4 missions
	- Reviewed by minister of science every five years
8. Project managers must be capable, empowered, and given genuine authority
9. Minimal Bureaucracy
	- Project managers should need to convince 2 people
10. Government must also develop commercial pull-through to bring ARPA tech to market
	- Reform of public sector procurement
	- Single strategic customer for an area
	- Establishment for tech to market group
- Structure
	- Offices (2-4)
		- Projects (5-10 per office) 
			- Grant Allocations ￡500k-5m per grant
### A Brief History of the UK Science and Innovation System
- Department for Scientific and Industrial Research established in 1916
- DSIR didn’t raise level of industrial research in UK 
	- [[What can be learned from government and research policy in the United Kingdom, 1914-1965]]
- Medical research council formed in 1920
- Agricultural research council formed in 1931
- “Loss” of penicillin - it was discovered in the UK but commercialized in the US
- National Research Development Corporation in ~1950 to commercialize discoveries from publically funded ideas to 'prevent foreigners from filching ideas’
- Minister for science 1959
- Ministry of Supply (which handled purchase of military equipment) and Ministry of Aviation controlled several laboratories. 
- Ministry of Technology formed in 1964 
	- Took over Atomic Energy Authority, NRDC and parts of DSR
	- Dismantled in 1970
- Lord Rothschild pushed customer/contractor principle
- Labour returned to office and set up National Enterprise Board in 1974
	- Created Inmos semiconductor and Celltech biotech
- Thatcher in 1979 sold off Inmos, and Celltech
- NEB+NRDC -> private British Technology Groups - got first refusal rights on discoveries coming from publically funded research
	- NRDC failed to patent monoclonal antibodies
- Thatcher approach was to use public money only for basic research
- Research Assessment Exercise/Research Excellence Framework created to make sure research directed towards areas where "peer review established research as outstanding”
- Policies of 1960’s/1970’s “failed”
- 1997 tried to figure out how to create more university-private linkages (p16)
- 2013 government moved more towards helping tech get from lab to marketplace (p17)
- Put Research Councils and Innovate UK into a new org - UK Research and Innovation (UKRI) (p18)
- UKRI focus on
	- Healthcare and Medicine
	- Robotics and AI
	- Batteries
	- Self driving vehciles
	- Manufacturing and materials 
	- Satellites and Space
- Unclear whether support for this system will continue

### Lessons from US Science and Innovation System
- 1969 R&D in Germany+Japan+UK+France = $11.3B
- 1969 R&D in US = $25.6B
- US has porous boundaries between universities and industry
- More antimonopoly in US
	- Cites IBM software unbundling
- Funds to VC increased after 1979 when government pension funds could invest
- NASDAQ was more suited to young companies
- Uncoordinated government agencies help fund in US
- DARPA initial funding for networking had no obvious commercial payoff (p22)
- Trump administration wanted to dismantle ARPA-E
- Balance between freedom and accountability - as more freedom is given to individuals, the talent of the individuals matters more.

### The government’s proposals
- “new funding agency broadly modeled on ARPA”
- ￡800 over five years
Less formally
- Offer academics funding >10 years 
- Attributeds idea to [[Dominic Cummings]]

### A summary of the DARPA Model
- Cites [[Science: The Endless Frontier]] as creating the ‘Pipeline Model' for science
- Technology push/technology supply model
- [[The (idea) valley of death]] between ‘front end’ - university/R&D etc and 'back end' - usually private.
- Describes DARPA as an intermediary 
- Says DARPA passes off research
- DARPA uses a Technology visioning process
- Looks for revolutionary breakthroughs 
- Principles From 39. DARPA (2005), ‘DARPA - Bridging the Gap, Powered by Ideas,’ DARPA: Arlington, Va. ; DARPA (2003), ‘DARPA Over the Year (p29)
	- Small and flexible 100 program managers and office directors
	- Flat 
	- Entrepreneurial 
		- Program managers are there only 3-5 years
	- No Labs
	- Focus on impact not risk
	- Seed and Scale - funding to a bunch of different things 
	- Autonomy from civil service hiring process
	- Hybrid model - taps firms and universities
	- Teams and networks - tries to create great groups
	- Acceptance of failure
	- Orientation to revolutionary breakthroughs in a connected approach
- None of those principles are written down
- Catalyze formation of great groups - focus on people as innovators and connect them
- Apply the Island/Bridge Model - put innovation entity on an island and protect from 'suits'
	- 	Cites [[Lockheed Skunkworks]], [[Xerox PARC]], and [[IBM PC Project]]
	- Skunkworks and PC Project had strong bridges, PARC didn’t
	- DAPRA uses connections to Secretary of defense as bridge
- Create a community of people both inside the organization and outside of it that creates legacy of thought
	- Cites [[J.C.R. Licklider]]
- Connecting Technologists and operators 
	- DARPA TTO stays in constant contact with DoD to make sure inventions are relevant
- Transitioning technology
	- DARPA can build directly into the DoD procurement process
	- Claims DARPA computer tech was originally civilian and supported by VC
	- Claims VC is not working for “hard tech” that requires more manufacturing or 
	- ARPA-E created ‘tech to market’ group
		- Tech to market group tried things like ‘creating alliances with established companies' and 'follow on funding
- IARPA had >70% of projects that passed midpoint be adopted by an intelligence agency

### ARPA: A critical addition to our innovation landscape
- ARPA should aid transition for ideas from lab to market
- Network of ‘Catapults’ working closely with industry
- ‘Technology pull’ comes from big companies
- Reasons industry hasn’t picked up research from universities
Communication related
	- Industry tends to focus on higher TRLs and engages less with science
	- Academics are not motivated or rewarded for having ideas applied
	- People who think broadly about where research could go are not rewarded in the current systems
Risk, Customer, and Industrial Landscapre related
	- Hard to find funding for high risk ideas with significant capital costs to demonstration
	- UK doesn’t have supply chain in some areas like materials
	- No market for some ideas
- - Henry Royce Institute is UK’s national institute for advanced materials
- ARPA doesn’t need to stimulate good research
- ARPA needs to ‘pick winners and take risks’ 
- Things an ARPA will need
	- Great program managers
	- Few constraints
	- Be able to risk public money
	- Concept of a customer
	- Universities that can deal with tranches of funding that can stop
- Places an ARPA might be effective
	- Graphene
	- Photonic materials
	- Photon multiplying materials 
(Energy trnaisiton, clean water, sensing and monitoring)
- “Crown shaped molecules to sieve CO2 out of the air”

### UK ARPA: An experiment in science policy? 
#### Author
- [[Richard Jones]]
- Science funding is based on one of three philosophies
	1. Competitive funding for projects - groups propose work to be done and is judged by other scientists
	2. Funding for people
	3. Funding for program managers/leaders
- ARPA customer shouldn’t be academia or industry
- Contract R&D labs like [[SRI International]]don’t exist in the UK
- ARPA should not be part of UKRI

### ARPA’s place in the UK’s Public Research Landscape
- ARPA should be part of UKRI

### Navigating high scientific risk and high innovation risk to reap the rewards of ARPA funding
- Getting science funding is inefficient 
- Funding bodies don’t want to take on risk
- Claims ARPA-E has been less successful with commercialization
- Areas that should be funded: clean growth, AI, healthy aging, future mobility
- Could provide funding to early researchers who want to work on crazy things
- Public procurement is too afraid to take risks
- Focus on more local procurement

### Designing a British DARPA
- Estimate that 85% of DARPA projects don’t work
- Lots of stuff 

### When failure is the key to success
- Need to make the org risk tolerant 
- Kill projects that aren’t doing well and ones that aren’t going fast enough
- Mission is important
- Key pieces - focus, long-term backing, high failure tolerance, sufficient funding
- Where does the customer come from?
- Mandate that 5% of public sector budgets need to be spent on innovative solutions
- Public procurement as an innovation driver

## Questions
- Do people actually know what they’re saying about commercialization or is it a [[Suitcase Handle Word]]
- [[How successful did DARPA look ten years in?]]
- How much of the structure that we analyze and praise about DARPA was actually in place during the time period when DARPA created the outputs that we praise?
- Why is it necessarily true that markets like energy take a long time? Didn't electrificatin happen privately and 
- Does Royce institute have a list of companies that want advanced materials and which ones they want?
- Would researchers accept grants with “no go” gates?
- How much funding do you *actually* need
- What if companies would spend more on new solutions
## Thoughts
- Factually incorrect that ARPA became DARPA in 1996
- Could you measure burocracy by the average number of people you need to convince/number of people who need to look at something to get permission for something?
- The review of British policy never talks about WHAT WORKED
- General tech areas are bullshit
- Uncoordinated government agencies leading to good research is related to [[More Uncorrelated Mistakes lead to more success]] 
- There is something valuable in an organizations desire for self-preservation (contra in the government how it can just be killed at will)
- They say that DARPA is an intermediary bridging valley of death but it's really not
- Maybe should use revolutionary breakthroughs instead of [[Quantum Leaps, Innovations, and Discrete Improvements]]
- Island/Bridge Model seems to be the same as [[Separate soldiers and scientists]]+ [[Manage the Transfer not the Technology]]
- Bridges need to be back to senior people in the production org
	- This person needs to be actually involved in the bridging not generally handwaving
		- This model is how old school pharma companies work
- The concept of “Ideas stuck in lab” is lazy thinking. The reason ideas are "stuck in labs" is because they weren't created to ever leave the lab. 
- So fucking little conviction about the things that should be funded without the balls to say "leave it up the porgram managers"
- [[“Failure is good” is bullshit]]

## Related
## Authors
Editors
- [[Ian Mansfield]]
- [[Geoffrey Owen]]
- [[William Bonvillian]]
- [[Julia King]]
- [[Richard Jones]]
- [[Jo Johnson]]
## Link
https://policyexchange.org.uk/wp-content/uploads/Visions-of-Arpa.pdf

[Web URL for this note](http://notes.benjaminreinhardt.com/Visions_of_ARPA)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Visions_of_ARPA)






Benʼs working notes

How successful did DARPA look ten years in?

How successful did DARPA look ten years in

Links to this note