The best DARPA program managers are the ones who can look at an entire literature in an area and notice a systemic bias

Exploratory Program Organization

DARPA Program managers pull control and risk away from both researchers and directors

DARPA PMs need to think for themselves, be curious, and have low ego

It is hard for a private ARPA to capture value without hamstringing impact

By definition outlier analysis can’t rely on data as comparison to other organizations

Where do ideas come from?

Who is the ARPA Model for and what do I hope to achieve

DeepMind combines bottom-up research work with top-down strike teams

Lessons learned from publishing ARPA Model

ARPA Model Conclusions

Startups and Academia are bad at addressing structural constraints

Why try to emulate DARPA and not Bell Labs

Why doesn’t ARPA use prizes instead of grants?

ARPA-E faces different industry incentives than DARPA

DARPA works on a mix of Big H and Little H hard problems

# §ARPA model
This is not meant to be a primer on *how* the ARPA model works or a history of DARPA. See references for plenty of good writing on that. It is *not* meant to be an unbiased presentation of the facts or argue for a general thesis about the world (though it hints at some.) The explicit goal of this analysis is to acknowledge that DARPA is a massive outlier and to figure out which of the attributes of DARPA contributed to its outlier results with the explicit intention of creating more organizations that can enable more outlier results.  [[§How do we get more awesome sci-fi shit?]]

The nature of outliers means that you can’t do a data-based analysis. Perhaps controversially, I think it’s foolish to try to find patterns among outliers because outliers are outliers in *different* ways except in the most broad strokes. Instead, I dig into the things that seem distinctive and ask why they might lead to outlier results.  At the end of the day, this is storytelling. But stories are powerful.  Of course, it’s easy to take this approach too far and create a story about how  *anything* weird about an outlier contributed to their success. “Steve Jobs only wore one outfit which is important because it allowed him to have EXTREME FOCUS.”  I try to avoid this by focusing on why the pieces lead to outlier performance *in combination* - many people who try to replicate outliers fail by picking and choosing the pieces they want to replicate.

There is some relevant history to deciding what to pay attention to without [[Cargo cult]]ing.

- [[The definition of the ARPA model has changed over time]]
- [[ARPA became DARPA in 1972 because of the increased scrutiny on military spending both in the government and outside of it]]
- [[Most of DARPA’s outlier output happened before 1972 when it was ARPA]]
- [[Was the shift from ARPA to DARPA a focus change or a process change?]]
Upshot:
- [[Pay attention to DARPA’s informal process and ignore formal process]]

### Program Managers
Anything written about DARPA agrees that it’s all about the program managers.
* [[DARPA Program managers pull control and risk away from both researchers and directors]]
* [[The best DARPA program managers are the ones who can look at an entire literature in an area and notice a systemic bias]]
*  [[DARPA is incredibly flexible with who it hires to be program managers]]
* [[DARPA Program managers have a tenure of four to five years]] 
* [[Why do people become DARPA Program managers?]]
* [[A large part of a DARPA program manager’s job is focused network building]] 
	* [[Maverick work is lonely and DARPA provides peers and community]] ::new::
* [[DARPA PMs need to think for themselves, be curious, and have low ego]]
*  [[The dependence of DARPA on high quality program managers mirrors the obsession with “talent” in other disciplines]] ::Updated::

### Incentives and Structure
* [[The government’s influence on DARPA is buffered by opacity and the director]]
* [[Opacity is important to DARPA’s outlier success]]
* [[DARPAs director is important]] ::needs work::
* [[DARPA does multiple levels of top-down problem generation and bottom-up solution generation]]
* [[The transient nature of most people at DARPA enables ideas to be revisited]]
* [[DARPA is relatively tiny and flat]] 
* [[DARPA Employees aren’t paid very much compared to what they could be]]
* [[DARPA’s aversion to people with a web presence may be how they avoid asymmetric career risk]]
* [[DARPA doesn’t do any research in house]] ::Updated::
* [[DARPA works on programs that go against the established paradigm]] ::new:: ::needs work::
* [[The ARPA Model is an unbuffered system]]::new::
* [[DARPA Staff are Important]] ::new:: ::needs work::
* [[Negative results are important to DARPA]]

### Funding
* [[DARPA is ~0.5% of the Department of Defense Budget and ~12% of that is basic research]]
* [[DARPA is more ideas limited than money limited]] 
* [[DARPA funds wacky things that go nowhere]]
* [[It is relatively easy for DARPA PMs to re-deploy funding]]
*  [[Program Managers have the ability to deploy money without much overhead]] 

### Process 
* [[The DARPA execution framework boils down to showing that thing is not impossible, showing that thing is possible, and then making that thing possible]] 
* [[There are many tight feedback loops built into the ARPA model]] 
*  [[The initial exploratory tranche of a DARPA program is approximately $1.5m]]
*  [[DARPA PMs use seedling programs to ‘acid test’ the riskiest pieces of a program idea]]
* [[The ‘framework’ DARPA PMs use to create a program is by modeling the tech council]]
* [[Every program at DARPA is intensely technically scrutinized by the tech council]] 
* [[DARPA facilitates idea sex both between PMs and Performers]]
* [[DARPA provides a derisking role for people in other organizations]]
* [[Decoupling from market discipline is like cave diving]] ::new:: ::Needs work::
* [[DARPA works where markets dont]] ::New::

### Riffing on ARPA
- [[A handful of organizations have riffed on the ARPA Model with mixed success]]::New::
	- Answers:  [[How does IARPA work? What are its successes?]]
* [[It’s easy to set out to build a DARPA but end up building a Skunkworks]]
* [[Most DARPA clones start too big or with heavy process]]
* [[Can the ARPA model exist outside of the government?]]

### Closed Questions
-  [[How successful did DARPA look ten years in?]]
	- Pretty successful

## Open Questions
Because a lot of this is just storytelling there are a lot of open questions about different knobs, what happens when you turn them and what they mean in different domains.  

* [[When is externalized research better than internalized research and vice versa?]]
*  [[Is it essential that a research org is a small fraction of the money factory’s budget?]]
*  [[Why doesn’t ARPA use prizes instead of grants?]]
* [[Is there a minimum effective budget to generate outlier results?]]
- [[How could the ARPA Model take advantage of the internet?]]

## Conclusions
* [[ARPA Model Conclusions]]
* [[Things about ARPA that can be improved]]

## Related

## References
* [[Funding Breakthrough Research]]
* [[“Special Forces” Innovation: How DARPA Attacks Problems]]
* [[Sam Moleneux convo 16/1/20]]
* [[Sam Moleneux Conversation 31/10/2019]]
* [[Know When to Fold ‘Em: An Empirical Description of Risk Management in Public Research Funding]]
* [[What makes DARPA tick]]
* [[What else makes DARPA tick]]
* [[Cloning DARPA Successfully - Article]]
* [[fuchsRethinkingRoleState2010]]
* [[Mark Micire Podcast]]

 
          
[Web URL for this note](http://notes.benjaminreinhardt.com/§ARPA_model)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/§ARPA_model)




Graduate students are the labor in academic science and engineering

People giving out grants try to derisk them as much as possible

Manufacturing is important and often ignored

The word ‘novel’ is used as an idea bludgeon in academia

All the incentives in research drive towards conservatism

Peer review doesn’t catch duplications

Bell Labs is one of the few examples of an innovation org that was aligned with its money factory

A new professor needs about $1.5m to get going

Academic culture prizes individual recognition

Younger professors have to spend more time getting money

Academics are incentivized to do peer-reviewed publishable work

Things that are not paper-worthy enough for academia and not product-focused enough for a startup

Pay-it-forward tithing could resolve the tension between the potential public-good and potential profit in research.

The idea that academia is for crazy exploration and companies are for low risk exploration is bullshit

Three myths about scientific peer review

A PI needs to provide evidence that they have access to a lab facility

Different axes of where research gets done in the US

What was the real incentive for people to work on the DARPA Grand Challenge?

It is easier to have a multipoint mutation in technology than in evolution

NS Notes on Officers Techniques

Necrobotics illustrates how academia forces engineering

The boundaries between industry government and academia are fuzzy

The Architected Innov􏰂ation Institute - Engineering Scientific Revolutions

There are very few senior individual contributors in academia

Is it more impactful to find and train undiscovered diamonds or to unlock well-positioned but constrained researchers?

Wolfgang Merkt conversation 12 Oct 2021

An institution riffing on the ARPA Model could enable paradigm shifts that are not happening

Academic administrators and professors are playing very different games

An Institute for Creative Technology - Riffing on ARPA

Funding structures lead to several layers of indirection on what even gets proposed

Is it possible that we’re actually getting less research because we have more researchers?

Running list of relevant science news?

# §Academia Constraints
- [[Academia is the game where you gain status by getting attention for new knowledge]]

- [[Universities have multiple conflicting missions]]
- [[Universities have been pushed more towards applied research since the end of WWII]]
- [[Academic credit is a first-to-the post system]]
-  [[Graduate students are the labor in academic science and engineering]]
- [[People giving out grants try to derisk them as much as possible]]
- [[Professors do three or more jobs]]
- [[Successful professors are not necessarily the best researchers]]
- [[University professorships are the highest status jobs in academia]]
- [[National labs and other government research centers are a research alternative to universities]]
- [[The game in national labs is roughly the same as academia but with different constraints]]
- [[In academia more grant money leads to more power]]
- [[Academic research is often just the concept half of creating a heuretic]]
- [[Professors are constantly doing the calculation of whether a grant is worth the time to apply for it]]
- [[Most grant money is tagged so unrestricted money is like gold for professors]]
- [[Universities can take more than half of grant money as overhead]]
- [[A PI needs to provide evidence that they have access to a lab facility]]
- [[Physical experiments are expensive]]
- [[Younger professors have to spend more time getting money]]
- [[The government grant process depends on politics and committees]]
- [[Any technology that comes out of a university needs to go through their tech transfer office]]
- [[Months of grant writing is one of the things that makes people think twice about academia]]
- [[Academia incentivizes novelty, not focus]]
- [[Academia is not a good environment for systems engineering]]
- [[Academics are incentivized to do peer-reviewed publishable work]]
- [[Going outside of the approved process can be a black mark against people in academia]]
- [[People proposing work that they have already done is a sign that feedback loops are broken]]
- [[Necrobotics illustrates how academia forces engineering]]
- [[Labs implicitly judge research through the sense of dollars per paper]]

### Related
- [[§Startup Constraints]]
- [[§Corporate R&D Constraints]]
- [[§Philanthropy and Non-Profit Constraints]]
- [[Market Failures in Science]]

[Web URL for this note](http://notes.benjaminreinhardt.com/§Academia_Constraints)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/§Academia_Constraints)






§Academia Constraints

Decoupling from market discipline is like cave diving

Current value capture mechanisms are crude

When should an idea that smells like research be a startup?

§Corporate R+D Constraints

Startups are good at point changes

Systemic decay theory of stagnation

Ideally corporate R+D enables work on a general purpose technology before it’s specialized

There are many tight feedback loops built into the ARPA model

One function of a private ARPA is to fill in the gap once filled by corporate R+D

There are many important innovations that are just bad investments

Startup Constraints - essay

The market in for-profit organizations is fairly efficient

Profit-maximizing organizational structures can hamstring technologies whose impact depends on them

PARPA could walk the line between profit and non profit

Startups  need to spend effort on sales, marketing, and productization

# §Startup Constraints
- [[Startups need to make a lot of money or have a lot of users]]

- [[Startups need to conform to investor timescales]]
- [[Venture capital doesn’t work for things that only realize their value on long timescales]]
- [[Startups need an exit strategy]]
- [[Startups need a great story]]
- [[Startups need to produce a scalable product]] 
- [[Startups need to break development into correctly sized stages]]
- [[Startups need specific growth curves]]
- [[If a project has massive possible upside and low enough risk it is venture fundable]]
- [[Scalability and venture fundability happens when you turn a process innovation into a tool innovation]]
- [[Startups are good at point changes]]

- [[Startup Constraints - essay]]

[Web URL for this note](http://notes.benjaminreinhardt.com/§Startup_Constraints)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/§Startup_Constraints)






Peer review has failed as an institution

Human capital theory of stagnation

People are incentivized to work on nodes higher up the tech tree

Peer review gives peers veto power over each other

# Academics are incentivized to do peer-reviewed publishable work
Peer review published papers both determine an academics livelihood and status. Therefore, they are incentivized to spend their time doing work they think can be published. 

On top of this [[Academic credit is a first-to-the post system]]

[[People and organizations are all playing some game that has different ways of gaining status and power]] In Academia there are a couple of long term games:
- Gain [[Kleos]] through contribution to the [[Knowledge frontier]]
- Just get to scratch your itch to investigate certain things
- ….

[[People create proxy games for long term or hard-to-measure games]] and in academia the entwined proxy games are:
- Peer reviewed Published papers
- Citations on those papers
- Grants (which are usually based on papers) 
- Awards (which are usually based on who knows you which comes from papers)

It’s a safer strategy to publish lots of papers than to try to write one super-cited paper because it’s very hard to know which papers will be highly cited. Additionally everybody is impressed when you say “I’ve published a bajillion papers” while only a subgroup might know your super-cited paper by title unless you’ve [[hit the top of the columnade]]. 

Tenure is based on the number of papers you’ve published and the amount of grant money you’ve brought in.  Without tenure, academia is extremely rough. People who don’t think they can get tenure leave universities. [[National labs and other government research centers are a research alternative to universities]].

[[The peer review and citation system incentivizes people to work on things that other people think is interesting]]

### Related
- [[§Academia Constraints]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Academics_are_incentivized_to_do_peer-reviewed_publishable_work)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Academics_are_incentivized_to_do_peer-reviewed_publishable_work)




Corporate R+D orgs are expensive

# Problems that require system-level solutions often require massive teams
This is especially true when you’re talking about systems *within* an organization - you need to replace-in-place, backwards compatibility etc.
This may be one of the [[§Corporate R+D Constraints]] and a reason why [[Corporate R+D orgs are expensive]].
From [[Conversation with apple research guy 12/2018]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Problems_that_require_system-level_solutions_often_require_massive_teams)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Problems_that_require_system-level_solutions_often_require_massive_teams)




DARPA has a mixed record on transitioning

When is externalized research better than internalized research and vice versa?

DARPA doesn’t do any research in house

The higher a TRL the more internalized research should be

Externalized research is a buffer against B players

Ways an ARPA-riff could improve on Things about DARPA that can be improved

It’s easy to set out to build a DARPA but end up building a Skunkworks

What is the role of organizational boundaries?

§Axes for Looking at Funding and Organizations

Paul Merolla conversation 4 Oct 2021

Executing on multiple paths towards the same goal is hard within an organization

Key differences between FROs and different research institutional structures

# R+D orgs lie on a spectrum between externalized and internalized execution and benefits
Externalized research is done by giving grants or contracts to external organizations. Internalized research is done on equipment owned by the organization by employees of that organization. In reality, most organizations do a combination of both.  DARPA is on the extreme externalization end. [[DARPA doesn’t do any research in house]].  [[DOE National Labs]] are probably near the middle - doing a lot of work but then collaborating heavily with university labs.  Corporate R&D are more towards the right end of the spectrum. 

The benefits of the work can be captured externally or internally as well. This creates a nice 2x2:
* Internal execution, internal benefit: most companies and R&D labs
* Internal execution, external benefit: contractors
* External execution, internal benefit: a company hiring contractors
* External execution, external benefit: a coordinating org like DARPA

Internalization involves hiring people you want to work with, while externalization involves contracting or issuing grants. This dichotomy is one way you could conceivably measure degrees of externalization vs internalization - money spent outside vs inside an org. It almost smells like balance of trade calculations for countries. 

### Ways organizations externalize research
1. Rent time on equipment along with skilled operators
2. Get specific things done by cloud labs like [[Emerald Cloud Lab]]
3. Hire contractors to do things like build software, electronics 
4. Hire contractors to do larger parts of a project like “figure out how to do X”
5. ‘Collaborate’ with other organizations (which in reality usually means co-applying for a grant and dividing up the work)
6. Give out grants to do entire research projects
7. Prizes - everything from the [[X-Prize]] to platforms like [[Innocentive]]

Most of these externalization methods work through a price mechanism (1,2,3,~4,~6,7) and some of them do not (5.) Looking at these methods raises another aspect of externalization - the degree to which you’re externalizing conception ([[Conception is the imaginative design of a heuretic]] ) and/or implementation ([[Implementation is the actualization of the heuretic in the real world]].) Roughly, the amount of externalized conception increases with the method number - renting time on equipment involves zero conceiving of new ideas while giving out a prize means you don’t even know what the idea is until you see it in action. 

### Advantage of externalization
- [[There are many pieces of equipment or tacit knowledge that only exist  in one or two places in the world]]
- You don’t have to make long term commitments to someone
- You don’t need to buy equipment you might need for only one project
- Externalization minimizes politics, which allows you to do things like pursue parallel approaches to the same thing. 
- You don’t risk having people with specialized skills around who need to be retrained or let go
	- [[Alex Rider]] pointed out that this happens at [[SRI International]] quite a bit

### Advantages of internalization
- Internalization can reduce [[Transaction costs]]. [[Robert Coase]] goes into more detail but you don’t need to haggle over the price of every piece of work, it’s easier to coordinate because you are in the same building (or at least you were before coronavirus), and you presumably all have an incentive to see the project succeed as a whole instead of just your little part. 
- Its easier to change direction quickly.
- You don’t have to specify interfaces as clearly.
- Tacit knowledge/[[Process Knowledge]] created during the project stays with the organization.
- If you need to maintain something, you keep the people who can do it around. (More [[Process Knowledge]].) 
- Nominally people have more buy-in to an organization, so they are more likely to go above and beyond the call of duty
- Taxes?
- People you’re working with become more known quantities - you need to do less vetting.
	- This becomes more important the smaller the units of work become. 
- You’re effectively playing repeated games with people so there is less incentive for them to screw you over.
- It’s easier to capture value from the research
	- You can immediately start commercializing 
	- You don’t need to police an external entity to see whether they’ve created something valuable 
	- External entities may want to own IP
	- If the value of the research is spread over several organizations it may be difficult to coordinate them. Each one individually may think their contribution is more valuable than an equitable division of value could handle. 
	- IP is usually coupled to tacit knowledge 
	- One upshot from this is [[It is especially hard to capture value from externalized research]]

### Examples
- A famous example of startups doing externalized research [[Genentech was started by bringing together several pieces of existing research and subcontracting to university labs]].
	- Did this only work because the founders of Genentech 
* [[Microsoft Research]]
*  [[Bell Labs didn’t have a unique model]] was primarily internalized but researchers would collaborate with academics and hire them. 
* [[Xerox PARC]] was almost entirely internalized - they would hire people away from 

## Questions
- [[When is externalized research better than internalized research and vice versa?]]

## Related
* [[It’s easy to set out to build a DARPA but end up building a Skunkworks]]
* [[The Nature of the Firm]]
* [[Designed Serendipity]]
* [[The dominant paradigm of R&D is for it to be internalized]]
* [[The higher a TRL the more internalized research should be]]
* [[What is the role of organizational boundaries?]]

## References
[[Sam Moleneux Conversation 31/10/2019]]
[[Mark McGranaghan]]



[Web URL for this note](http://notes.benjaminreinhardt.com/R+D_orgs_lie_on_a_spectrum_between_externalized_and_internalized_execution_and_benefits)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/R+D_orgs_lie_on_a_spectrum_between_externalized_and_internalized_execution_and_benefits)




Addressing structural constraints often requires drawing on more than one discipline

# Startups and Academia are bad at addressing structural constraints 
Both systems encourage making one single clear contribution at a time. 
- Papers are supposed to be discrete building blocks that add one brick to the wall that is ‘the literature’
	- [[Academics are incentivized to do peer-reviewed publishable work]]
- It is common wisdom that startups need to focus on changing one thing at a time.

In each case, there are rational good reasons for these constraints but it makes it hard for either system to address [[Structural Constraints]].

Additionally,  [[Problems that require system-level solutions often require massive teams]] and both startups and single labs do not have those teams. Interestingly, the [[§ARPA model]] may be able to cheat around this constraint by using externalized research. [[R+D orgs lie on a spectrum between externalized and internalized execution and benefits]]

### Related
- [[§Startup Constraints]]
- [[§Academia Constraints]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Startups_and_Academia_are_bad_at_addressing_structural_constraints)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Startups_and_Academia_are_bad_at_addressing_structural_constraints)




Systems must take performance hits to get out of local optima

aroraChangingStructureAmerican2020

Wicked Problem

Local Optima

Boundary conditions between different institutions affect how ideas move between them

The Value of Vision in Radical Technological Innovation

Pamella Vegata Conversation 2020-04-10

Successful  new organizations only change one thing about the world at a time

# Structural Constraints
Structural constraints occur when a constraint on a system can be traced to several nodes with interlocking dependencies. By definition you can’t fix it by just “swapping out the pieces” otherwise you could consider those pieces a module, ie one piece. 

Fixing structural constraints is relatively easy in software because you can leave the old system running, replicate the whole system, rip out all the pieces that need replacing and then swap it out. 

Fixing structural constraints in a physical system 

### Related
- [[Modularization allows different groups to work on different pieces]]
- [[Modularization enables the higher level technology to improve continuously even if pieces improve discretely]]
---
### References
[[mokyrLeverRichesTechnological1992]]


            

[Web URL for this note](http://notes.benjaminreinhardt.com/Structural_Constraints)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Structural_Constraints)


