The best DARPA program managers are the ones who can look at an entire literature in an area and notice a systemic bias

Exploratory Program Organization

DARPA Program managers pull control and risk away from both researchers and directors

DARPA PMs need to think for themselves, be curious, and have low ego

It is hard for a private ARPA to capture value without hamstringing impact

By definition outlier analysis can’t rely on data as comparison to other organizations

Where do ideas come from?

Who is the ARPA Model for and what do I hope to achieve

DeepMind combines bottom-up research work with top-down strike teams

Lessons learned from publishing ARPA Model

ARPA Model Conclusions

Startups and Academia are bad at addressing structural constraints

Why try to emulate DARPA and not Bell Labs

Why doesn’t ARPA use prizes instead of grants?

ARPA-E faces different industry incentives than DARPA

DARPA works on a mix of Big H and Little H hard problems

# §ARPA model
This is not meant to be a primer on *how* the ARPA model works or a history of DARPA. See references for plenty of good writing on that. It is *not* meant to be an unbiased presentation of the facts or argue for a general thesis about the world (though it hints at some.) The explicit goal of this analysis is to acknowledge that DARPA is a massive outlier and to figure out which of the attributes of DARPA contributed to its outlier results with the explicit intention of creating more organizations that can enable more outlier results.  [[§How do we get more awesome sci-fi shit?]]

The nature of outliers means that you can’t do a data-based analysis. Perhaps controversially, I think it’s foolish to try to find patterns among outliers because outliers are outliers in *different* ways except in the most broad strokes. Instead, I dig into the things that seem distinctive and ask why they might lead to outlier results.  At the end of the day, this is storytelling. But stories are powerful.  Of course, it’s easy to take this approach too far and create a story about how  *anything* weird about an outlier contributed to their success. “Steve Jobs only wore one outfit which is important because it allowed him to have EXTREME FOCUS.”  I try to avoid this by focusing on why the pieces lead to outlier performance *in combination* - many people who try to replicate outliers fail by picking and choosing the pieces they want to replicate.

There is some relevant history to deciding what to pay attention to without [[Cargo cult]]ing.

- [[The definition of the ARPA model has changed over time]]
- [[ARPA became DARPA in 1972 because of the increased scrutiny on military spending both in the government and outside of it]]
- [[Most of DARPA’s outlier output happened before 1972 when it was ARPA]]
- [[Was the shift from ARPA to DARPA a focus change or a process change?]]
Upshot:
- [[Pay attention to DARPA’s informal process and ignore formal process]]

### Program Managers
Anything written about DARPA agrees that it’s all about the program managers.
* [[DARPA Program managers pull control and risk away from both researchers and directors]]
* [[The best DARPA program managers are the ones who can look at an entire literature in an area and notice a systemic bias]]
*  [[DARPA is incredibly flexible with who it hires to be program managers]]
* [[DARPA Program managers have a tenure of four to five years]] 
* [[Why do people become DARPA Program managers?]]
* [[A large part of a DARPA program manager’s job is focused network building]] 
	* [[Maverick work is lonely and DARPA provides peers and community]] ::new::
* [[DARPA PMs need to think for themselves, be curious, and have low ego]]
*  [[The dependence of DARPA on high quality program managers mirrors the obsession with “talent” in other disciplines]] ::Updated::

### Incentives and Structure
* [[The government’s influence on DARPA is buffered by opacity and the director]]
* [[Opacity is important to DARPA’s outlier success]]
* [[DARPAs director is important]] ::needs work::
* [[DARPA does multiple levels of top-down problem generation and bottom-up solution generation]]
* [[The transient nature of most people at DARPA enables ideas to be revisited]]
* [[DARPA is relatively tiny and flat]] 
* [[DARPA Employees aren’t paid very much compared to what they could be]]
* [[DARPA’s aversion to people with a web presence may be how they avoid asymmetric career risk]]
* [[DARPA doesn’t do any research in house]] ::Updated::
* [[DARPA works on programs that go against the established paradigm]] ::new:: ::needs work::
* [[The ARPA Model is an unbuffered system]]::new::
* [[DARPA Staff are Important]] ::new:: ::needs work::
* [[Negative results are important to DARPA]]

### Funding
* [[DARPA is ~0.5% of the Department of Defense Budget and ~12% of that is basic research]]
* [[DARPA is more ideas limited than money limited]] 
* [[DARPA funds wacky things that go nowhere]]
* [[It is relatively easy for DARPA PMs to re-deploy funding]]
*  [[Program Managers have the ability to deploy money without much overhead]] 

### Process 
* [[The DARPA execution framework boils down to showing that thing is not impossible, showing that thing is possible, and then making that thing possible]] 
* [[There are many tight feedback loops built into the ARPA model]] 
*  [[The initial exploratory tranche of a DARPA program is approximately $1.5m]]
*  [[DARPA PMs use seedling programs to ‘acid test’ the riskiest pieces of a program idea]]
* [[The ‘framework’ DARPA PMs use to create a program is by modeling the tech council]]
* [[Every program at DARPA is intensely technically scrutinized by the tech council]] 
* [[DARPA facilitates idea sex both between PMs and Performers]]
* [[DARPA provides a derisking role for people in other organizations]]
* [[Decoupling from market discipline is like cave diving]] ::new:: ::Needs work::
* [[DARPA works where markets dont]] ::New::

### Riffing on ARPA
- [[A handful of organizations have riffed on the ARPA Model with mixed success]]::New::
	- Answers:  [[How does IARPA work? What are its successes?]]
* [[It’s easy to set out to build a DARPA but end up building a Skunkworks]]
* [[Most DARPA clones start too big or with heavy process]]
* [[Can the ARPA model exist outside of the government?]]

### Closed Questions
-  [[How successful did DARPA look ten years in?]]
	- Pretty successful

## Open Questions
Because a lot of this is just storytelling there are a lot of open questions about different knobs, what happens when you turn them and what they mean in different domains.  

* [[When is externalized research better than internalized research and vice versa?]]
*  [[Is it essential that a research org is a small fraction of the money factory’s budget?]]
*  [[Why doesn’t ARPA use prizes instead of grants?]]
* [[Is there a minimum effective budget to generate outlier results?]]
- [[How could the ARPA Model take advantage of the internet?]]

## Conclusions
* [[ARPA Model Conclusions]]
* [[Things about ARPA that can be improved]]

## Related

## References
* [[Funding Breakthrough Research]]
* [[“Special Forces” Innovation: How DARPA Attacks Problems]]
* [[Sam Moleneux convo 16/1/20]]
* [[Sam Moleneux Conversation 31/10/2019]]
* [[Know When to Fold ‘Em: An Empirical Description of Risk Management in Public Research Funding]]
* [[What makes DARPA tick]]
* [[What else makes DARPA tick]]
* [[Cloning DARPA Successfully - Article]]
* [[fuchsRethinkingRoleState2010]]
* [[Mark Micire Podcast]]

 
          
[Web URL for this note](http://notes.benjaminreinhardt.com/§ARPA_model)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/§ARPA_model)




§ARPA model

When is externalized research better than internalized research and vice versa?

R+D orgs lie on a spectrum between externalized and internalized execution and benefits

There are DARPA-style ideas that DARPA doesn’t pursue

Bell Labs enabled free radicals

DARPA would not have a positive return

Externalized research is a buffer against B players

New kinds of ARPAs need new kinds of performers

What is the role of organizational boundaries?

Key differences between FROs and different research institutional structures

# DARPA doesn’t do any research in house
Some subtle advantages of externalizing research:
- It enables [[DARPA is relatively tiny and flat]] 
- Actual cutting edge research may require rare equipment or knowledge. [[There are many pieces of equipment or tacit knowledge that only exist  in one or two places in the world]] 
- It enables strong accountability because for any program there is exactly one responsible person
- It enables program managers to have multiple teams working on the same objective without internal politics. 
- You don’t have to lay anyone off or find new work for people when you change direction
- You can get people to work on projects who might not want to work for the org full time
- [[Externalized research is a buffer against B players]]

Of course there are also disadvantages:
- There are higher [[Transaction costs]] in making sure deliverables are created, finding the right people in the first place, paying administrative overhead at the other org, and convincing them to work on projects
- DARPA doesn’t have any control over the “intellectual exhaust” created during projects. 
- [[Feedback loops and play are important for breakthroughs]] and externalized research can hamper both of these points. Even if you have good communication channels, you’re not in the same building. Play requires a feeling of safety and if a performer knows there’s a chance their funding will be cut, it may make them less playful. 
([[When is externalized research better than internalized research and vice versa?]]) 

It’s easy to conflate [[Publishing feels more like hardware development than software development]] with [[Bell Labs didn’t have a unique model]], [[Lockheed Skunkworks]], or [[Xerox PARC]] but they’re fundamentally different because [[R+D orgs lie on a spectrum between externalized and internalized execution and benefits]] and DARPA is all the way on the externalized end, while those other orgs are much closer to the internalized end. [[It’s easy to set out to build a DARPA but end up building a Skunkworks]]

[Web URL for this note](http://notes.benjaminreinhardt.com/DARPA_doesn’t_do_any_research_in_house)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/DARPA_doesn’t_do_any_research_in_house)






Innovation orgs need a money factory

It takes roughly 20 years to exploit a phenomenon at scale

# List of progress studies questions
- What is the distribution of timescales it took people to exploit  phenomena over time and the reasons behind those timescales would be good for 
	- See [[It takes roughly 20 years to exploit a phenomenon at scale]]
- What is the role of expectations in progress? Sometimes expectations push things to be done faster, but sometimes they bury a project under hype. 
- [[Could you model dispersion as a second order linear differential equation like a LRC circuit or a spring-mass-damper system?]]
- [[When is externalized research better than internalized research and vice versa?]]
- [[Why aren’t people building more institutions?]]
- [[Do programs that fall out of scope for present institutions have a large threshold cost? If so what is it?]]

- [ ] Go through questions and fill them in here

### Related
- [[What is the ontology of constraints on progress?]]


[Web URL for this note](http://notes.benjaminreinhardt.com/List_of_progress_studies_questions)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/List_of_progress_studies_questions)




neumannOneProcess2020

Structural Constraints

Modularization requires legible interfaces

The DARPA execution framework boils down to showing that thing is not impossible, showing that thing is possible, and then making that thing possible

Specialists create knowledge within fields

The process of creating a new heuretic node is like building something out of legos

baldwinModularityDesignComplex2006

How can you describe the difference between learning by doing and idea sex?

The difference between specialists and generalists is not clean but depends on which nodes they can extend

It is hard to change interfaces

Composability

Systems research is necessary when a system is not modular

All else held equal modularization makes systems less efficient

# Modularization allows different groups to work on different pieces
There are several reasons why enabling different groups to work on different pieces is important. 

 [[Modularization requires legible interfaces]] and [[Creating legible interfaces is expensive]] so the parallelization that modularization enables must be worth the cost. 

Modularization leads to specialization and the creation of disciplines ([[Specialists create knowledge within fields]].) The danger of modularization is that the interfaces will decay. This is why [[Program and project management is important and valuable]]. 

Additionally, good program management can “hack” interfaces that are not super clean. 

It is not always obvious where the lines between groups should be.  There are also politics around who works on what. Some things create more prestige or money if you work on them so groups will naturally try to take over some things and avoid working on other things. [[Defining clear timescales and scopes for collaborations is a tool to minimize politics]]. 

### Related
- [[Modularization enables pieces to be used in many different technologies]]
- [[Modularization enables the higher level technology to improve continuously even if pieces improve discretely]]
- [[Design for roadmapping system]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Modularization_allows_different_groups_to_work_on_different_pieces)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Modularization_allows_different_groups_to_work_on_different_pieces)




Modularization allows different groups to work on different pieces

Modularization enables the higher level technology to improve continuously even if pieces improve discretely

Modularization enables pieces to be used in many different technologies

The creation of a heuretic has two steps - conception and implementation

Modularizing ideas and specialization are the same thing

Leaky Abstraction

The Feynman path may look like shrinking modular components

Interaction surface area

Legible novelty is the worst

# Modularization requires legible interfaces
Legible interfaces enable someone to build on top of a module without understanding what is inside of it.

[[Creating legible interfaces is expensive]]

### Ways that an interface can be legible
- Interfaces can remain legible by not changing, but this is rare. 
- Interfaces can remain legible if the people who change them update the ‘documentation if you will’ but this is usually an afterthought. 
- Interfaces can remain legible if there is a party whose job is to maintain the interface. This requires coordination that often doesn’t happen. [[Program and project management is important and valuable]]

I’m sure that there is a lot of thinking and literature around this in the computer science and possibly program management communities. (

The real test of an interface is how much work it takes to combine a node with another node. [[Connections create new knowledge]].  Interface quality creates the endogenous part of the friction in combining nodes that [[Jerry Neumann]] mentions in [[neumannOneProcess2020]].

### Related
- [[The Architecture of Complexity]]
- [[Modularization enables pieces to be used in many different technologies]]
- [[Modularization enables the higher level technology to improve continuously even if pieces improve discretely]]
- [[Modularization allows different groups to work on different pieces]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Modularization_requires_legible_interfaces)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Modularization_requires_legible_interfaces)




DARPA has a mixed record on transitioning

DARPA doesn’t do any research in house

The higher a TRL the more internalized research should be

Ways an ARPA-riff could improve on Things about DARPA that can be improved

It’s easy to set out to build a DARPA but end up building a Skunkworks

§Axes for Looking at Funding and Organizations

Paul Merolla conversation 4 Oct 2021

Executing on multiple paths towards the same goal is hard within an organization

An Institute for Creative Technology - Riffing on ARPA

# R+D orgs lie on a spectrum between externalized and internalized execution and benefits
Externalized research is done by giving grants or contracts to external organizations. Internalized research is done on equipment owned by the organization by employees of that organization. In reality, most organizations do a combination of both.  DARPA is on the extreme externalization end. [[DARPA doesn’t do any research in house]].  [[DOE National Labs]] are probably near the middle - doing a lot of work but then collaborating heavily with university labs.  Corporate R&D are more towards the right end of the spectrum. 

The benefits of the work can be captured externally or internally as well. This creates a nice 2x2:
* Internal execution, internal benefit: most companies and R&D labs
* Internal execution, external benefit: contractors
* External execution, internal benefit: a company hiring contractors
* External execution, external benefit: a coordinating org like DARPA

Internalization involves hiring people you want to work with, while externalization involves contracting or issuing grants. This dichotomy is one way you could conceivably measure degrees of externalization vs internalization - money spent outside vs inside an org. It almost smells like balance of trade calculations for countries. 

### Ways organizations externalize research
1. Rent time on equipment along with skilled operators
2. Get specific things done by cloud labs like [[Emerald Cloud Lab]]
3. Hire contractors to do things like build software, electronics 
4. Hire contractors to do larger parts of a project like “figure out how to do X”
5. ‘Collaborate’ with other organizations (which in reality usually means co-applying for a grant and dividing up the work)
6. Give out grants to do entire research projects
7. Prizes - everything from the [[X-Prize]] to platforms like [[Innocentive]]

Most of these externalization methods work through a price mechanism (1,2,3,~4,~6,7) and some of them do not (5.) Looking at these methods raises another aspect of externalization - the degree to which you’re externalizing conception ([[Conception is the imaginative design of a heuretic]] ) and/or implementation ([[Implementation is the actualization of the heuretic in the real world]].) Roughly, the amount of externalized conception increases with the method number - renting time on equipment involves zero conceiving of new ideas while giving out a prize means you don’t even know what the idea is until you see it in action. 

### Advantage of externalization
- [[There are many pieces of equipment or tacit knowledge that only exist  in one or two places in the world]]
- You don’t have to make long term commitments to someone
- You don’t need to buy equipment you might need for only one project
- Externalization minimizes politics, which allows you to do things like pursue parallel approaches to the same thing. 
- You don’t risk having people with specialized skills around who need to be retrained or let go
	- [[Alex Rider]] pointed out that this happens at [[SRI International]] quite a bit

### Advantages of internalization
- Internalization can reduce [[Transaction costs]]. [[Robert Coase]] goes into more detail but you don’t need to haggle over the price of every piece of work, it’s easier to coordinate because you are in the same building (or at least you were before coronavirus), and you presumably all have an incentive to see the project succeed as a whole instead of just your little part. 
- Its easier to change direction quickly.
- You don’t have to specify interfaces as clearly.
- Tacit knowledge/[[Process Knowledge]] created during the project stays with the organization.
- If you need to maintain something, you keep the people who can do it around. (More [[Process Knowledge]].) 
- Nominally people have more buy-in to an organization, so they are more likely to go above and beyond the call of duty
- Taxes?
- People you’re working with become more known quantities - you need to do less vetting.
	- This becomes more important the smaller the units of work become. 
- You’re effectively playing repeated games with people so there is less incentive for them to screw you over.
- It’s easier to capture value from the research
	- You can immediately start commercializing 
	- You don’t need to police an external entity to see whether they’ve created something valuable 
	- External entities may want to own IP
	- If the value of the research is spread over several organizations it may be difficult to coordinate them. Each one individually may think their contribution is more valuable than an equitable division of value could handle. 
	- IP is usually coupled to tacit knowledge 
	- One upshot from this is [[It is especially hard to capture value from externalized research]]

### Examples
- A famous example of startups doing externalized research [[Genentech was started by bringing together several pieces of existing research and subcontracting to university labs]].
	- Did this only work because the founders of Genentech 
* [[Microsoft Research]]
*  [[Bell Labs didn’t have a unique model]] was primarily internalized but researchers would collaborate with academics and hire them. 
* [[Xerox PARC]] was almost entirely internalized - they would hire people away from 

## Questions
- [[When is externalized research better than internalized research and vice versa?]]

## Related
* [[It’s easy to set out to build a DARPA but end up building a Skunkworks]]
* [[The Nature of the Firm]]
* [[Designed Serendipity]]
* [[The dominant paradigm of R&D is for it to be internalized]]
* [[The higher a TRL the more internalized research should be]]
* [[What is the role of organizational boundaries?]]

## References
[[Sam Moleneux Conversation 31/10/2019]]
[[Mark McGranaghan]]



[Web URL for this note](http://notes.benjaminreinhardt.com/R+D_orgs_lie_on_a_spectrum_between_externalized_and_internalized_execution_and_benefits)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/R+D_orgs_lie_on_a_spectrum_between_externalized_and_internalized_execution_and_benefits)




DARPA Program managers have a tenure of four to five years

Nodes deeper in the tech tree take more time and work to unlock

# The Management of Innovation
Asks about theoretical tradeoffs between internalized research vs externalized research.  [[When is externalized research better than internalized research and vice versa?]]

Ridiculously theoretical and not much use but highly cited

### Published
1993
## Questions
## Thoughts
## Related
## Authors
## Link

[Web URL for this note](http://notes.benjaminreinhardt.com/The_Management_of_Innovation)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_Management_of_Innovation)






Interactions inside an org vs outside of it comes down to how the interactions are mediated by contracts

Coordination and Coercion in Research Funding

# The higher a TRL the more internalized research should be
[[When is externalized research better than internalized research and vice versa?]]
[[R+D orgs lie on a spectrum between externalized and internalized execution and benefits]]


[Web URL for this note](http://notes.benjaminreinhardt.com/The_higher_a_TRL_the_more_internalized_research_should_be)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_higher_a_TRL_the_more_internalized_research_should_be)




Creating legible interfaces is expensive

aroraChangingStructureAmerican2020

Manufacturing is important and often ignored

It is important for big things to start small

Bell Labs was only profitable in conjunction with AT+T

Bell Labs was not a business

# Transaction costs
[[Michael - Mike - Munger]]’s big thing.

Originally? From [[The Nature of the Firm]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Transaction_costs)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Transaction_costs)




Project-based funding reduces organizational agency

Directly acknowledging alignment and misalignment can be powerful

# What is the role of organizational boundaries?
# What is the role of organizational boundaries
To a large extent, these are just expansions on [[coaseNatureFirm1937]], however, it’s worth digging into specifics in the context of considering whether to create a new organization to accomplish a task and how we fund organizations. 

People tend to be much freer with their time towards other people within an organization. When someone reaches out to you over an organizational slack, the reaction is very different than when someone reaches out to you over a community slack. I’ve found a shocking difference in my own commitedness to literally the same project when my collaborator and I were nominally part of the same organization (even though we were working on completely different projects most of the time). 

On the funding side: A benefit of organizational boundaries  is that they enable the organization to shuffle around effort and money without needing to justify it. [[It takes a shocking amount of effort to justify things]]. Imagine if startups had to raise money to do each different part of what they wanted to do. 

You can view even projects with a single nominal goal through many different lenses. Any project also has many subgoals and it’s fractal. 

[[Project-based funding reduces organizational agency]]

### When you shouldn’t create organizational boundaries

At the same time, organizational boundaries are not all upside. Everybody in the organization is, to some extent, competing with each over the same resources. And because communication is easier inside organizations, intra-organizational politics is easier than extra-organizational politics. 

### Related
- [[R+D orgs lie on a spectrum between externalized and internalized execution and benefits]]
- [[When is externalized research better than internalized research and vice versa?]]
- [[DARPA doesn’t do any research in house]]
- [[Institutional affiliation matters for coordination]]



[Web URL for this note](http://notes.benjaminreinhardt.com/What_is_the_role_of_organizational_boundaries)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/What_is_the_role_of_organizational_boundaries)




List of progress studies questions

The Management of Innovation

# When is externalized research better than internalized research and vice versa?
# When is externalized research better than internalized research and vice versa
[[DARPA doesn’t do any research in house]], which has several advantages. However even in the domain of [[Mission-focused research]] examples like [[Lockheed Skunkworks]], [[Bell Labs didn’t have a unique model]], and others have demonstrated that internalized research can produce excellent results and has clear advantages. Internalized research lowers some set of [[Transaction costs]], enables closer collaboration between disciplines, smooths transition to manufacturing, and makes capturing value easier.  The fact that internalized vs externalized work is not a binary distinction but a spectrum makes the situation even more complicated. My hunch is that there is some class of work where the advantages of externalized research dominates and another class of work where the advantages of internalized research dominates.  I don’t have a good answer to where that line is, but it is an important question to address. 

### Related 
- [[Modularization allows different groups to work on different pieces]]
- [[Modularization requires legible interfaces]]
- [[The higher a TRL the more internalized research should be]]

- UX/low-requirement systems feel like something that’s very hard to externalize
	- Low-requirement systems are hard to modularize [[Modularization allows different groups to work on different pieces]]
- Related: externalization, modularization

* [[R+D orgs lie on a spectrum between externalized and internalized execution and benefits]]


[Web URL for this note](http://notes.benjaminreinhardt.com/When_is_externalized_research_better_than_internalized_research_and_vice_versa)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/When_is_externalized_research_better_than_internalized_research_and_vice_versa)




Benʼs working notes

When is externalized research better than internalized research and vice versa?

When is externalized research better than internalized research and vice versa

Related

Links to this note