§PARPA

You can create a program design discipline that enables better research and development

The distinction between areas, programs, and projects

Treating Science Fiction as case studies from a future history

Program design could be thought of as creating a causal graph of blockers and hypotheses about how to mitigate those blockers

Program design is a coordination tool

The value of planning is inversely proportional to the cost of experimenting

Program design is like creating the lego instruction book

Instead of planning not working as well as people in the past thought, as a society we’ve become worse at technological planning

How does money work in PARPA?

Physical experiments are expensive

Workshops could be a way to start small building a private ARPA

How do you measure real progress in research?

What role does scenario planning have in technological program design?

Maps help you figure out possibilities

Top of mind

There was a high return to technological roadmapping compared to not roadmapping in the past

fuEnablingMissionImpact2021

Efficiency biases systems towards false negatives

An ontology around peoples attitudes competency and roles is important for program design

Explaining things like I’m five helps avoid the Einstellung Effect

The tech tree model suggests that explicitly drawing out tech trees is important

An ARPA-like program could be the first step towards making general-purpose telerobotics inevitable

Funding is path dependent and funding curves matter

A predictably successful idea is composed of known unknowns that that a team can execute against

The crux question of program design is when to commit

Planning to Start vs Planning to Finish

Say ‘X is more important than Y’ instead of ‘X is important’

What are the different kinds of bottlenecks?

zwickyNewMethodsThought1967

What does a modified PARPA Heilmeier catachism look like?

There are two kinds of program design conversations — expansive and focused

Betting markets for research ideas could circumvent expert review killing ideas

The role of a visionary is to bootstrap the new from the old

Is there a way that python collab notebooks could be integrated into program design and roadmapping?

What is PARPA’s edge?

Roadmapping is related to the ability to create leading metrics for success

How do you think about the risk profile of PARPA programs?

Jim Matheson and Joshua Krieger conversation 14 Jul 2021

Hunting vs Farming Ideas

Work backwards with questions and work forward with answers

Joe Rollin conversation 10 May 2021

Precisely listing entrenched beliefs could be an important part of bottleneck analysis

Abstractly you want a good way of visualizing a bunch of chained vectors

Block Diagrams need to be in easily editable format

Geoff Anders conversation 7 Jun 2021

Geoff Anders conversation 9 Jun 2021

Getting people to talk about speculative research extensions is hard

Josh Greenberg Conversation 18 May 2021

Program design as a recursive response to the Heilmeier catechism

Roadmaps should have statements like “if we gain the ability to do X then we should do Y”

What are the pieces of a roadmapping architecture?

# §Program Design
- [[What are the goals of the program design process?]]]::needs work::
- [[Program design is like creating the lego instruction book]]
- [[Program design could be thought of as creating a causal graph of blockers and hypotheses about how to mitigate those blockers]]
- [[The crux question of program design is when to commit]]::needs work::

### Hypotheses
- [[The first step of roadmapping is frontier mapping]]
- [[A roadmapping framework may make the ARPA model more effective]]
- [[Talking to experts can expose bad equilibria, the moves necessary to get out of them, and considerations for doing them like cost constraints]]
- [[You can create a framework for talking to people that makes it easier to generate a roadmap]]
- [[You could build tools to make program design more efficacious]] 

### What are roadmaps
- [[What are the properties of maps?]]
- [[Roadmaps have two pieces - the map and the path]]
-  [[Roadmaps are a theoretical construct to escape inadequate equilibria]]
- [[List of Roadmapping Examples]]
-  [[Most roadmaps suck]]
	- [[Most roadmaps are not execution oriented]]

### Why Roadmapping
- [[Instead of planning not working as well as people in the past thought, as a society we’ve become worse at technological planning]]
- [[Roadmapping could be thought of as backwards-chaining together a series of conception-only heuretics]]
- [[Roadmaps can enable faster progress than blind exploration]]
- [[Technological roadmapping can give an idea of where technological evolution can go and how to get there]]
- [[A roadmapping framework may make the ARPA model more effective]]
- [[Program design is like performance debugging a vision]]
And of course we need to look at the possibility that this is all BS
- [[The roadmapping process is heavily context dependent and there may not be a generalized roadmapping process at all]]

### Building better roadmaps
- [[Design for roadmapping system]]
-  [[Person targeted questions for roadmapping]]
- [[What are the eight steps of Wigmore analysis translated to science?]]
- [[Technical Planning Principles]]
- [[Good roadmaps work backwards from a goal]]
- [[Good Simulations could be the ‘why now’ of roadmapping tools]]

### Glossary
- [[The distinction between areas, programs, and projects]]
- Bottlenecks -> Roadmap -> Execution
- Language
	- Binding Constraints
	- Affordances
	- Dependencies 
	- Project
	- Program
	- Area
	- Causal graph of blockers
- Steps
	- Frontier Mapping
	- Finding the right people
	- Comparing different paths
	- Goal setting
	- Finding serious contexts of use
	- Different institutional ‘moves’
	- Respectability cascades
	- Spending curves

### References
- [[And Suddenly the Inventor Appeared]]
- [[Wigmore Guide - Paper]]
- [[vincentTradeoffCentralConcept2017]]
- [[zwickyNewMethodsThought1967]]



[Web URL for this note](http://notes.benjaminreinhardt.com/§Program_Design)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/§Program_Design)




Uncertainty always involves risk but risk does not always involve uncertainty

Roadmaps can enable faster progress than blind exploration

# A predictably successful idea is composed of known unknowns that that a team can execute against
Let’s compare Instagram and SpaceX. Arguably both are extremely successful companies but Instagram is an unpredictably successful idea while SpaceX was a predictably successful. 

When Instagram started, there were tons of photo-sharing apps and the thing that would cause one to dominate all the others was totally unknown - there was no way to say “if the instagram team can figure out X, Y, Z then they will win.” (Beyond the obvious ‘if they can get lots of people to use their service’ - X, Y, Z need to be some sort of clear intermediate success factors - *leading metrics*.) 

When [[SpaceX]] started, they could say “it will be frakking hard but if we can get profitable launch price below $100/kg, we will win.” (June 2020 costs are $2700/kg and the fully-reusable starship will supposedly get costs below $10/kg.)  You can add up all the factors contributing to launch cost and systematically work on bringing them down. These are the known unknowns that a team can execute against.

Executing might have a high chance of failure - it might just be too expensive, take too long, be physically impossible, killed by regulation or danger or one of a million other things. 

The description of the inability to find *leading metrics* for an unpredictably successful idea sounds like  [[The Map is not the Territory]]  and why [[§Program Design]] - effectively the ability to describe the territory well with a map - is coupled to predictably successful ideas. To go one step further, because [[Technological roadmapping can give an idea of where technological evolution can go and how to get there]] you could argue that roadmapping is a way to turn ideas into predictably successful ideas. 

Another way to think about it: predictably successful ideas are risky but not uncertain. Or at least they constrain uncertainty. [[Uncertainty always involves risk but risk does not always involve uncertainty]].

### Related
- [[Roadmapping is related to the ability to create leading metrics for success]]



[Web URL for this note](http://notes.benjaminreinhardt.com/A_predictably_successful_idea_is_composed_of_known_unknowns_that_that_a_team_can_execute_against)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/A_predictably_successful_idea_is_composed_of_known_unknowns_that_that_a_team_can_execute_against)




§Program Design

ARPA Model Conclusions

# A roadmapping framework may make the ARPA model more effective


[Web URL for this note](http://notes.benjaminreinhardt.com/A_roadmapping_framework_may_make_the_ARPA_model_more_effective)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/A_roadmapping_framework_may_make_the_ARPA_model_more_effective)




Explain it like I’m Five

TRIZ is a framework for creating inventions

Describing an effect in an inventive manner is important for separating invention and discovery

Effect A can be used for purpose B entails describing the effect in an inventive manner

Each node on the tech tree can recursively be decomposed into more nodes

Module assembly and evolution are two ends of a spectrum

Michael Filler conversation 2 Sep 2020

# And Suddenly the Inventor Appeared
An inventive solution must improve a single part or characteristic of a system without impairing other parts or characteristics of the system or adjacent systems.

Solving an inventive problem involves solving a contradiction

A technical contradiction relates to the whole system or to several parts of the system vs. a physical contradiction related to only one part of the system

Try to find statements where something both *should be* and *shouldn’t be*

Engineers from beginning of 20th century saw world today, most machines would be familiar but scaled up or working differently.  (P34)

At some point just scaling up doesn’t make sense at which point you need to join two systems to create a new system 

Evolution of ships: paddles→ more paddles→ paddles + sails→ sails + paddles→ Sails→ more masts/sails→ sails + steam→ steam + sails→ steam (p35) 

System A + System B = System AB which has new properties that neither A nor B had.  ::[[Emergent behavior happens when a system has properties that none of its subsystems have]]::

Word traps often happen when trying to solve problems. They get you into the mindset of what a thing was intended for or a specific discipline. [^2]  ::[[Einstellung Effect]] and [[Explain it like I’m Five]]::

You can avoid word traps by replacing special terms with simple words[^3] 

Using ash and clinker through the same pipe to protect the pipe itself is an example of combining two systems into a new one. This is also an example of  (p37)

Animals are more than just a collection of cells - an amoeba cannot say “the era of cells continues!” Forest for the trees, systems for the cells. (P38)

Every technical system has a super system and one or more subsystem. Changes in a system effects systems above and below them in the hierarchy. 

Each system has an “interest” and solutions should not harm the interest of sub or super systems. (P39)

Systems at the beginning of their development actually do not perform their given function well (p40)
- First steamboat had huge inefficient paddlewheel and engine

Every system goes through four stages of development (p41)
1. Bringing together other systems to make the system (wings, engine, etc. for the airplane)
2. Improvement of the parts (Airplane begins to look like the modern version)
3. Dynamization of the system (Moving airfoils, swivel engines)
4. Self-development of the system (Self-adaptive systems)  [^4]

The stages of development are important because if you need to improve a system you can’t skip steps

#### Field Theory
**Fields act on substances**

Six basic “fields” [^5]
1. Gravitational
2. Electromagnetic
3. Strong nuclear
4. Weak nuclear
5. Mechanical
6. Thermal
We can really only control 2, 5,6

Sometimes you can add a substance to another substance in order to change how it is affected by a field (a generalization of method #11) If you can’t add a substance sometimes you can change the state of the substance (for example charging a liquid.)

S field diagrams:
- Wavy arrow: unsatisfied action
- Double arrow: has to make a transition to another system
- Broken arrow: has to introduce an action
- Fields acting on substances go above substance line
- Fields coming from substances go below line 

#### S Field Anaylis 
1. In order to solve a problem that has a partial S-Field the S-field must be completed
2. If there is a worthless S-Field, Introducing S_3 between S1 and S2 to improve the situation. S3 can be a modification of S1 or S2
3. You can modify an S field by
	1. Changing F1, S1 or S2
	2. Removing
	3. Introducing a second field
Ideally you want S3 to be a modification of S1 or S2

- When a system reaches its own limits it joins another system and a new more complex system emerges (p65) 
- Sometimes combining a system with another system is impossible, in which case you break up the existing systems and recombine parts into different systems.
- If neither of those options are open, you can start to look at the system as made up of many subsystems - ie. A spring is a system of atoms→ this is where Method 15 comes from
- Systems that seem to have used up all their resources of development can either join to other systems or transition to microstructure
(These still fit within the four stages of development framework)

*Describing an effect in an inventive manner* (P68)
“The substance will expand when heat is applied, therefore this phenomenon could be used in all cases when we need to control very small and precise movements” [^5]

Dry water(p73)

Effects can be stated in the frame of science or in the frame of inventive use. (P74)

[[Effect A can be used for purpose B entails describing the effect in an inventive manner]]

Ideal Final result (IFR)

#### Methods
- [[TRIZ Methods]]
1. Do it inversely 
	- Instead of pouring a liquid into a solid, form the solid around a frozen liquid, etc
2. Change the state of a physical property
	- Changing a state enables you to do things with a material that you couldn’t otherwise
3. Do it in advance
	- Putting a saw in a cast to be able saw your way out
4. Do a little less 
	- It may be possible to do an action partially that both enables the intended goal and solves the contradiction - ie. Scouring a pipe to be broken later instead of cutting it
5. Matreshka
	- It may be possible to put one subsystem inside of another subsystem - ie. Steel balls in wheels 
6. Conflicting requirements are separated in time and space
7. Replace special terms with simple words
8. Incorporating similar or different objects into one system
	- Measure single weevil temperature by putting a bunch of weevils in a cup
9. Fragmentation and/or consolidation
	- Instead of drilling lots of little holes through a chunk of glass, bundle together a bunch of glass rods
10. Dynamization -
	- Making fixed things adjustable
11. Add Magnetic powder to the substance and apply a magnetic field
12. S-Field Analysis
13. Self-service - pulling steel balls into the corner of a pipe so that they protect the pipe from other steel balls
14. Heat-expansion. This is an example of #15 below
15. Transition from Macrostructures to microstructures
16. Effects of the corona discharge - glow around high voltage objects depends upon gas pressure, voltage, and shape of object
17. Curie point of ferromagnetic materials - ferromagnetic materials demagnetize at specific temperatures
18. Combination of different effect - polarized light+changing polarization based on crystal angle + heat expansion = thermometer
19. Geometric effect of the Moebius Ribbon - double surface area 
20. Geometric effect of the rotating hyperboloid  - something made out of all right parts and straight lines that can change shape easily by torsioning

### Published
1994
## Questions
[[Can you apply TRIZ to bitland?]]
[[Can you apply TRIZ to people land?]]
What happened to Dry water?
Is it possible to use Klein bottles in the same way as a Möbius strip?
## Thoughts
- Still a ton of optimism about what can be done in the physical world  
- Feels like science fiction
- The requirements for inventive solutions are actually quite tight 
- Self-service protection seems related to [[Antifragile]]
- [^5]: Describing things in an inventive manner more generally entails
	- “When X happens, it produces the effect Y. This effect could be used in cases where we need A under B conditions.”  
## Related
- [[Technology relaxes constraints]] 
- [[Wigmore Guide - Paper]]
- [[TRIZ is a framework for creating inventions]]

[^2]:Related to the [[Einstellung Effect]]
[^3]:[[Explain it like I’m Five]] and [[Wittgenstein: Reality is Shaped by the Words we Use]]
[^4]: Computers are the thing that gets us to level 4 of a system - what other self-adaptive systems are there now as opossed to 1994. 
The hierarchy of development explains why things “look” almost the same since the 1960’s and also suggest how we make things look different. Need to bring together entirely different systems. 
[^5]: Note that the fields are fundamental forces + macroscopic instantiations of them

- 
## Authors
- [[Genrich Altshuller]]
## Link

[Web URL for this note](http://notes.benjaminreinhardt.com/And_Suddenly_the_Inventor_Appeared)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/And_Suddenly_the_Inventor_Appeared)






§Misalignment between funders and nonprofits in research

Systems research requires a lot of work that is expensive and uninteresting

Astro Teller conversation 1 Feb 2022

Some ideas take a large chunk of resources to assess

The efficiency imperative makes it tough for profit-seeking organizations to take on systems research

# Efficiency biases systems towards false negatives
Before you even undertake the work to create a thing, there’s some assessment of whether it’s possible and how valuable it will be.[^4] (For some extremely nebulous definition of value — I’ll talk about that later). [[Some ideas take a large chunk of resources to assess]] (**“fat” ideas**).[^3] In other cases, a spreadsheet and well-founded assumptions can make a strong case that “it will be hard, but if we pull off all these things that don’t violate physics along this critical path, it will be incredible.” (**“Lean” ideas**). 

SpaceX’s reusable rocket is a canonical lean idea: “if we can do X (which is very hard but has a clear critical path) then we will reduce/increase this key metric my an order of magnitude, which is a complete game changer.” [^1] Software’s low cost and fast iteration speeds make many bit-based ideas lean even without clear metrics or critical paths. “Basic science”[^5] is a classic example of a “fat” idea: the curiosity-driven study of tree-frog venom leads along a circuitous route to a miracle cure. However, fat ideas aren’t just curiosity-based discovery. Most “systems research”[^6] ideas — of the sort that were the precursors of a good chunk of modern technology —are also fat ideas.[^7]   

If all you care about is big returns per dollar  (in either financial terms or “impact”) there is a clear strategy: work to quickly kill ideas and bias towards a thousand false negatives over one expensive false positive. 

Imagine that lean ideas look good after 1 unit of work and will create 1000 units of value, and fat ideas only look good after 100 units of work but produce 1100 units of value. Each idea creates the same *net* value but if you care about returns on investment you would rather spend 1 unit of work every time, ignoring every single one of the fat ideas and their potential results. In order to make it worth your while, the fat ideas would need to on average create 100000 units of value (an increase proportional to the cost of evaluating them). 

Another framing is through the lens of scarce resources.  If you have 100 units of work to deploy, your choice is whether to spend that 100 units of work on 100 lean ideas or 1 fat idea. If each project has the same expected value, you would have 100x expected value by focusing 100% on lean ideas. Lean ideas are better to work on in expected value and ROI terms.

The advantage of the lean strategy has nothing to do with whether you’re trying to capture the value you create or not — as long as you prioritize efficiency, it makes sense for both investments and philanthropic spending. 

In reality, ideas don’t fall neatly into buckets that take one of two values of resources to evaluate. There’s some continuum from zero effort to [infinite effort? The value of executing on the idea itself? (You don’t know if it will be valuable and possible until you literally complete it)]. However, because we’re talking about a scalar effort to evaluate them, it’s reasonable to draw a line that is the maximum effort you’re willing to spend per idea — everything to the left is lean and everything to the right is fat. This bucketing system means that you lose out on the marginal idea immediately to the right of the line. 

 The fact that it’s hard to distinguish between many fat and lean ideas compounds the argument for a “lean assessment only” strategy.  As a result, many research organizations either implicitly or explicitly pursue the lean strategy — explicitly preferring many false negatives over an expensive false positive. Indeed, places like [[Google X]] play up the efficiency of their process. And given the assumptions and constraints they’re operating under, they’re not wrong. 

Let’s examine those assumptions:
1. Lean and fat ideas create roughly the same amount of value on average. 
2. The difference in cost between evaluating lean and fat ideas is several orders of magnitude. 
3. There are plenty of lean ideas.
4. It’s impossible to know a-priori how much effort it will take to evaluate an idea.
5. The results of lean and fat ideas are interchangeable.  
6.  Efficiency should be prioritized.

**1. Lean and fat ideas create, on average, roughly the same amount of value**  
This assumption seems pretty reasonable — I don’t know of a comprehensive study of innovations, the value they created, and whether it was easy for people to evaluate whether they were worth pursuing a priori but there are plenty of valuable examples of both types. SpaceX, Tesla, Transistors, airplanes, AI, artificial fertilizers, the structure of DNA, atomic power (after a certain point) are all lean ideas — generally they’re characterized by either a gold rush or criticisms of impossibility to pull off. Lasers, the internet, germ theory, and solar panels are all fat ideas — generally they’re characterized by people calling them niche or useless and are preceded by a lot of “piddling around.” 

While historically, lean and fat ideas do seem to create roughly the same amount of value if you force them into a framework of scalarized value, I’m going to push back against that framework in point #5. 

**2. The difference in cost between evaluating lean and fat ideas is several orders of magnitude.**
This assumption is also pretty reasonable. Lean ideas are characterized by being able to put a lot of assumptions into a spreadsheet and possibly identifying one or two key experiments to validate some assumptions. Fat ideas often involve years of piddling around (and thus years of salaries and expensive equipment) before even having a clear idea of what the valuable output even could be.   

**3. There are plenty of lean ideas.**

The number of fat ideas (even if they would only take slightly more effort to evaluate than the fat-lean cutoff) is irrelevant if all you care about is efficiency. The frequency of fat ideas is irrelevant unless the frequency of lean ideas is so low compared to the frequency of fat ideas that you hit a tipping point where pursuing a lean strategy forces you to spend more resources throwing away ideas quickly than you would spend evaluating fat ones. Thus, the lean strategy is rational if you believe there are plenty of lean ideas. 

It’s (almost? Actually?) impossible to evaluate this assumption. My personal hunch is that a component of the [[Low Hanging fruit theory of stagnation]] is that so many institutions (rationally) pursue the lean strategy that lean ideas have become depleted. You could call lean ideas “low hanging fruit” themselves, but generally the “low hanging fruit” is understood to refer to the total effort to realize the idea, not to evaluate whether it’s worthwhile. 

My other hunch is that while the distribution of evaluation costs is continuous, it is not flat. It likely has at least two peaks — one that’s lean-esque and one that’s fat-esque. If that’s the case, institutions pursuing the lean evaluation strategy can’t just incrementally increase their threshold for leanness as lean ideas are depleted. 

The first three sets of assumptions all fit within a nice numerical framework. You can play with numbers here: [False negative bias - Google Sheets](https://docs.google.com/spreadsheets/d/1j679FlpPgl4GT6Ohi9CR0yrZCLIuVrhT7rc36D_NpF0/edit?usp=sharing)

**4.  It’s impossible to know a-priori how much effort it will take to evaluate an idea.**
 The lean strategy implicitly assumes that it’s impossible to know how much effort it will take to evaluate an idea (meta evaluation!). If it were possible to know up front how much work it would take to at least know whether an idea was worth pursuing, you could know which ideas are marginal, taking two units of effort but not 100, and evaluate those ideas as well. The dream of [[§Program Design]] as a discipline is to be able to cheaply determine how much work it will be to evaluate an idea (in some senses, an ARPA-like program is an idea evaluation). 

**5. The results of lean and fat ideas are interchangeable.**  
Collapsing value into scalar quantities assumes that the outputs of pursuing different ideas are the same in kind. That is, the *types* of innovations that will come out of pursuing only lean ideas look the same as if you pursued both lean and fat ideas. Or at least the types of innovations created by lean and fat ideas are equally desirable. 

Here is where thinking in terms of monetary outcomes violently diverges from a more nebulous concept of “value.” If all you care about is cash in and cash out, the results of all ideas are interchangeable. However, I suspect that pursuing primarily lean ideas systemically biases towards certain types of innovations. As a result, whole classes of ideas are systematically *not* being explored. It will take a lot more work to enunciate what the systemic differences are between the outputs of lean and fat ideas, but my hunch is that a lot of software and point changes are lean ideas, while many [[General purpose technologies]] and multi-point changes are fat ideas.[^2] Even ignoring which specific classes of innovations we’re missing out on by pursuing primarily lean ideas, I suspect that pursuing an “unbalanced” distribution of lean/fat ideas is unhealthy for long-term innovation because ideas feed on each other — fat ideas generate lean ideas and vice versa, somewhat analogous to [[Phenomena-based cycles]]. The bias towards lean ideas may be a component of why [[Phenomena-based cycles are stuck]]. 

**6. Efficiency should be prioritized.**
The final assumption underlying lean strategies is that efficiency (value created/effort in) is the most important thing. When you have a fiduciary duty to maximize shareholder value, *this is correct*. [[There are many important innovations that are just bad investments]]. Speaking to several high-ranking people who run corporate research labs, they constantly need to justify, not the absolute cost of the work they’re doing, but the ROI they expect. 

You would expect the unbreakable coupling between profit and efficiency to suggest a clear opportunity for philanthropic money to step into the gap and enable the fat, inefficient assessments that can unlock entirely different classes of innovation. Disappointingly, I’ve found this to not be the case. Instead, the majority of philanthropic funding decisions I’ve run into are still implicitly or explicitly dominated by efficiency. Questions like “what is the most impactful way to use this dollar?” “How do we minimize risk of failure?” “How do you know it will work?” “What will the impact of this be?” Are all trying to maximize efficiency. Efficiency is not bad in a vacuum, but it is when it is killing fat ideas at a societal level (if the assumption that fat and lean ideas are interchangeable is wrong). 

Where does this all leave us? If the outputs of lean and fat ideas do indeed generate different types of innovations, a relentless prioritization of return on investment (whether in dollars or impact) has created a world of systemically skewed innovations. Many incredible discoveries and inventions have been sacrificed on the altar of efficiency. The lean-only approach is completely rational for profit-seeking businesses. However, institutions with other mandates, like philanthropy and governments, have also been captured by the same mindset. It’s a heretical notion in the current zeitgeist (and there are many ways it can go wrong), but it would be powerful for institutions that can get away with it to prioritize things besides efficiency. Perhaps [[Efficiency is overrated]].  

### Related 
- [[Efficiency is overrated]]
- [[Justification kills play]]
- [[alexanderStudiesSlack2020]]
- [[neumannOneProcess2020]]

Thanks to [[Matt Clancy]] and [[Luke Constable]] for reading drafts of this piece and amazing ideas. 

[^1]: See [[Startup by spreadsheet]]
[^2]: See [[Startups are good at point changes]], [[We don’t have good institutional ways to create multi-point changes]]
[^3]: The “fat” and “lean” framework comes from [[raoGUTSGrandUnified2018]]
[^4]: I’m implicitly talking about research ideas here, but I think this argument applies more broadly. 
[^5]: I despise this term but it is what it is
[^6]: See [[pikeSystemsSoftwareResearch2000]]
[^7]: Digging into how to tell whether an idea is fat or lean and whether there are systemic differences between them is for another time. 



[Web URL for this note](http://notes.benjaminreinhardt.com/Efficiency_biases_systems_towards_false_negatives)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Efficiency_biases_systems_towards_false_negatives)




# Explaining things like I’m five helps avoid the Einstellung Effect
[[And Suddenly the Inventor Appeared]] points out that you can fall into “word traps” that lock you into the mindset of a specific discipline. The words you use can make  the [[Einstellung Effect]] more severe because jargon can encode assumptions that may not be true in a new situations. [[Jargon is a mechanism for compressing information]]. 

A partial solution to word-traps is to [[Explain it like I’m Five]], which effectively decompresses information.  When you break a problem statement open to look at each of its parts, you might realize an assumption that can be shifted. 

The  first assertion of [[The Heilmeier Catechism]] is a bit like independent verification that this is a good idea.

### Related
- [[Jargon is mystifying to people without the relevant context]].
- [[§Program Design]]
- [[Naming things is Powerful]]
- [[Suitcase Handle Word]]
- [[People like to create shibboleths to distinguish their tribe]]
- [[§Demystification and Thought Leaders]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Explaining_things_like_I’m_five_helps_avoid_the_Einstellung_Effect)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Explaining_things_like_I’m_five_helps_avoid_the_Einstellung_Effect)




Most roadmaps suck

Good Simulations could be the ‘why now’ of roadmapping tools

# Good roadmaps work backwards from a goal
Working backwards from a goal prevents ideas from being locked-in by how people think about the problem right now. 

In a way, thinking backwards from a goal is what people mean when they talk about first-principles thinking. The usual interpretation is to start with physics but I would assert that while tying back to physical laws is important to ‘first principles’ thinking, it’s actually the end. Or perhaps ending each step with checking “does this violate physics?” Is an important part of the process.  Working backwards from a goal is ‘first principles’ thinking in the Aristotelian sense of not working off of the work of others, which *is* critical to working backwards. 

The working backwards approach contrasts with the map-and-then-roadmap approach, where you would survey the entire landscape and *then* create a path through it. [[Roadmaps have two pieces - the map and the path]] . In reality, the approach is probably a deeply impure mixture of the two approaches - jumping back and forth between working backwards and describing the present state of the world. [[Understanding a thing has to do with intuiting the affordances of a thing]] and in order to understand the affordances of the area you’re playing in you need to at least engage with the way it does work now.  

In a way there is a tension between working forward and working backwards. If you purely work backwards you ignore the accumulated wisdom of one or more disciplines, which is considerable if those disciplines are valuable in the first place. [[We stand on the shoulders of too many giants to give them all credit]]. And to compound the problem, often that wisdom (especially wisdom on the [[Knowledge frontier]]) is not well modularized and composable, so it’s hard to know what’s important and what’s ignorable without knowing it. At the same time, starting with “the state of a field” can be a trap in a couple of ways.  [[The tech tree is fractal]] so you could actually spend an infinite amount of time digging into exactly what the frontier looks like. Additionally if you spend a lot of time immersed in current paradigms you’ll start to see the world through those paradigms, which is problematic for going after goals that almost by definition have not been reached by the people who work in those paradigms. 

This tension between working backwards and forwards is embodied any time someone approaches a problem outside of their ‘area of expertise.’ It is perhaps best illustrated by physics. Physicists are constantly bombarded by theories from quacks who think that they have found a new theory of <life, the universe, and/or everything.> At the same time, people who in the past *have* found new theories have on occasion come from non-traditional backgrounds and *were* regarded as quacks, until they weren’t. 

Perhaps the correct way to think about roadmapping is actually similar to a backpropagation algorithm, where you do many passes going both backwards and forwards. You

Not working backwards from a clear goal is one of the ways that [[Most roadmaps suck]].  A big reason they don’t do this is that working backwards is *really hard.* It forces you to consider important questions about something that doesn’t actually exist. Asking questions about “made up things” isn’t a mode of thinking we’re used to. 

[[The Architected Innov􏰂ation Institute - Engineering Scientific Revolutions]] asserts that working backwards from a goal helps you evaluate possible paths to that goal with an unbiased eye. 

[[Treating Science Fiction as case studies from a future history]] might be one way to make considering important questions about things that don’t exist. 

Thinking about hard-to-think-about questions regarding things that don’t exist seems like an actual place for tools for thought. [[You could build tools to make program design more efficacious]]. 

Another framing besides ‘goals’ is answers to the question [[What does ‘working’ mean?]] for this particular program?

One thought is that perhaps you [[Work backwards with questions and work forward with answers]]?

### Related
- [[A lot of research feels like it is building lego pieces without any idea how they’ll be used]]
- [[Roadmapping could be thought of as backwards-chaining together a series of conception-only heuretics]]
- [[Technological roadmapping can give an idea of where technological evolution can go and how to get there]]
- [[Luke Durant Email 18 Sep 2020]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Good_roadmaps_work_backwards_from_a_goal)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Good_roadmaps_work_backwards_from_a_goal)




Perceived progress can unlock real progress

Timeboxed processes is a way to measure research

# How do you measure real progress in research?
# How do you measure real progress in research
One possibility is that [[Timeboxed processes is a way to measure research]]
### Related 
- [[§Program Design]]
- 
- 

[Web URL for this note](http://notes.benjaminreinhardt.com/How_do_you_measure_real_progress_in_research)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/How_do_you_measure_real_progress_in_research)






When is push or pull appropriate for innovation orgs?

# Hunting vs Farming Ideas
Ideas are like food for institutions in the innovation business. Accordingly, there are roughly two ways that those institutions find new ideas: hunting and farming. 

Organizations that *hunt ideas* go out in the world assuming that someone external to the organization has come up with the idea. The organization’s moves are to further enable that person (usually through funding) or acquire the idea. “Finding talent”, grants, investing, and portfolios are all hunting strategies. 

Organizations that *farm ideas* try to create ideas internally. [[§Program Design]], research labs, ::need more things in this list:: are all farming strategies. 

Hunting is a searching and filtering process. Farming is a creation process.

Hunting is [[Indefinite optimism]] while farming is [[Definite optimism]].  If you don’t know how to create a good idea or what one looks like until it can justify itself, hunting is the correct strategy. Hunting ideas assumes that idea creation is a mystical process that cannot be systemized. Ideas spring up in the world and you need to go find them. Farming ideas assumes that idea creation can, to some extent, be systemized. 

The world’s pendulum feels like it has swung strongly towards hunting — many people see startups and venture capital as the way to advance our material technology. The swing towards hunting reflects broader trends towards “Index Mindsets” [[luttigIndexMindset2021]] and [[Indefinite optimism]]. There are a number of good reasons towards the swing: hunting has been incredibly successful (especially when the farming was the dominant approach; (perhaps breaking the analogy with food hunting) farming is inefficient because you need to internalize the costs of all the ideas that go nowhere; many of the farming organizations of the 20th century degraded to innovation stifling waterfall development and top-down control.

Both hunting and farming can have both productive and unproductive forms. The two-sided nature of idea approaches is similar to [[Aristotle]]’s  observation of how democracies can degrade into mobs and monarchies can descend into tyrannies. In its productive form, hunting lets bottom-up emergent order work its magic, discovering and amplifying ideas that nobody could have predicted from unexpected sources. Hunting degenerates to endless buck-passing, the death of ideas that can’t justify themselves to an external organization, and herd behavior as everybody uses everybody else as their heuristic for good ideas.  In its productive form, farming keeps [[Safi Bahcall]]’s “warty babies”[^1] alive until they can justify themselves, it fosters ideas that require a higher level of coordination or planning, and enables an organization to hone a process of creating ideas over time. Farming degenerates to politics, pet projects, waste, rigid processes, and top-down control. 

Hunting ideas is efficient, but [[Efficiency biases systems towards false negatives]]. 

While the seas might be deep and idea fish might still be plentiful, at the end of the day hunting is an extractive and zero-sum process. Each idea that your organization grabs is one that another organization cannot (obviously this analogy is imperfect — multiple firms can fund a start up, but there is only so much room in a round). Hunting requires some external font of ideas to replenish the supply. Of course, hunting can be done in a sustainable manner and in some cases can actually benefit its targets (keeping deer populations from depleting their food supplies for animals and incentivizing people to act on their latent ambitions in the case of ideas). 

[[Research management matters]]

### Related
- [[DARPA is a coordination mechanism]]
- [[PARPA is a coordination mechanism]]
- [[When is push or pull appropriate for innovation orgs?]]
- [[Emergent systems can't be designed but they could be gardened]]
- [[The knowledge frontier is a high-dimensional garden]]
- [[luttigIndexMindset2021]]
- [[Instead of planning not working as well as people in the past thought as a society we’ve become worse at technological planning]]
- [[Research requires intrinsic motivation]]

[^1]: See [[bahcallLoonshotsHowNurture2019]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Hunting_vs_Farming_Ideas)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Hunting_vs_Farming_Ideas)




Program Evaluation Review Technique - PERT

Building Information Modeling - BIM

Untapped technical people near the end of their careers may have more potential than young people

# Instead of planning not working as well as people in the past thought, as a society we’ve become worse at technological planning
The hypothesis goes: in the past 30 years,  software and finance has created a disproportionate amount value, especially based on mindshare. Both software and finance are domains where the returns to doing a good job developing a precise, multi-year technical plan are low compared to not developing that plan. In fact, detailed technical plans in those domains can possibly even lead to *worse* outcomes. This low-or-negative return to long-term technical planning led to the individual and institutional ability to do long-term technical being selected out - ie. culturally we’ve become worse at long-term technical planning. I suspect that the failure of plans to create value both in domains where they could *potentially* create value and also in domains where they *can’t* create value injected an attitude that ‘plans don’t work’ into our culture. 

This narrative runs counter to a different narrative that we’ve always been just as bad at long-term technical planning and over time the alternatives have become more feasible. In this scenario, we develop both better alternative processes like agile development and technologies that make iteration cheaper and faster. 

In other words - *is roadmapping a lost art or is it a myth?*

In a way this is a facet of [[Peter Thiel]]’s indefinite/definiteness thesis. 

 The decline-of-planning-ability narrative depends on four core pieces of evidence: 
1. [[There was a high return to technological roadmapping compared to not roadmapping in the past]]
2. [[The returns to good technical planning have decreased]]
3. [[Plans are valuable compared to not planning for a certain class of problem]]

It’s also important to disentangle “long-term technical planning” from the many other kinds of planning.  Many ideas and actions have gotten lumped under the term ‘planning.’ For example, there is evidence that management practices have contributed massively to productivity over the past few decades and those practices in large part involve effective planning. I would call this ‘firm-level strategic planning.’ 

During the 20th century (and still today), “planning” connoted “government planning” specifically “government planning economies.” Fighting for or against the idea took on a philosophical and ideological tone. American economists who argued against planning were no longer just arguing for what was the best economic policy but instead fighting by proxy against the Marxist ideology. This conflation of idealogical battle and policy debate has been forgotten  As far as I can tell there is pretty strong that government economy planning has poor results. However, people generalize from this to “therefore planning doesn’t work and is bad in general” which is not necessarily true. 

There’s also the specter of “Big Design Up Front.” Are roadmapping and big design up front the same thing? I don’t think so. Roadmapping is concerned with constraints, dependencies, and sequencing. Big design up front is concerned with constraints and dependencies, but it’s less worried about sequencing and feels more rigid. ::This needs more work::

What is the relationship of technological roadmapping with systems engineering

In short - we should be open to the possibility that planning works well in some situations but not in other and instead of making universal statements about planning, we should instead try to tease apart the situations where it does and does not work and why that is the case. 

### Related
- [[Can we intentionally improve the world? Planners vs Hayekians]]
- [[Planning to Start vs Planning to Finish]]
- [[§Program Design]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Instead_of_planning_not_working_as_well_as_people_in_the_past_thought,_as_a_society_we’ve_become_worse_at_technological_planning)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Instead_of_planning_not_working_as_well_as_people_in_the_past_thought,_as_a_society_we’ve_become_worse_at_technological_planning)




# Is there a way that python collab notebooks could be integrated into program design and roadmapping?
# Is there a way that python collab notebooks could be integrated into program design and roadmapping
### Related
- [[Technological roadmapping can give an idea of where technological evolution can go and how to get there]]
- [[§Program Design]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Is_there_a_way_that_python_collab_notebooks_could_be_integrated_into_program_design_and_roadmapping)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Is_there_a_way_that_python_collab_notebooks_could_be_integrated_into_program_design_and_roadmapping)




What are the properties of maps?

Maps help you understand the state of something

Maps are multi-scale

Roadmaps need to be coupled to people

# Maps help you figure out possibilities
Even the medieval T-O map let you know that you could get to Africa from Europe by crossing the mediterranean.  Star maps help you figure out which constellations you could draw or where you could look in the night sky. 

If you can’t figure out possible (and unallowable moves) from a map, it’s not doing its job. Many ‘competitive’ landscapes fail on this front. 

A good map makes you aware of the tradeoffs between different possible moves.  Google Maps also helps you figure out possibilities by showing alternate possibilities and making you aware of the time and distance between those possibilities. The parameters of these tradeoffs vary drastically between maps - on a hiking map, elevation is important while on a driving map it is almost irrelevant. On a research map, cost is important while on a star map the concept of cost doesn’t even make sense. 

[[The Map is not the Territory]] - a good map will capture the information that you need to make decisions and exclude information that is distracted. 

*Roadmaps* ([[§Program Design]]) explicitly point out those possibilities in an opinionated way.  The difference between a map and a roadmap is the difference between google maps and the directions the map gives you to a location. 

![](../BearImages/57FE1074-F3ED-4A30-A4EB-C75575380032-456-00003424E76F85DB/9F042C35-37C6-4A14-8DCC-2066389EFB4E.png)
(Wardley)
![](../BearImages/D46F1E2D-1AC7-43B6-B703-46E7BA120E26-456-0000343CA6138B63/A9DE43E5-B06E-442A-8C30-4ED7643D2317.png)

### Related
- [[The less shared context you have the higher the coordination costs]]
- [[Context is important and underrated for knowledge transfer]]
- [[Maps are multi-scale]]
- [[Maps help you understand the state of something]]
- Wardley points out how useless a strengths/weaknesses/opportunities/threats analysis would be for a battle [On being lost - wardleymaps - Medium](https://medium.com/wardleymaps/on-being-lost-2ef5f05eb1ec)
	- [[Wardley Maps]]

[Web URL for this note](http://notes.benjaminreinhardt.com/Maps_help_you_figure_out_possibilities)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Maps_help_you_figure_out_possibilities)






Most predictions suck

List of Roadmapping Examples

Good roadmaps work backwards from a goal

Technical language should be treated like an actual other language

Jordan Blashek conversation 23 Feb 2023

Playbook for Writing a Roadmap

Technical Planning Principles

People often confuse whether they are or should be creating prescriptive vs descriptive work

Program design creates Schelling points for purpose maintenance

Engineering Biology Roadmap -ERBC

What is SpecTechs Edge?

Andy Matuschak Conversation 30 Aug 2020

Michael Filler Converstation 2 Sep 2020

# Most roadmaps suck
As the name suggests, roadmaps should have the properties of a map ([[What are the properties of maps?]]) [[Maps are multi-scale]], [[Maps help you understand the state of something]], [[Maps help you figure out possibilities]] and explicitly point out those possibilities and associated tradeoffs. 

Most roadmaps do a poor job on many of these points. Instead, they feel like a mush of keywords and key phrases that someone evaluating a grant could check against and say “aha! This grant is legitimate because they’re trying to do this this thing that’s mentioned in the roadmap.” 

Of course, it’s possible that creating a good roadmap is an impossible task. It may be impossible to create a roadmap that actually enables coordination and progress that would not happen otherwise. However, the assertion that most roadmaps suck is based on a list of points that are very possible to fix.  

- Most roadmaps are primarily text, which makes it hard for them to be multi-scale. You should be able to glance at a roadmap at at least get a sense of what you need to do. For most of these roadmaps you need to do a deep reading to actually get a full picture. The only thing you can get at a glance from a full-text roadmap is an isolated chunk of whatever you’re looking at. 
- None of the roadmaps I’ve seen talk about either who is most suited for a task and who is going to do it.  This people/organizational decoupling might work on for the semiconductor industry, where everything is highly modularized, interfaces are standardized, and each organization has a clear responsibility. For less mature industries this isn’t the case and there’s still no pointers from jobs to be done to people or organizations. [[Roadmaps need to be coupled to people]].
- Most roadmaps don’t have precise ideas of what would enable a capability step change. All aspects of a technology can be improved, but improving some metrics can actually make the difference between a tool being almost useless and super useful [[Continuous changes can lead to discrete differences]]. Sometimes these changes are going to be improved metrics, but I’m being careful not to over-focused on metrics because often the key blocker keeping a technology from being useful is more fuzzy. At the same time, sometimes numbers are necessary to enforce precision. 
- Most roadmaps don’t go into the precision needed to actually decide whether a project is on a critical path towards enabling something important. Following on the previous point, most of the intermediate steps in roadmaps are incredibly fuzzy “Step 5: build improved planning algorithms.” What does it mean to improve a planning? Calculating paths faster? Being able to recalculate smoothly on the fly? etc.
- Most roadmaps don’t precisely present the state of an area. A question I have constantly in response to “In five years, we hope to be able to X, Y, Z” is  “Ok, so why can’t we do those things today?” I suspect the way to answer that question is to work backwards from where you hope to be to where you are today. ie. [[Good roadmaps work backwards from a goal]].  In order to do that you need a clear idea of where you are on the things that matter.
- Most roadmaps don’t make it clear which constraints are major blockers and which are nice-to-haves from an application perspective. For example, do we need robots to have five fingered hands that are as dextrous as a human’s in order to take care of older adults? Probably not! Do they need to be able to locomote over carpet? Absolutely. 
- Most roadmaps don’t make it clear which metrics matter and which metrics don’t matter. Different metrics are important to different applications. 
- All the roadmaps I’ve looked at don’t distinguish between exogenous changes that they’re counting on and endogenous changes in the field that someone reading the roadmap needs to go do. Instead, the roadmaps just use the blanket statement “X should be possible or true in five years.” The litmus test is whether X would still happen if everybody reading the roadmap went and drank mai tais on the beach for five years. In the context of robotics, for example, Moore’s Law chugging away or shifts in the ratio of older adults to working-age people are exogenous changes. Faster path-panning algorithms are endogenous. It’s important to distinguish between endogenous and exogenous predictions because exogenous predictions can be core assumptions, while endogenous predictions need to be broken down into “X will happen in five years IF A, B, and C” so that if A is behind schedule you can update your expectations for X. 
- Most roadmaps don’t talk about different scenarios. It’s absurd to think that there will simultaneously be no exogenous changes in the world *and* you are absolutely right about each step of the plan. [[Plans can change in a definite worldview]]. I would hope to see statements like “The primary path to success looks like A -> B -> C, but if B fails to achieve X, an alternative path looks like E -> F -> G.” This sort of thinking allows you to assign some sort of weight to the chance that different scenarios would play out and spend effort effectively. Scenario planning has been a discipline since at least the 1970’s, so there isn’t much excuse. [[What role does scenario planning have in technological program design?]]
- Most roadmaps don’t break down why a timeline is what it is. Most timelines are in the form of “in the next five years, we could expect to get to this point.”  That sort of estimate doesn’t give you any room to come to your own conclusion. A bulk time estimate also doesn’t let you 
- [[Most roadmaps are not execution oriented]]. Concretely, most roadmaps don’t address jobs to be done. There’s a description of applications (like elder care) and ‘tools’ (like path planning)  ::need a better word here:: but not “these are the challenges in path planning that need to be done in order to enable elder care.” 
- Most roadmaps are dead documents. [[Make living documents]] is almost universally true but is doubly so for roadmaps. The whole point is to try to shift the frontier, so hopefully within a few months something is different about the world than when the roadmap was created - whether it’s an exogenous assumption, an endogenous capability, or even just different information about the requirements on a subsystem thanks to a failed experiment. 
- Most roadmaps do not acknowledge that [[Program design is a coordination tool]]. 

### Examples
[THIS IS NOT A ROADMAP](https://home.cern/news/opinion/physics/roadmap-future)
[The Rejuvenation Roadmap | Lifespan.io](https://www.lifespan.io/road-maps/the-rejuvenation-roadmap/) - reasonable
This can’t help you make any decision https://nowandnext.com/PDF/Mega%20Trends%20and%20Technologies%202017-2050%20(Web).png

### Examples of roadmaps that don’t suck
The [[International Roadmap for Devices and Systems]] is actually useful for guiding the semiconductor industry. I suspect this is because there are well-defined interfaces, most progress will be incremental, entities are fairly fungible, and there’s strong incentive for new entities to take on different pieces of the roadmap (like in the semiconductor industry .

### Related
- [[The returns to good technical planning have decreased]]
- [[Instead of planning not working as well as people in the past thought as a society we’ve become worse at technological planning]]
- [[List of Roadmapping Examples]]
- [[Most predictions suck]]
- [[What are the properties of maps?]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Most_roadmaps_suck)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Most_roadmaps_suck)




Design for roadmapping system

Efficiency type heuretics can be breakthroughs when they get a thing to cross some threshold

Is there a minimum effective budget to generate outlier results?

The first step of roadmapping is frontier mapping

# Person targeted questions for roadmapping
Before jumping into the exact questions — a few notes on the general philosophy on questions. I believe that while [[Thoughts should be written, Questions should be verbal]] for the express purpose of clarification and tangents. Given that philosophy, it’s important to not be too prescriptive about the wording of a question or getting to all of them. Similarly, questions should *enable* situations where you want to dig into something unexpected. To that end they should be short and open-ended. 

Another thing is to go in always having done some pre-work. <The law about being wrong on the internet> also works in conversations as well, so it’s much more productive to throw out a wrong hypothesis and be corrected than to ask the other person to come up with a hypothesis on the spot. Of course, this runs the danger of anchoring them too hard. 

People are very bad at answering questions like [[What are the most important problems in your field?]]. Most people  outside of fields think about fields but not how (most) people inside of them think. 
I suspect that Hamming didn’t randomly walk up to people he had never met before and ask them — he was asking people in the Bell Labs cafeterias who he probably had preexisting relationships with or didn’t ask in exactly that way. 

The goal of roadmapping questions is to figure out:
- In a given area, what are the compelling goals (in as much detail as possible) 
- (As MECE as possible) what are the paths to get to that goal. 
- What (roughly indivisible) projects/intermediate goals need to be done along each of those paths 
- What does the dependencies between each of those projects look like?
- Which people/organizations are best suited to do each of those projects
	- Note here that “best suited” is not just about skill, but about alignment about the end goal, willing to do it, etc. 

The trick is that in almost every situation, you *cannot ask those questions*. People do not think in this way, especially practitioners in a field. 

- Questions after doing this for almost two (!? Oh Lord) years
	- I was very impressed by <thing in paper that shows you read it> where did the idea for that come from?
		- Notes: this only works if you are *actually* impressed. People can smell fake interest a mile away. 
		- This question is much less about the answer than signaling that you actually did your homework, making them feel special, and helping them understand which aspect of their work you’re most interested in. 
	- It seems to me based on your work that <X> might be promising but I haven’t found anybody working on it. Have you thought about that?
	- Why isn’t anybody doing that work?
	- What would enable them to do it? 
	- Have you considered working with <other discipline informed>? why not?
		- Note that this question can make you sound extremely dumb in the bad way if you don’t have a strong hypothesis for why they would want to work with that sort of person. 
	- So it sounds like <X> is particularly <hard/expensive> is that right? Why do you think that is? 
	- Do you have a sense of the big audacious goal for your work? 
		- This question is less meant to answer the question but to see how people react to it. 
	- Other people have told me <devil’s advocate position> what do you think about that?
	- <at the end of a conversation> Who else do you think I should talk to about this? 

- Original questions from [[22 May 2020]]
	- [[What are the most important problems in your field?]]
		- Why haven’t they been solved?
	- What do you think is the most promising thing in your field that is not being worked on? 
		- Why not?
	- What is the limiting factor keeping the system from being more widely used?
	- Can this heuretic be connected to that heuretic? Why or why not?
	- If this piece were changed, would any other pieces need to change?
	- Do you have a sense of what the budget would be to tackle these big problems?
		- [[What is the minimum amount to fund a project in a lab?]]
		- [[Is there a minimum effective budget to generate outlier results?]]

### Blurb
A bit of an ask if I may be so bold. I’m working on a hypothesis (inspired in part by Adam Marblestone’s lecture on roadmapping biology http://web.mit.edu/amarbles/www/docs/marblestone_revven_lecture2.pdf) that you can map research paths that can then enable coordinated work that wouldn’t happen otherwise.

If you (or someone you know) is down to do a sort of “user interview” about the frontier of your discipline, please do let me know.

### Related
- [[Design for roadmapping system]]
- [[What causes people to give super fuzzy answers to precise questions?]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Person_targeted_questions_for_roadmapping)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Person_targeted_questions_for_roadmapping)




§Academia Constraints

Low Hanging fruit theory of stagnation

Using computers to simulate possible experiment targets is still primitive

It is cheaper to do experiments on a simulated system than a physical system

# Physical experiments are expensive
In any field, experiments can require custom equipment ([[There are many pieces of equipment or tacit knowledge that only exist  in one or two places in the world]],  possibly rare consumables, specialized rooms or buildings etc.  Physics experiments can require massive facilities in remote parts of the world or even space and huge amounts of energy.  Biology has many time-related costs: experiments can require waiting for cells to multiply or breeding animals  (which I suppose is like multiplying cells inside a skin bag.) 

Because [[Science enables tools enables science]] the tools that enable cutting edge invention and discovery will *always* be expensive and require specialized knowledge.  Miniaturization will continue to put tools that were previously cutting edge into the hands of many non experts and they may discover things that the serious experts did not like biohackers and computer hackers before them. ([[Focused play enables serendipity]]). However, by the time those tools become inexpensive and easy to use, the heuretics that they allow people to access will be fairly picked over.  Unless we stop creating useful new scientific tools there will always be a new tool that is custom (and thus expensive) and require specialized skills.  An upshot of this is that the proud statement “all science experts do can be done on these inexpensive democratized tools!” should scare the crap out of you unless we actually have something like a replicator that can seamlessly turn imagination into reality.

### Related
- [[Frontier Technology is tough to create]]
- [[You can only push the frontier on dimensions where you are at the frontier]]
- [[Corporate R+D orgs are expensive]]
- [[The value of planning is inversely proportional to the cost of experimenting]]
- [[§Program Design]]
- [[§Atomland, Bitland, and Peopleland]]
- [[Manufacturing is important and often ignored]]


 

[Web URL for this note](http://notes.benjaminreinhardt.com/Physical_experiments_are_expensive)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Physical_experiments_are_expensive)




Instead of planning not working as well as people in the past thought as a society we’ve become worse at technological planning

What does good technical planning look like?

# Planning to Start vs Planning to Finish
Originally from [[raoPlanningStartPlanning2019]]
- Setting the stage (French word?)
- Improvisation is natural but getting to the point where you can is not natural

### Related
- [[Plotters vs Pantsers]]
- [[Systems vs Goals]]
- [[§Program Design]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Planning_to_Start_vs_Planning_to_Finish)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Planning_to_Start_vs_Planning_to_Finish)




When is externalized research better than internalized research and vice versa?

Heilmeier — Licklider — Braben — Elon quadrants of Research Management

What are the goals of the program design process?

PARPA H1 2022 Update Outline

How to think about the differences between FROs and private ARPA programs

Katarzyna Kate Adamala conversation 5 Oct 2021

Detailed outline for nanomodular electronics program

3 Sep 2021

8 Sep 2021

Week of 6 Sep 2021

6 Sep 2021

# Program design is a coordination tool
From [[weaverUnderstandingProgramsProjects2010]]: **Programs** *focus on the coordination of a number of related projects and other activities, over time, to deliver benefits to the organization*. (Contra projects)

Unlike something like architecting a building, the point of doing[[§Program Design]][^1] is not to create a blueprint that encodes all the information of the process and can be handed off to someone to execute on. In this sense, it’s a misnomer to call it “program design” and to call the resulting artifact a “roadmap.” Both of these terms imply a process whose main goal is to produce something static that can be separated from the people involved. In reality the goal is to create a shared purpose (in the [[dancoWelcomeDancoland2021]]/[[Complex System]] sense)[^2] and then maintain that shared purpose. 

Program design coordinates people in three distinct ways: 
1. The process creates is a Schelling point for ongoing *purpose maintenance*. [[Program design creates Schelling points for purpose maintenance]]
2.  The design process’ output isn’t just the artifact but is also connectivity between people, a shared understanding about who has what role, and a shared context between the people who are going to participate in the program. [[The most important role of program design is to build a shared context and connectivity between stakeholders]]
3. The process of *doing* program design itself is a coordination mechanism both between levels of a hierarchy and within levels of that hierarchy. [[The process of program design itself surfaces information through iterative feedback loops]]

### Upshots
There are several takeaways from treating program design seriously as a coordination mechanism. 
- [[Roadmaps need to be coupled to people]] except in situations where there are well-defined interfaces, most progress will be incremental, entities are fairly fungible, and there’s strong incentive for new entities to take on different pieces of the roadmap (like in the semiconductor industry [[International Roadmap for Devices and Systems]].
- [[Speeding up program design requires sales or power]].  
- [[Good program design can’t exist in isolation]]

### Pushbacks
[[Andy Matuschak]] points out that this description of program design clashes with the story of  [[JCR Licklider]]. Instead of an iterated process of understanding the state of a field and finding a target point that aligns the various stakeholders <[[Stakeholders are sometimes the maximum level of precision for complex systems]]>, Lick (the story goes) had a vision and did a combination of funding the few people who shared it and forced other people to go along with it by threatening their other funding. 

It’s the difference between 
1. Going around a boat and finding a mutually acceptable way for everybody to row and 
2. Only letting the people who will to row the way you want to row on the boat in the first place. 

There are several corollary questions here:
- Is the program manager’s role primarily *selection* and then acting as a communication facilitator OR is it primarily *alignment* and then acting as a shepherd and communicator to maintain that alignment? Another framing might be “getting the right people in the room” vs “getting the relevant people in the room and then making them work together.” 
	- 
- Is there a spectrum between those two roles? It feels that way. [[Programs and Program Managers exist on several different spectra]]
- Does the state of the world and a program manager’s power affect how program management needs to happen? Lick was hooked into some of the only sources of funding and had a lot of money, which allowed him to both enable people to spin up entire labs from scratch and put pressure on existing labs. 
	- How hard would it be to spin up a lab that wouldn’t exist otherwise in a university? 
	- Is there a way to “spin up a lab” outside of the university?
	- Does this only work with low enough equipment costs that you don’t need to have accumulated equipment over years?
	- There’s something about the gap between what people say they’re doing and what they’re actually doing that makes this whole thing very sticky.
		- There are many more people doing research <[[Research can mean drastically different things]]> on many different areas so finding people who aren’t at least adjacent to a thing is harder. Which means that no matter what you want to do you’re probably going to be at least nominally adjacent to what someone  else is working on. <[[“There’s a startup for that” is a harmful dogma]]> 
	- Does this suggest that the sorts of programs PARPA can undertake is also constrained by how many people are working adjacent to the area?
- Does the roadmap-as-coordination only work for incremental things that already have momentum like Semiconductors? 
- Have there been successful programs where the program manager didn’t come in with a strong vision from day one? 
	- If it is the case that every successful program needs a Lick-style PM, where did the person create the vision from in the first place? 
		- The assumption would be that it was primarily brewed in their own head from experiences? Is there more value to visions that come from experiences vs. conversations that are specifically for the purpose?
		- [[Can you only come at powerful visions sideways?]]
	- [[DARPA’s Process for Creating New Programs]] suggests that the answer is yes. 

### References
- [[Stephen Dean podcast 20 Aug 2021]]
- The [[The International Technology Roadmap for Semiconductors]] is widely acclaimed as one of the most successful roadmaps *because* it enables this coordination. 

### Related
- [[DARPA is a coordination mechanism]]
- [[Plans are coordination mechanisms]]
- [[PARPA is a coordination mechanism]]
- [[Coordination Problems]]
- [[Centralized coordination makes sure effort is not being repeated in stupid ways]]
- [[The less shared context you have the higher the coordination costs]]
- [[Centralized coordination sometimes stops effort from being repeated in smart ways]]
- One thing that does need to be decided up front is what organizational form the program will take. 2021 culture is heavily biased towards programs being structured as high-growth startups so answering the question [[When should an idea that smells like research be a startup?]] is important.
- [[The distinction between areas, programs, and projects]]
- [[Programs and Program Managers exist on several different spectra]]

[^1]: Note that as of at least [[3 Sep 2021]] I’m still feeling around what program design actually *is* but you can still make assertions about a [[Nebulous]] thing.
[^2]: At least as of [[8 Sep 2021]] I find this to be a really powerful concept to think about the equivalent for a “goal” in complex systems to sidestep the pressure to either think about goals or simply the things that a productive system *does* like conversations and decisions.

[Web URL for this note](http://notes.benjaminreinhardt.com/Program_design_is_a_coordination_tool)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Program_design_is_a_coordination_tool)






A lot of research feels like it is building lego pieces without any idea how they’ll be used

Building a system out of completed pieces of research is itself research

ICT Distillation

6 Jan 2021

9 Jan 2021

ICT Outline

# Program design is like creating the lego instruction book
Modern research institutions tend to produce a lot of (ideally, but usually not fully) atomic contributions.[^1] These contributions resemble lego bricks in that you can intuit that while they are not useful on their own, they will be useful as part of a broader system. But like most lego bricks it’s not immediately clear what amazing thing the brick can eventually be part of (and in fact it can be part of many things!) Even a giant bucket of lego bricks often doesn’t immediately suggest what to build. This ‘bucket-of-bricks’ is the state of many disciplines. That description is extreme. The situation is more like we have the picture on the front of the box and the bricks that came in the box mixed with a bunch of extraneous ones. Thought leaders and big ideas people are talk about the different box-pictures. So everybody has a vague idea that their brick is a part of the spaceship, but if you’ve ever put together a lego set, it’s rarely clear where each part fits in until you see all the intermediate steps in the instruction book. 

The goal of [[§Program Design]] is to create these intermediate steps. For example, the painfully unsexy step 34 that has a bunch of tan and grey pieces in a rough octagon with some slopes on it that doesn’t look like anything at all. It is *hard* to know that step 34 is going to lead to the awesome spaceship on the box; it certainly doesn’t look like a spaceship. 

Getting step 34 to happen is hard. You need to create a sense of the interior of whatever the picture on the box depicts, which pieces are available, which pieces you would ideally have, and which substitutions are ok (it doesn’t really matter if you use a neon green brick instead of a grey one as an interior part.) If you really need a piece that doesn’t exist, you need to encourage that to happen. And then you need to get all the pieces together - just because you put together an actionable plan does not mean people will take that action!

Ideally, as many steps as possible are relatively self-contained ‘modules’: the cockpit part of the spaceship, a secondary landing craft, an escape pod. In real life, this corresponds to intermediate goals that are useful on their own or at least exciting milestones or demos. There are pages to be written about the tension between the real benefits of useful intermediate steps and the danger that undue focus on useful intermediate steps can pose to eventual outcomes.[^2] In hindsight it’s easy to both look at technological development and forget how many incremental useful steps there were - top of mind is that synthetic plastic was used as a nifty liquid by printers before it was accidentally cooked - and to forget how much work has gone into technologies before they get to *anything* useful - transistors needed eight years of work before the first completely unusable point-contact demonstration. 

Obviously, this analogy can only be pushed so far: research-heavy programs will inevitably have gaps between steps that would make any lego-assembler tear their hair out: the picture on the front of the box might be fuzzy or wrong; In real life, pieces don’t just snap together; [[Heuretics come out of people’s brains (TECHNOLOGY IS PEOPLE)]], so you need to deal with incentives, politics, and egos; etc.

You could reasonably object to this analogy with a counter-analogy that atomic pieces of work are less like legos and more like biological components. Separate components in a biological process have much more ‘agency’ - if you create the right environment and shake or [apply a jolt of electricity](https://en.wikipedia.org/wiki/Miller%E2%80%93Urey_experiment), they will self-assemble. The biological analogy would be much more apt than a lego analogy if paradigm shifting work cannot be planned or the innovation ecosystem is ‘directionally’ correct and just needs more funding and work along existing channels.  

I suspect the reality is a mix of both analogies - unlike in lego construction you can’t lay out every single step so ‘biological’ self-assembly between steps will be essential. Keep in mind that these analogies are meant as thinking tools, not evidence or predictive descriptions of the world. [[Analogies should be used as thinking tools but not as evidence]]. Hopefully, though, the Lego analogy is provocative. Playing with it is a more tangible way to describe the need for program design overall and suggests extensions, like the the the possible role of simulations <[[Good Simulations could be the ‘why now’ of roadmapping tools]]>, and legible intermediate steps. 

### Related
- [[A lot of research feels like it is building lego pieces without any idea how they’ll be used]]
- [[Academic-driven fields feel like a whole bunch of mixed legos]]
- [[The process of creating a new heuretic node is like building something out of legos]]

[^1]: At the same time, modularization is important! See:[[The Architecture of Complexity]] and [[arthurNatureTechnologyWhat2009]] [[Modularization enables pieces to be used in many different technologies]] etc. 
[^2] : See -  [[Most roadmaps suck]], [[Decoupling from market discipline is like cave diving]], [[Systems must take performance hits to get out of local optima]]

[Web URL for this note](http://notes.benjaminreinhardt.com/Program_design_is_like_creating_the_lego_instruction_book)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Program_design_is_like_creating_the_lego_instruction_book)






Adam Marblestone Podcast 2020-03-31

Roadmaps have two pieces - the map and the path

The structure of the tech tree can inform actions that are short term suboptimal but long term useful

# Roadmaps can enable faster progress than blind exploration
- Roadmapping ideally makes the goal very precise and chains together the unknowns you need to elucidate in order to get to the goal. 
- If you’re in a situation where [[Long term plans often involve decisions that look stupid over shorter time horizons]] roadmapping can prevent you from doing the “smart” decision that you will ultimately need to reverse. 
- If you know the unknowns and their dependencies, you always know the [[Highest order bit]], so you always know the most important thing to work on to win. 
- When is this not true?
	- If there is no clear goal, going in a specific direction is almost useless
		- Roadmapping is not particularly useful for a lot of software because it simultaneously figures out the goal 
		- It’s harder to know what the goal is if it’s not technical 
	- If there are many unknown unknowns 
	


### Related
- [[Plans can change in a definite worldview]]
- [[The value of planning is inversely proportional to the cost of experimenting]]
- [[Good plans have several steps]]
- [[Predictably successful ideas are engineering, unpredictably successful ideas are Science]]
- [[A predictably successful idea is composed of known unknowns that that a team can execute against]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Roadmaps_can_enable_faster_progress_than_blind_exploration)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Roadmaps_can_enable_faster_progress_than_blind_exploration)




# Roadmaps have two pieces - the map and the path
### The Map
The map needs to come before the path - if you create the path without the map you have no way of judging where it stands with respect to other possibilities. However, you only care about the map insofar as it can inform creating the roadmap. So you don’t really care about population density maps or topography if you’re in a car. 

What to include or not include in the map is tricky though because you *do* care about topography if you’re riding a bicycle or walking. This ‘vehicle dependence’ suggests that creating the map and the path needs to be an iterative process. 

Map detail is important for seeing path possibilities and path alternative possibilities, however it’s impossible to entirely map possibility space. There is a tension between the fact that you need a map to draw the road but could spend forever creating the map. Perhaps you can resolve this tension by analogy to occupancy maps, where you binarize at each level and only subdivide in places where you know there’s a hit. [[The first step of roadmapping is frontier mapping]].

In a way, science is cartography, engineering is highway engineering. Science describes what is possible, which engineering makes that thing possible. [[narayanamurtiCyclesInventionDiscovery2016]]. By analogy, part of the roadmapping process can be “well we need to send a surveying team here so we know whether we can build over or around that area.” 

The map is analogous to some kind of optimization landscape. While I normally like to think of maxima as the place you’re trying to get to in optimization problems, having minima be the places you’re seeking makes more sense with the analogy of a map because then it’s very hard to get over “mountains.”

The map is also isomorphic to the structure of [[The Idea Maze]]

### The Path
The path is a hypothetical traversal of the map to get to a specific point. This definition should suggest how important it is to actually understand where you are starting and where you want to end up. 

The path should suggest the answers to the [[The Heilmeier Catechism]] and also suggest alternative routes should one be blocked. 

### Related
- [[Roadmaps can enable faster progress than blind exploration]]
- [[Technological roadmapping can give an idea of where technological evolution can go and how to get there]]
- [[Roadmapping could be thought of as backwards-chaining together a series of conception-only heuretics]]
- [[The first step of roadmapping is frontier mapping]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Roadmaps_have_two_pieces_-_the_map_and_the_path)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Roadmaps_have_two_pieces_-_the_map_and_the_path)




Life Style Guide

# Say ‘X is more important than Y’ instead of ‘X is important’
For example [[How to Take Smart Notes]] states that it is important to be able to opportunistically change objective. 

For example, startups should be able to pivot quickly.
There aren’t any areas where it isn’t at least useful to be able to opportunistically change objectives. However, that will come at a cost. Which tradeoffs to make is the important chunk of knowledge.

### Related 
- [[Life Style Guide]]
- [[§Program Design]]
- [[Precise communication enables other people to make decisions]]
          


[Web URL for this note](http://notes.benjaminreinhardt.com/Say_‘X_is_more_important_than_Y’_instead_of_‘X_is_important’)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Say_‘X_is_more_important_than_Y’_instead_of_‘X_is_important’)




When should we defer to experts?

We need more metaexpertise

# Talking to experts can expose bad equilibria, the moves necessary to get out of them, and considerations for doing them like cost constraints
### Sub Hypotheses
- People in a discipline *know* where the bad equilibria are
- People in a discipline have hunches about possibilities for getting out of bad equilibria 
- People would talk to you about things if they think that it will enable them to work on things
	- This would suggest that we need to connect in their mind with the possibility of funding

The danger here is that [[Structuring knowledge is expensive]] 

The reason for the conversation piece is that [[Context is important and underrated for knowledge transfer]]

### Questions

### Experiments
- Have multiple ‘exploratory’ conversations in multiple disciplines with experts and play with different ways of asking questions, but make sure you’re targeting clear 

### Related
- [[You can create a framework for talking to people that makes it easier to generate a roadmap]]

Super hypothesis:[[You can create a program design discipline that enables better research and development]]
[Web URL for this note](http://notes.benjaminreinhardt.com/Talking_to_experts_can_expose_bad_equilibria,_the_moves_necessary_to_get_out_of_them,_and_considerations_for_doing_them_like_cost_constraints)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Talking_to_experts_can_expose_bad_equilibria,_the_moves_necessary_to_get_out_of_them,_and_considerations_for_doing_them_like_cost_constraints)






# Technical Planning Principles
Each of these principles must reference one or more of the idealized properties of [[What does good technical planning look like?]]
 
Since we’ve established that good technical plans resemble roadmaps, what happens if we invert why [[Most roadmaps suck]]?
- ***State a precise, realistic, but ambitious vision*** What does winning look like?
	- 1, 8.1, 8.4, 8.8, 15
-  **Make living documents**  “No plan survives first contact with the enemy.”  It takes work, but living plans are powerful coordination tools that helped us get to the Moon. Computers make living documents easier to create but the liveness needs to be designed in from the beginning - ways to keep track of what was changed when, what depends on what, etc.
	- 1, 2, 3, 5, 6, 7, 9, 15
- **Graphical information should be a first class citizen** Text is great for explaining, making arguments, and creating a narrative, but it is terrible at creating a holistic overview.  [[Maps are multi-scale]] and it is hard to achieve that with text. 
	- 3, 4, 14
- **Talk about who could work on what.** More detail is better, from precise skillsets down to specific people or groups.  The latter is especially important for work on the knowledge frontier because [[There are many pieces of equipment or tacit knowledge that only exist  in one or two places in the world]]. 
	- 1, 2, 10, 11, 12
- **Prioritize problems to go attack.*** It’s hard but critical to identify ‘need to haves’ vs ‘nice to haves’ to make a vision a reality.  
	- 1, 2, 8.1, 8.4, 8.5, 11, 15
- **Make it clear what has been tried before and why it didn’t work.*** Most visions are not new and the people who tried to achieve them weren’t stupid. These two facts mean that you’re likely to try something that someone else has tried before. Doing this is not *necessarily* bad - the world can change, but the burden of proof is on you to explain what those changes are and why they will make the difference between success and failure.  
	- 8.2, 8.3, 9, 13
- **Explain both technically and politically why the vision isn’t a reality today** Technical projects don’t exist in a vacuum. Sometimes the real reason a vision doesn’t exist is because of regulation, cost, or institutional bias. It’s not a party-killer if the primary blocker on a technology isn’t sheer technical capacity but it does change how to think about it and the work that needs to be done to make it a reality.  [[You can transform between technology constraints, regulatory constraints, and cultural constraints]]. Non-technical reasons don’t have to be broad macro explanations either - it could be that professor A and professor B hate each other so they never integrated their technologies. 
	- 7, 8.2, 8.3, 11, 13
- ***Use numbers but no more than necessary*** Numbers are great because they are the most unambiguous way to compare two things, set targets, and see improvement. However, [[Metrics can cause paradigm lock-in]],  the [[Streetlight Effect]], and they can give a false sense of progress (ie. going from 50% to 75% is unambiguously a 50% improvement but is actually worthless if the same methods won’t get you to 99.9% and nothing useful will happen until get there.)  If there are one or more numbers that people are trying to hit, even the most diligent person will focus on the number instead of the high level goal. The upshot is that numbers are powerful, in both good and bad ways so be judicious about their use. Numbers are just one way to represent constraints and goals - there are other low-ambiguity markers of success and progress: precisely articulated scenarios, reactions of people using a thing to do a real job, etc. 
	- 1, 2, 3, 4, 7, 8.1, 8.8, 11, 14
- **Draw a clear line between the parameters that matter in a given context and those that don’t.** Both technologies and scientific theories have many operating parameters. One difference between the two is that while a scientific theory wants to explain as many phenomena and be applicable in as many situations as possible, a technology just needs to work in a set of specific contexts. A technical plan should make it clear which parameters matter for the contexts a technology is intended for. (ie. Weight really matters for a technology meant to be used on a spacecraft but doesn’t matter at all for one meant to be used on a boat.)  
	- 2, 8.4, 9, 11, 12
- **Address multiple development scenarios.** Anybody who has does technical development work knows that the first thing you try often doesn’t work. Resorting to alternatives is not a bad thing, but not calling out what they might be (and the failure modes that might lead to them) means that the plan doesn’t reflect reality from the start.  Good statements look like like “The primary path to success looks like A -> B -> C, but if B fails to achieve X, an alternative path looks like E -> F -> G.” Additionally, external circumstances might change - funding could increase or decrease or external technologies could be invented. Acknowledging this and providing multiple “tracks” the project could follow make it more likely to succeed.
	- 1, 2, 7, 8.5, 9, 11, 12, 
- **Explain timelines and budgets in as much detail as possible but no more.** Demystifying timelines and budgets open the possibility that someone else can help figure out ways to decrease them “oh, you need to buy that machine? A friend has a spare” etc.  At the same time, be honest - if you don’t know, you don’t know. Pulling timelines and budgets out of thin air is far too common and doesn’t help anybody. 
	- 2, 3, 7, 8.6, 8,7, 12
- **Be execution-oriented.** Unless there is a great reason to believe that some work will happen exogenously to the scope of the plan, everything needs to be done by a real human. Focus on specifically what those experiments, build projects, and other actions are. 
	- 2, 3, 8.6, 8.7, 10, 12

We can do even better than  just inverting the things that suck, but by embracing [[What are the properties of maps?]] 

- **Make the plan useful at multiple levels of detail starting with just a glance.** You should be able to get some information about the plan just by glancing at it and then more and different information by engaging further. Even the people who care about minute details appreciate an easily digestible overview. This principle is one of the reasons that images should be first class - it’s very hard to get information from glancing at pure text. 
	- 3, 4, 5, 8.1, 14
- **Help people come to their own conclusions.**  Giving a reader the power to disagree forces you to show  them that your plan is the best plan instead of asserting it. Room for disagreement enables much stronger buy-in in the long run. How do you help people come to their own conclusions? …
	- 1, 2, 3, 5, 7, 9, 11, 15
- **Lay out constraints, assumptions, and affordances** Constraints look like “this technique doesn’t doesn’t work in temperatures about 5K.” A lot of technological development is about relaxing or finding ways around constraints. Affordances look like “this tool is great at doing X and Y” - they’re kind of the mirror of constraints. A lot of technological development is about expanding or finding clever uses for affordances. Assumptions look like “we’re assuming that we’ll be able to run everything on off the shelf parts.” Explicitly calling out constraints, assumptions, and affordances helps people understand tradeoffs. 
	- 1, 2, 7, 9, 11, 13
- **Draw lines between the exogenous and endogenous.** Exogenous changes are things you expect to happen external to the plan - for example, it’s a good bet that there will be better GPUs in two years. Exogenous are fine to incorporate into the plan as assumptions. However, it’s important to call them out and ideal to outline different scenarios for if they fail to happen. It’s important to draw a bright line between exogenous and endogenous changes because endogenous changes fall under the responsibility of whoever is executing on the plan while exogenous changes do not. 
	- 2, 7, 13, 15 

### Related
- [[There are scientific and technological problems that make other technology advances irrelevant by altering the environment]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Technical_Planning_Principles)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Technical_Planning_Principles)




Systems must take performance hits to get out of local optima

The DARPA execution framework boils down to showing that thing is not impossible, showing that thing is possible, and then making that thing possible

Constraints and affordances are more useful to think about than problems

PARPAs first evolutionary stage is individuals doing program design

How do you incentivize people to work on nodes farther down the tree?

# Technological roadmapping can give an idea of where technological evolution can go and how to get there
You can think of roadmapping as thinking multiple nodes up the [[§Tech Tree Model of Heuretics]].  Possibly pointing out where you need to take one step back to go two steps forward or work on something that isn’t immediately useful in order to create something that is even more useful.  [[Systems must take performance hits to get out of local optima]].

- Technological roadmapping might be one answer to [[What does technological husbandry look like?]]

### Related
- [[Road-mapping Biology - Article]]
- [[boydenArchitectingDiscoveryModel2019]]
- [[Design for roadmapping system]]
- [[Constraints and affordances are more useful to think about than problems]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Technological_roadmapping_can_give_an_idea_of_where_technological_evolution_can_go_and_how_to_get_there)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Technological_roadmapping_can_give_an_idea_of_where_technological_evolution_can_go_and_how_to_get_there)




raoGUTSGrandUnified2018

weaverUnderstandingProgramsProjects2010

Institution builders are initially judged based on their programs

olsonManagingModesEffort2020

A taxonomy of research organizations and activities

It is important to differentiate between the needs wants and activities of funders, institution designers, program managers, and researchers in the context of precommercial scitech

Research projects have a halting problem

# The distinction between areas, programs, and projects 
# The distinction between areas, programs, and projects 
- **Projects** *are chartered to create a speciﬁed “deliverable” as efﬁciently as possible*[^1]. They are are the ‘discrete’ unit of work. Of course, projects can still have [[Knightian Uncertainty]], but they have a clear definition of what “done” looks like (is this true?) 
- **Programs** *focus on the coordination of a number of related projects and other activities, over time, to deliver benefits to the organization*.[^2] 
- **Areas** are broad groups of interest like “climate” or “space exploration.”  Unlike programs, they are both not necessarily focused on delivering benefits and don’t require coordination.

Note that these distinctions can span all the quadrants in [[raoGUTSGrandUnified2018]]/[[obengAllChangeProject1996]]. 

### Related
- [[§Program Design]]
- [[Program design is a coordination tool]]

[^1]: From [[weaverUnderstandingProgramsProjects2010]]
[^2]: Also from  [[weaverUnderstandingProgramsProjects2010]]

[Web URL for this note](http://notes.benjaminreinhardt.com/The_distinction_between_areas,_programs,_and_projects)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_distinction_between_areas,_programs,_and_projects)






# The first step of roadmapping is frontier mapping
[[It’s hard to discover where the frontier is]]

This is what [[The Longevity FAQ]] and [[Climate Technology Primer]] are.

There may be two types of frontier map: a holistic frontier map and a purely technical map. 

Holistic maps don’t assume that the world will beat a path to your door if you build a better mousetrap. A holistic map tries to take into account the environment: all the different constraints around diffusion - alternatives, markets, regulation, etc. It tries to answer “what are literally all the obstacles and opportunities between this possibility and becoming a prevalent part of the world.”

Technical maps ignores the environment and only cares about better mousetraps. These maps are like an underwater cave system or one of those video game maps that’s full of  fire and you take damage every turn. You can use it to get somewhere you couldn’t ordinarily get to while coupled to the market system ([[Systems must take performance hits to get out of local optima]]) but you can’t stay there forever. [[Decoupling from market discipline is like cave diving]].

 [[Technological Whitespace]] is areas on the map that seem promising.

At first glance might seem like Technical maps are naïve and if you can you should always work with a holistic map. However! Holistic maps have many more assumptions about the world baked into them. Technical maps are less likely to change in surprising ways. 

### Related
- [[Person targeted questions for roadmapping]]
- [[Design for roadmapping system]]
- [[Technological S curves]]
- [[Good Simulations could be the ‘why now’ of roadmapping tools]]



[Web URL for this note](http://notes.benjaminreinhardt.com/The_first_step_of_roadmapping_is_frontier_mapping)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_first_step_of_roadmapping_is_frontier_mapping)




neumannOneProcess2020

§Tech Tree Model of Heuretics

A heuretic’s impact is not endogenous to the technology but depends on the environment

Technodiversity is important

# The tech tree model suggests that explicitly drawing out tech trees is important
Let’s make sure this isn’t a normative belief
1. If you draw out the tech tree, you argue why working on a suboptimal technology might actually lead to amazing things in the future.  The evolutionary analogy would be “Why should we keep around these little hairy mammal things?”  (See: [[alexanderStudiesSlack2020]])

2. [[Good plans have several steps]] , [[Long term plans often involve decisions that look stupid over shorter time horizons]]

3. Drawing out the tech tree enables you to explicitly see the state of different disciplines areas and constraints on them

4. Drawing out tech trees makes unknowns explicit

5. An explicitly drawn tech tree helps see what you *could* be doing that you aren’t doing instead of working on the immediate adjacency
	- [[The best DARPA program managers are the ones who can look at an entire literature in an area and notice a systemic bias]]

6. It biases work towards potentially more productive ends

At the end of the day this really does come down to a philosophical difference over whether you think that people can plan better than dumb chance.  [[A definite worldview means you believe the universe is deterministic]] and [[Indefinite worldviews and definite worldviews enable exploration of different parts of idea space]].

### Related
- [[List of Roadmapping Examples]]
- [[Most roadmaps are not execution oriented]]
- [[§Program Design]]



[Web URL for this note](http://notes.benjaminreinhardt.com/The_tech_tree_model_suggests_that_explicitly_drawing_out_tech_trees_is_important)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_tech_tree_model_suggests_that_explicitly_drawing_out_tech_trees_is_important)




Program and project management is important and valuable

Project management  is important if transitioning between TRLs is complex and expensive

We need to take a more engineering approach to enabling more awesome shit

The returns to good technical planning have decreased

Definite optimism requires both a vision and evidence that we are on a trajectory to reach it

Why did business plans fall out of favor?

# The value of planning is inversely proportional to the cost of experimenting
Both planning and experimenting are uncertainty-reducing activities. This means that in the face of uncertainty 
1. Lower the cost of experimentation
2. Plan more and better

In a way, plans are simulated experiments. The best experiments get rid of uncertainty *and* risk, while plans try to reduce uncertainty, but cannot get rid of risk.  [[Uncertainty always involves risk but risk does not always involve uncertainty]]. Therefore, it’s always better to do experiments when you can. However, experiments can cost time and resources, so the more expensive the experiments, the more important planning becomes. 

This would suggest that planning is more important for physical experiments because [[Physical experiments are expensive]].  ([[§Atomland, Bitland, and Peopleland]])

Another corollary is that definiteness in the Thielian sense loses value in cheap-to-experiment environments.  [[Definiteness means that you have a long-term plan]].  This corollary would imply that the lamented cultural loss of definiteness wasn’t the result of some moral decay. Instead it was correct move in a world where experiments were cheap (ie. High profit software.)  

### Related
- [[§Program Design]]
- [[Experiments can lead to false certainty]]
- [[Slack - concept]]



[Web URL for this note](http://notes.benjaminreinhardt.com/The_value_of_planning_is_inversely_proportional_to_the_cost_of_experimenting)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_value_of_planning_is_inversely_proportional_to_the_cost_of_experimenting)




# Top of Mind Questions
- [[When is externalized research better than internalized research and vice versa?]]
- [[Is there a minimum effective budget to generate outlier results?]]
- [[Can impact certificates work for research?]]
- [[Could we make a system that could spin up transformer production from nothing in a day?]]
- [[What does a modified Heilmeier catachism look like?]]
- [[How should SpecTech donations work?]]
- [[How can we make roadmaps living documents?]]
- [[What are the goals of the program design process?]]

[Web URL for this note](http://notes.benjaminreinhardt.com/Top_of_Mind_Questions)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Top_of_Mind_Questions)






28 Aug 2021

24 Aug 2021

25 Aug 2021

26 Aug 2021

27 Aug 2021

About these notes

# Top of mind
(Updated 26 Nov 2022) 

The intention of this page is to serve the function of  [a /now page](http://nownownow.com/about)  - shamelessly stealing the idea of a notes-based now page from the indomitable Andy Matuschak.

Most of my time is spent building the new [new institution](spec.tech) I outlined [in the piece I wrote on building a private ARPA-riff.](https://benjaminreinhardt.com/parpa). (More details soon!)

This entails:
- Finding [potential program managers](https://benjaminreinhardt.com/parpa#its_possible_to_find_people_who_are_both_willing_and_capable_to_be_program_managers) and working with them on the fist steps of [[§Program Design]].
- Working on our first two programs. Specifically [[Nanomodular Electronics]] and [[Molecular Additive Manufacturing core]]
- Finding the money to fund these things. 

The nature of trying to create the organization still raises a number of research-y questions: [[Top of Mind Questions]]

And of course, I can’t help occasionally durdling around with some other ideas:
- [[In Defense of Academia]]
- [[Research is not a commodity]]
- [[We have no good metrics for successful research]]
- [[Directly acknowledging alignment and misalignment can be powerful]]
- [[Periodic table of research organizations OR Innovation Engineering]]
- Recording an occasional [Idea Machines Podcast](ideamachinespodcast.com)as a way to talk to fascinating people and share it with the world.

On the physical front, I’m pretty plateaued on climbing :-/ I’m not quite sure how to break through it — probably need to get a coach. 



[Web URL for this note](http://notes.benjaminreinhardt.com/Top_of_mind)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Top_of_mind)




Bottlenecks Workshop June 2021

Real bottlenecks make a situation feel intractable or hopeless

# What are the different kinds of bottlenecks?
# What are the different kinds of bottlenecks
Periodic table of bottlenecks? [[Periodic tables are a useful way of categorizing]]

### Related
- [[Program design could be thought of as creating a causal graph of blockers and hypotheses about how to mitigate those blockers]]
- [[§Program Design]]

[Web URL for this note](http://notes.benjaminreinhardt.com/What_are_the_different_kinds_of_bottlenecks)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/What_are_the_different_kinds_of_bottlenecks)






# What are the pieces of a roadmapping architecture?
# What are the pieces of a roadmapping architecture
For example, one could imagine a database of assumption numbers (which could be a spreadsheet, a list of numbers in a jupyter notebook) etc 

- [[Is there a way that python collab notebooks could be integrated into program design and roadmapping?]]
- [[§Program Design]]



[Web URL for this note](http://notes.benjaminreinhardt.com/What_are_the_pieces_of_a_roadmapping_architecture)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/What_are_the_pieces_of_a_roadmapping_architecture)




Most roadmaps are not execution oriented

# What are the properties of maps?
# What are the properties of maps
“Maps”, “Mapping”, and “Roadmaps” feel like [[Suitcase Handle Word]]s but  there is a nugget of desire in using them. What are we really yearning for from a map?

- [[Maps are multi-scale]]. You can get some information just by glancing at a map and then more and different information by zooming in.  Fractals were literally discovered by someone mapping coastlines.[[Every time you zoom in on a fractal, it presents the same pattern on a different scale]]. 
	- If a map doesn’t give you something useful on first glance, it’s not doing its job.
		- Most pure-text roadmaps fail on this point - it’s very hard to get something useful on first glance from pure text unless it has a well crafted outline. 
	- If you can’t get different useful things when you investigate a map more closely it’s not doing its job. 
-  [[Maps help you understand the state of something]].
	- If you don’t understand the state of something better after looking at a map, it’s not doing its job. 
- [[Maps help you figure out possibilities]]. Even the medieval T-O map let you know that you could get to Africa from Europe by crossing the mediterranean.  
	- *Roadmaps* explicitly point out those possibilities in an opinionated way. 
	- If you can’t figure out possible (and unallowable moves) from a map, it’s not doing its job.

![](BearImages/A1F27DD7-DF6B-467B-96E4-599D26CC393D-32352-00023ECE922974A5/DEE0FD40-B247-4C7C-82BD-5CB1EB003417.png)

### Types of maps
- Conceptual 

### Related
- [[Wardley Maps]]



[Web URL for this note](http://notes.benjaminreinhardt.com/What_are_the_properties_of_maps)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/What_are_the_properties_of_maps)




Scientists, Engineers, and Economists should write more science fiction

Narratives are how we deal with uncertainty

A series of ARPA-like programs and commercialization could make general-purpose telerobotics inevitable

Tail risk analysis should specify first, second, third order effects

# What role does scenario planning have in technological program design?
# What role does scenario planning have in technological program design
### Related
- [[§Program Design]]

[Web URL for this note](http://notes.benjaminreinhardt.com/What_role_does_scenario_planning_have_in_technological_program_design)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/What_role_does_scenario_planning_have_in_technological_program_design)






And Suddenly the Inventor Appeared

Wardley Maps

Can you apply TRIZ to people land?

Michael Goldblatt Conversation 18/3/20

Proof Discovery Tree - PDT

# Wigmore Guide - Paper
::The paper doesn’t seem to exist anymore::
![](BearImages/F1A49B1D-77BE-47F9-8505-4AB1CB11E30A-68756-00029E9C4F8155A4/81F9376C-6EED-4091-BA2F-DA901D337F2B.png)

1. Read statement of facts
2. Read what person is accused of and definition of that fact
3. List possible defenses against the charge
4. Lay out the ultimate question to be proved and the penultimate questions
	- Each question has a evidential fact counterpart - *factum probans*
	- You know that you have found all the lower level questions if all of them answered positively would undoubtably prove the higher level question
	
5. Consider the evidence in relation to the probanda and formulate questions
6. Connect the ultimate fact to the penultimate facts
7. Link evidence to the penultimate facts directly or through deduction or induction
8. Defense possibilities go to the side and target different parts
### Published
## Questions

## Thoughts
## Related
- [[What are the eight steps of Wigmore analysis translated to science?]]
- [[Stretchable Actuators Questioning Doc]]
- [[Design for roadmapping system]]
- [[wigmoreTreatiseSystemEvidence1908]]

## Authors
## Link
[(PDF) WIGMORE GUIDE](https://www.researchgate.net/publication/331198370_WIGMORE_GUIDE)

[Web URL for this note](http://notes.benjaminreinhardt.com/Wigmore_Guide_-_Paper)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Wigmore_Guide_-_Paper)






Benʼs working notes

§Program Design

Hypotheses

What are roadmaps

Why Roadmapping

Building better roadmaps

Glossary

References

Links to this note