# Quantized Heilmeier Catechism DARPA famously uses [[The Heilmeier Catechism]] as a basis to judge programs. However, going through the catechism is an exteremely involved process, requiring both the interlocutor and the answerer to do a lot of work. The trick with the Heilmeier Catechism is that it’s an information-extraction and thinking tool, not a decision making tool. In many situations, the ability to give precise answers to the questions is more important than the questions themselves. But the answers to the questions *do* matter to some extent - a program that costs precisely $1,877, 811, 952.04 still wouldn’t cut it. Good answers to the questions about cost and time are contingent on the rest of the program. At the end of the day, whether or not a program is good is left at the discretion of the arbitrator. One way to turn the Heilmeier Catechism into a decision-making tool would be to binarize it (turn it into a sequence of yes/no questions) or quantize it (assign points) a la the [[Braben Venture Research Index]]. Shifting from e.g. “How long will it take?” To “Does the proposal have a precise, believable timeline?” Or “0 points for vague timeline with little justification, 1 point for precise timeline with justifications, 2 points for multiple conditional timelines” would still encourage the same outputs as the original Heilmeier Catechism, but also highlight and enable discussion around “what does good look like.” A danger with quantizing anything is that once you start putting numbers on a thing, it is tempting to start treating it as an objective metric - see happiness research. A quantized Heilmeier Catechism must absolutely avoid this. There are a ton of examples of how metrics screw up systems ([[Goodhart's Law]], [[Fair games can be gamed]], etc.) and how good research management is based on trust ([[Research requires more trust than other disciplines]].) So I would argue that a quantized Catechism should explicitly be a ‘thinking’ guide more than a decision guide. There is still tons of built-in discretion: it’s impossible (and inadvisable) to tightly define ‘precise’ or ‘believable.’ And as Braben notes for the BraVRI, overriding personal discretion should be totally allowed. Idea from [[Arnaud Schenk]] ### Related * [[The Heilmeier Catechism]] * [[Molecular 3D printer Heilmeier Proposal]] <!-- #evergreen #project/spectech --> [Web URL for this note](http://notes.benjaminreinhardt.com/Quantized+Heilmeier+Catechism) [Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Quantized+Heilmeier+Catechism)