There are two coupled pieces here - DARPA works on things that are market failures and the activity that DARPA undertakes is a market failure.
From What is a market failure? - the market failure Questions are:
The rational expectation value of many DARPA programs is net negative negative because the chances of success are so low despite an amazing outcome if successful. Negative expected net-value programs correspond to #2 on the market failure list — specifically that the risk-weighted value of the thing is below what people would pay. More generally, from a pure expected-value point of view isn’t irrational to work on high variance, high potential upside but low mean things? For example, many tech entrepreneurs also fall into this category. On average, starting a company has lower lifetime value than an engineering job at BigCo. In the entrepreneur’s case you could make an argument around ‘fulfillment’ being part of the expected value calculation, but once you open that door, it starts to smell like Epicycles. COVID tests are an interesting case study here - they would be an incredibly valuable thing for the world, but people might not pay as much for them as it would cost to develop them.
DARPA works on programs whose value cannot necessarily be captured by the group coordinating it. (#3 on the market failure list) Value-capture failures could look like autonomous cars. DARPA pushed autonomous cars through a series of competitions, but at the end of the day the different participating orgs owned the work. My hunch is that if DARPA had demanded that they hand over all IP from the competition, fewer people would have participated and there wouldn’t have been an industry flourishing. In general, these cases are problems that need a bunch of different groups to coordinate and/or compete, each of whom needs a large enough chunk of ownership or low enough friction to motivate them that trying to capture value would kill the impact.
DARPA works on hard-to-monetize intermediate products which are another value-capture type market failure (#3) . In this case, the work to be done creates an intermediate result result still needs a lot of work by another group to become a sellable product. For example, a prototype that needs a lot more work by an organization with a different set of expertise to become manufacturable. These cases are value-capture failures because so many different heuretics go into the final product that’s actually valuable. The multitude creates a situation where 1. It’s hard to credit everybody who contributed pieces 2. It’s hard to assign value to each contribution 3. Even if you could perfectly figure out the value of every piece and who the money should go to there is massive friction in the system if everybody is trying to get their piece of the pie: either the final organization needs to spend time and effort licensing every piece of the puzzle or the previous folks need to sue them which burns tons of money on lawyers. Instead, DARPA pays the previous folks directly for the research and then gets the thing out into the world.
DARPA works on things that require funding on a combined time-risk scale that doesn’t jam with financial markets (#4). Venture capitalists need to fund things that increase valuation in a few years and return in 5-7 years, which just doesn’t work for projects that can take more than ten years to create a product. Long-term financial instruments like mortgages and loans require much lower risk. You also need to beat the stock market - financial markets don’t just care about the absolute expected value of the project, but what the alternatives for that capital are. If the stock market is expected to give you a 5% rate of return, you need to do much better than that because the risk is higher.
Even if criteria 1 through 4 are met, enough people need to believe that they are true and the benefits outweigh their alternatives. (#5) DARPA works on systems where several things need to be changed at once. However changing any one of those things would actually make the system work worse, so any rational person only wants it when the whole thing is done. The need to build multiple pieces of the same system to create any value creates compounding risk, which puts programs back into market failure scenario 1.
The fact that DARPA has a mixed record on transitioning emphasizes the fact that they work where markets don’t. Some of the translation failure is perhaps a failure on DARPAs part. But it can also be because even when they’ve brought a program to a point where it has produced awesome things, people don’t see a way to make money doing it. In the analogy that Decoupling from market discipline is like cave diving this is the grim situation where there’s no air bubble where you thought there would be.
Sometimes when you’re trying to change a paradigm, you’re just straight up threatening people’s jobs or reputations and so you need to outlast them. DARPA works on programs that go against the established paradigm. Markets don’t care why someone doesn’t want the thing - people are not rational and their acquisition of something is based on their expected value of it, not the actual value. And that expected value
DARPA works on programs that go against the established paradigm but markets sometimes can support something that is paradigm changing. For example cars. On the other hand, I would argue that the market would not have supported personal computing. ( Fiction / Science Fiction/ Historical Fiction are a way of exploring counterfactuals.)