The systemic decay theory of stagnation is a loose collection of assertions that changes in institutions - government, universities, culture - are responsible for Stagnation.
Where’s my Flying Car - A memoir of futures past points to ever-increasing regulation as the knife in the throat of innovation. Regulation doesn’t need to explicitly ban innovations - it just needs to make the hurdles they have to clear to be adopted so high that they never make it out of the lab/prototype stage. Regulations create paradigm lock-in, and outside of economic metrics (Science is getting less Bang for its Buck) the nebulous but real evidence for stagnation is that we are not seeing as many paradigm shifts in atom-based technology. Additionally, there are several technologies we can point to that are not commonplace primarily because of regulations (nuclear power, most genetically modified things, many helpful drugs) so how many don’t we know about. Arguably cars wouldn’t exist under the current regulatory regime.
Some people blame the fact that Universities have been pushed more towards applied research since the end of WWII at least in part because of the tight coupling between government funding and academia. Most politicians see government spending on research as an investment and applied research has a much clearer ROI. I find this argument a little weak because lots of non-applied research still happens in universities. Yes, a smaller fraction is unfettered discovery, but I haven’t seen any compelling reason to believe that the fraction matters.
While the composition of university research might not be to blame, there are a lot of theories that center around the coupled dependence on and mismanagement of government funding for research. Everything from risk aversion ( A case study of the NSF Small Grants for Exploratory Research programme) to the bureaucratic apparatus that makes it extremely painful to apply for grants (Giving Away Money Costs More Than You Think), for the most part only gives small or absolutely huge grants. Where’s my Flying Car - A memoir of futures past goes so far as to point out that GDP growth is in fact inversely correlated with non-military government R&D funding. Most foundations are not much better.
One contributing factor might be the death of corporate R&D. We can point to so many impactful innovations that came out of corporate R&D in the middle of the 20th century. The changing structure of American Innovation - some cautionary remarks for economic growth notes that incumbent companies have shifted towards expanding markets by mergers and acquisitions.^1 This strategy shift means that most innovations need to either work within all the constraints of a startup (§Startup Constraints).
There’s a mini-nebula of theories around The (idea) valley of death. What would ideally be a communication-rich channel going both ways between people trying to innovate and improve on existing technologies and the people who use and sell them is instead a high friction, leaky pipeline with very little feedback. Academic research is often just the concept half of creating a heuretic, but a proof of concept is often not enough to derisk a for-profit company. Radical Abundance argues that many innovations don’t happen because Academia is not a good environment for systems engineering, which is necessary and underrated for building actual functional products. There is another class of theories that the valley of death has widened because globalization and other forces have separated manufacturing from research. Definite Optimism as Human Capital.
The discussions around The Valley of Death invariably touch on the patent system. These theories go that The relationship of government, academia, and technology fundamentally changed in 1980 with the Bayh-Dole act. Now technology transfer offices are all trying to grab their share of the profit (despite the fact that Most tech transfer offices are not profitable), making it hard to combine patents into impactful innovations, demanding a large enough share in companies that it makes it hard for some to get funded, and generally increasing the friction to turn university research into useful products. Any technology that comes out of a university needs to go through their tech transfer office. Corporations also abuse the patent system - taking out defensive patents that mean paradigm shifts that require several new pieces are all potential lawsuits.
The patent issues point to a more Nebulous systemic problem: declining Slack - concept across institutions: Academia, Incumbent Corporations, and Startups. This idea is a massive rabbit hole so I will touch on it only briefly, but in general Systems must take performance hits to get out of local optima but without enough slack, any system (or individual) that takes a performance hit will get competed out of existence. The oversaturation of academia with people, potential returns to investing in software, patent obsession, regulations, and decline of R&D labs may all have contributed to this declining slack.^2
Declining slack is coupled to an even more Nebulous issue: culture. In part, the world is so competitive because the pie doesn’t feel like it’s growing anymore in the first place, so it’s unclear which way the causality runs between Stagnation and culture. Peter Thiel’s critique is mostly a cultural one (Back to the Future (Thiel)) that we don’t make and support big, precise plans (Definiteness means that you have a long-term plan and generally don’t have a definite optimistic stance, embracing indefinite optimism which leads to incremental improvements and short-term optimization. To make new things you need to take risks and there’s a sense that we are just more risk averse (see the previous paragraph about regulation.) Perhaps related is the phenomena of increasing leader age across institutions. And there’s a sense that with the decline of religion and nationalism we aren’t striving for anything bigger than ourselves or our lifetimes.
The systemic decay theory is almost the dual of the Human capital theory of stagnation, which stands to reason: systems both are composed of individuals and drive their incentives. The difference may be in the flavor of solutions that each theory suggests - whether you focus on shifting individual incentives or institution building/reform. Even then, the lines are blurry.
All of these varied causes could be why Low Hanging fruit theory of stagnation is correct. Perhaps there is a limit to how coordinated humans can be and we’ve picked most of the fruit that can be easily grasped within those constraints. Similarly, all of these effects could be because we’ve run out magical growth pixie dust.
I need to do more work to understand the counterarguments more deeply. They boil down to the fact that as a society we are spending a lot of money and manpower on research. It’s not like nothing new is being created, and we’ve tried a lot of things that just haven’t worked.
^1: AI research is an exception to the decline of corporate R+D
^2: Note that I’m not saying competition is bad. Simply that more competition is not always better. Studies on Slack - SSC.