The human capital theory of stagnation is that specialization among researchers and the incentives they experience have increased coordination costs and led to effort allocation that has decreased the rate of breakthroughs.
We do know that specialization and division of labor drives Smithian growth. (Smithian Growth vs Schumpetarian Growth) The scientific economy is subject to those same forces and researchers have become increasingly specialized. However, this specialization may be contributing to Stagnation in several ways.
One argument goes that since New ideas need to come out of a single mind and new innovations don’t actually require fewer knowledge areas than they did in the past. Combined with specialization, this requirement means that innovations need increasingly large teams of people working closely, which leads to higher coordination costs per invention in order for each specialist to understand the others enough to contribute usefully. Both The Burden of Knowledge and the ‘Death of the Renaissance Man’ Is Innovation Getting Harder? and The decline of unfettered research hint at this argument.
Another fact is that the number of researchers has exploded along with the specialization. However, despite the internet, the explosion of researcher numbers and specializations means that their ability to find the right person to collaborate with has not kept up. So while there might exist a productive division of labor on a problem, it takes a ton of effort to find the person to collaborate with. The Economics of Scientific Collaboration touches on this theory.
The decline of unfettered research argues that the exploding number of researchers means that they just can’t operate a mode they did in the past. The explosion of research has led innovations becoming effectively a commodity and expectations of continuously improving systems. These outcomes in turn both make scientists fungible and expected to churn out improvement at a certain rate that need to work with existing systems. My personal take is that these incentives aren’t conducive to deep work or paradigm-shifting innovations.
Other incentives on scientists have also pushed them to work on more incremental shorter term projects. Stagnation and Scientific Incentives argues out that two of the biggest incentives driving this incrementalism are citation-obsession and grant-giving processes. The number of papers a research has published and the number of citations those papers garnered are a key metric for measuring a professor’s performance when considering tenure, hiring, and prestige. Thus, anybody playing the academic research game (which includes people in universities, national labs have incentives to publish things that will get a lot of citations. (Academia is the game where you gain status by getting attention for new knowledge. ) Keep in mind that these citation incentives also include National Labs (The game in national labs is roughly the same as academia but with different constraints) and many people in corporate R&D labs, as papers are the primary shared currency between those worlds.
In order for something to be publishable, work needs to be sufficiently novel (Academia incentivizes novelty, not focus), so individuals are pushed away from grungy engineering efforts to implement already-described ideas. A publication also needs to get through peer review (Academics are incentivized to do peer-reviewed publishable work) which punishes excessive divergence from current paradigms The peer review and citation system incentivizes people to work on things that other people think is interesting.
In order to get lots of citations, a publication needs to hit a sweet spot where it is both new but not far outside of current paradigms. Of course, if you publish a paper that kicks off a new paradigm, it will get many citations. However, paradigm-shifting papers also are likely to get close to zero citations if they’re even published (note the above point about peer review filtering for ideas that are in-paradigm.) This tradeoff sets up a choice between a high risk, high reward option and a lower-risk medium reward option. The fact that there are two options doesn’t necessarily lead to stagnation, but combined with citation-obsession and the incentives of grad school, the potential downsides of the high-risk option are too great.
Remember, most academic researchers are grad students who would like to graduate and get a good job in a finite amount of time and There is an increasingly wide gap between the number of graduate students training for PhDs and academic positions for them to fill.
The majority of grants are given out by foundations or government agencies. In these situations, People giving out grants try to derisk them as much as possible. There is relatively little upside to funding wildly successful research, but substantial downside to wasting someone else’s money. This Asymmetric career risk for grant givers mean that researchers applying for those grants need to convince the grant giver that
Navigating the publication-grant game is a skillset in and of itself, which means that Successful professors are not necessarily the best researchers. It’s hand-wavey, but you could see how a research world that rewards good managers more than good researchers might lead to a shift of human capital that could impact the sorts of innovations that happen.
On top of all of this, Academic culture prizes individual recognition which is one of many contributors to The (idea) valley of death that prevents many innovations that require a bunch of grungy work to scale up from making an impact.
Overall, the human capital theory of stagnation feels like stagnation by a thousand cuts. However, the uniting cause seems to be the massive growth in the number of people doing research and the resulting cascade of locally rational adaptations. If this is the case, fixing each of these issues one at a time may be like playing whack-a-mole and you definitely can’t decimate the number of people doing research.
The human capital theory of stagnation (especially the flavor in Stagnation and Scientific Incentives) is closely tied to the Systemic decay theory of stagnation because the systems drive individual’s incentives and would ideally be acting to attenuate individual incentives that harm the system on-net. Institutions are the second level of a group selection evolutionary system. Specifically, citation-obsession and grant-gaming are results of the bureaucratization of research.
Arguably the Low Hanging fruit theory of stagnation could be driving the specialization because as new knowledge gets harder to find, people are forced to specialize more and more. I’m not sure if it’s possible to tease apart how much of the specialization is driven by the nature of research itself and how much is driven by sociological forces.
There’s also a general sense that (partially driven by specialization) people don’t know how to build things anymore. That is, if someone had an idea about a flying car in the 50’s he might be able to just talk to his neighbor who knew how to build cars and airplanes and they could whip together a prototype. Definite Optimism as Human Capital.
Is Science Slowing Down? - SSC argues convincingly that logarithmic technological progress should be the null hypothesis (remember, we have no immortal god-emperor Eisenhower or immortal god-emperor Marcus Aurelius.) In this frame, the human capital theory of stagnation is simply how The gods of straight lines are enforcing that logarithmic growth.
A more mundane counterargument is that there is actually a lot of good science being published and funded. Sure, it could be better, but on the margin, people who would do crazy research are still doing that research and if they’re good at it, they can still find money to do it.