Specialization has a complicated relationship with new technologies. New technology will never^1 be as good as good as old technology along every dimension. This failure of Pareto-superiority is why Frontier technologies need to start in niche markets where they are especially valuable. It’s rare that a technology can do well in a niche market without some specialization. If you imagine the straightest path that a technology can take to its most powerful or general use^2, the specialization work to fit into niches can require larger or smaller diversions from this path. These diversions take the form of everything from technology development, to marketing, to building sales channels for a company. Going from niche to niche is essential, but you can also get stuck in a niche or jump onto a different development path altogether. Elon is the master of charting paths between niches. More time and resources (Slack - concept) can enable you to pick an “optimal” sequence of niches or search around for yet-unknown niches instead of being forced to hop into the nearest or most obvious ones. In other words, if Decoupling from market discipline is like cave diving, corporate R&D can act like extra oxygen tanks.
I don’t know how often I’ve seen a presentation from a grad student or professor that abstractly goes like “we’ve invented this incredible technology that could potentially do <amazing thing> however it will take a lot of work to get there, so we’re starting a company to build a product to do <kind of pathetic thing that will still require a lot of specialization and company-building> in <domain totally unrelated to <amazing thing>>. While I’m usually sure it’s doomed to failure, it’s unfortunately the best move given history and §Startup Constraints. First, many successful startups do look like they started in a tiny scoffable niche - Tesla (weird rich people), NVIDIA (gamers), PayPal (Beanie Babies on eBay?), Apple (hobbyists), Twitch (life live-streaming), Amazon (books) … the list goes on. However, each of these weird niches didn’t require huge diversions from the critical path.^3 Second, ‘start in a niche’ is folk startup wisdom for a reason. History is littered with the corpses of startups that were going to be the next platform or general purpose technology as soon as they launched. General Magic, NeXT, Quibi, Atrium, Rethink Robotics, and of course Magic Leap [[Magic Leap is a good case study of a potentially amazing technology killed by startup constraints. Many others that started with grand ambitions were forced to crawl into a niche to survive - Kindred Robotics comes to mind. At the same time, startup investors want to see massive potential returns. ‘Find a compelling story about how a niche becomes a billion dollar market’ puts a technology between a rock and a hard place.^4
In most cases, it wouldn’t have been feasible to do the necessary work to find or develop for a better niche in an academic setting either. While that work still requires “research” it also requires focus and systems engineering, both things that academia does not support for a slew of reasons. Academia is not a good environment for systems engineering. Academia incentivizes novelty, not focus.
It’s of course possible that I and others just overestimated the potential of many of these technologies. However, the examples of technologies that were massively impactful only after “hanging out” in a corporate research labs for many years would seem to argue otherwise. The Transistor, Public key cryptography and the Graphical User Interface (and personal computing in general) come to mind. Ideally corporate research labs provide the environment for to do the work to get a technology to a point where it’s viable for a ‘useful’ niche and enable it to find that niche in the first place.
In addition to simply having longer time scales, less existential risk, and larger budgets than most startups, there are several specific ways that corporate R&D labs (used to) help technologies find good niches that startups and academia didn’t provide.
When you’re still trying to figure a technology out, it’s not clear which skillsets you want in the room. Corporate R&D allows there to be people floating between different projects loosely creating and breaking collaborations. Bell Labs enabled free radicals. To get the same effect at a Startup, you need to either find a magical individual who has gone deep on multiple areas (M shaped individual?) or hire people who might ultimately turn out to be useful.
Bell labs asked people to do useful things but left them room to say huh that’s funny - often that moment of “huh that’s funny” can lead to the useful niche.
Healthy R&D orgs feel like they have very smooth project rampups with high ceilings - the discontinuous rampup implicit in a high-growth startup can force premature committal to a specific niche and the ceiling imposed by academia can prevent you from doing the work to accurately test a niche in the first place.
Of course, this is all talking about an idealized corporate R&D organization, which arguably has become exceedingly rare. The need to produce demos for corporate higher-ups may serve the same ‘specialization forcing function’ as startups. One way to look at this would be to ask ‘how has the management of corporate R&D changed over time? On the hypothesis that People who have done a thing should be in charge of a thing. On the other hand, The Mother of All Demos was in fact a demo. Same with SpaceX live fire/other Elon musk things. So the upshot might not be that “demos necessarily force specialization” but that Demos are like afterburners - applied strategically along a planned path they can allow you to jump gaps you would otherwise be unable to cross, but applied whenever you feel pressure will leave you without any fuel.
^1: I will say this very aggressively in the hopes that I am wrong. Cunningham's Law
^2: This idea might be a fallacy. Arguably there is no platonic “ultimate stage” of a technology and whatever a technology evolves into is indelibly touched by the different specialized forms it needed to pass through.
^3:There are a number of just-so stories I could tell about why these university spin-offs are different than the list of successes - most of those on the list were not deeply new technology, their founders were brilliant and plotted the exact path to success, it’s easier to go into new niches with software and chips, they got lucky, I’m a moron etc. However, the empirical evidence is that so many niche-seeking startups built around new (genuinely cool) technologies fail to help that technology realize its potential.
^4: I’m not trying to say VC and startups suck and are broken! There are many amazing things that have come out of the system. What I am asserting is the incorrectness of the idea that “if it wasn’t a viable startup it would never have been able to be amazing for the world in the long run”