Graduate students are the labor in academic science and engineering

People giving out grants try to derisk them as much as possible

Manufacturing is important and often ignored

The word ‘novel’ is used as an idea bludgeon in academia

All the incentives in research drive towards conservatism

Peer review doesn’t catch duplications

Bell Labs is one of the few examples of an innovation org that was aligned with its money factory

A new professor needs about $1.5m to get going

Academic culture prizes individual recognition

Younger professors have to spend more time getting money

Academics are incentivized to do peer-reviewed publishable work

Things that are not paper-worthy enough for academia and not product-focused enough for a startup

Pay-it-forward tithing could resolve the tension between the potential public-good and potential profit in research.

The idea that academia is for crazy exploration and companies are for low risk exploration is bullshit

Three myths about scientific peer review

A PI needs to provide evidence that they have access to a lab facility

Different axes of where research gets done in the US

What was the real incentive for people to work on the DARPA Grand Challenge?

It is easier to have a multipoint mutation in technology than in evolution

NS Notes on Officers Techniques

Necrobotics illustrates how academia forces engineering

The boundaries between industry government and academia are fuzzy

Startups and Academia are bad at addressing structural constraints

The Architected Innov􏰂ation Institute - Engineering Scientific Revolutions

There are very few senior individual contributors in academia

Is it more impactful to find and train undiscovered diamonds or to unlock well-positioned but constrained researchers?

Wolfgang Merkt conversation 12 Oct 2021

An institution riffing on the ARPA Model could enable paradigm shifts that are not happening

Academic administrators and professors are playing very different games

An Institute for Creative Technology - Riffing on ARPA

Funding structures lead to several layers of indirection on what even gets proposed

Is it possible that we’re actually getting less research because we have more researchers?

Running list of relevant science news?

# §Academia Constraints
- [[Academia is the game where you gain status by getting attention for new knowledge]]

- [[Universities have multiple conflicting missions]]
- [[Universities have been pushed more towards applied research since the end of WWII]]
- [[Academic credit is a first-to-the post system]]
-  [[Graduate students are the labor in academic science and engineering]]
- [[People giving out grants try to derisk them as much as possible]]
- [[Professors do three or more jobs]]
- [[Successful professors are not necessarily the best researchers]]
- [[University professorships are the highest status jobs in academia]]
- [[National labs and other government research centers are a research alternative to universities]]
- [[The game in national labs is roughly the same as academia but with different constraints]]
- [[In academia more grant money leads to more power]]
- [[Academic research is often just the concept half of creating a heuretic]]
- [[Professors are constantly doing the calculation of whether a grant is worth the time to apply for it]]
- [[Most grant money is tagged so unrestricted money is like gold for professors]]
- [[Universities can take more than half of grant money as overhead]]
- [[A PI needs to provide evidence that they have access to a lab facility]]
- [[Physical experiments are expensive]]
- [[Younger professors have to spend more time getting money]]
- [[The government grant process depends on politics and committees]]
- [[Any technology that comes out of a university needs to go through their tech transfer office]]
- [[Months of grant writing is one of the things that makes people think twice about academia]]
- [[Academia incentivizes novelty, not focus]]
- [[Academia is not a good environment for systems engineering]]
- [[Academics are incentivized to do peer-reviewed publishable work]]
- [[Going outside of the approved process can be a black mark against people in academia]]
- [[People proposing work that they have already done is a sign that feedback loops are broken]]
- [[Necrobotics illustrates how academia forces engineering]]
- [[Labs implicitly judge research through the sense of dollars per paper]]

### Related
- [[§Startup Constraints]]
- [[§Corporate R&D Constraints]]
- [[§Philanthropy and Non-Profit Constraints]]
- [[Market Failures in Science]]

[Web URL for this note](http://notes.benjaminreinhardt.com/§Academia_Constraints)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/§Academia_Constraints)






§Academia Constraints

Peer review has failed as an institution

Human capital theory of stagnation

People are incentivized to work on nodes higher up the tech tree

Peer review gives peers veto power over each other

# Academics are incentivized to do peer-reviewed publishable work
Peer review published papers both determine an academics livelihood and status. Therefore, they are incentivized to spend their time doing work they think can be published. 

On top of this [[Academic credit is a first-to-the post system]]

[[People and organizations are all playing some game that has different ways of gaining status and power]] In Academia there are a couple of long term games:
- Gain [[Kleos]] through contribution to the [[Knowledge frontier]]
- Just get to scratch your itch to investigate certain things
- ….

[[People create proxy games for long term or hard-to-measure games]] and in academia the entwined proxy games are:
- Peer reviewed Published papers
- Citations on those papers
- Grants (which are usually based on papers) 
- Awards (which are usually based on who knows you which comes from papers)

It’s a safer strategy to publish lots of papers than to try to write one super-cited paper because it’s very hard to know which papers will be highly cited. Additionally everybody is impressed when you say “I’ve published a bajillion papers” while only a subgroup might know your super-cited paper by title unless you’ve [[hit the top of the columnade]]. 

Tenure is based on the number of papers you’ve published and the amount of grant money you’ve brought in.  Without tenure, academia is extremely rough. People who don’t think they can get tenure leave universities. [[National labs and other government research centers are a research alternative to universities]].

[[The peer review and citation system incentivizes people to work on things that other people think is interesting]]

### Related
- [[§Academia Constraints]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Academics_are_incentivized_to_do_peer-reviewed_publishable_work)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Academics_are_incentivized_to_do_peer-reviewed_publishable_work)




How many important things barely squeezed through with statistics?

# How many undetected frauds in science
### Published
## Questions
## Thoughts
## Related
 - [[Peer review doesn’t catch duplications]]
## Authors
## Link
https://fantasticanachronism.com/2020/08/11/how-many-undetected-frauds-in-science/



[Web URL for this note](http://notes.benjaminreinhardt.com/How_many_undetected_frauds_in_science)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/How_many_undetected_frauds_in_science)




The peer review and citation system incentivizes people to work on things that other people think is interesting

Lets just get rid of peer review - article

# Is peer review a good idea - paper
- Assumes a primary motivation for scientists is credit
- Sharing
	- Priority rule incentivizes scientist to publish asap
		- [[Academic credit is a first-to-the post system]]
	- Well-connected scientists already preprint so killing peer review could be equalizing
- Time allocation
	- Advantages of not press ganging scientists into reviewing
- Gender skew
	- Women hold their work to higher standard because they perceive gender bias regardless of whether it exists
- Library resources
	- Cost of publishing and distributing an article = 4000lbs
	- Journals become collections of existing articles
	- arXive maintainence cost = $10
- Scientific careers
	- Metrics would shift towards citations rather than impact factors of journals ::Isn’t this the same as online articles?::
- Gatekeepers
	- Gatekeepers would have less power
- Epistemic sorting from peer review wouldn’t be a loss to science
- Abolishing peer review wouldn’t change level of malpractice detection
- Scientists will still do things that they expect other scientists to think is impressive
- Abolishing peer review won’t shift who gets or does not get one run credit
- Peer review provides a quality guarantee to outsiders
	- Peer review definitely lets through shit
- Concerns about runaway rich get richer effect due to citation cascades
	- Especially bad because funding is based on citations
	- ::This may be the biggest piece::

### Published
	
5/2020
## Questions
## Thoughts
- It’s interesting that nobody writes this sort of paper about art
- Why do we think different things would happen than what’s happened to internet art - most people get overwhelmed by choice
## Related
## Authors
- [[Remco Heesen]]
- [[Liam Kofi Bright]]
## Link
https://academic.oup.com/bjps/advance-article/doi/10.1093/bjps/axz029/5526887



[Web URL for this note](http://notes.benjaminreinhardt.com/Is_peer_review_a_good_idea_-_paper)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Is_peer_review_a_good_idea_-_paper)




# Lets just get rid of peer review - article
Ref: [[Is peer review a good idea - paper]]
- 13 arguments in favor of getting rid of preprint peer review
1. Sharing results *would not* decrease without peer review
2. Peer review takes unpaid time away from researchers to help for-profit journals for free
3. There is bias in peer review
4. Academic journals charge tons of money
	1. Interesting ideas to have library budgets go towards helping edit papers
5. Peer review is a regressive tax on researchers because to get through the gate keepers they need good brand access which they get by working for more established labs
	1. Paper uses week thesis - peer review is an imperfect judge of work quality
6. Peer review involves gatekeepers
	1. ::Still need a mechanism to filter::
7. Ideally peer review keeps science quality high - instead it forces a quality hierarchy - shitty things are still published
	1. ::Still need a mechanism to filter::
8. Ideally peer review provides malpractice detection but it’s failed at that. 
9. Removing peer review will increase herding frequency, which he argues is a good thing
	1. ::is higher frequency herding a good thing?::
10. Getting rid of peer review will fix credit a little because people won’t have to hitch their wagons to a brand and will work on what they actually want to do 
	1. ::maybe, but there’s also the thing online where people become so overwhelmed by uncharted content that they just go for the most popular thing::
11. Quality guarantees for outsiders - high end research will still get reviewed, low end research isn’t reviewed anyway - problem in the middle
12. [[The Matthew Effect]] - rich get richer. Does this already happen with peer review?
### Published
5/2020
## Questions
## Thoughts
- Could we compare to maybe the music industry? Other de-gatekeepered worlds?
- How would “we” pull it off?
- [[Coordination Problems]]
## Related
- [[Fair games have legible rules]]
## Authors
- [[Alex Danco]]
## Link
[Let’s just get rid of peer review – alexdanco.com](https://alexdanco.com/2020/05/28/lets-just-get-rid-of-peer-review/)



[Web URL for this note](http://notes.benjaminreinhardt.com/Lets_just_get_rid_of_peer_review_-_article)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Lets_just_get_rid_of_peer_review_-_article)




# Peer Review- Rodney Brooks
Of [[Rodney Brooks]]’s most cited papers, all were rejected by peer review
- Papers with lots of theory are more impressive than ones with experiments
- Some reviewers always reject, some reviewers always accept
- The peer review system can’t handle the scale of modern science
- Editors are really important and many have abdicated that responsibility
### Published
2020
## Questions
## Thoughts
## Related
## Authors
[[Rodney Brooks]]
## Link
[Peer Review – Rodney Brooks](https://rodneybrooks.com/peer-review/)



[Web URL for this note](http://notes.benjaminreinhardt.com/Peer_Review-_Rodney_Brooks)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Peer_Review-_Rodney_Brooks)




How many undetected frauds in science

Peer review fails to catch blatantly false things

# Peer review doesn’t catch duplications
[Elisabeth #StayingAtHome #StayingAlive Bik on Twitter: “Update!This is a set of (at least) EIGHT papers, from different authors, different hospitals, different cancers, different protein expressions. But with identical Kaplan-Meier curves, tables values, line graphs. See e.g. https://t.co/SmHEMPpEm5… https://t.co/grcaEE5tEI”](https://twitter.com/MicrobiomDigest/status/1266140721716719616)

### Related
- [[When you see a position being advocated with no pushback but no action - it means there are a bunch of little frictions in the system]]
- [[§Academia Constraints]]
- [[Academics are incentivized to do peer-reviewed publishable work]]
- [[The peer review and citation system incentivizes people to work on things that other people think is interesting]]
- [[Is peer review a good idea - paper]]
- [[Problems With Peer Review And Alternatives]]
- [[Three myths about scientific peer review]]
- [[Peer Review- Rodney Brooks]]
- [[Lets just get rid of peer review - article]]
- [[The Philosophical Basis of Peer Review and the Suppression of Innovation]]
- [[The fact that large-scale automated machine reading discovers new cancer-driving mechanisms doesn’t cite big mechanism is a clear failing of peer review]]
- [[Unlock ways to share data on peer review]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Peer_review_doesn’t_catch_duplications)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Peer_review_doesn’t_catch_duplications)




Research requires more trust than other disciplines

True Peer Peer Review

The journal of half baked ideas

# Peer review has failed as an institution
Let’s be clear that the words “peer review” have baggage that comes from outside the words themselves. You hear ‘peer review’ and think ‘oh this is legit because other scientists read it’ which is technically true. But nobody has any idea if they would actually consider the people who reviewed the paper to be actual peers.

One solution to this might be [[True Peer Peer Review]]. 

[[Want to See My Data? I Thought We Were Friends]] gives a good highlight of the problems, but the solutions are all incremental steps away from an inadequate equilibrium. It doesn’t address the meta-problem that there isn’t [[Slack - concept]] in the system.  [[Most institutions have become cancerous]]. 

### Related
- [[Peer review doesn’t catch duplications]]
- [[Lets just get rid of peer review - article]]
- [[The Philosophical Basis of Peer Review and the Suppression of Innovation]]
- [[The peer review and citation system incentivizes people to work on things that other people think is interesting]]
- [[Is peer review a good idea - paper]]
- [[Problems With Peer Review And Alternatives]]
- [[Academics are incentivized to do peer-reviewed publishable work]]
- [[Three myths about scientific peer review]]
- [[Peer Review- Rodney Brooks]]
- [[Goodhart's Law]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Peer_review_has_failed_as_an_institution)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Peer_review_has_failed_as_an_institution)




# Problems With Peer Review And Alternatives
Offers up a bunch of alternative metrics which then play out badly

Bias towards institutions - resubmitting highly cited papers with changed institutions: didn’t get caught for copying but for poor quality work

No real consensus about what good work looks like

Bias toward pure research instead of building real things

Costs so much

Origination of the [[Szilard point]]
### Published
1988
## Questions
## Thoughts
## Related
## Authors
[[Richard Smith]]
## Link
[Problems With Peer Review And Alternatives on JSTOR](https://www.jstor.org/stable/29530055?read-now=1&refreqid=excelsior:7cdbee1eabdb53671ec85b09f53f66b5&seq=1)



[Web URL for this note](http://notes.benjaminreinhardt.com/Problems_With_Peer_Review_And_Alternatives)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Problems_With_Peer_Review_And_Alternatives)




# The Philosophical Basis of Peer Review and the Suppression of Innovation
- Tries to address was peer review is for
- Peer review isn’t just for quality control - it needs to both be for quality control and innovation
- #case-study of peer review failing
- `It also requires an awareness that even some of the most distinguished of scientists may display sophisticated Behavior that can only be described as  pathological.`
- Editors need to take on more responsibility 
### Published
## Questions
## Thoughts
## Related
## Authors
- [[David Horrobin]]
## Link
https://jamanetwork.com/journals/jama/article-abstract/380984



[Web URL for this note](http://notes.benjaminreinhardt.com/The_Philosophical_Basis_of_Peer_Review_and_the_Suppression_of_Innovation)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_Philosophical_Basis_of_Peer_Review_and_the_Suppression_of_Innovation)




Stagnation

The way to get around journals is to focus on non-paper artifacts

Einstein would have been stuck in a patent office - failures of paradigm-shifting science

People do fundable work

Maverick work is lonely and DARPA provides peers and community

DARPA provides a derisking role for people in other organizations

The systemization of science since the 1950s treats all fields the same

A system of credit and funding for non-Scientific Paper artifacts should be artifact-type agnostic

Using citations to measure science is insidious

# The peer review and citation system incentivizes people to work on things that other people think is interesting
You could argue that this is a good mechanism that forces people to work on relevant work. On the other hand it enforces memesis. ([[Memetic Theory explains a lot of human nature]])

Put in a more extreme way [[Peer review gives peers veto power over each other]]. If something is too heretical, it just won’t make it through peer review. 

Even if there is funding available to do crazy shit you wouldn’t do it if you couldn’t get the results published and cited because [[Academia is the game where you gain status by getting attention for new knowledge]].

![](../BearImages/D70F62F9-2CA8-4B86-851D-505094A86D26-61320-000774DAF820EE3A/FD1B16C6-91A1-44CF-98EC-44CDE6D5AD96.png)

As a [[Stagnation]] theory this could be attributed t 

### Related
- [[Peer review doesn’t catch duplications]]
- [[Peer review has failed as an institution]]
- [[True Peer Peer Review]]
- [[People do fundable work]]

### References
- [[Peer Review- Rodney Brooks]]
- [[Is peer review a good idea - paper]]
- [[Problems With Peer Review And Alternatives]]
- [[Three myths about scientific peer review]]



[Web URL for this note](http://notes.benjaminreinhardt.com/The_peer_review_and_citation_system_incentivizes_people_to_work_on_things_that_other_people_think_is_interesting)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_peer_review_and_citation_system_incentivizes_people_to_work_on_things_that_other_people_think_is_interesting)




# Three myths about scientific peer review
1. Scientists have always used peer review
	- Most journals didn’t use it until mid 20th century
	- Only 1/300 Einstein paper was subject to peer review
	- 1936 Physical review had recently instituted peer review
`Einstein’s indignant reply to the editor is amusing to modern scientific sensibilities, and suggests someone quite unfamiliar with peer review:`
`*Dear Sir,*`
`*We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorized you to show it to specialists before it is printed. I see no reason to address the in any case erroneous comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.*`
`*Respectfully,*`
`*P.S. Mr. Rosen, who has left for the Soviet Union, has authorized me to represent him in this matter.*`
Nature didn’t have formal peer review system until 1967
Three factors led to peer review
1. Specialization
2. Increase in number of publications
3. Ability to copy papers
In short Editors could no longer review on their own and could get other people to do it

2. Peer review is reliable
- Not much evidence that peer review works well
- Doesn’t catch deliberately inserted errors
- Discoveries almost destroyed by peer review [The Philosophical Basis of Peer Review and the Suppression of Innovation | JAMA | JAMA Network](http://jama.ama-assn.org/cgi/content/abstract/263/10/1438)

3. Peer review is  how we determine what is right and wrong in science
	- Science worked fine before peer review
### Published
2009
## Questions
## Thoughts
- Isn’t peer review basically decision by committee?
## Related
- [[§Academia Constraints]]
## Authors
- [[Michael Nielsen characterizes institution building as making previously illegible things legible]]
## Link



[Web URL for this note](http://notes.benjaminreinhardt.com/Three_myths_about_scientific_peer_review)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Three_myths_about_scientific_peer_review)




# Unlock ways to share data on peer review
- Of 11,000 researchers 98% said peer review was extremely important
- It’s hard to study peer review
- Journals have gone from dissemination to curation roles
- Reviewers are reluctant to reveal identity and only when its a positive review
- Pharmeceutical companies pool data through
	- Vivli 
	- YODA 
	- Open Targets
- Propose the PEERE collaboration protocol for sharing peer review data in 2017 after 3(?!) years of work

### Published
Feb 2020
## Questions
## Thoughts
## Related
## Authors
## Link
https://www.nature.com/articles/d41586-020-00500-y



[Web URL for this note](http://notes.benjaminreinhardt.com/Unlock_ways_to_share_data_on_peer_review)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Unlock_ways_to_share_data_on_peer_review)




Benʼs working notes

Peer review doesn’t catch duplications

Related

Links to this note