§Academia Constraints

Notes and statements should be bloody

Academics are incentivized to do peer-reviewed publishable work

Is peer review a good idea - paper

DARPA facilitates idea sex both between PMs and Performers

# Academic credit is a first-to-the post system
Everybody in academia pays attention to ‘who was the first person to discover this thing.’ The way that it counts is by publishing something about it.  

The first-to-the-post system and the existence of preprint servers means encourages people to create a lot of false positives. This would not be a problem if there was a good way of alerting the world that something was a false positive but there is no good system for that. 

The first-to-the-post system makes everybody obsessed with being “scooped” because the first person to publish an idea gets the credit even if someone comes along afterwards and explains it better. Of course, it doesn’t always happen like this and you also get into fights where there is a better known expression of an idea and the person who originally came up with it spends an inordinate amount of time yelling that they came up with it *cough* Schmidthuber *cough* 

This is why [[The word ‘novel’ is used as an idea bludgeon in academia]] - extending an existing idea is not in high esteem even if it’s an important reframing that can unlock something important. 

The first-to-the-post system doesn’t encourage dispersing a concept beyond making sure everybody knows that you came up with it and will cite you as such. ([[Dispersion is the process of a heuretic impacting the world]]) But [[Just having the piece of literature doesn’t solve the problem]]. 

### Related
- [[weinsteinBretWeinsteinPrediction]]
-  [[Manage the Transfer not the Technology]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Academic_credit_is_a_first-to-the_post_system)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Academic_credit_is_a_first-to-the_post_system)




The peer review and citation system incentivizes people to work on things that other people think is interesting

Peer review has failed as an institution

Peer review doesn’t catch duplications

Lets just get rid of peer review - article

Peer review gives peers veto power over each other

# Is peer review a good idea - paper
- Assumes a primary motivation for scientists is credit
- Sharing
	- Priority rule incentivizes scientist to publish asap
		- [[Academic credit is a first-to-the post system]]
	- Well-connected scientists already preprint so killing peer review could be equalizing
- Time allocation
	- Advantages of not press ganging scientists into reviewing
- Gender skew
	- Women hold their work to higher standard because they perceive gender bias regardless of whether it exists
- Library resources
	- Cost of publishing and distributing an article = 4000lbs
	- Journals become collections of existing articles
	- arXive maintainence cost = $10
- Scientific careers
	- Metrics would shift towards citations rather than impact factors of journals ::Isn’t this the same as online articles?::
- Gatekeepers
	- Gatekeepers would have less power
- Epistemic sorting from peer review wouldn’t be a loss to science
- Abolishing peer review wouldn’t change level of malpractice detection
- Scientists will still do things that they expect other scientists to think is impressive
- Abolishing peer review won’t shift who gets or does not get one run credit
- Peer review provides a quality guarantee to outsiders
	- Peer review definitely lets through shit
- Concerns about runaway rich get richer effect due to citation cascades
	- Especially bad because funding is based on citations
	- ::This may be the biggest piece::

### Published
	
5/2020
## Questions
## Thoughts
- It’s interesting that nobody writes this sort of paper about art
- Why do we think different things would happen than what’s happened to internet art - most people get overwhelmed by choice
## Related
## Authors
- [[Remco Heesen]]
- [[Liam Kofi Bright]]
## Link
https://academic.oup.com/bjps/advance-article/doi/10.1093/bjps/axz029/5526887



[Web URL for this note](http://notes.benjaminreinhardt.com/Is_peer_review_a_good_idea_-_paper)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Is_peer_review_a_good_idea_-_paper)




# Lets just get rid of peer review - article
Ref: [[Is peer review a good idea - paper]]
- 13 arguments in favor of getting rid of preprint peer review
1. Sharing results *would not* decrease without peer review
2. Peer review takes unpaid time away from researchers to help for-profit journals for free
3. There is bias in peer review
4. Academic journals charge tons of money
	1. Interesting ideas to have library budgets go towards helping edit papers
5. Peer review is a regressive tax on researchers because to get through the gate keepers they need good brand access which they get by working for more established labs
	1. Paper uses week thesis - peer review is an imperfect judge of work quality
6. Peer review involves gatekeepers
	1. ::Still need a mechanism to filter::
7. Ideally peer review keeps science quality high - instead it forces a quality hierarchy - shitty things are still published
	1. ::Still need a mechanism to filter::
8. Ideally peer review provides malpractice detection but it’s failed at that. 
9. Removing peer review will increase herding frequency, which he argues is a good thing
	1. ::is higher frequency herding a good thing?::
10. Getting rid of peer review will fix credit a little because people won’t have to hitch their wagons to a brand and will work on what they actually want to do 
	1. ::maybe, but there’s also the thing online where people become so overwhelmed by uncharted content that they just go for the most popular thing::
11. Quality guarantees for outsiders - high end research will still get reviewed, low end research isn’t reviewed anyway - problem in the middle
12. [[The Matthew Effect]] - rich get richer. Does this already happen with peer review?
### Published
5/2020
## Questions
## Thoughts
- Could we compare to maybe the music industry? Other de-gatekeepered worlds?
- How would “we” pull it off?
- [[Coordination Problems]]
## Related
- [[Fair games have legible rules]]
## Authors
- [[Alex Danco]]
## Link
[Let’s just get rid of peer review – alexdanco.com](https://alexdanco.com/2020/05/28/lets-just-get-rid-of-peer-review/)



[Web URL for this note](http://notes.benjaminreinhardt.com/Lets_just_get_rid_of_peer_review_-_article)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Lets_just_get_rid_of_peer_review_-_article)




How many undetected frauds in science

Peer review fails to catch blatantly false things

# Peer review doesn’t catch duplications
[Elisabeth #StayingAtHome #StayingAlive Bik on Twitter: “Update!This is a set of (at least) EIGHT papers, from different authors, different hospitals, different cancers, different protein expressions. But with identical Kaplan-Meier curves, tables values, line graphs. See e.g. https://t.co/SmHEMPpEm5… https://t.co/grcaEE5tEI”](https://twitter.com/MicrobiomDigest/status/1266140721716719616)

### Related
- [[When you see a position being advocated with no pushback but no action - it means there are a bunch of little frictions in the system]]
- [[§Academia Constraints]]
- [[Academics are incentivized to do peer-reviewed publishable work]]
- [[The peer review and citation system incentivizes people to work on things that other people think is interesting]]
- [[Is peer review a good idea - paper]]
- [[Problems With Peer Review And Alternatives]]
- [[Three myths about scientific peer review]]
- [[Peer Review- Rodney Brooks]]
- [[Lets just get rid of peer review - article]]
- [[The Philosophical Basis of Peer Review and the Suppression of Innovation]]
- [[The fact that large-scale automated machine reading discovers new cancer-driving mechanisms doesn’t cite big mechanism is a clear failing of peer review]]
- [[Unlock ways to share data on peer review]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Peer_review_doesn’t_catch_duplications)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Peer_review_doesn’t_catch_duplications)




Research requires more trust than other disciplines

The word ‘novel’ is used as an idea bludgeon in academia

True Peer Peer Review

The journal of half baked ideas

# Peer review has failed as an institution
Let’s be clear that the words “peer review” have baggage that comes from outside the words themselves. You hear ‘peer review’ and think ‘oh this is legit because other scientists read it’ which is technically true. But nobody has any idea if they would actually consider the people who reviewed the paper to be actual peers.

One solution to this might be [[True Peer Peer Review]]. 

[[Want to See My Data? I Thought We Were Friends]] gives a good highlight of the problems, but the solutions are all incremental steps away from an inadequate equilibrium. It doesn’t address the meta-problem that there isn’t [[Slack - concept]] in the system.  [[Most institutions have become cancerous]]. 

### Related
- [[Peer review doesn’t catch duplications]]
- [[Lets just get rid of peer review - article]]
- [[The Philosophical Basis of Peer Review and the Suppression of Innovation]]
- [[The peer review and citation system incentivizes people to work on things that other people think is interesting]]
- [[Is peer review a good idea - paper]]
- [[Problems With Peer Review And Alternatives]]
- [[Academics are incentivized to do peer-reviewed publishable work]]
- [[Three myths about scientific peer review]]
- [[Peer Review- Rodney Brooks]]
- [[Goodhart's Law]]



[Web URL for this note](http://notes.benjaminreinhardt.com/Peer_review_has_failed_as_an_institution)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/Peer_review_has_failed_as_an_institution)




Stagnation

The way to get around journals is to focus on non-paper artifacts

Einstein would have been stuck in a patent office - failures of paradigm-shifting science

People do fundable work

Maverick work is lonely and DARPA provides peers and community

Human capital theory of stagnation

DARPA provides a derisking role for people in other organizations

The systemization of science since the 1950s treats all fields the same

A system of credit and funding for non-Scientific Paper artifacts should be artifact-type agnostic

Using citations to measure science is insidious

Is it possible that we’re actually getting less research because we have more researchers?

# The peer review and citation system incentivizes people to work on things that other people think is interesting
You could argue that this is a good mechanism that forces people to work on relevant work. On the other hand it enforces memesis. ([[Memetic Theory explains a lot of human nature]])

Put in a more extreme way [[Peer review gives peers veto power over each other]]. If something is too heretical, it just won’t make it through peer review. 

Even if there is funding available to do crazy shit you wouldn’t do it if you couldn’t get the results published and cited because [[Academia is the game where you gain status by getting attention for new knowledge]].

![](../BearImages/D70F62F9-2CA8-4B86-851D-505094A86D26-61320-000774DAF820EE3A/FD1B16C6-91A1-44CF-98EC-44CDE6D5AD96.png)

As a [[Stagnation]] theory this could be attributed t 

### Related
- [[Peer review doesn’t catch duplications]]
- [[Peer review has failed as an institution]]
- [[True Peer Peer Review]]
- [[People do fundable work]]

### References
- [[Peer Review- Rodney Brooks]]
- [[Is peer review a good idea - paper]]
- [[Problems With Peer Review And Alternatives]]
- [[Three myths about scientific peer review]]



[Web URL for this note](http://notes.benjaminreinhardt.com/The_peer_review_and_citation_system_incentivizes_people_to_work_on_things_that_other_people_think_is_interesting)

[Comment on this note](http://via.hypothes.is/http://notes.benjaminreinhardt.com/The_peer_review_and_citation_system_incentivizes_people_to_work_on_things_that_other_people_think_is_interesting)




Benʼs working notes

Is peer review a good idea - paper

Published

Questions

Thoughts

Related

Authors

Link

Links to this note