Peer review is a great system for preventing waste

Peer review is good at preventing waste because it gives many people the ability to say “no” < Analyze by who can tap the breaks and who can open the throttle> and therefore you have multiple independent checks on whether a project will be wasteful. If each of m reviewers has an n% chance of noticing a wasteful proposal, then then a wasteful proposal has a (1-n)^m chance of making it through.

Peer review is effectively a distributed committee and thus has most of the properties of committee based decision making. Thinking about it, I’m tempted to ask whether anything would be different if we replaced the words “peer review” with “committee” everywhere they appear. <Decisions made by committees lead to median results>

Of course, I don’t think we culturally have a good handle on what wasteful means in the context of research. ‘Wasteful’ is implicitly tied to efficiency which requires a metric on the input and output of a process. Assuming efficiency is the thing you want to maximize^1 what are the metrics on research? Money is probably a reasonable metric on input at an organizational level but output? Citations or number of papers with a lot of citations is a common output metric, but do most people or funding agencies actually care about citations - no, especially organizations like the military. <Using citations to measure science is insidious>


Web URL for this note

Comment on this note

^1: Which shouldn’t be a given! Some wonderful things in the world are wildly inefficient! Efficiency is overrated