Peer Review
I’m in my metascience era and just read a neat paper by Wray (2025).
To assess whether a scientific article ought to be published, scientists use a peer review process. This process is important because peer reviewers are supposed to scrutinize articles to ensure that good research is published and bad research is not.
Unfortunately, peer review has a well-known inter-rater reliability problem. Say you send a paper to a scientific journal. You would expect reviewers to generally agree about whether it should, or should not, get published. As it happens, peer reviewers don’t agree very much, leading some to question whether peer review is better than chance.
Wray (2025) claims that this skeptical perspective on peer review is too skeptical. He says:
We should not think of peer review as providing a stamp of approval, as if the referees can reasonably determine which scientific claims are true. Instead, we should think of peer review as a means of filtering out errors. Peer-review works because it effectively identifies mistakes in manuscripts.
Despite the best intentions of referees when refereeing manuscripts, whatever they may be, the key function referees serve is to identify flaws in research. As such, they save journals from publishing egregious mistakes. In this way, they also save other researchers from inadvertently reading papers with such mistakes, which in turn can save them from building their research on these mistakes.
The argument is persuasive but I’m not sure if it vindicates peer review, at least as traditionally conceived. I think back to William James in “The Will to Believe” when he says:
We must know the truth; and we must avoid error — these are our first and great commandments as would-be knowers; but they are not two ways of stating an identical commandment, they are two separable laws.
Some people would rather avoid error than know the truth. There will be many important truths that come bundled with less-important errors. Those who are error-averse will, however, abandon important truths if they cannot be detangled from errors (no matter how small).