Why we shouldn’t take peer review as the ‘gold standard’
In July, India’s government dismissed a research paper finding that the country’s economic growth had been overestimated, saying the paper had not been “peer reviewed.” At a conference for plastics engineers, an economist from an industry group dismissed environmental concerns about plastics by claiming that some of the underlying research was “not peer reviewed.” And the Trump administration — not exactly known for its fealty to science — attempted to reject a climate change report by stating, incorrectly, that it lacked peer review.
Researchers commonly refer to peer review as the “gold standard,” which makes it seem as if a peer-reviewed paper — one sent by journal editors to experts in the field who assess and critique it before publication — must be legitimate, and one that’s not reviewed must be untrustworthy. But peer review, a practice dating to the 17th century, is neither golden nor standardized. Studies have shown that journal editors prefer reviewers of the same gender, that women are underrepresented in the peer review process, and that reviewers tend to be influenced by demographic factors like the author’s gender or institutional affiliation. Shoddy work often makes it past peer reviewers, while excellent research has been shot down. Peer reviewers often fail to detect bad research, conflicts of interest and corporate ghostwriting.
That’s a really interesting point that the research paper makes that peer-review confers a false sense of security to a paper, which results in less subsequent scrutiny.
It also sounds like the paper is pushing for involving the authors more in the peer-review process, but I couldn’t quite understand how that was meant to proceed. Can you elaborate at all?
Yep, absolutely. See the discussion here on Topic 4, with Feynman and the ‘culture of doubt’: https://www.mdpi.com/2304-6775/7/2/34/htm
One possible way to do peer review more efficiently is to have it as more of an interactive and iterative discussion. Copernicus, Frontiers, and Elife are all publishers which try this in one form or another at the moment. I proposed a simple model for this recently too https://academic.oup.com/femsle/article/365/19/fny204/5078345