Do we need a new research field of Peer Review Studies?
Today, Tony Ross-Hellauer and I have a new preprint shared on SocArXiv. It’s about the limits to our understanding of peer review. Rather than simply being another meta-analysis of all of the things we do have evidence for about peer review, we tried to think of it a different way: what are all the things we DO NOT know about peer review? And well, it turns out, quite a bit!
This was quite a difficult cognitive exercise, as the potential for what we don’t know is well, vast. There are likely to be gaps and errors throughout, and if anyone has the time, we would love any feedback. Criticisms and kind words equally appreciated! Either way, we hope that you find this useful, and it will be interesting to see where our proposal leads. Here’s the abstract below:
Peer review is embedded in the core of our scholarly knowledge generation systems, conferring legitimacy on research while distributing academic capital and prestige on individuals. Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have programmatically analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie. We distill this into core themes around editorial accountability, the subjectivity and bias of reviewers, the function and quality of peer review, the role of the reviewer, the social and epistemic implications of peer review, and regarding innovations in open peer review platforms and services. We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record. This will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this. We recognise that many of the criticisms attributed to peer review might reflect wider issues within academia and wider society, and future care will be required in order to carefully demarcate and address these.