Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy! This section describes the role and purpose of modern peer review. Next week, criticisms of the modern system of peer review!

1. Peer Review: An Introduction.
2. Peer Review: An Early History
3. Peer Review: The Modern Revolution
4. Peer Review: Recent Studies

The role and purpose of modern peer review

The systematic use of external peer review has become entwined with the core activities of scholarly communication. Without approval through peer review to assess importance, validity, and journal suitability, research articles do not become part of the body of scientific knowledge. While in the digital world the costs of dissemination are very low, the marginal cost of publishing articles is far from zero (e.g., due to time and management, hosting, marketing, and technical and ethical checks). The economic motivations for continuing to impose selectivity in a digital environment, and applying peer review as a mechanism for this, have received limited attention or questioning, and are often simply regarded as how things are done. Use of selectivity is now often attributed to quality control, but may be more about building the brand and the demand from specific publishers or venues. Proprietary reviewer databases that enable high selectivity are seen as a good business asset. In fact, the attribution is based on the false assumption that peer review requires careful selection of specific reviewers to assure a definitive level of adequate quality, termed the “Fallacy of Misplaced Focus” by Kelty et al.(2008).

In addition to being used to judge submitted material for acceptance at a journal, review comments provided to the authors serve to improve the work and the writing and analysis skills of the authors. This feedback can lead to improvements to the submitted work that are iterated between the authors, reviewers, and editor, until the work is either accepted or the editor decides that it cannot be made acceptable for their specific scientific journal. In other cases, it allows the authors to improve their work to prepare for a new submission to another venue. In both cases, a good (i.e., constructive) peer review should provide general feedback that allows authors to improve their skills and competency at preparing and presenting their research. In a sense, good peer review can serve as distributed mentorship.

In many cases, there is an attempt to link the goals of peer review processes with Mertonian norms (Lee et al., 2013; Merton, 1973) (i.e., universalism, communalism, disinterestedness, and organized scepticism) as a way of showing their relation to shared community values. The Mertonian norm of organized scepticism is the most obvious link, while the norm of disinterestedness can be linked to efforts to reduce systemic bias, and the norm of communalism to the expectation of contribution to peer review as part of community membership (i.e., duty). In contrast to the emphasis on supposedly shared social values, relatively little attention has been paid to the diversity of processes of peer review across journals, disciplines, and time (an early exception is Zuckerman & Merton (1971)). This is especially the case as the (scientific) scholarly community appears overall to have a strong investment in a “creation myth” that links the beginning of scholarly publishing—the founding of The Philosophical Transactions of the Royal Society—to the invention of peer review. The two are often regarded to be coupled by necessity, largely ignoring the complex and interwoven histories of peer review and publishing. This has consequences, as the individual identity of a scholar is strongly tied to specific forms of publication that are evaluated in particular ways (Moore et al., 2017). A scholar’s first research article, doctoral thesis, or first book are significant life events. Membership of a community, therefore, is validated by the peers who review this newly contributed work. Community investment in the idea that these processes have “always been followed” appears very strong, but ultimately remains a fallacy.

As mentioned above, there is an increasing quantity and quality of research that examines how publication processes, selection, and peer review evolved from the 17th to the early 20th century, and how this relates to broader social patterns (Baldwin, 2017a; Baldwin, 2017b; Fyfe et al., 2017; Moxham & Fyfe, 2017). However, much less research critically explores the diversity of selection of peer review processes in the mid- to late-20th century. Indeed, there seems to be a remarkable discrepancy between the historical work we do have (Baldwin, 2017a; Gupta, 2016; Rennie, 2016; Shuttleworth & Charnley, 2016) and apparent community views that “we have always done it this way,” alongside what sometimes feels like a wilful effort to ignore the current diversity of practice. The result of this is an overall lack of evidence about the mechanics of peer review (e.g., time taken to review, conflict resolution, demographics of engaged parties, acceptance rates, quality of reviews, inherent biases, impact of referee training), both in terms of the traditional process and ongoing innovations, that obfuscates our understanding of the functionality and effectiveness of the present system (Jefferson et al., 2007). However, such a lack of evidence should not be misconstrued as evidence for the failure of these systems, but interpreted more as representing difficulties in empirically assessing the effectiveness of a diversity of practices in peer review.

Such a discrepancy between a dynamic history and remembered consistency could be a consequence of peer review processes being central to both scholarly identity as a whole and to the identity and boundaries of specific communities (Moore et al., 2017). Indeed, this story linking identity to peer review is taught to junior researchers as a community norm, often without the much-needed historical context. More work on how peer review, alongside other community practices, contributes to community building and sustainability would be valuable. Examining criticisms of conventional peer review and proposals for change through the lens of community formation and identity may be a productive avenue for future research.

Reference

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 2; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.2)