Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy!

Introduction to Peer Review

Peer review is a core part of our self-regulating global scholarship system. It defines the process in which professional experts (peers) are invited to critically assess the quality, novelty, theoretical and empirical validity, and potential impact of research by others, typically while it is in the form of a manuscript for an article, conference, or book (Daniel, 1993; Kronick, 1990; Spier, 2002; Zuckerman & Merton, 1971). For the purposes of this article, we are exclusively addressing peer review in the context of manuscript selection for scientific research articles, with some initial considerations of other outputs such as software and data. In this form, peer review is becoming increasingly central as a principle of mutual control in the development of scholarly communities that are adapting to digital, information-rich, publishing-driven research ecosystems. Consequently, peer review is a vital component at the core of research communication processes, with repercussions for the very structure of academia, which largely operates through a peer reviewed publication-based reward and incentive system (Moore et al., 2017). Different forms of peer review beyond that for manuscripts are also clearly important and used in other contexts such as academic appointments, measurement time, research ethics or research grants (see, e.g., Fitzpatrick, 2011b, p. 16), but a holistic discussion of all forms of peer review is beyond the scope of the present article.

Peer review is not a singular or static entity. It comes in various flavors that result from different approaches to the relative timing of the review in the publication cycle, the reciprocal transparency of the process, and the contrasting and disciplinary practices (Ross-Hellauer, 2017). Such interdisciplinary differences have made the study and understanding of peer review highly complex, and implementing any systemic changes to peer review is fraught with the challenges of synchronous adoption between heterogeneous communities often with vastly different social norms and practices. The criteria used for evaluation, including methodological soundness or expected scholarly impact, are typically important variables to consider, and again vary substantially between disciplines. However, peer review is still often perceived as a “gold standard” (e.g., D’Andrea & O’Dwyer (2017); Mayden (2012)), despite the inherent diversity of the process and never intended to be used as such. Peer review is applied inconsistently both in theory and practice (Casnici et al., 2017; Pontille & Torny, 2015), and generally lacks any form of transparency or formal standardization. As such, it remains difficult to know precisely what a “peer reviewed publication” means.

Traditionally, the function of peer review has been as a vetting procedure or gatekeeper to assist the distribution of limited resources—for instance, space in peer reviewed print publication venues. With the advent of the internet, the physical constraints on distribution are no longer present, and, at least in theory, we are now able to disseminate research content rapidly and at relatively negligible cost (Moore et al., 2017). This has led to the innovation and increasing popularity of digital-only publication venues that vet submissions based exclusively on the soundness of the research, often termed “mega-journals” (e.g., PLOS ONE, PeerJ, the Frontiers series). Such a flexibility in the filter function of peer review reduces, but does not eliminate, the role of peer review as a selective gatekeeper, and can be considered to be “impact neutral.” Due to such digital experimentations, ongoing discussions about peer review are intimately linked with contemporaneous developments in Open Access (OA) publishing and to broader changes in open scholarship (Tennant et al., 2016).