Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy! This section describes the development of Open Peer Review! Next time we’ll delve into the issue of crediting peer review.

1. Peer Review: An Introduction.
2. Peer Review: An Early History
3. Peer Review: The Modern Revolution
4. Peer Review: Recent Studies
5. Peer Review: Modern Role and Purpose
6. Peer Review: Criticisms of the Conventional System
7. Peer Review: Modern Trends and Traits

The development of open peer review

New suggestions to modify peer review vary, between fairly incremental small-scale changes, to those that encompass an almost total and radical transformation of the present system. A core question is how to transform traditional peer review into a process that is aligned with the latest advances in what is now widely termed “open science”. This is tied to broader developments in how we as a society communicate, thanks to the inherent capacity that the Web provides for open, collaborative, and social communication. Many of the suggestions and new models for opening peer review up are geared towards increasing different levels of transparency, and ultimately the reliability, efficiency, and accountability of the publishing process. These traits are desired by all actors in the system, and increasing transparency moves peer review towards a more open model.

Novel ideas about “Open Peer Review” (OPR) systems are rapidly emerging, and innovation has been accelerating over the last several years (Figure 2; Table 3). The advent of OPR is complex, as the term can refer to multiple different parts of the process and is often used inter-changeably or conflated without appropriate prior definition. Currently, there is no formally established definition of OPR that is accepted by the scholarly research and publishing community (Ford, 2013). The most simple definitions by McCormack (2009) and Mulligan et al. (2008) presented OPR as a process that does not attempt “to mask the identity of authors or reviewers” (McCormack, 2009, p.63), thereby explicitly referring to open in terms of personal identification or anonymity. Ware (2011, p.25) expanded on reviewer disclosure practices: “Open peer review can mean the opposite of double blind, in which authors’ and reviewers’ identities are both known to each other (and sometimes publicly disclosed), but discussion is complicated by the fact that it is also used to describe other approaches such as where the reviewers remain anonymous but their reports are published.” Other authors define OPR distinctly, for example by including the publication of all dialogue during the process (Shotton, 2012), or running it as a publicly participative commentary (Greaves et al., 2006).

However, the context of this transparency and the implications of different modes of transparency at different stages of the review process are both very rarely explored. Progress towards achieving transparency has been variable but generally slow across the publishing system. Engagement with experimental open models is still far from common, in part perhaps due to a lack of rigorous evaluation and empirical demonstration that they are more effective processes. A consequence of this is the entrenchment of the ubiquitously practiced and much more favored traditional model (which, as noted above, is also diverse). However, as history shows, such a process is non-traditional but nonetheless currently held in high regard. Practices such as self-publishing and predatory or deceptive publishing cast a shadow of doubt on the validity of research posted openly online that follow these models, including those with traditional scholarly imprints (Fitzpatrick, 2011a; Tennant et al., 2016). The inertia hindering widespread adoption of new models of peer review can be ascribed to what is often termed “cultural inertia”, and affects many aspects of scholarly research. Cultural inertia, the tendency of communities to cling to a traditional trajectory, is shaped by a complex ecosystem of individuals and groups. These often have highly polarized motivations (i.e., capitalistic commercialism versus knowledge generation versus careerism versus output measurement), and an academic hierarchy that imposes a power dynamic that can suppress innovative practices (Burris, 2004; Magee & Galinsky, 2008).

How and where we inject transparency has implications for the magnitude of transformation required and, therefore, the general concept of OPR is highly heterogeneous in meaning, scope, and consequences. A recent survey by OpenAIRE found 122 different definitions of OPR in use, exemplifying the extent of this issue. This diversity was distilled into a single proposed definition comprising seven different traits of OPR: participation, identity, reports, interaction, platforms, pre-review manuscripts, and final-version commenting (Ross-Hellauer, 2017). The various parts of the “revolutionary” phase of peer review undoubtedly have different combinations of these OPR traits, and it remains a very heterogeneous landscape. Table 3 provides an overview of the advantages and disadvantages of the different approaches to anonymity and openness in peer review.

The ongoing discussions and innovations around peer review (and OPR) can be sorted into four main categories, which are examined in more detail below. Each of these feed into the wider core issues in peer review of incentivizing engagement, providing appropriate recognition and certification, and quality control and moderation: