Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy!

This section describes part 3 the lively debate around whether peer review reports should be anonymised or not. Previous sections:

1. An Introduction.
2. An Early History
3. The Modern Revolution
4. Recent Studies
5. Modern Role and Purpose
6. Criticisms of the Conventional System
7. Modern Trends and Traits
8. Development of Open Peer Review
9. Giving Credit to Referees
10. Publishing Review Reports
11. Anonymity Versus Identification
12. Anonymity Versus Identification (II)

The impact of identification and anonymity on bias.

One of the biggest criticisms levied at peer review is that, like many human endeavours, it is intrinsically biased and not the objective and impartial process many regard it to be. Yet, the question is no longer about whether or not it is biased, but to what extent it is in different social dimensions – a debate which is very much ongoing (e.g., (Lee et al., 2013; Rodgers, 2017; Tennant, 2017)). One of the major issues is that peer review suffers from systemic confirmatory bias, with results that are deemed as significant, statistically or otherwise, being preferentially selected for publication (Mahoney, 1977). This causes a distinct bias within the published research record (van Assen et al., 2014), as a consequence of perverting the research process itself by creating an incentive system that is almost entirely publication-oriented. Others have described the issues with such an asymmetric evaluation criteria as lacking the core values of a scientific process (Bon et al., 2017).

The evidence on whether there is bias in peer review against certain author demographics is mixed, but overwhelmingly in favor of systemic bias against women in article publishing (Budden et al., 2008; Darling, 2015; Grivell, 2006; Helmer et al., 2017; Kuehn, 2017; Lerback & Hanson, 2017; Lloyd, 1990; McKiernan, 2003; Roberts & Verhoef, 2016; Smith, 2006; Tregenza, 2002) (although see also Blank (1991); Webb et al. (2008); Whittaker (2008)). After the journal Behavioural Ecology adopted double blind peer review in 2001, there was a significant increase in accepted manuscripts by women first authors; an effect not observed in similar journals that did not change their peer review policy (Budden et al., 2008). One of the most recent public examples of this bias is the case where a reviewer told the authors that they should add more male authors to their study (Bernstein, 2015). More recently, it has been shown in the Frontiers journal series that women are under-represented in peer-review and that editors of both genders operate with substantial same-gender preference (Helmer et al., 2017). The most famous, but also widely criticised, piece of evidence on bias against authors comes from a study by Peters & Ceci (1982) using psychology journals. They took 12 published psychology studies from prestigious institutions and retyped the papers, making minor changes to the titles, abstracts, and introductions but changing the authors’ names and institutions. The papers were then resubmitted to the journals that had first published them. In only three cases did the journals realize that they had already published the paper, and eight of the remaining nine were rejected—not because of lack of originality but because of the perception of poor quality. Peters & Ceci (1982)concluded that this was evidence of bias against authors from less prestigious institutions, although the deeper causes of this bias remain unclear at the present. A similar effect was found in an orthopaedic journal by Okike et al. (2016), where reviewers were more likely to recommend acceptance when the authors’ names and institutions were visible than when they were redacted. Further studies have shown that peer review is substantially positively biased towards authors from top institutions (Ross et al., 2006; Tomkins et al., 2017), due to the perception of prestige of those institutions and, consequently, of the authors as well. Further biases based on nationality and language have also been shown to exist (Dall’Aglio, 2006; Ernst & Kienbacher, 1991; Link, 1998; Ross et al., 2006; Tregenza, 2002).

While there are relatively few large-scale investigations of the extent and mode of bias within peer review (although see Lee et al. (2013) for an excellent overview), these studies together indicate that inherent biases are systemically embedded within the process, and must be accounted for prior to any further developments in peer review. This range of population-level investigations into attitudes and applications of anonymity, and the extent of any biases resulting from this, exposes a highly complex picture, and there is little consensus on its impact at a system-wide scale. However, based on these often polarised studies, it is inescapable to conclude that peer review is highly subjective, rarely impartial, and definitely not as homogeneous as it is often regarded.

Applying a single, blanket policy across the entire peer review system regarding anonymity would greatly degrade the ability of science to move forward, especially without a wide flexibility to manage exceptions. The reasons to avoid one definite policy are the inherent complexity of peer review systems, the interplay with different cultural aspects within the various sub-sectors of research, and the difficulty in identifying whether anonymous or identified works are objectively better. As a general overview of the current peer review ecosystem, Nobarany & Booth (2016) recently recommended that, due to this inherent diversity, peer review policies and support systems should remain flexible and customizable to suit the needs of different research communities. For example, some publishers allow authors to opt in to double blinded review Palus (2015), and others could expand this to offer a menu of peer review options. We expect that, by emphasizing the differences in shared values across research communities, we will see a new diversity of OPR processes developed across disciplines in the future. Remaining ignorant of this diversity of practices and inherent biases in peer review, as both social and physical processes, would be an unwise approach for future innovations.

Reference

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.3)