Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy! This section describes some of the main criticisms of the modern system of peer review. The next post will explore some of the traits and trends affecting modern peer review.

1. Peer Review: An Introduction.
2. Peer Review: An Early History
3. Peer Review: The Modern Revolution
4. Peer Review: Recent Studies
5. Peer Review: Modern Role and Purpose

Criticisms of the conventional peer review system

In spite of its clear relevance, widespread acceptance, and long-standing practice, the academic community does not appear to have a clear consensus on the operational functionality of peer review, and what its effects in a diverse modern research world are. There is a discrepancy between how peer review is regarded as a process and how it is actually performed. While peer review is still generally perceived as key to quality control for research, it has been argued that mistakes are becoming more frequent in the process (Margalida & Colomer, 2016; Smith, 2006), and that peer review is not being applied as rigorously as generally perceived. As a result, it has become the target of widespread criticism, with a range of empirical studies investigating the reliability, credibility and fairness of the scholarly publishing and peer review process (e.g., (Bruce et al., 2016; Cole, 2000; Eckberg, 1991; Ghosh et al., 2012; Jefferson et al., 2002; Kostoff, 1995; Ross-Hellauer, 2017; Schroter et al., 2006; Walker & Rocha da Silva, 2015)). In response to this, initiatives like the EQUATOR network (equator-network.org) have been important to improve the reporting of research and its peer review according to standardised criteria. Another response has been COPE, the Committee on Publication Ethics (publicationethics.org), established in 1997 to address potential cases of abuse and misconduct during the publication process (specifically regarding author misconduct), and later created specific guidelines for peer review. Yet, the effectiveness of this initiative at a system-level remains unclear. A popular editorial in The BMJ made some quiter serious allegations at peer review, stating that it is “slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless at detecting fraud” (Smith, 2006). In addition, beyond editorials, a substantial corpus of studies has now critically examined the technical aspects of conventional journal article peer review (e.g., (Armstrong, 1997; Bruce et al., 2016; Jefferson et al., 2007; Overbeke, 1999; Pöschl, 2012; Siler et al., 2015a)), with overlapping and some times contrasting results.

The issue is that, ultimately, this uncertainty in standards and implementation can, at least in part, potentially lead to widespread failures in research quality and integrity (Ioannidis, 2005; Jefferson et al., 2002), and even the rise of formal retractions in extreme cases (Steen et al., 2013). Issues resulting from peer review failure range from simple gate-keeping errors, based on differences in opinion of the perceived impact of research, to failing to detect fraudulent or incorrect work, which then enters the scientific record (Baxt et al., 1998; Gøtzsche, 1989; Haug, 2015; Moore et al., 2017; Pocock et al., 1987; Schroter et al., 2004; Smith, 2006). A final issue regards peer review by and for non-native English speaking authors, which can lead to cases of linguistic inequality and language-oriented research segregation, in a world where research is increasingly becoming more globally competitive (Salager-Meyer, 2008, Salager-Meyer, 2014). Such criticisms should be a cause for concern given that traditional peer review is still viewed by some, almost by concession, as a gold standard and requirement for the publication of research results (Mayden, 2012). All of this suggests that, while the concept of peer review remains logical and required, it is the practical implementation of it that demands further attention.

Peer review needs to be peer reviewed.

Attempts to reproduce how peer review selects what is worthy of publication demonstrate that the process is generally adequate for detecting reliable research, but often fails to recognize the research that has the greatest impact (Mahoney, 1977; Moore et al., 2017; Siler et al., 2015b). Many critics now view traditional peer review as sub-optimal and detrimental to research because it causes publication delays, with repercussions on the dissemination of novel research (Armstrong, 1997; Bornmann & Daniel, 2010a; Brembs, 2015; Eisen, 2011; Jubb, 2016; Vines, 2015b). Reviewer fatigue and redundancy when articles go through multiple rounds of peer review at different journal venues (Breuning et al., 2015; Fox et al., 2017; Jubb, 2016; Moore et al., 2017) are just some of the major criticisms levied at the technical implementation of peer review. In addition, some view many common forms of peer review as flawed because they operate within a closed and opaque system. This makes it impossible to trace the discussions that led to (sometimes substantial) revisions to the original research (Bedeian, 2003), the decision process leading to the final publication, or whether peer review even took place. By operating as a closed system, it protects the status quo and suppresses research viewed as radical, innovative, or contrary to the theoretical or established perspectives of referees (Alvesson & Sandberg, 2014; Benda & Engels, 2011; Horrobin, 1990; Mahoney, 1977; Merton, 1968; Siler et al., 2015a; Siler & Strang, 2017), even though it is precisely these factors that underpin and advance research. As a consequence, questions arise as to the competency, effectiveness, and integrity, as well as participatory elements, of traditional peer review, such as: who are the gatekeepers and how are the gates constructed; what is the balance between author-reviewer-editor tensions and how are these power relations and conflicts resolved; what are the inherent biases associated with this; does this enable a fair or structurally inclined system of peer review to exist; and what are the repercussions for this on our knowledge generation and communication systems?

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 2; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.2)