Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy! This section describes the debate about how to credit peer review. Next time we’ll have a look at whether peer review reports should be published.

1. Peer Review: An Introduction.
2. Peer Review: An Early History
3. Peer Review: The Modern Revolution
4. Peer Review: Recent Studies
5. Peer Review: Modern Role and Purpose
6. Peer Review: Criticisms of the Conventional System
7. Peer Review: Modern Trends and Traits
8. Peer Review: Development of Open Peer Review

Giving credit to peer reviewers

A vast majority of researchers see peer review as an integral and fundamental part of their work Mulligan et al. (2013). They often consider peer review to be part of an altruistic cultural duty or a quid pro quo service, closely associated with the identity of being part of their research community. To be invited to review a research article can be perceived as a great honor, especially for junior researchers, due to the recognition of expertise—i.e., the attainment of the level of a peer. However, the current system is facing new challenges as the number of published papers continues to increase rapidly (Albert et al., 2016), with more than one million articles published in peer reviewed, English-language journals every year (Larsen & Von Ins, 2010). Some estimates are even as high as 2–2.5 million per year (Plume & van Weijen, 2014), and this number is expected to double approximately every nine years at current rates (Bornmann & Mutz, 2015). Several potential solutions exist to make sure that the review process does not cause a bottleneck in the current system:

Of these, the latter two can both potentially reduce the quality of peer review and therefore affect the overall quality of published research. Paradoxically, while the Web empowers us to communicate information virtually instantaneously, the turn around time for peer reviewed publications remains quite long by comparison. One potential solution is to encourage referees by providing additional recognition and credit for their work. The present lack of bona fide incentives for referees is perhaps one of the main factors responsible for indifference to editorial outcomes, which ultimately leads to the increased proliferation of low quality research (D’Andrea & O’Dwyer, 2017; Jefferson et al., 2007; Wang et al., 2016).

Traditional methods of recognition.

One current way to recognize peer reviewers is to thank anonymous referees in the Acknowledgement sections of published papers. In these cases, the referees will not receive any public recognition for their work, unless they explicitly agree to sign their reviews. Generally, journals do not provide any remuneration or compensation for these services. Notable exceptions are the UK-based publisher Veruscript (veruscript.com/about/who-we-are) and Collabra (collabra.org/about/our-model), published by University of California Press, as well as most statistical referees (Barbour, 2017). Other journals provide reward incentives to reviewers, such as free subscriptions or discounts on author-facing open access fees. Another common form of acknowledgement is a private thank you note from the journal or editor, which usually takes the form of an automated email upon completion of the review. In addition, journals often list and thank all reviewers in a special issue or on their website once a year, thus providing another way to recognise reviewers. Some journals even offer annual prizes to reward exceptional referee activities (e.g., the Journal of Clinical Epidemiology; www.jclinepi.com/article/S0895-4356(16)30707-7/fulltext). Another idea that journals and publishers have tried implementing is to list the best reviewers for their journal (e.g., by Vines (2015a) for Molecular Ecology), or, on the basis of a suggestion by Pullum (1984), naming referees who recommend acceptance in the article colophon (a single blind version of this recommendation was adopted by Digital Medievalist from 2005–2016; see Wikipedia contributors, 2017, and bit.ly/DigitalMedievalistArchive for examples preserved in the Internet Archive). Digital Medievalist stopped using this model and removed the colophon as part of its move to the Open Library of Humanities; cf. journal.digitalmedievalist.org). As such, authors can then integrate this into their scholarly profiles in order to differentiate themselves from other researchers or referees. Currently, peer review is poorly acknowledged by practically all research assessment bodies, institutions, granting agencies, as well as publishers, in the process of professional advancement or evaluation. Instead, it is viewed as expected or normal behaviour for all researchers to contribute in some form to peer review.

Increasing demand for recognition.

These traditional approaches of credit fall short of any sort of systematic feedback or recognition, such as that granted through publications. A change here is clearly required for the wealth of currently unrewarded time and effort given to peer review by academics. A recent survey of nearly 3,000 peer reviewers by the large publisher Wiley showed that feedback and acknowledgement for work as referees are valued far above either cash reimbursements or payment in kind (Warne, 2016) (although Mulligan et al. (2013) found that referees would prefer either compensation by way of free subscriptions, or the waiver of colour or other publication charges). Wiley’s survey reports that 80% of researchers agree that there is insufficient recognition for peer review as a valuable research activity and that researchers would actually commit more time to peer review if it became a formally recognized activity for assessments, funding opportunities, and promotion (Warne, 2016). While this may be true, it is important to note that commercial publishers have a vested interest in retaining the current, freely provided service of peer review, since this is what provides their journals the main stamp of legitimacy and quality (“added value”) as society-led journals. Therefore, one of the root causes for the lack of appropriate recognition and incentivization is publishers with have strong motivations to find non-monetary forms of reviewer recognition. Indeed, the business model of almost every scholarly publisher is predicated on free work by peer reviewers, and it is unlikely that the present system would function financially with market-rate reimbursement for reviewers. Other research shows a similar picture, with approximately 70% of respondents to a small survey done by Nicholson & Alperin (2016) indicating that they would list peer review as a professional service on their curriculum vitae. 27% of respondents mentioned formal recognition in assessment as a factor that would motivate them to participate in public peer review. These numbers indicate that the lack of credit referees receive for peer review is likely a strong contributing factor to the perceived stagnation of traditional models. Furthermore, acceptance rates are lower in humanities and social sciences, and higher in physical sciences and engineering journals (Ware, 2008), as well as differences based on relative referee seniority (Casnici et al., 2017). This means there are distinct disciplinary variations in the number of reviews performed by a researcher relative to their publications, and suggests that there is scope for using this to either provide different incentive structures or to increase acceptance rates and therefore decrease referee fatigue (Fox et al., 2017; Lyman, 2013).

Progress in crediting peer review.

Any acknowledgement model to credit reviewers also raises the obvious question of how to facilitate this model within an anonymous peer review system. By incentivizing peer review, much of its potential burden can be alleviated by widening the potential referee pool concomitant with the growth in review requests. This can also help to diversify the process and inject transparency into peer review, a solution that is especially appealing when considering that it is often a small minority of researchers who perform the vast majority of peer reviews (Fox et al., 2017; Gropp et al., 2017); for example, in biomedical research, only 20 percent of researchers perform 70–95 percent of the reviews (Kovanis et al., 2016). In 2014, a working group on peer review services (CASRAI) was established to “develop recommendations for data fields, descriptors, persistence, resolution, and citation, and describe options for linking peer-review activities with a person identifier such as ORCID” (Paglione & Lawrence, 2015). The idea here is that by being able to standardize the description of peer review activities, it becomes easier to attribute, and therefore recognize and reward them.

The Publons platform provides a semi-automated mechanism to formally recognize the role of editors and referees who can receive due credit for their work as referees, both pre- and post-publication. Researchers can also choose if they want to publish their full reports depending on publisher and journal policies. Publons also provides a ranking for the quality of the reviewed research article, and users can endorse, follow, and recommend reviews. Other platforms, such as F1000 Research and ScienceOpen, link post-publication peer review activities with CrossRef DOIs and open licenses to make them more citable, essentially treating them equivalent to a normal open access research paper. ORCID (Open Researcher and Contributor ID) provides a stable means of integrating these platforms with persistent researcher identifiers in order to receive due credit for reviews. ORCID is rapidly becoming part of the critical infrastructure for open OPR, and greater shifts towards open scholarship (Dappert et al., 2017). Exposing peer reviews through these platforms links accountability to receiving credit. Therefore, they offer possible solutions to the dual issues of rigor and reward, while potentially ameliorating the growing threat of reviewer fatigue due to increasing demands on researchers external to the peer review system (Fox et al., 2017; Kovanis et al., 2016).

Whether such initiatives will be successful remains to be seen However, Publons was recently acquired by Clarivate Analytics, suggesting that the process could become commercialized as this domain rapidly evolves (Van Noorden, 2017). In spite of this, the outcome is most likely to be dependent on whether funding agencies and those in charge of tenure, hiring, and promotion will use peer review activities to help evaluate candidates. This is likely dependent on whether research communities themselves choose to embrace any such crediting or accounting systems for peer review.

Reference

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.3)