10. Peer Review: Publishing Review Reports
This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy!
This section describes the debate around whether peer review reports should be published. Next time we’ll have a look at the issues surrounding anonymity in peer review. Previous sections:
- Peer Review: An Introduction.
- Peer Review: An Early History
- Peer Review: The Modern Revolution
- Peer Review: Recent Studies
- Peer Review: Modern Role and Purpose
- Peer Review: Criticisms of the Conventional System
- Peer Review: Modern Trends and Traits
- Peer Review: Development of Open Peer Review
- Peer Review: Giving Credit to Referees
Publishing peer review reports
The rationale behind publishing referee reports lies in providing increased context and transparency to the peer review process, and can occur irrespective of whether or not the reviewers reveal their identities. Often, valuable insights are shared in reviews that would otherwise remain hidden if not published. By publishing reports, peer review has the potential to become a supportive and collaborative process that is viewed more as an ongoing dialogue between groups of scientists to progressively assess the quality of research. Furthermore, the reviews themselves are opened up for analysis and inspection, including how authors respond to reviews, adding an additional layer of quality control and a means for accountability and verification.
There are additional educational benefits to publishing peer reviews, such as training purposes or for journal clubs. Given the inconclusive evidence regarding the training of referees (Galipeau et al., 2015; Jefferson et al., 2007), such practices might be further useful in highlighting our knowledge and skills gaps. At the present, some publisher policies are extremely vague about the re-use rights and ownership of peer review reports (Schiermeier, 2017). The Peer Review Evaluation (PRE) service (www.pre-val.org) was designed to breathe some transparency into peer review, and provide information about the peer review itself without exposing the reports (e.g., mode of peer review, number of referees, rounds of review). While it describes itself as a service to identify fraud and maintain the integrity of peer review, it remains unclear whether it has achieved these objectives in light of the ongoing criticisms of the conventional process.
In a study of two journals, one where reports were not published and another where they were, Bornmann et al. (2012)found that publicized comments were much longer by comparison. Furthermore, there was an increased chance that they would result in a constructive dialogue between the author, reviewers, and wider community, and might therefore be better for improving the content of a manuscript. On the other hand, unpublished reviews tended to have more of a selective function to determine whether a manuscript is appropriate for a particular journal (i.e., focusing on the editorial process). Therefore, depending on the journal, different types of peer review could be better suited to perform different functions, and therefore optimized in that direction. Transparency of the peer review process can also be used as an indicator for peer review quality, thereby potentially enabling the tool to predict quality in new journals in which the peer review model is known, if desired (Godlee, 2002; Morrison, 2006; Wicherts, 2016). Journals with higher transparency ratings were less likely to accept flawed papers and showed a higher impact as measured by Google Scholar’s h5-index (Wicherts, 2016).
Assessments of research articles can never be evidence-based without the verification enabled by publication of referee reports. However, they are still almost ubiquitously regarded as having an authoritative, and uniform, stamp of quality. The issue here is that the attainment of peer reviewed status will always be based on an undefined, and only ever relative, quality threshold due to the opacity of the process. This is in itself quite an unscientific practice, and instead, researchers rely almost entirely on heuristics and trust for a concealed process and the intrinsic reputation of the journal, rather than anything legitimate. This can ultimately result in what is termed the “Fallacy of Misplaced Finality”, described by Kelty et al. (2008), as the assumption that research has a single, final form, to which everyone applies different criteria of quality.
Publishing peer review reports appears to have little or no impact on the overall process but may encourage more civility from referees. In a small survey, Nicholson & Alperin (2016) found that approximately 75% of survey respondents (n=79) perceived that public peer review would change the tone or content of the reviews, and 80% of responses indicated that performing peer reviews that would be eventually be publicized would not require a significantly higher amount of work. However, the responses also indicated that incentives are needed for referees to engage in this form of peer review. This includes recognition by performance review or tenure committees (27%), peers publishing their reviews (26%), being paid in some way such as with an honorarium or waived APC (24%), and getting positive feedback on reviews from journal editors (16%). Only 3% (one response) indicated that nothing could motivate them to participate in an open peer review of this kind.
Leek et al. (2011) showed that when referees’ comments were made public, significantly more cooperative interactions were formed, while the risk of incorrect comments decreased, suggesting that prior knowledge of publication encourages referees to be more constructive and careful with their reviews. Moreover, referees and authors who participated in cooperative interactions had a reviewing accuracy rate that was 11% higher. On the other hand, the possibility of publishing the reviews online has also been associated with a high decline rate among potential peer reviewers, and an increase in the amount of time taken to write a review, but with a variable effect on review quality (Almquist et al., 2017; van Rooyen et al., 2010). This suggests that the barriers to publishing review reports are inherently social, rather than technical.
When BioMed Central launched in 2000, it quickly recognized the value in including both the reviewers’ names and the peer review history (pre-publication) alongside published manuscripts in their medical journals in order to increase the quality and value of the process. Since then, further reflections on OPR (Godlee, 2002) led to the adoption of a variety of new models. For example, the Frontiers series now publishes all referee names alongside articles, EMBO journals publish a review process file with the articles, with referees remaining anonymous but editors being named, and PLOS added public commenting features to articles they published in 2009. More recently launched journals such as PeerJ have a system where both the reviews and the names of the referees can optionally be made public, and journals such as Nature Communications and the European Journal of Neuroscience have also started to adopt this method.
Unresolved issues with posting review reports include whether or not it should be conducted for ultimately unpublished manuscripts, and the impact of author identification or anonymity alongside their reports. Furthermore, the actual readership and usage of published reports remains ambiguous in a world where researchers are typically already inundated with published articles to read. The benefits of publicizing reports might not be seen until further down the line from the initial publication and, therefore, their immediate value might be difficult to convey and measure in current research environments. Finally, different populations of reviewers with different cultural norms and identities will undoubtedly have varying perspectives on this issue, and it is unlikely that any single policy or solution to posting referee reports will ever be widely adopted. Further investigation of the link between making reviews public and the impact this has on their quality would be a fruitful area of research to potentially encourage increased adoption of this practice.
Reference
Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.3)
BMJ Open is an example of an open access journal which is publishing alongside all articles the peer-review reports of all versions, together with the name of the reviewer and with the comments of the authors on all peer-review reports.
I had noted some issues in http://bmjopen.bmj.com/content/6/11/e012047 and I had therefore contacted the Editor-in-Chief of BMJ Open with a request for a copy of the ICMJE disclosure form of the guarantor of this article. The Editor-in-Chief responded with an invitation to submit an eletter. BMJ Open is a journal which is very willing to accept and publish eletters (responses / letters-to-the-editor / comments, etc.). There are no strict restrictions about the size of the text and/or about a deadline. See http://bmjopen.bmj.com/eletters
I have therefore conducted an exhaustive post-publication peer-review of this article. It was during the preparation of this post-publication peer-review very valuable that I was able to have access to the peer-review reports. The peer-review reports even contained details which I was unable to locate in the article in question. I have subsequently submitted my post-publication peer-review as eletter to BMJ Open. It got soon accepted and it was soon afterwards publised at http://bmjopen.bmj.com/content/6/11/e012047.responses#commentary-on-a-study-about-retraction-notices-in-journals-of-biomed-central
A copy of the ICMJE form was not received, also not after reminders. I have thus decided to prepare a manuscript about the unavailability of this ICMJE form. This manuscript was rejected by 3 different preprint servers. It was therefore not possible to get feedback / reviews / comments from readers of a preprint version of this manuscript.
The manuscript deals with a highly controversial topic. I have thus decided to submit a new version to a DOAJ-listed open access journal with no ties to any of the parties who are listed in my manuscript. This turned out to be a good strategy. My first attempt was immediately successful. The manuscript was quickly accepted by ‘Roars Transactions, a Journal on Research Policy and Evaluation’. This journal has no APC (so free to publish) and it is published by the ‘Università degli Studi di Milano’ (in Italy). See https://riviste.unimi.it/index.php/roars/article/view/9073
A short eletter to BMJ Open in which I refer to this new publication was quicky accepted and can be read through http://bmjopen.bmj.com/content/6/11/e012047.responses BMJ and both authors have until now not communicated with me about my paper at https://riviste.unimi.it/index.php/roars/article/view/9073 A copy of the ICMJE form was until now never received.
So yes, publishing the peer-review reports with the names of the reviewers and with the comments of the authors on all peer-review reports of all versions of an article is very valuable.
Awesome! Thanks for sharing this story, Klaas – appreciated!
hi Jon, thanks alot for your encouraging words. My post-publication peer-review contained for example the following sentences. “The review of reviewer 1 contains only two words. A response from the authors to reviewer 3 reveals that reviewer 1 has reviewed earlier versions. Details are lacking.” This post-publication peer-review was published on 18 January 2017.
It can be read at http://bmjopen.bmj.com/content/6/11/e012047.responses that BMJ Open has published on 13 February 2017 an apology about this issue and that since that day the full prepublication history has been updated. http://bmjopen.bmj.com/content/bmjopen/6/11/e012047.draft-revisions.pdf is the direct link to all versions of this paper and http://bmjopen.bmj.com/content/bmjopen/6/11/e012047.reviewer-comments.pdf is the direct link to all peer-reviews of all versions.
The review of R. Grant Steen on version 1 is harsh. “This paper reads like an advertisement for BioMed Central (BMC). The manuscript touts BMC’s adherence to COPE guidelines, its low overall rate of retraction, and its low rate of retraction specifically for dishonesty. Yet a low rate of retraction can arise in two ways; either editors are vigilant and weed out bad work pre-publication or editors are lax about oversight post-publication. It is impossible to know which process led to the low rate of retraction reported here. The fact that the paper was written by two people from the BMC Editorial office naturally led them to conclude that editorial oversight cannot be at fault. My experience with BMC journals has not been so positive that I want to help BMC conquer the world.”
This review was thus not yet available when I was conducting my post-publication peer-review of the article by Elizabeth Moylan and Maria Kowalczuk. Would you mind to reveal if you have ever received from a peer-reviewer such a harsh review on any of your submitted manuscripts?
Hi Klaas, as I know you’re aware of, I’m not exactly happy with the current peer review system either! 🙂 https://f1000research.com/articles/6-1151/v3
My experience of peer review has been largely positive, however. By this, I mean it has almost always invariably helped to improve the MS. While comments have sometimes been written in a manner that wasn’t perhaps as civil as could be, the motivations of the reviewers has always been positive and constructive, and to me that’s what matters. Palaeontology is a very small community though, and I suspect that we have different practices in review.
In several cases, however, I have received what I would call inadequate review – several lines of comments from a single referee on very large studies. But never what I would consider a ‘harsh’ review, except for depth of criticism of my research. Which is a good thing.