Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy!

This section describes some of the ways which peer review has been decoupled from traditional journals, including via preprints and overlay journals. Previous sections:

1. An Introduction.
2. An Early History
3. The Modern Revolution
4. Recent Studies
5. Modern Role and Purpose
6. Criticisms of the Conventional System
7. Modern Trends and Traits
8. Development of Open Peer Review
9. Giving Credit to Referees
10. Publishing Review Reports
11. Anonymity Versus Identification
12. Anonymity Versus Identification (II)
13. Anonymity Versus Identification (III)
14. Decoupling Peer Review from Publishing
15. Preprints and Overlay Journals
    

    ---

Registered Reports represent a significant departure from conventional peer review in terms of relative timing and increased rigour (Chambers et al., 2014; Chambers et al., 2017; Nosek & Lakens, 2014). Here, peer review is split into two stages. Research questions and methodology (i.e., the study design itself) are subject to a first round of evaluation prior to any data collection or analysis taking place (Figure 4). Such a process is analogous to clinical trials registrations for medical research, the implementation of which became widespread many years before Registered Reports, and is a well-established specialised process that innovative peer review models could learn a lot from. If a protocol is found to be of sufficient quality to pass this stage, the study is then provisionally accepted for publication. Once the research has been finished and written-up, completed manuscripts are then subject to a second-stage of peer review which, in addition to affirming the soundness of the results, also confirms that data collection and analysis occurred in accordance with the originally described methodology. The format, originally introduced by the psychology journals Cortex and Perspectives in Psychological Science in 2013, is now used in some form by more than 70 journals (Nature Human Behaviour, 2017) (see cos.io/rr/ for an up-to-date list of participating journals). Registered Reports are designed to boost research integrity by ensuring the publication of all research results, which helps reduce publication bias. As opposed to the traditional model of publication, where “positive” results are more likely to be published, results remain unknown at the time of the first review stage and therefore even “negative” results are equally as likely to be published. Such a process is designed to incentivize data-sharing, guard against dubious practices such as selective reporting of results (via so-called “p-hacking” and “HARKing”—Hypothesizing After the Results are Known) and low statistical power, and also prioritizes accurate reporting over that which is perceived to be of higher impact or publisher worthiness.

Reference

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.3)