Green Tea and Velociraptors

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy!

This section is the introduction to the next major part of the paper on potential different models of peer review. Next time, what would Reddit-style peer review look like? Previous parts in this series:

1. An Introduction.
2. An Early History
3. The Modern Revolution
4. Recent Studies
5. Modern Role and Purpose
6. Criticisms of the Conventional System
7. Modern Trends and Traits
8. Development of Open Peer Review
9. Giving Credit to Referees
10. Publishing Review Reports
11. Anonymity Versus Identification
12. Anonymity Versus Identification (II)
13. Anonymity Versus Identification (III)
14. Decoupling Peer Review from Publishing
15. Preprints and Overlay Journals
16. Two-stage peer review and Registered Reports
17. Peer review by endorsement
18. Limitations of decoupled Peer Review

As we have discussed in detail above, there has been considerable innovation in peer review in the last decade, which is leading to widespread critical examination of the process and scholarly publishing as a whole (e.g., (Kriegeskorte et al., 2012)). Much of this has been driven by the advent of Web 2.0 technologies and new social media platforms, and an overall shift towards a more open system of scholarly communication. Previous work in this arena has described features of a Reddit-like model, combined with additional personalized features of other social platforms, like Stack Exchange, Netflix, and Amazon (Yarkoni, 2012). Here, we develop upon this by considering additional traits of models such as Wikipedia, GitHub, and Blockchain, and discuss these in the context of the rapidly evolving socio-technological environment for the present system of peer review. In the following section, we discuss potential future peer review platforms and processes in the context of the following three major traits, which any future innovation would greatly benefit from consideration of:

• 1. Quality control and moderation, possibly through openness and transparency;

• 2. Certification via personalized reputation or performance metrics;

• 3. Incentive structures to motivate and encourage engagement.

While discussing a number of principles that should guide the implementation of novel platforms for evaluating scientific work, Yarkoni (2012) argued that many of the problems researchers face have already been successfully addressed by a range of non-research focused social Web applications. Therefore, developing next-generation platforms for scientific evaluations should focus on adapting the best currently used approaches for these rather than on innovating entirely new ones (Neylon & Wu, 2009; Priem & Hemminger, 2010; Yarkoni, 2012). One important element that will determine the success or failure of any such peer-to-peer reputation or evaluation system is a critical mass of researcher uptake. This has to be carefully balanced with the demands and uptakes of restricted scholarly communities, which have inherently different motivations and practices in peer review. A remaining issue is the aforementioned cultural inertia, which can lead to low adoption of anything innovative or disruptive to traditional workflows in research. This is a perfectly natural trait for communities, where ideas out-pace technological innovation, which in turn out-paces the development of social norms. Hence, rather than proposing an entirely new platform or model of peer review, our approach here is to consider the advantages and disadvantages of existing models and innovations in social services and technologies (Table 4). We then explore ways in which such traits can be adapted, combined, and applied to build a more effective and efficient peer review system, while potentially reducing friction to its uptake.

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.3)