Green Tea and Velociraptors

Peer review is broken, but we hold the technological and social solutions to fix it. Right now.

We just submitted a monster paper on the history and present diversity of peer review practices to F1000 Research. It’s available in advance here, and soon will be open to public commenting from anyone as it undergoes formal peer review. We wrote it in a similar manner to another paper published on Open Access last year.

I don’t want to beat around the bush here. We have 33(4) top of the range authors from across a huge range of disciplines. We spent more than a year whacking and moulding this into shape, bringing in an enormous amount of research and expertise. It started as an idea during a Mozilla Sprint in Berlin with myself and Jonathan Dugan in 2016, and evolved into this huge analysis of the history of peer review, the issues with the process versus the ideal, and an in depth examination of how it could function based on modern Web technologies.

It’s worth the read. I promise.

It’s also 44 pages long. Now, I realise this can be off-putting, so I have a request. Please, bear with it. Peer review is such an important process, that we cannot be ignorant of it any more. This paper does not hold all the answers, and it is certainly not perfect. But what we have created is an enormous resource of research and thought, and from a huge diversity of perspectives. Grab a coffee, and have a read! It was made for everyone out there, and is an opportunity for you all to learn, and to give us your feedback on what you think.

Something unexpected which came out of this can be found in Section 4. We speculated about what an ‘ideal’  hybridised peer review platform would look like if we brought all the different aspects we had discussed together, in order to create an optimal system fit for a modern research environment. It’s actually not a bad idea at all, and could potentially overhaul our entire scholarly communication system, if we were to build and implement it at a large scale.

Throughout the entire manuscript, we have attempted to be balanced and evidence-informed. If this is not the case anywhere, please, let me know. I want this to be the best available resource for learning about peer review that exists.

Abstract

Peer review of research articles is a core part of our scholarly communication system. In spite of its clear importance, the status and purpose of peer review is often contested. What is its role in our modern, digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? In addition, the historical evolution and present diversity of peer review is not appreciated by the research community. Differing peer review processes developed over several hundred years in a response to the needs of journals, scholarly societies, and research communities. Today, it is viewed as an inherent and altruistic responsibility among researchers, but also a common point of contention with commercial publishers. Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of Web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review, with new concepts and models emerging at the same times as the broader development of open research. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally any new systems will demonstrate that they out-perform current models while avoiding as many of the biases of existing systems as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that, at least partially, resolves many of the technical and social issues associated with peer review, and can potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments. A major part of this relies on research communities recognizing that peer review is not a static or singular process, but a constantly evolving, and ideally improving, mechanism of quality assurance. This could greatly reduce friction to the cultural adoption of new models, and lead to a peer review process fit for a modern scholarly communication system.