22: A Stack Exchange/Overflow-style model

July 31, 2018protohedgehogPeer Review

This is adapted from our recent paper in F1000 Research, entitled “A multi-disciplinary perspective on emergent and future innovations in peer review.” Due to its rather monstrous length, I’ll be posting chunks of the text here in sequence over the next few weeks to help disseminate it in more easily digestible bites. Enjoy!

This section outlines what would a model of Stack Exchange-style peer review could look like. Previous parts in this series:

——————————————————————————

Stack Exchange (stackexchange.com) is a collective intelligence system comprising multiple individual question and answer sites, many of which are already geared towards particular research communities, including maths and physics. The most popular site within Stack Exchange is Stack Overflow, a community of software developers and a place where professionals exchange problems, ideas, and solutions. Stack Exchange works by having users publish a specific problem or question, and then others contribute to a discussion on that issue. This format is considered to be a form of dynamic publishing by some (Heller et al., 2014). The appeal of Stack Exchange is that threaded discussions are often brief, concise, and geared towards solutions, all in a typical Web forum format. Highly regarded answers are positioned towards the top of threads, with others concatenated beneath. Like the Amazon model of weighted ratings, voting in Stack Exchange is more of a process that controls relative visibility. The result is a library of topical questions with high quality discussion threads and answers, developed by capturing the long tail of knowledge from communities of experts. The main distinction between this and scholarly publishing is that new material rarely is the focus of discussion threads. However, the ultimate goal remains the same: to improve knowledge and understanding of a particular issue. As such, Stack Exchange is about creating self-governing communities and a public, collaborative knowledge exchange forum based on software (Begel et al., 2013).

Existing Overflow-style platforms.

Some subject-specific platforms for research communities already exist that are similar to or based on Stack Exchange technology. These include BioStars (biostars.org), a rapidly growing Bioinformatics resource, the use of which has contributed to the completion of traditional peer reviewed publications (Parnell et al., 2011). Another is PhysicsOverflow, an open platform for real-time discussions between the physics community combined with an open peer review system (Pallavi Sudhir & Knöpfel, 2015) (physicsoverflow.org/). PhysicsOverflow forms the counterpart forum to MathOverflow (Tausczik et al., 2014) (https://mathoverflow.net/), with both containing a graduate-level question and answer forum, and an open problems section for collaboration on research issues. Both have a reviews section to complement formal journal-led peer review, where peers can submit preprints (e.g., from arXiv) for public peer evaluation, considered by most to be an “arXiv-2.0”. Responses are divided into reviews and comments, and given a score based on votes for originality and accuracy. Similar to Reddit, there are moderators but these are democratically elected by the community itself. Motivation for engaging with these platforms comes from a personal desire to assist colleagues, progress research, and receive recognition for it (Kubátová et al., 2012) – the same as that for peer review for many. Together, both have created successful open community-led collaboration and discussion platforms for their research disciplines.

Community-granted reputation and prestige.

One of the key features of Stack Exchange is that it has an inbuilt community-based reputation system, karma, similar to that for Reddit. Identified peers rate or endorse the contributions of others and can indicate whether those contributions are positive (useful or informative) or negative. Karma provides a point-based reputation system for individuals, based not just on the quantity of engagement with the platform and its peers alone, but also on the quality and relevance of those engagements, as assessed by the wider engaging community (stackoverflow.com/help/whats-reputation). Peers have their status and moderation privileges within the platform upgraded as they gain reputation. Such automated privilege administration provides a strong social incentive for constructively engaging within the community. Furthermore, peers who asked the original questions mark answers considered to be the most correct, thereby acknowledging the most significant contributions while providing a stamp of trustworthiness. This has the additional consequence of reducing the strain of evaluation and information overload for other peers by facilitating more rapid decision making, a behavior based on simple cognitive heuristics (e.g., social influences such as the “bandwagon effect” and position bias) (Burghardt et al., 2017). Threads can also be closed once questions have been answered sufficiently, based on a community decision, which enables maximum gain of potential karma points. This terminates further contribution but ensures that the knowledge is captured for future needs.

Karma and reputation can thus be achieved and incentivized by building and contributing to a growing community and providing knowledgeable and comprehensible answers on a specific topic. Within this system, reputation points are distributed based on social activities that are akin to peer review, such as answering questions, giving advice, providing feedback, sharing data, and generally improving the quality of work in the open. The points directly reflect an individual’s contribution to that specific research community. Such processes ultimately have a very low barrier to entry, but also expose peer review to potential gamification through integration with a reputation engine, a social bias which proliferates through any technoculture (Belojevic et al., 2014).

Badge acquisition on Stack Overflow.

An additional important feature of Stack Overflow is the acquisition of merit badges, which provide public stamps of group affiliation, experience, authority, identity and goal setting (Halavais et al., 2014). These badges define a way of locally and qualitatively differentiating between peers, and also symbolize motivational learning targets to achieve (Rughiniş & Matei, 2013). Stack Overflow also has a system of tag badges to attribute subject-level expertise, awarded once a peer achieves a certain voting score. Together, these features open up a novel reputation system beyond traditional measurements based on publications and citations, that can also be used as an indication of expertise transferable beyond the platform itself. As such, a Stack Exchange model can increase the mobility of researchers who contribute in non-conventional ways (e.g., through software, code, teaching, data, art, materials) and are based at non-academic institutes. There is substantial scope in creating a reputation platform that goes beyond traditional measurements to include social network influence and open peer-to-peer engagement. Ultimately, this model can potentially transform the diversity of contributors to professional research and level the playing field for all types of formal contribution.

Tennant JP, Dugan JM, Graziotin D et al. A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; referees: 2 approved]. F1000Research 2017, 6:1151 (doi: 10.12688/f1000research.12037.3)

Green Tea and Velociraptors