Editor’s Note: Today’s guest post is by Sandra Franco Iborra, Jessica Polka, and Iratxe Puebla, members of the ASAPbio preprint review cultural norms Working Group, which has recently developed principles for a positive and constructive culture of review and commenting on preprints. Sandra is the Events Coordinator for the Danish Data Science Academy. Jessica Polka is Executive Director at ASAPbio, Iratxe Puebla is Director of Strategic Initiatives & Community at ASAPbio. 

The potential to get feedback on a preprint is a major benefit of posting one. However, when that feedback is shared publicly, it benefits not only the author, but the entire community. Nowhere has this been more obvious than in COVID-19 preprints. During the pandemic, non-specialized readers are flocking to preprints, creating an environment in which preprints can be misinterpreted or errors can be inappropriately propagated. However, rapid, public responses on the preprinted work can reduce the risk of this happening.

Take the infamous example of the “uncanny similarity” preprint. Posted in the early days of 2020, it suggested a potential link between HIV and SARS-CoV-2. As reported in STAT News, within hours of its appearance, dozens of researchers weighed in via Twitter and the bioRxiv commenting section with their concerns, which gave journalists and general readers context to help prevent the spread of misinformation. The authors subsequently decided to withdraw the paper within 48 hours of its original posting.

Most public preprint feedback is nowhere near as dramatic, but it can still be beneficial to help researchers understand the key debates in the field. Moreover, if one reader has a concern or question, it’s likely that the issue will come up again as others evaluate the paper. Addressing those issues in a public forum is to the author’s advantage: providing a response can prevent concerns from becoming deal-breakers as a reviewer passes judgment without considering the author’s point of view.

The discussion thread for this preprint clearly demonstrates the power of post publication peer review. As a member of the field, seeing agreements and disagreements is way more valuable than reading any review.#openaccess #preprint @FORsymp @ASAPbio_https://t.co/ENopT5Itv8

— Vincent Boudreau (@viboud) March 14, 2018

Despite the benefits, this kind of public dialog is still unusual for preprints, and researchers are apprehensive about giving and receiving it. Since we view constructive feedback as necessary for preprints to reach their full potential, we wanted to ensure that the nascent culture grows in the most constructive way possible. Toward that end, ASAPbio looked for ways to identify and codify productive behaviors.

FAST principles for preprint feedback

Last year, we convened a Working Group to reflect upon and develop a set of best practices for public preprint feedback. Our rationale was to provide a framework that could benefit and support authors, reviewers, and the community to engage in public and open scientific discussion of preprints, while ensuring a thriving and welcoming environment for everyone. The group worked on a set of “norms” that reflect the behaviors and culture we would like to promote to increase participation and acceptance of preprint public commenting. The result is the FAST principles for preprint feedback, a set of 14 principles clustered around four broad themes:

• Focused: assess whether our comments and feedback are targeted towards relevant and actionable parts of the preprint (e.g., the current focus of the paper or the scientific work, rather than suitability for a particular journal).
• Appropriate: ensure that before engaging in any kind of scientific discussion, we have reflected on our biases and behave with the same level of integrity as in any other scientific exchange.
• Specific: similar to reviewing a manuscript for a journal, preprint feedback should also evaluate the study’s claims against the data and clarify whether the critiques are major or just meant to tackle minor issues that don’t affect the overall findings.
• Transparent: as with any type of scientific discussion, it is key to be as open and transparent as possible, embracing any oversights and crediting everyone who participated in the work. However, we acknowledge that not everyone is comfortable signing their comments. In such cases, we provide alternative options for reviewers to disclose their background or expertise to contextualize the comments they post.

What’s unique about the FAST principles is that they acknowledge and incorporate the particularities of preprint feedback: it takes place in the open; it is journal agnostic; and the exchange can be under the scrutiny of several actors who, at the same time, can also participate in the process. As a result, the FAST principles apply not only to reviewers, but also to authors, who can have more direct and seamless communication with reviewers, and to the community, who can engage with the feedback and should ensure that the exchange takes place under a constructive and appropriate framework.

‘My journal already has reviewer guidelines, why should I care?’

We view the FAST principles as relevant to all involved in science dissemination and scientific discourse: this, of course, includes journals. Many journals already have reviewer guidelines, and there is also guidance for reviewers by organizations such as the Council of Science Editors and the Committee on Publication Ethics. We do not view the FAST principles as supplanting reviewer guidelines for individual journals, but rather as a complementary resource. Editors can use the FAST principles to check for any gaps in their existing guidelines e.g., regarding the consideration of potential biases or credit for co-reviewers. In addition, because the FAST principles apply to all involved in feedback and not only the reviewers, they can provide a foundation for journal guidance to authors on how to respond to or rebut reviewer requests, or on expectations for readers who comment on published articles. Many of those who participate in preprint feedback already review for journals, or will likely do so in the future, so we hope that encouraging positive behaviors in preprint feedback will also result in those behaviors reflected in journal reviews, and vice versa.

It is common to hear journal editors lament that it is becoming increasingly difficult to find reviewers. Preprint reviewers provide a pool of potential experts for editors to tap into. Editors can check public preprint reviews to evaluate whether the reviewer has the expertise, thoroughness, and attention to detail they expect. This can enable editors to include a wider range of voices — more junior researchers, those based in different regions, or affiliated with different types of institutions – in their reviewer pool, helping to address some of the diversity issues we currently see in journal peer review (Publons report ‘Global State of Peer Review’). To bridge the gap between preprint review and journal review, ASAPbio is coordinating the Preprint Reviewer Recruitment Network. Through this initiative researchers can submit their preprint review activity for consideration by a group of over 40 participating journals, which will evaluate the preprint reviewers for inclusion in their reviewer pools or editorial boards. In this context, the FAST principles provide a framework for journals to evaluate preprint reviews for the expectations of tone and approach that their reviewers should meet.

Several journals are also exploring how they can bring preprint reviews into their editorial process. A couple of notable examples include the affiliate journals involved with Review Commons, which have agreed to consider the reviews provided by Review Commons on papers prior to journal submission (for example posted as preprints); and the partnership between Peer Community In Registered Reports and 15 journals, where the journals will accept the peer review recommendations by Peer Community In on registered reports not yet submitted for journal publication. The FAST principles could be included in the contributor guidelines for partnerships between journals and preprint review communities to signal the expectations for ‘journal-compatible’ preprint reviews.

Be part of culture change in public preprint feedback

With a growing number of preprints, we are also seeing new initiatives around the commentary and review of preprints. We welcome experimentation in this space, as it has the potential to bring greater transparency and diversity in review activities across scholarly communication. The FAST principles are one step in what we hope will be a broader conversation toward a more positive and diverse review culture, and we discussed this topic further at a recent session at NISO Plus 2022 (the slides from the session are available). The nascent preprint feedback environment will continue to develop and the needs and expectations of the community around preprint feedback are likely to evolve. At this early stage, we all have an opportunity to contribute to positive change, as we move towards the culture of preprint feedback we’d like to see.