Editor’s Note: Today’s post is by Daniel S. Katz and Arfon M. Smith. Dan leads the FORCE11 Software Citation Implementation Working Group’s journals task force, and is the Chief Scientist at NCSA and Research Associate Professor in Computer Science, Electrical and Computer Engineering, and the School of Information Sciences at the University of Illinois. Arfon is the Editor-in-Chief of the Journal of Open Source Software, and Staff Product Manager for Data at GitHub.

As the scientific world changes and makes an effort to become more open and more collaborative, new models for scientific peer review are being examined and trialed, including open peer review, as implemented by F1000Research and others, and open and collaborative peer review, as implemented by the journal we edit, The Journal of Open Source Software (JOSS). The intent of this post is not to argue the pros and cons of open peer review, but rather, to look at the changing role of the editor from single- or double-anonymized peer review to open peer review and open collaborative peer review.

To some extent, this reflects changes in how journals interact with the internet in general. The way that most journals use the internet hasn’t changed in the last 20 years, but the internet itself has changed, in particular making it easier to both produce and interact with content. The model we created at JOSS in 2016, and that others have piloted, reacts to these changes, making it easier for authors to submit and for reviewers to interact with the submission. But this also has dangers, as seen in unmoderated comment forums in much of the media, which often quickly degenerate to non-productive discussions at best (cf Godwin’s Law.)

Key differences in the experiences of the reviewers and author are:
Anonymizing – do the reviewers know the author’s identity, does the author know the reviewers’ identities, and to a lesser extent, do the reviewers know one another’s identities?
Purpose – to what extent do reviewers judge the author’s work towards making a recommendation to the editor about it, to what extent do they constructively suggest general changes, and to what extent do they constructively suggest specific changes that can be incorporated into the work under review (known as pull requests in open source)?
Communication granularity – what is the granularity of communication between the reviewer(s), editor, and author? Is there a fixed review period where all reviewers work, then provide feedback, and a fixed response/rebuttal period for the author? Or does the reviewer provide feedback in pieces, so that the author can respond while the review is ongoing?

How authors and reviewers experience these different peer review models is important. But crucially, editors also have a different role with different expectations and experiences in these different models of peer review.

In a single- or double-anonymized peer review system, as implemented in traditional journals, the editor selects reviewers, acting as guarantor of their suitability, and then has two functions: intermediary and judge. As an intermediary, the editor translates and summarizes the reviews for the author, so that the author can understand the importance of individual review comments, as well as moderating all communication between the reviews and authors to ensure communication is reasonable and professional. As a judge, the editor interprets the reviewer comments (and perhaps the authors response to them) and decides whether the work should be accepted, modified, or rejected.

In open peer-review, as implemented in F1000Research, some of the editor’s intermediary/moderator function is no longer necessary, as reviewers and authors know each others’ identities and are communicating publicly, which may lead to more professional exchanges, though it also may demotivate reviewers from making necessary but unpopular or controversial comments, particularly when the authors are in a position of power or authority, which adds another aspect to the selection of appropriate reviewers. The editor may still have the role of judge.

In a collaborative open peer-review model, the editor has the same important role in reviewer selection, but is no longer an intermediary. The editor may still need to moderate or dampen frequent and interactive discussions between the reviewers and author to ensure that they are professional and constructive, as well as to guide discussion to consider important factors and to provide helpful examples to further the discussion.

The editor of the open collaborative review process thus needs many of the same skills as the editor in the anonymized peer-review process (selecting appropriate reviewers, though no longer needing to guarantee their suitability; providing guidance to them on review criteria, appropriate comments, and required elements of a review; understanding their reviews; being able to translate them for the author if needed; and summarizing multiple reviews into a single decision and set of guidance for the author). The editor needs additional skills to facilitate, guide, and moderate the interactive discussion. In some sense, the editor of the open collaborative review process is simply one participant of the discussion, and thus needs to have a softer touch and work more through influence and less through control.

At JOSS, where our goal is to improve submissions through the review process to the point that they can ultimately be accepted, an editor invests significant effort in facilitating the constructive discussion between the reviewers and authors with frequent communication among all parties. While the editor does have ‘hard’ power, i.e., is able to make decisions on behalf of the journal, it is rarely used, and instead ‘soft power’ tactics such as encouragement are often all that is required to effectively steer the review process. In addition, for the review process to be productive, the editor needs to help the communication, which means not blocking it, being responsive to any potential problems before they become serious, and steering it to a consensus decision. This means that the editor needs to stay tightly involved in the review process, and cannot just check on reviews once a week or less frequently; ideally the editor will be as responsive to issues in the review as they are to their other email/communications work. This does potentially lead to scalability issues for the editors, though in JOSS we now have about 70 editors, each of whom handles 3 to 4 papers at a time during a renewable two-year term, and this appears to be sustainable.

We believe that this open collaborative model of editing is an important opportunity for authors, reviewers, and editors to work together towards strong publications, and that editors are capable of supporting it. Unsolicited public feedback from authors has confirmed that this mode of working is a positive and novel experience, although we haven’t systematically surveyed authors.

*I hope all other publishers are paying attention. Focus was on the resource output more than the article, where normally journals care more about the type setting of the bibliography than about the actual research. Highly recommended! (Egon Willinghagen)
*I loved this transparent, professional and friendly review process! (Fabian Hofmann)
*I was really impressed with your reviewing system which is very smart! (Sacha Schutz)
*We appreciate what a positive and helpful experience this was. (Jeffrey Barrick)
*Thank you everyone very much for this awesome review process and publication. (Edward Rusu)