Editor’s Note: Today’s post is by Daniel S. Katz and Arfon M. Smith. Dan leads the FORCE11 Software Citation Implementation Working Group’s journals task force, and is the Chief Scientist at NCSA and Research Associate Professor in Computer Science, Electrical and Computer Engineering, and the School of Information Sciences at the University of Illinois. Arfon is the Editor-in-Chief of the Journal of Open Source Software, and Staff Product Manager for Data at GitHub.

As the scientific world changes and makes an effort to become more open and more collaborative, new models for scientific peer review are being examined and trialed, including open peer review, as implemented by F1000Research and others, and open and collaborative peer review, as implemented by the journal we edit, The Journal of Open Source Software (JOSS). The intent of this post is not to argue the pros and cons of open peer review, but rather, to look at the changing role of the editor from single- or double-anonymized peer review to open peer review and open collaborative peer review.

person editing document

To some extent, this reflects changes in how journals interact with the internet in general. The way that most journals use the internet hasn’t changed in the last 20 years, but the internet itself has changed, in particular making it easier to both produce and interact with content. The model we created at JOSS in 2016, and that others have piloted, reacts to these changes, making it easier for authors to submit and for reviewers to interact with the submission. But this also has dangers, as seen in unmoderated comment forums in much of the media, which often quickly degenerate to non-productive discussions at best (cf Godwin’s Law.)

Key differences in the experiences of the reviewers and author are:
Anonymizing – do the reviewers know the author’s identity, does the author know the reviewers’ identities, and to a lesser extent, do the reviewers know one another’s identities?
Purpose – to what extent do reviewers judge the author’s work towards making a recommendation to the editor about it, to what extent do they constructively suggest general changes, and to what extent do they constructively suggest specific changes that can be incorporated into the work under review (known as pull requests in open source)?
Communication granularity – what is the granularity of communication between the reviewer(s), editor, and author? Is there a fixed review period where all reviewers work, then provide feedback, and a fixed response/rebuttal period for the author? Or does the reviewer provide feedback in pieces, so that the author can respond while the review is ongoing?

How authors and reviewers experience these different peer review models is important. But crucially, editors also have a different role with different expectations and experiences in these different models of peer review.

In a single- or double-anonymized peer review system, as implemented in traditional journals, the editor selects reviewers, acting as guarantor of their suitability, and then has two functions: intermediary and judge. As an intermediary, the editor translates and summarizes the reviews for the author, so that the author can understand the importance of individual review comments, as well as moderating all communication between the reviews and authors to ensure communication is reasonable and professional. As a judge, the editor interprets the reviewer comments (and perhaps the authors response to them) and decides whether the work should be accepted, modified, or rejected.

In open peer-review, as implemented in F1000Research, some of the editor’s intermediary/moderator function is no longer necessary, as reviewers and authors know each others’ identities and are communicating publicly, which may lead to more professional exchanges, though it also may demotivate reviewers from making necessary but unpopular or controversial comments, particularly when the authors are in a position of power or authority, which adds another aspect to the selection of appropriate reviewers. The editor may still have the role of judge.

In a collaborative open peer-review model, the editor has the same important role in reviewer selection, but is no longer an intermediary. The editor may still need to moderate or dampen frequent and interactive discussions between the reviewers and author to ensure that they are professional and constructive, as well as to guide discussion to consider important factors and to provide helpful examples to further the discussion.

The editor of the open collaborative review process thus needs many of the same skills as the editor in the anonymized peer-review process (selecting appropriate reviewers, though no longer needing to guarantee their suitability; providing guidance to them on review criteria, appropriate comments, and required elements of a review; understanding their reviews; being able to translate them for the author if needed; and summarizing multiple reviews into a single decision and set of guidance for the author). The editor needs additional skills to facilitate, guide, and moderate the interactive discussion. In some sense, the editor of the open collaborative review process is simply one participant of the discussion, and thus needs to have a softer touch and work more through influence and less through control.

At JOSS, where our goal is to improve submissions through the review process to the point that they can ultimately be accepted, an editor invests significant effort in facilitating the constructive discussion between the reviewers and authors with frequent communication among all parties. While the editor does have ‘hard’ power, i.e., is able to make decisions on behalf of the journal, it is rarely used, and instead ‘soft power’ tactics such as encouragement are often all that is required to effectively steer the review process. In addition, for the review process to be productive, the editor needs to help the communication, which means not blocking it, being responsive to any potential problems before they become serious, and steering it to a consensus decision. This means that the editor needs to stay tightly involved in the review process, and cannot just check on reviews once a week or less frequently; ideally the editor will be as responsive to issues in the review as they are to their other email/communications work. This does potentially lead to scalability issues for the editors, though in JOSS we now have about 70 editors, each of whom handles 3 to 4 papers at a time during a renewable two-year term, and this appears to be sustainable.

We believe that this open collaborative model of editing is an important opportunity for authors, reviewers, and editors to work together towards strong publications, and that editors are capable of supporting it. Unsolicited public feedback from authors has confirmed that this mode of working is a positive and novel experience, although we haven’t systematically surveyed authors.

*I hope all other publishers are paying attention. Focus was on the resource output more than the article, where normally journals care more about the type setting of the bibliography than about the actual research. Highly recommended! (Egon Willinghagen)
*I loved this transparent, professional and friendly review process! (Fabian Hofmann)
*I was really impressed with your reviewing system which is very smart! (Sacha Schutz)
*We appreciate what a positive and helpful experience this was. (Jeffrey Barrick)
*Thank you everyone very much for this awesome review process and publication. (Edward Rusu)

Daniel S. Katz

Daniel Katz is the Chief Scientist at National Center for Supercomputing Applications (NCSA) and Research Associate Professor in Computer Science, Electrical and Computer Engineering, and the School of Information Sciences, at the University of Illinois Urbana-Champaign. He is working to make research software more sustainable by encouraging scholarly incentives and academic career paths for those who work on it, policy changes for its recognition and support, and best practices to make it easier to develop and maintain. He is an Associate-Editor-in-Chief at the Journal of Open Source Software, leads the FORCE11 Software Citation Implementation Working Group's journals task force, and is the Steering Committee Chair for the Research Software Alliance (ReSA).

Arfon M. Smith

Arfon M. Smith is the Editor-in-Chief of the Journal of Open Source Software, and Staff Product Manager for Data at GitHub. Previously he was the founding Head of the Data Science Mission Office at Space Telescope Science Institute in Baltimore – the home of the Hubble Space Telescope and the MAST NASA archive. He is interested in broadening the range of creditable activities in modern research, especially recognizing open source software contributions. 


1 Thought on "Guest Post — The Evolving Role of Scientific Editing"

Open peer-review; interactive peer-review are all important and evolving approaches to assessing the quality of a manuscript. And although transparency should always be an important goal – the ultimate question is does this lead to more accurate / better manuscripts being published. Given the huge increase in the scientific literature and general lack of payment for peer-reviewers, any system that requires more time of peer-reviewers is going to be a huge challenge – less so for the more influential journals. Social media has created another challenge – sadly, it has been reported to me that some authors have taken to social media to criticize peer-reviewers for their review, further reducing the likelihood that peer-reviewers will participate in open-review. And ultimately someone – usually the Editor in Chief or that person’s designee must make a decision about whether to publish a manuscript. That does not change regardless of the approach to peer-review.

Comments are closed.