It’s that time of the year again — Peer Review Week 2024 starts on Monday and, as usual, Jasmine Wallace, Karin Wulf, and I (Alice Meadows)have been putting together a week of posts based on this year’s theme of innovation and technology in peer review. We’re delighted to once again be kicking things off with some thoughts from the Scholarly Kitchen Chefs in answer to our question for them:

What is, or would be, the most valuable innovation in peer review for your community?

We hope you’ll enjoy their responses — and please share your own in the comments!

Avi Staiman

I recently met with a startup working on an AI solution for peer review for a quick demo of their product. When we got to talking towards the end, they shared with me that they actually could do a lot more with AI than they were currently, but the publishers had pumped the brakes.

And I want to ask one simple question: Why?

If we hypothesize (based on gut?) that AI systems can’t give us the same level of constructive feedback as people, then shouldn’t we be subjecting this theory to rigorous academic studies? Are we willing to see if AI can, in fact, outperform human review?

Some early studies give us reason to think that AI can provide feedback and comments that are much more detailed and clearer to implement than human review, while other studies show us that humans often can’t tell the difference.

So let’s clarify the question even further: Do we prefer human review over AI even if we find it to be less accurate, more vague and harder to implement?

Another argument might take issue with handing over the keys to determining which science is more valuable than others to machines. I would agree that we should keep the final decision-making process of what we, as a scientific community, determine to be innovative, novel, and significant, to the realm of humans. However, this is only a very small portion of what reviewers actually do and doesn’t need to preclude the use of AI for the remainder of the review.

So I’d like to toss out an idea for what an innovative AI workflow for peer review might look like.

Step 1- Human desk review

Step 2- Automated AI review feedback to help authors improve their research (researchers generally know when to disregard bad suggestions the same way they do with human review)

Step 3- Author revisions

Step 4- Reviewers read through the revised research to make a determination regarding innovation, novelty, and significance. They then make a recommendation for acceptance, rejection, or sending it back for further review (which can also be AI-assisted).

Hong Zhou

Innovation in peer review for the scholarly community would likely involve leveraging advanced digital transformation technologies such as AI, Big data, and cloud to handle a growing number of submissions effectively and rapidly, without compromising on the quality of their processes today.

The most valuable innovation in peer review, now and in the near future, is to automatically detect integrity issues by leveraging AI and global publication information. For example:

• Detecting AI-generated content, which is one of the most demanding and challenging topics in era of Generatvie AI (GenAI)
• Automatically detecting manipulated images and unusual patterns in datasets
• Verifying accuracy of citations and detecting any integrity issues in references such as sneaky references and citation stacking

However, AI also brings challenges to research integrity especially in the era of GenAI, like the potential increases in plagiarism, image manipulation, and paper mills. Its ability to create human-like text and alter images can make fraud harder to spot. As deceptive tactics grow, so will the strategies to detect and stop them.

Another valuable innovation in peer review in the future would be to leverage GenAI to automatically conduct initial reviews to assist human editors/reviewers.  For example:

• Identifying relevancy and suggesting relevant prior research,
• Identifying potential biases, flagging areas needing further scrutiny, and measuring novelty, quality, and readability.
• Drafting review reports & summarizing and evaluating reports

This would streamline the review process and ensure a more thorough evaluation, but would need careful management by humans, who would still be responsible for making final decisions. Actually, research shows that between 6% and 17% of text submitted as peer reviews to many AI conferences could have been substantially modified by LLMs. That number is going to increase as GenAI is becoming more powerful and has gained more domain-specific knowledge. There might be some errors, but even human editors/reviewers make some subjective/wrong decisions

Karin Wulf

I’m writing here solely now as a researcher and author, and not as someone on the other side of the screen/desk who manages publication processes and intake. In my history and humanities world, we wondered more than a decade ago what would happen as a consequence of society-published journals being incorporated into the bigger publishers’ offerings. I remember worrying about retaining brand identity on the one hand, and on the other how copyediting, so essential for close word work fields, might suffer from being deployed across disciplines. I don’t remember talking as much about how the human scale of humanities fields might be affected by the online systems for processing submissions and peer review, but as an author and reviewer this now feels profound. I get automatic, rather than personalized solicitations now to do review work – even from really quite small and specialized publications, almost always nonprofit publications associated with societies and managed by larger publishers – and then if I accept, I get launched into a system where first I must accept some conditions of use. Not unusually, I get a direction to categorize my assessment using a check box, and to post my extended work within modestly-sized text boxes. I work around this, as I ponder whether and how this facilitates my agreement to read and respond substantively to 8-10,000 words on an early modern topic. Sure, my reaction to the anonymized structures can be read as an artifact of nostalgia, or even a resistance to the efficiencies of these now-ubiquitous, ever in refinement mode, now stalked by the promise of AI, systems. But I also think these anonymized systems can create more distance in research communities, which despite historical bias and discrimination, more informal and personalized structures can alleviate. So what would be a useful innovation? Low-cost ways for small nonprofit publications in under-resourced fields to adapt peer review systems that could not just simulate, but actually facilitate human interaction and connection around research would be useful.

Tim Vines

The most valuable innovation in peer review would – in my opinion – be to scrap ‘number of manuscripts submitted’ as a journal Key Performance Indicator (KPI). Bringing in ever more submissions may seem like the journal is winning at being an attractive destination for researchers, but a monomaniacal focus on boosting submissions has damaging side-effects:

1) There are a lot of poor quality manuscripts out there. Boosting submissions inevitably means bringing more bad articles through your door. If the journal wants to avoid publishing low quality or fraudulent research, the journal peer review system must now identify and reject all of these bad articles. Since peer review is fallible and only a percentage of these bad articles will get spotted, getting more through the door means that more bad articles will get published. Each of these bad articles adds to the risk of mass retractions or other embarrassing editorial debacles. It’s far better for the journal that these articles never get submitted in the first place.

2) Having ’number of manuscripts submitted’ as a KPI that must keep rising also blocks multiple editorial innovations that would improve academic publishing. Anything that might deter authors from submitting their manuscripts – such as open data requirements or submission fees – is nixed on the grounds that submissions might decrease. The question that apparently never gets asked: if the authors are not willing to share their datasets or pay a modest sum to submit their article, why do we want their article?

3) Getting in as many articles as possible overburdens the Editorial Office and the peer review system, which drives up the cost of publishing and exhausts the editor and reviewer community.

Journals would be far better off focusing on getting in a higher number of publishable submissions rather than more articles overall. Have clear, strict editorial criteria that force authors to demonstrate their commitment to quality research before they click ’submit’ would lead to fewer submissions, but a much higher proportion of those would be good articles.

Lisa Hinchliffe

Over the course of my career, I’ve seen peer review from every faculty angle — author, reviewer, and editor. I have had very positive experiences and very frustrating experiences — also across all of those many roles. What I’m most excited about right now is the “Peer Review Terminology Standardization” effort (disclosure: I am a member of the NISO Standing Committee overseeing this work). What I’ve come to realize over the years is that most people have a particular experience with a peer review approach (e.g., double anonymized) and then come to think that particular “flavor” of peer review is peer review and that everything else that others might call peer review is deficient, lacks rigor, etc. This is particularly confusing to students who, in the courses of their studies, get introduced to peer review repeatedly with no one sorting out the variability and pros/cons of the different models. Additionally, it can be very unclear what model one’s work will be subjected to upon submission and this can create a great deal of stress, particularly for early career researchers. The terminology standardization process introduces a vocabulary that is not perfect, but is useful in establishing recommended language for labeling the different approaches to peer review. As the use of the terminology spreads across the publishing industry (thank you early adopters!), we will save time and decrease confusion by being able to better understand at a glance the mechanisms in place.

Roohi Ghosh

For me, the most valuable innovation would be when a reviewer is able to spend time on their core role, to improve the scientific output with expert inputs rather than spending time on routine tasks like detecting AI generated content, plagiarism, and other checks that can be easily automated. Peer review as a system traditionally was set up to bring in this expertise and to spot flaws in logic and improve the value of research. I hope that with the help of AI we can return to that goal.

AI can also help facilitate the integration of metadata standards within the peer review process, ensuring both authors and reviewers get credit for the work they are doing. By tagging each stage of the review process with metadata, we could provide more transparency to authors, helping them track the status of their submissions and understand decision-making processes. This would promote equity, especially for English as an additional language authors, and foster greater trust in the peer review process.

Going forward, I also hope that peer review itself becomes more inclusive. Imagine a system where review comments can be translated in the author’s native language so that they find it easier to understand and incorporate the comments. Similarly, can responses to the reviewer also be written in the local language and translated back to the reviewer’s own language? This will reduce language barriers and allow for more reviewers to participate in the review process. Perhaps this vision sounds utopian, but with advancements in AI and language technologies, I believe it is not out of reach.

Haseeb Irfanullah

When it comes to scholarly communication, I wear several hats. Let me start off this post as an author. Since Peer Review Week (PRW) 2023, my co-authors and I submitted several articles to different Scopus-indexed social science journals. We faced three major challenges: 1) Editors taking a long time to decide if the manuscript would go out to reviewers. On one occasion they desk rejected our manuscript after almost three months. 2) Peer reviewers not accepting journals’ invitations to review. Our highest number of non-responding reviewers was 10 over a couple of weeks, as told to us by the editorial office. And, 3) Peer reviewers not responding to editors despite promising to review. The longest silence one manuscript faced was almost six months. All these are too much for one year!

As a journal editor, I can vouch for the last two situations. But, as a reviewer, I’m with many other peers on understanding why a review invitation can’t be entertained, given the many other personal and professional priorities. Frankly, the list of points that interest me to review a paper is getting shorter day by day. One of the disappointing aspects of being an editor and a peer reviewer might be, if you reject a manuscript before or after a review, it may still find its way into another journal and get published. So, we are not quite the “Guardians of the Publishing Galaxy” as we think we might be. On the other extreme of the peer reviewers’ interest spectrum, as an editor, I’ve seen authors declining my request for review, even though they published a paper on a similar topic just three months back in our journal! For many, peer review is not a ‘good karma’, after all.

A survey conducted during the PRW 2022 showed why the changing peer-review landscape needs AI to do more. But, the surge of AI-based innovations we now see to fight paper mills, is far behind in the journal review arena.

During my 26 years as an author of journal articles, I have interacted with many fantastic reviewers, improving my manuscripts superbly. Still, if I wear all the three hats above for a moment, I strongly believe the (near) future is of 100% AI-based review. Before it happens in a few years’ time, we need to redefine the role of human reviewers by ‘only’ asking them to do things that AI isn’t capable of doing effectively, for now.

Last year, I wrote a piece on five stages to transition from 100% human review to 100% AI-based review. By the time we celebrate the PRW 2025, I would really love to see we are well into the Third Stage: “…we train AI to assess the quality of human review reports and add complementary notes.” Or, even at the Fourth Stage “… journals engage AI as one of the reviewers of a manuscript, and the authors respond to both human and AI reviewers.”

Alice Meadows

An innovation that I’d really like to see is one that’s already well within our grasp — in theory at least. The unpaid labor of peer review is one component of the recent antitrust lawsuit brought against Elsevier, Springer Nature, Taylor and Francis, Sage, Wiley, Wolters Kluwer, and STM by UCLA researcher, Lucina Uddin (covered this week in the Kitchen). I’m not going to get into the rights and wrongs of that case, but the lack of recognition for peer review has been a known issue for many years. We’re often told that recognition is challenging because it’s not possible to track an individual’s peer review activities and/or because peer review is often anonymous. However, we have the building blocks we need to address the first issue: adding peer reviews (anonymized or not) to ORCID records has been possible for nearly 10 years now, and adding peer review service more generally (for example, as a peer reviewer or associate editor) was introduced in 2018. With regard to the second issue, while it will probably (rightly) never be universal, open peer review — including of preprints — is now more established, especially in some communities and disciplines. Peer review reports can then be assigned a DOI and potentially become objects of evaluation in their own right. After all, a really constructive, well-written peer review can constitute a significant contribution to a piece of research and should be recognized as such.

Unfortunately, as is so often the case, it’s the socio-cultural change required to improve recognition, rather than the technology, that is the challenge. But I remain hopeful that funders, institutions, and publishers will (hopefully sooner rather than later) use the tools at their disposal to enable better recognition for the invaluable peer review service their researchers are providing.