We’re excited to be back with another full week of posts in celebration of Peer Review Week!  

The theme of Peer Review Week 2019 is quality in the peer review process. There’s no question that researchers see peer review as essential.  At its best it helps them ensure that their work is clear, free of errors or biases, and ready to share with the wider community. However, as plenty of social media and other discussions, including the Facebook group “”Reviewer 2 Must Be Stopped” suggests, there are also plenty of examples of unhelpful — or even offensive — peer review.  

peer review week 2019 logo

So what is a high-quality peer review process? How can we define quality? Measure it? Improve it? Do different individuals and communities view quality in the same way? Is it equally important across all disciplines and publication types? And what about peer review in other research workflows, such as grant applications, conference submissions, promotion and tenure applications, and more?

Some efforts to create an instrument for the evaluation of peer review have proved effective in some disciplinary contexts, and in assessing features of peer review. The Review Quality Index, for example, was used to explore differences between author-suggested and blind review assignment. But there are undoubtedly many more questions than easy answers and, during the course of this week, we will be hearing a number of different perspectives on quality in peer review. In this post, we’d like to focus on two main elements: the practices and principles of peer review, and how both are needed to achieve improved scholarship, which ought to be, in our view, the primary goal of high-quality review.

Let’s start with the practices of peer review. In its ideal sense, peer review is the collective commitment to the community production of knowledge. It is designed to bring expertise to bear in improving scholarship. To that end, each participant in the review process has a role to play. Let’s take the classic example of peer review, a submission to a journal. In this case the author(s), editor, and reviewer(s) each make essential contributions to a successful process.

On the editorial side the process involves identifying appropriate reviewers, collecting and then sharing reviews with authors, and communicating expectations and decisions based on those reviews. Editors need to take expertise and inclusion into account in their selection of reviewers, and then to be mindful of the quality of the reviews communicated to authors. On the reviewer’s side, the process involves careful consideration of the scholarship — content, style, originality, fit for the journal, etc — and a clear communication of needed improvements, along with a recommendation for publication immediately, after some revision, or not at this time. For the author(s), the process involves reading and reflecting on reviews, determining how the reviews and any other feedback from the editor can help improve the work, and responding accordingly.

Most researchers will play at least two of these roles during their careers: as reviewers themselves, or as the person being reviewed or the author of work being reviewed. Many researchers also engage in some level of editorial work during their career. So, for many of us, simply thinking through how our work in one role advances the work of the participants in the other roles can be helpful. Each relies on the others.

Training, feedback, new processes and structures aim to help individuals fulfill each of these roles. But the key is really whether, by each of these roles functioning together, peer review achieves its intention — leading to improved scholarship. Peer review is intended to be a productive process. 

Good principles are just as important to high quality peer review as good practices. Qualities such as respect, thoughtfulness, expertise, efficiency, and transparency are essential for ensuring that reviews achieve their purpose. Let’s look at each of these in turn.

    • Respect. Writing a review that is respectful of both the research and the researcher(s) is arguably more important than anything else. A disrespectful review can be harmful to the researcher, but also to the wider community. Anyone who has been on the receiving end of a review that is on the spectrum between condescending and rude will know that, even if the reviewer’s specific feedback is helpful, it’s their tone that often carries the most weight. This may especially be the case for early career researchers or those who already feel marginalized. And the consequences for the individual and the community can be significant — from someone choosing not to publish in the specific journal in question future to abandoning their work altogether.
    • Thoughtfulness. This goes hand in hand with respect. There is some evidence that reviews are getting shorter, and it’s hard not to think that this may mean they’re also becoming less thoughtful. A thoughtful review, one where it’s clear that the reviewer has taken the time to read and consider the full work, highlight both positives and negatives, and make specific suggestions for improvement, is essential for the research community  — even if the reviewer does not recommend publication at that time.
    • Expertise. It’s important to recognize that there are many different types of expertise, and that any one person may have multiple areas of expertise. Thus reviewers need to be clear about why they are being asked to review a work, and an editor must be clear about which type of perspective a reviewer is being asked to provide. For example, our newest Scholarly Kitchen colleague, Haseeb Irfanullah, is a biologist, has served as an editor of the Bangladesh Journal of Plant Taxonomy, and now works in development, focusing on scholarly communications and in particular helping develop and support the research ecosystem in Bangladesh. It could be appropriate, therefore, for him to be asked to review in a variety of areas, as long as it’s clear which of his hats he is wearing.   
    • Efficiency. Researchers are busy and, although they are strongly motivated to act as peer reviewers, it’s unfair to expect them to do so in ways that are inefficient and time-consuming. Making the peer review process as streamlined and easy as possible for reviewers means they’ll be much more likely to agree to review again in future. And reviewers can also help increase efficiency in peer review by responding promptly to invitations to be reviewers. It’s much more helpful to send an immediate “no” than to prevaricate for weeks and then say “yes.” (See also Respect!)
    • Transparency. As we’ve written before, transparency in peer review doesn’t have to mean having an open peer review process — though it might. Rather, it’s about ensuring that your processes are clearly defined and publicly accessible. This includes the type of peer review (with as much detail as possible), number of reviewers and how they are selected, average length of review, and more. This ensures that both researchers and reviewers know what to expect during the review process. 

It’s important that both reviewers and editors/publishers are equally  committed to these principles. Reviewers who do not invest appropriate care in their reviews undermine the process.  Editors who allow poor-quality reviews to be shared with the author(s) and/or to inform their decision-making are just as responsible as the “bad” reviewers themselves. Equally, authors who do not engage productively with reviews risk missing what potential for improvement even a tough review can provide. Above all, in principle and the practices they inform, mutual dedication to the mission of high quality in peer review is vital.

“Quality” is multi-dimensional. It can be measured in a variety of ways, and from a variety of perspectives. As we have written in previous Peer Review Weeks (see below), diversity and transparency are crucial for peer review and, we would argue, essential for achieving quality by any measure.  

To find out more about previous Peer Review Weeks, we suggest starting with these introductory posts:

 You can also look in The Scholarly Kitchen Peer Review Collection, which captures posts from Peer Review Week as well as other posts of interest and relevance.

Alice Meadows

Alice Meadows

I am a Co-Founder of the MoreBrains Cooperative, a scholarly communications consultancy with a focus on open research and research infrastructure. I have many years experience of both scholarly publishing (including at Blackwell Publishing and Wiley) and research infrastructure (at ORCID and, most recently, NISO, where I was Director of Community Engagement). I’m actively involved in the information community, and served as SSP President in 2021-22. I was honored to receive the SSP Distinguished Service Award in 2018, the ALPSP Award for Contribution to Scholarly Publishing in 2016, and the ISMTE Recognition Award in 2013. I’m passionate about improving trust in scholarly communications, and about addressing inequities in our community (and beyond!). Note: The opinions expressed here are my own

Karin Wulf

Karin Wulf

Karin Wulf is the Beatrice and Julio Mario Santo Domingo Director and Librarian at the John Carter Brown Library and Professor of History, Brown University. She is a historian with a research specialty in family, gender and politics in eighteenth-century British America and has experience in non-profit humanities publishing.

Discussion

4 Thoughts on "Quality is Multi-Dimensional: How Many Ways Can You Define Quality in Peer Review?"

With increasing attention being paid to post-publication review we should consider this when considering quality and the purposes of the review. The targets of a traditional review is the editor and the author. A key target of the post-publication review is the reader. Why should they read the paper and which parts are of particular merit and to which they should pay particular attention. Increasingly readers are using post-publication review journals such as AMEE’s second journal MedEdPublish. Articles can attract more than 15,000 views.

You rightly point to the need for efficiency in pre-publication peer review: “Researchers are busy and, although they are strongly motivated to act as peer reviewers, it’s unfair to expect them to do so in ways that are inefficient and time-consuming.” Here you appear to be asking editors to address their own processes so that reviewers will “be much more likely to agree to review again in future.” But editors should also be addressing agencies such as the NCBI whose PubMed processes greatly ease reviewing tasks.
.
For example, reviewers are confronted with long lists of references. Usually some need to be checked. A quick way to do this is first to visit corresponding PubMed abstracts. Should abstracts be insufficient, PubMed often provides quick links to actual papers. Furthermore, as discussed previously in The Scholarly Kitchen, from 2013 to 2018 an intermediate option was available that could precede, and hence possibly eliminate, the labour of visiting full papers. Having expert post-publication reviews a click away in the now defunct PubMed Commons (PMC) has been discussed previously in The Scholarly Kitchen.
.
PMC was not a free-for-all. It required that contributors have demonstrable qualifications (unlike PubPeer) and the names of well-known experts were often seen. Sadly, it seems that without any input from journal editors, PMC was deemed by the NCBI a failed “experiment.”

Thank you for your interesting article. I’ve also been interested in questions of quality in peer review, and have tried to evaluate it from the author’s perspective. Last summer, my student and I conducted an author survey across a large cross-section of peer-reviewed conferences in a single field.
Authors were asked to ranked review quality along several dimensions, including its helpfulness, fairness, and understanding of the paper. These three aspects appear to be very strongly correlated to each other, and to a lesser degree, also correlated to the length of the review. Another confounding factor that appears to be associated with satisfaction with reviews is conference reputation.
You can find an initial analysis of the results here: http://sysconf.review/survey/. (A more detailed analysis is also forthcoming).

Comments are closed.