We’re excited to be back with another full week of posts in celebration of Peer Review Week!
The theme of Peer Review Week 2019 is quality in the peer review process. There’s no question that researchers see peer review as essential. At its best it helps them ensure that their work is clear, free of errors or biases, and ready to share with the wider community. However, as plenty of social media and other discussions, including the Facebook group “”Reviewer 2 Must Be Stopped” suggests, there are also plenty of examples of unhelpful — or even offensive — peer review.
So what is a high-quality peer review process? How can we define quality? Measure it? Improve it? Do different individuals and communities view quality in the same way? Is it equally important across all disciplines and publication types? And what about peer review in other research workflows, such as grant applications, conference submissions, promotion and tenure applications, and more?
Some efforts to create an instrument for the evaluation of peer review have proved effective in some disciplinary contexts, and in assessing features of peer review. The Review Quality Index, for example, was used to explore differences between author-suggested and blind review assignment. But there are undoubtedly many more questions than easy answers and, during the course of this week, we will be hearing a number of different perspectives on quality in peer review. In this post, we’d like to focus on two main elements: the practices and principles of peer review, and how both are needed to achieve improved scholarship, which ought to be, in our view, the primary goal of high-quality review.
Let’s start with the practices of peer review. In its ideal sense, peer review is the collective commitment to the community production of knowledge. It is designed to bring expertise to bear in improving scholarship. To that end, each participant in the review process has a role to play. Let’s take the classic example of peer review, a submission to a journal. In this case the author(s), editor, and reviewer(s) each make essential contributions to a successful process.
On the editorial side the process involves identifying appropriate reviewers, collecting and then sharing reviews with authors, and communicating expectations and decisions based on those reviews. Editors need to take expertise and inclusion into account in their selection of reviewers, and then to be mindful of the quality of the reviews communicated to authors. On the reviewer’s side, the process involves careful consideration of the scholarship — content, style, originality, fit for the journal, etc — and a clear communication of needed improvements, along with a recommendation for publication immediately, after some revision, or not at this time. For the author(s), the process involves reading and reflecting on reviews, determining how the reviews and any other feedback from the editor can help improve the work, and responding accordingly.
Most researchers will play at least two of these roles during their careers: as reviewers themselves, or as the person being reviewed or the author of work being reviewed. Many researchers also engage in some level of editorial work during their career. So, for many of us, simply thinking through how our work in one role advances the work of the participants in the other roles can be helpful. Each relies on the others.
Training, feedback, new processes and structures aim to help individuals fulfill each of these roles. But the key is really whether, by each of these roles functioning together, peer review achieves its intention — leading to improved scholarship. Peer review is intended to be a productive process.
Good principles are just as important to high quality peer review as good practices. Qualities such as respect, thoughtfulness, expertise, efficiency, and transparency are essential for ensuring that reviews achieve their purpose. Let’s look at each of these in turn.
- Respect. Writing a review that is respectful of both the research and the researcher(s) is arguably more important than anything else. A disrespectful review can be harmful to the researcher, but also to the wider community. Anyone who has been on the receiving end of a review that is on the spectrum between condescending and rude will know that, even if the reviewer’s specific feedback is helpful, it’s their tone that often carries the most weight. This may especially be the case for early career researchers or those who already feel marginalized. And the consequences for the individual and the community can be significant — from someone choosing not to publish in the specific journal in question future to abandoning their work altogether.
- Thoughtfulness. This goes hand in hand with respect. There is some evidence that reviews are getting shorter, and it’s hard not to think that this may mean they’re also becoming less thoughtful. A thoughtful review, one where it’s clear that the reviewer has taken the time to read and consider the full work, highlight both positives and negatives, and make specific suggestions for improvement, is essential for the research community — even if the reviewer does not recommend publication at that time.
- Expertise. It’s important to recognize that there are many different types of expertise, and that any one person may have multiple areas of expertise. Thus reviewers need to be clear about why they are being asked to review a work, and an editor must be clear about which type of perspective a reviewer is being asked to provide. For example, our newest Scholarly Kitchen colleague, Haseeb Irfanullah, is a biologist, has served as an editor of the Bangladesh Journal of Plant Taxonomy, and now works in development, focusing on scholarly communications and in particular helping develop and support the research ecosystem in Bangladesh. It could be appropriate, therefore, for him to be asked to review in a variety of areas, as long as it’s clear which of his hats he is wearing.
- Efficiency. Researchers are busy and, although they are strongly motivated to act as peer reviewers, it’s unfair to expect them to do so in ways that are inefficient and time-consuming. Making the peer review process as streamlined and easy as possible for reviewers means they’ll be much more likely to agree to review again in future. And reviewers can also help increase efficiency in peer review by responding promptly to invitations to be reviewers. It’s much more helpful to send an immediate “no” than to prevaricate for weeks and then say “yes.” (See also Respect!)
- Transparency. As we’ve written before, transparency in peer review doesn’t have to mean having an open peer review process — though it might. Rather, it’s about ensuring that your processes are clearly defined and publicly accessible. This includes the type of peer review (with as much detail as possible), number of reviewers and how they are selected, average length of review, and more. This ensures that both researchers and reviewers know what to expect during the review process.
It’s important that both reviewers and editors/publishers are equally committed to these principles. Reviewers who do not invest appropriate care in their reviews undermine the process. Editors who allow poor-quality reviews to be shared with the author(s) and/or to inform their decision-making are just as responsible as the “bad” reviewers themselves. Equally, authors who do not engage productively with reviews risk missing what potential for improvement even a tough review can provide. Above all, in principle and the practices they inform, mutual dedication to the mission of high quality in peer review is vital.
“Quality” is multi-dimensional. It can be measured in a variety of ways, and from a variety of perspectives. As we have written in previous Peer Review Weeks (see below), diversity and transparency are crucial for peer review and, we would argue, essential for achieving quality by any measure.
To find out more about previous Peer Review Weeks, we suggest starting with these introductory posts:
- 2018 – Diversity in Peer Review: Clear, Blue Sky: The Environment for Diversity in Peer Review
- 2017 – Transparency in Peer Review: Peer Review in a World of “Alternative Facts”
- 2016 – Recognizing Review: Peer Review Week 2016 #RecognizingReview
- 2015 (no theme): Welcome to Peer Review Week!
You can also look in The Scholarly Kitchen Peer Review Collection, which captures posts from Peer Review Week as well as other posts of interest and relevance.