Next week is Peer Review Week 2018. Asking the Chefs a peer review question has become a bit of a tradition for us. In 2016 we asked: What is the future of peer review? Last year we contemplated: Should peer review change?
This year the theme is diversity in peer review. So we’ve asked the Chefs: How would you ensure diversity in peer review?
Lisa Hinchliffe: Given I have served as a journal editor multiple times in my career, allow me to respond to this question from that perspective. First, you must have a commitment to diversity in peer review as a non-negotiable facet of your process and investing the time and effort needed in order achieve your goal of diversity. Second, you must do the work to identify a diverse reviewer corps and solicit the commitment of reviewers who you wish to be part of your team. Third, you must ensure that the experience of serving as a peer reviewer is a positive experience. You may need to do additional outreach and offer additional support to overcome the impact of reviewers’ past negative experiences as a peer reviewer. Fourth, you must check your own biases and privileges when you review the assessments submitted by the peer reviewers and not discount the feedback and evaluations submitted from the diversity of perspectives you have recruited. Fifth, you must ensure that peer reviewers receive recognition for their labor in ways that are valued in the performance review (tenure/promotion) schemes under which they are evaluated. No one owes their diversity to our peer review processes but many are willing. It is our responsibility as editors to invite, recognize, and reward them.
Tim Vines: What is diversity in the context of peer review? At minimum, given a community of researchers who vary in seniority, country of origin, gender etc., the makeup of our reviewer pool should mirror the makeup of the broader community. It’s well established that humans are not good at generating unbiased lists of names, so there’s a big role here for technology. Fortunately, the relevant tech already exists, and is improving all the time. For example, Publons’ Reviewer Connect tool should pull up qualified reviewers that wouldn’t normally spring to mind, and bring them into the review process. Similar tools have been created by Elsevier, Uber Research, and Aries.
The next steps are encouraging Editors to make use of these tools, and have them invite the people the tools suggest. I think this can be achieved with a minor workflow change. Instead of providing Editors with the online equivalent of a blank sheet of paper, they could be presented with a list of potential reviewers generated by these tools. The Editors would then select the reviewers they wanted to invite. The proposed reviewer list must inspire confidence that it is finding the most appropriate people, which requires the tool to learn which reviewers the Editor prefers for a particular type of article, and then suggest others based on those preferences. On the surface, automating reviewer suggestions might not seem a powerful tool for promoting diversity, but it does open a path for a more representative subset of the community to get noticed as reviewers. Good reviewers often go on to become good Editors, and good Editors become Chief Editors in due course. Opening the door for reviewer diversity should thus promote better diversity throughout the peer review process.
It’s well established that humans are not good at generating unbiased lists of names, so there’s a big role here for technology. Fortunately, the relevant tech already exists, and is improving all the time.
Karin Wulf: Diversity begins with, and is best ensured by, inclusion. Inclusive practices have many dimensions. I’m inspired by Kent Anderson’s post from June about the BMJ Patient Review Initiative, and the reminder of the ways that a rigid definition of expertise often leaves peer review to a narrow – and historically not diverse — group of participants. This is not to say that expertise isn’t important — of course it is. It’s to note that we should be conscious of how we have defined what constitutes expertise, and therefore who can be expert.
In history, an important development is the intensifying call for recognition of diverse kinds of expertise. Two recent examples point to the ways, not only that professional training was long a bar to the histories of people excluded from elite institutions and/ or for whom expertise isn’t resident in those institutions, but that those people’s histories were often excluded altogether. One is the now famous, and embraced by Monticello, but long denied history of Sally Hemings with Thomas Jefferson, the man who enslaved her and her family including their children. Historian Annette Gordon-Reed has written extensively about the ways that the oral histories of the Hemings family, and the memoir of one of those children, Madison Hemings, published in 1873, were ignored and derided. As she noted in a New York Times editorial this summer when Monticello opened its exhibit about Sally Hemings, “a document that would otherwise have been devoured by scholars desperate to find any tidbit of information about Jefferson was ignored, except as a statement to refute or ridicule. White historians were determined not to listen to Madison Hemings, a historical figure who said things they did not want to hear.”
Another key example is in Native American history, and the leadership of the Native American and Indigenous Studies Association. Decades of work to draw attention to a dominant discourse about Native American disappearance, an over focus on a small handful of tribal groups rather than the “infinity of nations,” and the reliance on European-produced archival sources rather than Native-produced materials is joined by an appeal to recognize these patterns and their consequences. A recent essay that analyzes these developments (“Materials and Methods in Native American and Indigenous Studies: Completing the Turn“) notes that scholars must “reach outside the academy (as it is traditionally defined) to consult tribal experts and archives and to consider the impact of their work on Native communities…. [to] help us to interpret the past in ways that center Indigenous intellectual traditions, textualities, and temporalities.”
Diversity in peer review – just like diversity in authorship – is important to ensure that there aren’t gaps, biases and blind spots in the research that gets published.
Sian Harris: I think it’s useful to take a step back for a moment and consider the question: Why should we ensure diversity in peer review? Diversity in peer review – just like diversity in authorship – is important to ensure that there aren’t gaps, biases and blind spots in the research that gets published. From my perspective – and the perspective of INASP, which I work for – this means geographical diversity, and ensuring inclusion of researchers from the Global South as reviewers as well as authors. Researchers from the Global South sometimes tell us that they feel journals from the Global North are biased against them because editors and reviewers don’t understand or appreciate the context and importance of their research, or don’t see it as ‘globally relevant’.
In Peer Review Week, we will be publishing a series of blog posts written by reviewers and editors from the Global South on our AuthorAID website. At the time of writing this answer, we are still collecting responses but there are few recommendations that have already emerged. A key one is improving training and guidelines for peer reviewers. This is important for facilitating more researchers from low and middle income countries (LMIC) to take part in peer review but also for reviewers from high-income countries to have more understanding of the research contexts in which LMIC researchers work.
Kent Anderson: I think there are multiple ways of approaching diversity — life experience, intellectual diversity, phenotypic diversity, and genotypic diversity among them. I’m encouraged mostly by pursuits of diversity that are based on life experience and intellectual diversity, because people who look different can think the same as their social counterparts. Degrees of social separation are more informative of differences for me, and I think you’re more likely to get a unique perspective when you pay attention to these aspects. So, initiatives like BMJ’s patient peer review, and journals that include professionals or journalists related to the topic but not necessarily academic experts show the potential here for me. I think editors who take this kind of approach get more practical, pragmatic insights, and also are less likely to get reviewers who fall prey to academic mind viruses, buzzwords, and mental fads.
So, initiatives like BMJ’s patient peer review, and journals that include professionals or journalists related to the topic but not necessarily academic experts show the potential here for me.
- Making the peer review process more transparent. That doesn’t necessarily mean having a fully open process (though this has been shown, in some cases, to increase diversity), but it does mean being open about how your peer review process works. How are reviewers selected? What criteria are they given for their assessment? Are authors allowed — or encouraged — to suggest some or all of their reviewers? And so on…
- Being more inclusive. Given the continuing increase in the number of papers published — never mind the many other types of review that researchers are called on to do — it’s essential to increase the number of reviewers for those papers. Inviting reviews from a wider, more diverse group of researchers, and maybe even looking beyond researchers, as some publishers are starting to do — will increase diversity as well as addressing the practical issue of reviewer burnout.
- Calling for change. Initiatives such as this, have found that simply alerting editors and review editors to be more inclusive can lead to a significant increase in diversity. That’s great news, because there’s nothing to stop any publication or organization from proactively signaling that they want and expect to utilize a more diverse reviewer pool.
- Highlighting the benefits. There is ample evidence, from across multiple sectors, that diversity is essential for success. Whether you’re evaluating a product or service, or reviewing a book, article, or grant, you’ll get better results by incorporating a range of perspectives. So make sure your authors, editors, reviewers, and readers know about the benefits of diversity in peer review (which brings me neatly back to transparency!)
Alison Mudditt: Many factors contribute to gender, geographic and other inequities in scholarly publishing, but disparities in peer review outcomes undermine the robustness of and confidence in peer review itself. Just this week a new preprint on bioRxiv examining submissions to eLife between 2012 and 2017 found that gatekeepers favor manuscripts from authors of the same gender and from the same country.
And while we all know that the peer review process is often a bottleneck for publishers and editors, I worry about the consequences of trying to automate this for efficiency. For example, Publons’ new Reviewer Connect tool links its database of reviewers to the Web of Science to trawl for reviewers. The risk for this and similar approaches is that we simply reinforce built-in bias (that authors and reviewers are overwhelmingly white men from the northern hemisphere), continuing to limit diversity.
So, what to do? Our past experience with double-blind peer review at PLOS is that it’s almost impossible to properly anonymize authorship so that probably isn’t the answer. Publishers and journals should actively increase representation of women and scientists from more diverse backgrounds at editorial board level to expand the networks tapped to review (and perhaps go as far as some of us do in the hiring process and require a diverse pool).
But ultimately, given the challenges of picking apart overt and unconscious bias, the best solution is to introduce a far greater level of transparency into the review process. This can and should take different forms but publishing reviewer reports, signed or not, seems a critical step.
But ultimately, given the challenges of picking apart overt and unconscious bias, the best solution is to introduce a far greater level of transparency into the review process. This can and should take different forms but publishing reviewer reports, signed or not, seems a critical step. Apart from anything else, it will allow us to better research and understand a little about how lack of diversity and bias affect the review process and outcomes. Transparent review also creates a pathway to greater incentives and credit for reviewing which, along with better training and mentoring for reviewers, feels like another piece of the puzzle.
Now it’s your turn! How would you ensure diversity in peer review?
Author’s Note: (October 22-26 is Open Access week. Let me know if you have a question you’d like us to ask the Chefs!)