Next week is Peer Review Week 2018. Asking the Chefs a peer review question has become a bit of a tradition for us. In 2016 we asked: What is the future of peer review? Last year we contemplated: Should peer review change?

This year the theme is diversity in peer review. So we’ve asked the Chefs: How would you ensure diversity in peer review?

peerreviewweek_2018

Lisa Hinchliffe: Given I have served as a journal editor multiple times in my career, allow me to respond to this question from that perspective. First, you must have a commitment to diversity in peer review as a non-negotiable facet of your process and investing the time and effort needed in order achieve your goal of diversity. Second, you must do the work to identify a diverse reviewer corps and solicit the commitment of reviewers who you wish to be part of your team. Third, you must ensure that the experience of serving as a peer reviewer is a positive experience. You may need to do additional outreach and offer additional support to overcome the impact of reviewers’ past negative experiences as a peer reviewer. Fourth, you must check your own biases and privileges when you review the assessments submitted by the peer reviewers and not discount the feedback and evaluations submitted from the diversity of perspectives you have recruited. Fifth, you must ensure that peer reviewers receive recognition for their labor in ways that are valued in the performance review (tenure/promotion) schemes under which they are evaluated. No one owes their diversity to our peer review processes but many are willing. It is our responsibility as editors to invite, recognize, and reward them.

Tim Vines: What is diversity in the context of peer review? At minimum, given a community of researchers who vary in seniority, country of origin, gender etc., the makeup of our reviewer pool should mirror the makeup of the broader community. It’s well established that humans are not good at generating unbiased lists of names, so there’s a big role here for technology. Fortunately, the relevant tech already exists, and is improving all the time. For example, Publons’ Reviewer Connect tool should pull up qualified reviewers that wouldn’t normally spring to mind, and bring them into the review process. Similar tools have been created by Elsevier, Uber Research, and Aries.

The next steps are encouraging Editors to make use of these tools, and have them invite the people the tools suggest. I think this can be achieved with a minor workflow change. Instead of providing Editors with the online equivalent of a blank sheet of paper, they could be presented with a list of potential reviewers generated by these tools. The Editors would then select the reviewers they wanted to invite. The proposed reviewer list must inspire confidence that it is finding the most appropriate people, which requires the tool to learn which reviewers the Editor prefers for a particular type of article, and then suggest others based on those preferences. On the surface, automating reviewer suggestions might not seem a powerful tool for promoting diversity, but it does open a path for a more representative subset of the community to get noticed as reviewers. Good reviewers often go on to become good Editors, and good Editors become Chief Editors in due course. Opening the door for reviewer diversity should thus promote better diversity throughout the peer review process.

It’s well established that humans are not good at generating unbiased lists of names, so there’s a big role here for technology. Fortunately, the relevant tech already exists, and is improving all the time.

Karin Wulf: Diversity begins with, and is best ensured by, inclusion. Inclusive practices have many dimensions. I’m inspired by Kent Anderson’s post from June about the BMJ Patient Review Initiative, and the reminder of the ways that a rigid definition of expertise often leaves peer review to a narrow – and historically not diverse — group of participants. This is not to say that expertise isn’t important — of course it is. It’s to note that we should be conscious of how we have defined what constitutes expertise, and therefore who can be expert.

In history, an important development is the intensifying call for recognition of diverse kinds of expertise. Two recent examples point to the ways, not only that professional training was long a bar to the histories of people excluded from elite institutions and/ or for whom expertise isn’t resident in those institutions, but that those people’s histories were often excluded altogether. One is the now famous, and embraced by Monticello, but long denied history of Sally Hemings with Thomas Jefferson, the man who enslaved her and her family including their children. Historian Annette Gordon-Reed has written extensively about the ways that the oral histories of the Hemings family, and the memoir of one of those children, Madison Hemings, published in 1873, were ignored and derided. As she noted in a New York Times editorial this summer when Monticello opened its exhibit about Sally Hemings, “a document that would otherwise have been devoured by scholars desperate to find any tidbit of information about Jefferson was ignored, except as a statement to refute or ridicule. White historians were determined not to listen to Madison Hemings, a historical figure who said things they did not want to hear.”

Another key example is in Native American history, and the leadership of the Native American and Indigenous Studies Association. Decades of work to draw attention to a dominant discourse about Native American disappearance, an over focus on a small handful of tribal groups rather than the “infinity of nations,” and the reliance on European-produced archival sources rather than Native-produced materials is joined by an appeal to recognize these patterns and their consequences. A recent essay that analyzes these developments (“Materials and Methods in Native American and Indigenous Studies: Completing the Turn“) notes that scholars must “reach outside the academy (as it is traditionally defined) to consult tribal experts and archives and to consider the impact of their work on Native communities…. [to] help us to interpret the past in ways that center Indigenous intellectual traditions, textualities, and temporalities.”

Diversity in peer review – just like diversity in authorship – is important to ensure that there aren’t gaps, biases and blind spots in the research that gets published.

Sian Harris: I think it’s useful to take a step back for a moment and consider the question: Why should we ensure diversity in peer review? Diversity in peer review – just like diversity in authorship – is important to ensure that there aren’t gaps, biases and blind spots in the research that gets published. From my perspective – and the perspective of INASP, which I work for – this means geographical diversity, and ensuring inclusion of researchers from the Global South as reviewers as well as authors. Researchers from the Global South sometimes tell us that they feel journals from the Global North are biased against them because editors and reviewers don’t understand or appreciate the context and importance of their research, or don’t see it as ‘globally relevant’.

In Peer Review Week, we will be publishing a series of blog posts written by reviewers and editors from the Global South on our AuthorAID website. At the time of writing this answer, we are still collecting responses but there are few recommendations that have already emerged. A key one is improving training and guidelines for peer reviewers. This is important for facilitating more researchers from low and middle income countries (LMIC) to take part in peer review but also for reviewers from high-income countries to have more understanding of the research contexts in which LMIC researchers work.

Kent Anderson: I think there are multiple ways of approaching diversity — life experience, intellectual diversity, phenotypic diversity, and genotypic diversity among them. I’m encouraged mostly by pursuits of diversity that are based on life experience and intellectual diversity, because people who look different can think the same as their social counterparts. Degrees of social separation are more informative of differences for me, and I think you’re more likely to get a unique perspective when you pay attention to these aspects. So, initiatives like BMJ’s patient peer review, and journals that include professionals or journalists related to the topic but not necessarily academic experts show the potential here for me. I think editors who take this kind of approach get more practical, pragmatic insights, and also are less likely to get reviewers who fall prey to academic mind viruses, buzzwords, and mental fads.

So, initiatives like BMJ’s patient peer review, and journals that include professionals or journalists related to the topic but not necessarily academic experts show the potential here for me.

Alice Meadows: As with pretty much everything, there’s unlikely to be a simple, one-size-fits-all answer to this question. Different disciplines and communities have different norms and expectations — and different types of peer review (e.g., for grant application versus a book proposal) require different approaches. However, we could all make some general changes, which I believe would have a positive impact in terms of increasing diversity in peer review. These include:
  1. Making the peer review process more transparent. That doesn’t necessarily mean having a fully open process (though this has been shown, in some cases, to increase diversity), but it does mean being open about how your peer review process works. How are reviewers selected? What criteria are they given for their assessment? Are authors allowed — or encouraged — to suggest some or all of their reviewers? And so on…
  2. Being more inclusive. Given the continuing increase in the number of papers published — never mind the many other types of review that researchers are called on to do — it’s essential to increase the number of reviewers for those papers. Inviting reviews from a wider, more diverse group of researchers, and maybe even looking beyond researchers, as some publishers are starting to do — will increase diversity as well as addressing the practical issue of reviewer burnout.
  3. Calling for change. Initiatives such as this, have found that simply alerting editors and review editors to be more inclusive can lead to a significant increase in diversity. That’s great news, because there’s nothing to stop any publication or organization from proactively signaling that they want and expect to utilize a more diverse reviewer pool.
  4. Highlighting the benefits. There is ample evidence, from across multiple sectors, that diversity is essential for success. Whether you’re evaluating a product or service, or reviewing a book, article, or grant, you’ll get better results by incorporating a range of perspectives. So make sure your authors, editors, reviewers, and readers know about the benefits of diversity in peer review (which brings me neatly back to transparency!)

Alison Mudditt: Many factors contribute to gender, geographic and other inequities in scholarly publishing, but disparities in peer review outcomes undermine the robustness of and confidence in peer review itself. Just this week a new preprint on bioRxiv examining submissions to eLife between 2012 and 2017 found that gatekeepers favor manuscripts from authors of the same gender and from the same country.

And while we all know that the peer review process is often a bottleneck for publishers and editors, I worry about the consequences of trying to automate this for efficiency. For example, Publons’ new Reviewer Connect tool links its database of reviewers to the Web of Science to trawl for reviewers. The risk for this and similar approaches is that we simply reinforce built-in bias (that authors and reviewers are overwhelmingly white men from the northern hemisphere), continuing to limit diversity.

So, what to do? Our past experience with double-blind peer review at PLOS is that it’s almost impossible to properly anonymize authorship so that probably isn’t the answer. Publishers and journals should actively increase representation of women and scientists from more diverse backgrounds at editorial board level to expand the networks tapped to review (and perhaps go as far as some of us do in the hiring process and require a diverse pool).

But ultimately, given the challenges of picking apart overt and unconscious bias, the best solution is to introduce a far greater level of transparency into the review process. This can and should take different forms but publishing reviewer reports, signed or not, seems a critical step.

But ultimately, given the challenges of picking apart overt and unconscious bias, the best solution is to introduce a far greater level of transparency into the review process. This can and should take different forms but publishing reviewer reports, signed or not, seems a critical step. Apart from anything else, it will allow us to better research and understand a little about how lack of diversity and bias affect the review process and outcomes. Transparent review also creates a pathway to greater incentives and credit for reviewing which, along with better training and mentoring for reviewers, feels like another piece of the puzzle.

____________________________________________

Now it’s your turn! How would you ensure diversity in peer review?

Author’s Note: (October 22-26 is Open Access week. Let me know if you have a question you’d like us to ask the Chefs!)

Ann Michael

Ann Michael

Ann Michael is President of Delta Think, a business and technology consulting and advisory firm focused on innovation and growth in membership organizations, scholarly publishers, and professional information providers. Ann is Past-President of SSP.

View All Posts by Ann Michael

Discussion

5 Thoughts on "Ask The Chefs: How Would You Ensure Diversity In Peer Review?"

Interesting discussion! One other idea about how ensure or improve diversity in peer review is to ask a truly diverse group of people how to ensure or improve diversity in peer review. Sian Harris says, “Diversity in peer review – just like diversity in authorship – is important to ensure that there aren’t gaps, biases and blind spots in the research that gets published.” That rings true to me, perhaps because it is obvious, but what about the gaps and blind spots in this very discussion due to the all-white and (probably) all Anglo-American crew of respondents? I’m not saying there aren’t good ideas in the responses–surely there are some–but that’s beside the point. When you ask a homogeneous and historically empowered group how to ensure that their privilege will no longer be, well, privileged, you necessarily get a skewed set of responses. For example, Tim Vines says, “At minimum, given a community of researchers who vary in seniority, country of origin, gender etc., the makeup of our reviewer pool should mirror the makeup of the broader community.” Is the (minimum?) difference that defines the “broader community” merely one of age/seniority, gender, and country of origin? Isn’t that a blind spot that is the product of the nature of the question and the group you ask it of?

I saw a quote the other day that I liked: “Policies that fail in the same way over and over are not failing. Someone is lying about their intent” (Jacob Bacharach, “Capitalism is beyond Saving, and America is Living Proof”). In that spirit, maybe we can say that efforts to improve diversity by assigning the task to only whites do not fail because ensuring diversity is all that hard; these efforts fail because you are asking people who benefit from making the project (seem) hard. Humble suggestion: maybe the next time you want to have a round table about diversity, consider assembling a diverse group.

Hi John — we’ve talked about our efforts to add more diverse voices to the conversation a few times, as well as some of the roadblocks that are slowing us down (here, for example https://scholarlykitchen.sspnet.org/2018/02/27/scholarly-kitchen-last-ten-years-next-ten-years/). One way we’ve been able to bring in a wider range of opinions has been through our Guest Post efforts, where we ask an author to just do one post, rather than commit to ongoing work. On the subject of diversity, you can find a lot of those posts collected here: https://scholarlykitchen.sspnet.org/collection/diversity-and-inclusion/

SSP is also part of a larger diversity and inclusion effort going on across the community, described here:
https://scholarlykitchen.sspnet.org/2017/11/29/diversity-inclusion-ssp-interview-executive-director-melanie-dolechek/

But in the end, you’re right, there’s always more that can be done and we are always open to ways to improve representation from the entire community.

Hi John, I completely agree with you. My response to this question was a summary of comments from researchers from Africa, Asia and Latin America taken from a series of posts on peer review that we are publishing on the http://www.authoraid.info blog this week. The first, today, is from Dr Farooq Rathore, a medical researcher from Pakistan and makes a very interesting read – https://www.authoraid.info/en/news/details/1324/. Later this week we will be sharing perspectives from Rwanda, Tanzania and Bangladesh so please do keep an eye on our blog and share these perspectives. Thank you.

As mentioned in my response to Ann’s question, the AuthorAID blog (www.authoraid.info) is publishing a series of posts on peer review this week from peer reviewers and editors from Africa and Asia. We have now published the first two of these posts so I thought they would be interesting to readers of this post.

Yesterday’s post was from Dr Farooq Rathore, a medical researcher from Pakistan – https://www.authoraid.info/en/news/details/1324/.

Today’s post is from Dr Brenda Asiimwe Kateera, originally from Uganda and currently working at AIDS Healthcare Foundation Rwanda – https://www.authoraid.info/en/news/details/1323/

What role do journals specifically geared towards diversity have to encourage the journals who are not geared towards diversity to become more diverse? For example, everyone who works on the journal Feminist Theory have certain expectations. In fact, it may be that these journals are more welcoming.

Comments are closed.