The lack of diversity and inclusion in scholarly communications is increasingly well-documented. As the recent Workplace Equity Project survey shows, we are an overwhelmingly white and cis-female industry, with a leadership that is disproportionately white male dominated. And this isn’t just the case in scholarly publishing. There’s a “diversity crisis in academia” according to Times Higher Education; “a feedback loop in scholarship that privileges and publishes the majority voice, which is often white and male,” per College and Research Libraries News; or, as Safiya Noble put it, writing here on the Kitchen, “the whole system reproduces and consolidates power among a relatively small group of people.” What’s more, as Sara Rouhi pointed out in her 2017 Scholarly Kitchen guest post:

These folks created the paradigm, its rules of engagement (Study with this professor! Postdoc at that institution! Publish or perish!), its costs for entry (JIF! Citations!), and its prizes (Tenure! Grant money! Status!). Everyone else – the global south and non-white, non-western countries – starts at a disadvantage by not belonging to the dominant group and constantly confronting the structural inequities of that exclusion.

Sadly, but not unexpectedly, it’s becoming more and more apparent that peer review is likewise less diverse and more biased than it could or should be, which is hindering our efforts to ensure an inclusive, ethical, trustworthy scholarly communications ecosystem.

How can we tackle this? As a starting point, I decided to review some recent studies on diversity and bias issues in peer review in order to distill the main learning points. I used some simple Google searches (peer review + bias, gender bias, ethnic bias, country bias, age bias) to identify the articles, and selected those that were both relevant and openly available. I also found some additional papers by following the references from the initial ones.

Most of the articles discovered in this way focus primarily on gender representation and bias in peer review; a few look at age/career stage, geographic and/or language bias. Unfortunately I was unable to find anything openly available that specifically addresses racial or ethnic bias, perhaps because this data is not being routinely collected (if anyone can point to research on this topic, please do!). However, my assumption is that, since there is clear evidence of other forms of bias, some form of racial and ethnic bias is also inevitable.

In the spirit of many heads being better than one, I’m sharing here some practical suggestions for how to tackle diversity and inclusion in peer review, based on the results and recommendations of the papers I discovered.

1. Be aware of your own biases. In “Gender differences in patterns of authorship do not affect peer review outcomes at an ecology journal”, authors Charles W. Fox et al., note that, although they found no widespread evidence of gender bias in the Functional Ecology publication process: “men were less likely to respond to review invitations if the authors were female…[and that] men (but not women) invited to review for Functional Ecology were less likely to respond to review invitations, and less likely to agree to review, if the editor was female.” We think (hope) we can safely assume that those men weren’t deliberately rejecting invitations from women because they’re women — but that unconscious bias was almost certainly a (the?) cause.
2. Take action to address those biases at an individual level. A 2014 paper in Trends in Pharmacological Science, “Threats to Objectivity in Peer Review: The Case of Gender”, by Anna Kaatz et al., provides some great advice on how to do this. To paraphrase their recommendations and generalize them for other forms of bias, we should:
   • Be (very) aware that we are all susceptible to bias in judgment
   • Allow dedicated time (no multi-tasking) when undertaking peer review
   • Use our imagination to picture in detail a research leader who is a woman/person of color/from a different type of institution or background
   • Review the assessment criteria before evaluating manuscripts or applications
   • Challenge ourselves to consider whether our evaluation of their work would change if, for example, the researcher were male/white/from a major research institute
3. And do the same at the journal/publisher level. One of the better known recent studies of bias in peer review is Jory Lerback and Brooks Hanson’s 2017 article in Nature, “Journals invite too few women to referee”, reporting on their analysis of peer reviewers for the American Geophysical Union’s 20 journals. Focusing primarily on gender and age, they found that, “women of all ages have fewer opportunities to take part in peer review…women were used less as reviewers than expected…The bias is a result of authors and editors, especially male ones, suggesting women as reviewers less often, and a slightly higher decline rate among women in each age group when asked.” Depressing, right? However, the good news is that, simply by adding a note to the submission form for Geophysical Research Letters, asking authors to help improve diversity in peer review, they saw a significant increase in the number of women reviewers in just three months. You can read more about their findings in this interview. Along similar lines, in the article mentioned above, Kaatz et al., also recommend that: “Journals and funding agencies should make sure that review criteria do not implicitly prime the selection of male over female scientists (e.g., avoid the use of male-typed terms such as ‘high-risk’ or ‘aggressive’).
4. Consider blind(er) review. A study of Nature journals, by Barbara McGillivray and Elisa de Ranieri, presented at the Peer Review Congress in 2017, looked at the impact of double versus single blind review on bias. They found that double blind review had: “no overall difference in the acceptance rates of papers according to gender, while at the same time reporting a strong effect of number of authors and country of affiliation on manuscripts’ acceptance rates.” Writing in the Proceedings of the National Academy of Sciences, Tomkins et al., similarly concluded that: “two contributions of roughly equal merit might be scored differently by single-blind reviewers, in favor of the one from a top school, while double-blind reviewers may not show this bias as strongly.” And Christophe Bernard, in an eNeuro editorial, notes that the journal’s: “female/male ratio at submission is 30% … and the ratio at acceptance is 30%” which in his view is because: “the double-blind review process minimizes gender bias during the evaluation of the paper. It avoids adding an additional layer of bias to the dissemination of research findings.”
5. Conversely, use open review. In her paper, “Publishing while female: Are women held to higher standards? Evidence from peer review”, Erin Hengel makes the case (among other things) for open peer review as a way of reducing bias. She quotes this report of a randomized controlled trial at the British Journal of Psychiatry that suggests referee reports are better quality and less abusive when identities are known. According to Hengel: “Posting [reviews] online — as the British Medical Journal does — virtually guarantees continuous, independent audits by outside researchers. Worries that reviews are less critical and/or relationships are strained are either unfounded or alleviated by the deep pool of referees common to general interest journals.”
6. Collect the data. A recent editorial on Nature’s under-representation of women found that, “Our poorest outcome is in the refereeing of research papers. Counting only individuals whose gender we can attribute from their first names, the proportion of female referees has increased from 12% in 2011 to 16% in 2017.” Their approach to date has been “to counter unconscious bias, by getting senior staff and editors to ask themselves, ‘Who are the outstanding women for this task?’, before commissioning an author or a referee.” Clearly they have not been as successful as they — or any of us — would like, but at least they are collecting the data, which is a critical first step toward being able to understand, and ultimately resolve, the issue.
7. Make use of available tools. As well as collecting data, look out for other tools that can help address bias in the peer review process. “Research: Gender bias in scholarly peer review”, published in eLife, presents the findings of a large study of 9,000 editors and 43,000 reviewers from the Frontiers series of journals. Authors Markus Helmer, Manuel Schottdorf, Andreas Neef, and Demian Battaglia found that, “women are underrepresented in the peer-review process, that editors of both genders operate with substantial same-gender preference (homophily), and that the mechanisms of this homophily are gender-dependent…[and] that homophily will persist even if numerical parity between genders is reached.” Their preferred solution is the use of editorial management software and statistics to help editors spot and reduce the risk of bias in the selection of reviewers.
8. Encourage training and/or mentorship. Hannah Buckley et al., writing in the New Zealand Journal of Ecology, highlight the journal’s reviewer mentoring initiative as “an excellent way for associate editors to improve the gender ratio of their reviewer selection through increased visibility of female potential reviewers, including both female mentees and mentors.” Andrew Preston of Publons makes a similar point in this Scientific American article on the future of peer review: “Training needs to be built into PhD and postdoctoral programs, and supervisors need to recognize their roles in stewarding in the next generation of reviewers.”

I welcome and encourage you to share your own suggestions, as well as other papers you recommend on the topic of tackling diversity, inclusion, and bias in peer review!

With many thanks to Lauren Kane (BioOne) and fellow chef Karin Wulf for their thoughtful input and feedback on this post.