Peer review — as befitting its importance in scholarly communications — is one of the most highly-covered subjects on The Scholarly Kitchen, and one of the 13 topics to have its own (growing!) collection. We recommend The Peer Review Collection in its entirety to anyone looking to educate themselves about this important topic but, as a starting point, we have gathered what we think are some of the signature posts, and added a summary as well as some contextual links (often, but not always, to other Kitchen posts). Together they provide a helpful look at the topics that most concern authors, editors, publishers, and reviewers.

In particular, we’ve highlighted posts that cover the issues which framed our first post of the week, on a healthy environment for diversity in peer review. This includes the role of transparency in editorial and reviewer roles and responsibilities, and in credit for review. In keeping with Peer Review Week 2018’s theme of diversity and inclusion, we also wanted to highlight diverse perspectives; in the posts we selected we found not only among the authors but also in the (often robust) comment sections a good variety of experiences and views. We also recommend The Diversity and Inclusion Collection as a great resource for posts on this important topic.

Improving Peer Review: Let’s Provide an Ingredients List for Our Readers
In this early (2010) call for more transparency around the peer review process, Kent Anderson proposes an “ingredients list” approach, including:
Number of outside reviewers
Degree of blindedness (names and institutions eliminated from the manuscript, for instance)
Number of review cycles needed before publication
Duration of the peer review portion of editorial review
Other review elements included (technical reviews, patent reviews, etc.)
Editorial board review
Editorial advisers review
Statistical review
Safety review
Ethics and informed consent review
Kent’s advice is as sound now as it was back in 2010 — but have we made any progress? Transparency is still not the norm in peer review, as shown in Jason Roberts and Donald Samulack’s guest post earlier this week, and we wrote about this topic ourselves during Peer Review Week last year, noting in Peer Review in a World of “Alternative Facts” that: “Irrespective of the form of review, providing transparent (i.e., clear, accurate, and easy-to-find) information on their peer review process should be a no-brainer for any journal or organization.” And some progress is being made. Robert Harington’s recent post Peer Review – Authors and Reviewers – our “North Star” highlights EMBO Journal and Nature Communications as two good examples of journals who have adopted transparent peer review. Mark Edington’s recent guest post proposes taking a Creative Commons type approach and using “a system of symbols, or icons, that convey both the kind of review undertaken and the focus of that review.” on on their peer review process should be a no-brainer for any journal or organization.”

Sexism in Peer Review
One of the benefits of increased transparency in the peer review process is the opportunity it affords to better detect and address bias — in all its forms. Bias is a topic that has been written about quite a bit on the Kitchen, including in this 2015 post by Angela Cochran. She notes that: “Much of the gender bias in the publishing side could be eliminated by diversifying editorial boards, diversifying reviewer pools, zero-tolerance policies against gender bias in review, and review processes that protect authors from unfair treatment (going totally dark with double-blind review or shedding light on all participants with open review).” Also worth a read is this interview with Jory Lerback and Brooks Hanson on AGU’s study of bias in their peer review process. Spoiler alert: it exists! However, simply adding “a specific statement to the author instructions reminding authors about our results and asking that they consider diversity in their recommendations” led to a significant improvement.

Revisiting: The Problem(s) with Credit for Peer Review
In this post and the extensive comments section, author David Crotty revisits the thorny question of credit for reviewers, who wants it, and how to calculate it. As David notes, a key problem with credit is inherent in all metrical systems of evaluation for scholarly work, namely what gets counted and who uses that accounting to what ends. David also points out, and several comments reiterate, the importance of reviewing as an aspect of researcher community building and participation. Making sure that the means to participation is clear, and invitations to participate are widely shared, will make for more diverse communities and thus more diverse peer review.

Guest Post — When Metrics and Politics Collide: Reflections on Peer Review, the JIF and Our Current Political Moment
Sara Rouhi’s guest post also addresses bias, but places it in a broader context — that of “the dominant power paradigm of the global research landscape: white, western, English-speaking men and the prestige institutions they represent.” Bringing together themes from three 2017 events — the Eighth Peer Review Congress, the Charleston Library Conference Hyde Park Debate, and the Politics and Power Plays exhibit at the Chicago Museum of Contemporary Art, Sara powerfully demonstrates the social inequities that are all around us, and calls on us to “unflinchingly acknowledge where we sit within [the] hierarchy”. While not specifically about peer review, if you’re interested in learning more about social inequity in scholarly publishing we also recommend another guest post: Safiya Umoja Noble and the Ethics of Social Justice in Information (Part 2). Safiya notes, among other things, that: “Gate-keeping functions often are performed by men, white men, many of whom serve as key editors around the country at presses. So we have to think about what they recognize as valuable or meaningful, and how their interests may or may not converge with voices that are underrepresented in the field.”

Interview:  BMJ’s Patient Review Initiative
In this post, Kent Anderson interviews a BMJ team to ask about another dynamic in peer review diversity — bringing diverse forms of expertise to bear. They reported that:

In 2013 The BMJ committed to setting up an international panel of patients and patient advocates, to co-produce a new patient partnership strategy to advance the “patient revolution” in healthcare. The journal has been actively advocating for patient partnership for over 25 years, but we agreed that a co-produced strategy which saw us “walking the talk” on patient partnership in our own editorial process would make our advocacy for it both more credible and effective.

BMJ have created a registry, actively solicited and supported patient participation, and found that review time is not slowed. They plan a formal study of authors’ response to patient reviewers (though informally they have found most authors have responded well), and may look at whether acceptance rates are different for papers with patient reviewers. The project raises questions about how and in what ways diverse kinds of expertise can be sought for peer review.

A Curious Blindness Among Peer Review Initiatives
A key challenge in creating innovation in peer review is that peer review itself may not be well or fully understood. Ironically, lack of transparency about process flying under the banner of blindedness has hobbled our efforts to create workable, sustainable alternatives. In this post Tim Vines points to the essential work of editorial offices in managing the flow of reviews and, very crucially, arresting errors in the process and/or in the papers themselves. Tim proposed a couple of examples of this skilled labor: identifying conflict of interest among reviewers, or spotting a missing data set. The comments on this post are especially lively, include the fate of PubMed Commons, an argument that copyediting could be optional, and the role of library or institutional publishing. Angela Cochran’s comment especially struck us: “I am all for innovation, but step one is to actually know what you are trying to innovate.” Working toward more inclusive and diverse peer review depends on a full reckoning with every step of the process. We are reminded, though, of Phil Davis’s post from 2010 about the European Molecular Biology Association and other experiments in “opening the black box” of review process, which suggested significant extra costs associated with doing so.

Peer Review in the Humanities and Social Sciences: If It Ain’t Broke, Don’t Fix It?
HSS disciplines and their publication format of choice — books — are often overlooked in scholarly communications (including here on the Kitchen), in favor of STEM disciplines and journals. This post from last year’s Peer Review Week goes some way toward redressing that balance. In the form of a conversation between Mary Francis, Editorial Director at the University of Michigan Press, and Chefs Alison Mudditt and Karin Wulf — both of whom have HSS backgrounds — it provides a good overview of the differences between the review process in these disciplines, and of the (then) recently published AUP Best Practices for Peer Review handbook. Intended to: “advocate for the value of peer review to those whose support is essential: deans and provosts, foundations and funders, regents, elected representatives”, the handbook itself is another invaluable and detailed “how-to” guide, which we also highly recommend.

We hope this week’s posts along with this brief introduction to peer review on the Kitchen will inspire you to get reading, get thinking, and get sharing your own favorite posts and articles on the topic with everyone in the Comments section!