Peer review — as befitting its importance in scholarly communications — is one of the most highly-covered subjects on The Scholarly Kitchen, and one of the 13 topics to have its own (growing!) collection. We recommend The Peer Review Collection in its entirety to anyone looking to educate themselves about this important topic but, as a starting point, we have gathered what we think are some of the signature posts, and added a summary as well as some contextual links (often, but not always, to other Kitchen posts). Together they provide a helpful look at the topics that most concern authors, editors, publishers, and reviewers.

peer review week logo

In particular, we’ve highlighted posts that cover the issues which framed our first post of the week, on a healthy environment for diversity in peer review. This includes the role of transparency in editorial and reviewer roles and responsibilities, and in credit for review. In keeping with Peer Review Week 2018’s theme of diversity and inclusion, we also wanted to highlight diverse perspectives; in the posts we selected we found not only among the authors but also in the (often robust) comment sections a good variety of experiences and views. We also recommend The Diversity and Inclusion Collection as a great resource for posts on this important topic.

Improving Peer Review: Let’s Provide an Ingredients List for Our Readers
In this early (2010) call for more transparency around the peer review process, Kent Anderson proposes an “ingredients list” approach, including:
Number of outside reviewers
Degree of blindedness (names and institutions eliminated from the manuscript, for instance)
Number of review cycles needed before publication
Duration of the peer review portion of editorial review
Other review elements included (technical reviews, patent reviews, etc.)
Editorial board review
Editorial advisers review
Statistical review
Safety review
Ethics and informed consent review
Kent’s advice is as sound now as it was back in 2010 — but have we made any progress? Transparency is still not the norm in peer review, as shown in Jason Roberts and Donald Samulack’s guest post earlier this week, and we wrote about this topic ourselves during Peer Review Week last year, noting in Peer Review in a World of “Alternative Facts” that: “Irrespective of the form of review, providing transparent (i.e., clear, accurate, and easy-to-find) information on their peer review process should be a no-brainer for any journal or organization.” And some progress is being made. Robert Harington’s recent post Peer Review – Authors and Reviewers – our “North Star” highlights EMBO Journal and Nature Communications as two good examples of journals who have adopted transparent peer review. Mark Edington’s recent guest post proposes taking a Creative Commons type approach and using “a system of symbols, or icons, that convey both the kind of review undertaken and the focus of that review.” on on their peer review process should be a no-brainer for any journal or organization.”

Sexism in Peer Review
One of the benefits of increased transparency in the peer review process is the opportunity it affords to better detect and address bias — in all its forms. Bias is a topic that has been written about quite a bit on the Kitchen, including in this 2015 post by Angela Cochran. She notes that: “Much of the gender bias in the publishing side could be eliminated by diversifying editorial boards, diversifying reviewer pools, zero-tolerance policies against gender bias in review, and review processes that protect authors from unfair treatment (going totally dark with double-blind review or shedding light on all participants with open review).” Also worth a read is this interview with Jory Lerback and Brooks Hanson on AGU’s study of bias in their peer review process. Spoiler alert: it exists! However, simply adding “a specific statement to the author instructions reminding authors about our results and asking that they consider diversity in their recommendations” led to a significant improvement.

Revisiting: The Problem(s) with Credit for Peer Review
In this post and the extensive comments section, author David Crotty revisits the thorny question of credit for reviewers, who wants it, and how to calculate it. As David notes, a key problem with credit is inherent in all metrical systems of evaluation for scholarly work, namely what gets counted and who uses that accounting to what ends. David also points out, and several comments reiterate, the importance of reviewing as an aspect of researcher community building and participation. Making sure that the means to participation is clear, and invitations to participate are widely shared, will make for more diverse communities and thus more diverse peer review.

Guest Post — When Metrics and Politics Collide: Reflections on Peer Review, the JIF and Our Current Political Moment
Sara Rouhi’s guest post also addresses bias, but places it in a broader context — that of “the dominant power paradigm of the global research landscape: white, western, English-speaking men and the prestige institutions they represent.” Bringing together themes from three 2017 events — the Eighth Peer Review Congress, the Charleston Library Conference Hyde Park Debate, and the Politics and Power Plays exhibit at the Chicago Museum of Contemporary Art, Sara powerfully demonstrates the social inequities that are all around us, and calls on us to “unflinchingly acknowledge where we sit within [the] hierarchy”. While not specifically about peer review, if you’re interested in learning more about social inequity in scholarly publishing we also recommend another guest post: Safiya Umoja Noble and the Ethics of Social Justice in Information (Part 2). Safiya notes, among other things, that: “Gate-keeping functions often are performed by men, white men, many of whom serve as key editors around the country at presses. So we have to think about what they recognize as valuable or meaningful, and how their interests may or may not converge with voices that are underrepresented in the field.”

Interview:  BMJ’s Patient Review Initiative
In this post, Kent Anderson interviews a BMJ team to ask about another dynamic in peer review diversity — bringing diverse forms of expertise to bear. They reported that:

In 2013 The BMJ committed to setting up an international panel of patients and patient advocates, to co-produce a new patient partnership strategy to advance the “patient revolution” in healthcare. The journal has been actively advocating for patient partnership for over 25 years, but we agreed that a co-produced strategy which saw us “walking the talk” on patient partnership in our own editorial process would make our advocacy for it both more credible and effective.

BMJ have created a registry, actively solicited and supported patient participation, and found that review time is not slowed. They plan a formal study of authors’ response to patient reviewers (though informally they have found most authors have responded well), and may look at whether acceptance rates are different for papers with patient reviewers. The project raises questions about how and in what ways diverse kinds of expertise can be sought for peer review.

A Curious Blindness Among Peer Review Initiatives
A key challenge in creating innovation in peer review is that peer review itself may not be well or fully understood. Ironically, lack of transparency about process flying under the banner of blindedness has hobbled our efforts to create workable, sustainable alternatives. In this post Tim Vines points to the essential work of editorial offices in managing the flow of reviews and, very crucially, arresting errors in the process and/or in the papers themselves. Tim proposed a couple of examples of this skilled labor: identifying conflict of interest among reviewers, or spotting a missing data set. The comments on this post are especially lively, include the fate of PubMed Commons, an argument that copyediting could be optional, and the role of library or institutional publishing. Angela Cochran’s comment especially struck us: “I am all for innovation, but step one is to actually know what you are trying to innovate.” Working toward more inclusive and diverse peer review depends on a full reckoning with every step of the process. We are reminded, though, of Phil Davis’s post from 2010 about the European Molecular Biology Association and other experiments in “opening the black box” of review process, which suggested significant extra costs associated with doing so.

Peer Review in the Humanities and Social Sciences: If It Ain’t Broke, Don’t Fix It?
HSS disciplines and their publication format of choice — books — are often overlooked in scholarly communications (including here on the Kitchen), in favor of STEM disciplines and journals. This post from last year’s Peer Review Week goes some way toward redressing that balance. In the form of a conversation between Mary Francis, Editorial Director at the University of Michigan Press, and Chefs Alison Mudditt and Karin Wulf — both of whom have HSS backgrounds — it provides a good overview of the differences between the review process in these disciplines, and of the (then) recently published AUP Best Practices for Peer Review handbook. Intended to: “advocate for the value of peer review to those whose support is essential: deans and provosts, foundations and funders, regents, elected representatives”, the handbook itself is another invaluable and detailed “how-to” guide, which we also highly recommend.

We hope this week’s posts along with this brief introduction to peer review on the Kitchen will inspire you to get reading, get thinking, and get sharing your own favorite posts and articles on the topic with everyone in the Comments section!

Karin Wulf

Karin Wulf

Karin Wulf is the Beatrice and Julio Mario Santo Domingo Director and Librarian at the John Carter Brown Library and Professor of History, Brown University. She is a historian with a research specialty in family, gender and politics in eighteenth-century British America and has experience in non-profit humanities publishing.

Alice Meadows

Alice Meadows

I am a Co-Founder of the MoreBrains Cooperative, a scholarly communications consultancy with a focus on open research and research infrastructure. I have many years experience of both scholarly publishing (including at Blackwell Publishing and Wiley) and research infrastructure (at ORCID and, most recently, NISO, where I was Director of Community Engagement). I’m actively involved in the information community, and served as SSP President in 2021-22. I was honored to receive the SSP Distinguished Service Award in 2018, the ALPSP Award for Contribution to Scholarly Publishing in 2016, and the ISMTE Recognition Award in 2013. I’m passionate about improving trust in scholarly communications, and about addressing inequities in our community (and beyond!). Note: The opinions expressed here are my own


3 Thoughts on "Peer Review in a Baker’s Dozen"

I am wondering how close is the the rhetoric vs the reality as to how the academics, editor thru reviewers, actually process submitted articles. At one time, the academics handled this without the external pressure imposed by the publishers to meet publishing time lines. As we know, but have not adequately addressed, obtaining a reliable stable of volunteer reviewers is often thin and getting more so as the increase in submissions have upped the pressure. Yet these “employee” in practice have almost no say in the core issues, increasingly, since the monetization of journals (Maxwell’s insights).

From being a service to the academic community, to ownership whether in fact or practice, it’s clear from this discussion of peer review standards, that the publishers are driven by maintaining credibility- impact factors, fake journals and “all that”- while there is little effort to recall their pre Maxwell relationship with the academic community at large, particularly those who moil in the editorial mines. The “service” to community argument is wearing thin as is the rationale of pricing as the default value of impact factors and related persiflage is starting to wax thin in a world where content wants to be free.

Those who work at the editorial level within the publishing industry need to understand that, yes, standards are nice and important, but for whom do they benefit when academics under pub/perish with little recourse, regardless of such standards, are driven into a production mode and the “inspectors” driven to increase timely volume outputs for an increasingly reluctant purchaser.

Somewhere there needs to be a decrease in the cognitive dissonance between the journal owners and the academic community. This would go a long way to reduce the need to pin a proclamation on the company bulletin board by management.

Tom: We should go back to those heady premonitization days when it took about 18 months from submission to publication. In 1950 there were about 10,000 journals and now there are over 70,000. I am sure there are at least 70,000 scientists willing and able to volunteer their time. I am sure there are copy editors, production folks, printers and catalogers/indexers who are willing to provide free services. In short, I am sure with a little encouragement the dissonance between journal owners and the academic community that a harmonious state will come to be.

Hi Harvey
I agree that “with a little encouragement” change might occur. I believe that the publishers need to open up the discussion to explore how this can happen. Here is an editorial in University World News regarding the proliferation of articles: It’s a good place to start since it impacts at many levels including the number of journals that can be created and marketed.

To address your “70,000”, I think this would be an interesting issue for the publishers to work on with the editors to increase reviewers, maintain reputations, and simultaneously move forward with the thesis that they want to tack to the “Church” to guarantee quality. That, of course, could include more than honorarium level of compensation and a certificate of service.

Comments are closed.