Ask a researcher what matters most to them in their work, and effective peer review will always be among their top three. At the recent STM conference in Philadelphia, Judy Verses, Senior Vice President, Research at Wiley, clearly articulated a vision for publishers. She focused on the need for publishers to recognize who we are in business to serve. She exhorted us to consider the researcher as our “North Star”, and that everything we do be directed to serve their needs. Publishers recognize that peer review is a paramount concern for researchers, and yet have not addressed some of the key concerns that authors and reviewers face. In this post, I suggest that publishers need to do more for researchers to help authors, and to help reviewers understand their role as a reviewer and be recognized for their work. We need to focus on our “North Star”.
Perhaps a good place to start is to ask the question, why is peer review so important to a researcher? Peer review is a key mechanism for ensuring rigor and establishing community standards on quality. Peer review in itself does not merely exist to filter good papers from bad. Peer review is a valuable service researchers provide on behalf of other researchers that allows for possible improvement of a paper. Peer review that leads to rejection is still valuable for the insights provided to an author. The ecosystem of review, in other words, allows the community to discuss research, and depending on the model (something we will get into later), allows for opinions to be shared without judgement or fear of retribution. A reviewer who participates in peer review gains mightily from this ecosystem, participating in the evolution of the research itself, playing role in encouraging the career path of their colleagues, and stimulating their own development in their field.
A reviewer will tell you though that their role can sometimes be quite confusing, and — depending on the expectations put upon a reviewer and how their work is treated — may decide not to participate in the review process. Publishers can do better here. Journals vary in their expectations for a reviewer. A journal that is perceived to be of high quality will likely expect its reviewers to take a much tougher approach than a journal that is looking for good research, but is not as selective. Publishers and Journal editors would do well to think about how to guide their reviewers. A reviewer may well take an entirely different approach depending on the level of review required. What would be wonderful is if a journal articulated expectations to reviewers, perhaps even providing a series of parameters and types of question a reviewer could tackle when acting as a reviewer for their journal.
A reviewer is often a silent, anonymous actor in a journal’s ecosystem, so — apart from an internal sense of growth for a reviewer — it is important for publishers to find ways to overtly involve and reward their reviewers. Some publishers do this of course, providing lists of top reviewers, or badges that a researcher can apply to their emails identifying them as an active reviewer for a publisher, or journal. Scholarly societies also have a role to play here. Part of society life is the interaction with others in your field – a chance to participate in a wider community and encourage the career development of those around you. This societal aspect of peer review has largely been ignored by scholarly societies, even in a time of aging and declining memberships. Perhaps this should be a key area of focus for scholarly societies looking to engage their communities as active members and contributors to the field.
The elephant in the room when considering peer review is the issue of implicit bias. It is quite clear that there is bias ingrained in the system. I doubt anyone could argue that the issue of bias is not real.
A recent study highlights how deeply ingrained gender bias is in peer review: The American Geophysical Union (AGU) publishes around 20 journals and 600 papers per year, with a membership of 60,000. Their study (by Lerback and Hanson, Nature 541, 455-457, 26th January 2017) used self-reported age and gender data from 2013 to 2015, as well as data associated with anyone who had an AGU account since 2011. AGU combined this data with publishing data to produce age and gender data for 25,000 authors. 27% of published first authors were women, and yet only 20% of reviewers were women.
An author in an emerging nation may have a harder time getting their work accepted. An author from an R1 institution may benefit from assumptions of quality over someone from a less well-funded institution. We look at the gender of an author, and know that in an era where editorial boards are still mainly male dominated, there will be implicit bias at work.
There are a number of problems to solve here, boiling down to issues of trust in the ecosystem. If an author trusts the peer review model, then that author will be well served, and indeed likely to act as a good reviewer when the time comes. If a reviewer understands there is equity in the system and expectations are set accordingly, the quality of reviews will more likely be consistently high.
Much rides on the model of peer review being used – and there are quite a few. Broadly speaking, the humanities and social sciences use double blind peer review, and the sciences use single blind peer review. Then there are a plethora of attempts to open the review process to varying degrees.
At the American Mathematical Society (AMS), for example, we use single blind peer review and have done so for as long as I can find records. Single blind review means that the reviewer’s identity is anonymous, but the author’s name and affiliation is transparent. It is probably about time we have a discussion about implicit bias in peer review in the single blinded model. We have not yet taken this on within mathematics, but I imagine there will be a range of opinions. On the one hand, there are many who say that the system is fine, and that in a field such as ours, you know who the authors are, not just from the approach taken, but from the iterative reference lists that define the progress of a mathematical field. One can add to this argument, noting that much content published in a math journal exists already in a preliminary form from the author, on the arXiv preprint server. On the other hand, bias is at work, and so to ignore the problem is clearly not a good solution.
Double blind peer review is standard practice in the social sciences. Double blind review means that the reviewer does not know the identity of the author, and the author does not know the indent of the reviewer. From the point of view of implicit bias, double blind peer review appears to have more to offer the research community than single blind, even providing for the sort of concerns I articulate when considering a field like mathematics. The discussion at least need to be had rather than dismissed out of hand.
One society publishing organization that is embracing a discussion of double blind peer review is the Institute of Physics Publishing (IOPP) in the UK. IOPP launched their double blind pilot in January 2017, running through December of 2017.They offered an option of double blinded peer review to authors of two journals, Materials Research Express, and Biomedical Physics & Engineering Express. Results from this study were highly positive. IOPP reports an author uptake of the double blind option of around 20% on each journal, with highly positive feedback from both authors and reviewers, with comments indicating that double blind review was seen as being fairer than single blind review. Simon Harris, Managing Editor at IOPP says,
We have seen significant uptake from authors of the double-blind option in our pilot, and received very positive feedback from authors too, most of whom saw it as fairer than single-blind review. We believe that the results of the pilot prove there is a demand to be met for double-blind review in these research communities. The double-blind offering has also worked well from an operational point of view.
It appears that this option will now form a permanent offering for these two journals, with the option being extended to other selected journals in the IOPP portfolio.
I recently sat around a dinner table discussing peer review with colleagues from a range of publishers – both society, corporate and independent non-profit. I was surprised at the pushback I received when I suggested we should consider double blind peer review. Part of this push back is to be understood in terms of the wider publishing landscape of openness. My colleagues were not advocating for open peer review, but rather for a model that combines elements of the blinded review with openness – termed transparent review. Essentially, a review from an anonymous reviewer is published alongside the paper as part of the journal published record, sometimes along with author responses, and the decision letter. Two examples come to mind of journals currently deploying transparent peer review: the EMBO Journal and Nature Communications. My Scholarly Kitchen Chef colleague Alice Meadows posted an excellent article last year on this topic entitled, “What does transparent peer review mean, and why is it important?” One of the arguments I like about transparent peer review is that by publishing anonymous reviews alongside the articles themselves, it is clear whether or not the journal in question is a reputable, or predatory. The sheer number of predatory open journals, and the spamming of the research community is one of the real problems researchers face across all disciplines.
Transparent peer review is not synonymous with open peer review. Open peer review means the identity of the author and reviewer are publicly revealed, and more often than not published alongside the article openly. Advocates of increased openness in research gravitate towards this model, and indeed open access journal such as Springer Nature’s BMC journals have been using this model across their portfolio for some time. Open peer review has its problems. Some researchers will tell you that if they are asked to do a review anonymously, they will be much more likely to participate in the review process, not fearing retribution, and being able to express opinion freely – an important part of the review process. Some may say that a reviewer should take the same approach, whether anonymous, or open, but this ignores a psychological reality, that anonymity is important for allowing free expression without fear of judgement. This is especially likely to affect early career researchers who may well just decline to review.
A path some are taking is to combine elements of peer review models. The Royal Society, for example, encourages reviewers and authors to identify themselves, but will abide by the wishes of authors and reviewers and will keep both anonymous as they publish both the article, and review in CC-BY form.
I am not offering answers. I do think that publishers need to engage in a discussion about peer review with their “North Star” — the author – who is often, at the same time, a reviewer. We need to understand what service we as publishers and societies may provide to researchers to ensure that the role of reviewer is lauded, encouraged and respected as perhaps the most valuable part of the publishing ecosystem.
Editor’s Note: This post originally included an inappropriate image, which has been removed and replaced. This was a failure in editorial diligence, and we apologize to our readers. Thanks go out to those readers that brought it to our attention, we pledge to do better in the future.