As we do every year, we like to warm you up for Peer Review Week (PRW), next week (20-24 September). This year’s theme is Exploring the multifaceted role of identity in peer review.

So we asked the Chefs: How does identity influence peer review?

Charlie Rapple: How does identity influence peer review? It doesn’t. In an anonymous peer review scenario, at least. And…in theory. In practice, anonymity creates a vacuum ready to be filled by bias. In theory, I don’t know anything about the authors of the work I’m reviewing. In practice, we make all sorts of assumptions about authors based on nuances in their manuscript.

For example, I have an interest in how patterns of speech, grammatical constructs, or even just vocabulary choices reflect different language groups — so I’ll be gaining a sense of where an author comes from based on the writing style (on a related note, see Amer Abukhalaf’s recent Research Information article on linguistic monopoly). Sometimes the manuscript formatting will further influence that, for example a font might suggest an author is from Japan, or a punctuation mark might indicate a native Spanish speaker. Monetary units are a real giveaway. I’m not actively playing sleuth, but I struggle to switch off the part of my brain that is interested in and thus registers these things.

In theory, I don’t know anything about the authors of the work I’m reviewing. In practice, we make all sorts of assumptions about authors based on nuances in their manuscript.

The instinct that I’m working with an author who may be from a disadvantaged group – whether in terms of language, research culture, career stage, etc. – then leads me to try and be extra supportive, from offering up some language polishing, through providing more detailed feedback, even to allowing greater leeway (e.g., in terms of ’is this really novel / significant’) out of an attempt to practice what I preach in terms of diversity, equity and inclusivity.

Let me be very clear: I’m not saying all manuscripts by disadvantaged author groups need any kind of leeway or extra support. And I do see that my ’extra support’ may be misplaced, patrician and / or discriminatory (is it helpful or counterproductive to give ‘leeway’? is that fair to other authors I review? etc.) So, to say identity does not influence peer review is to ignore the risk of how we subconsciously ‘fill in the blanks’ around authors’ anonymity. We need to be alert to that risk to ensure peer review is consistent and fair.

We need to be alert to that risk to ensure peer review is consistent and fair.

Tim Vines: Identity is the fulcrum of peer review. By obscuring it with anonymization, we promote one set of peer review behaviors, and by exposing it we promote another. Anonymization reduces the risk of author retaliation, and therefore promotes franker and more objective comments from reviewers. Some worry that anonymous reviewers are nastier and less conscientious. Reviewers whose identities are shared with the authors are less likely to agree to review but seem to produce comments that are more appreciated by the authors. Double-anonymized peer review may hide author Conflicts of Interest that would otherwise be obvious to the reviewers. Double-anonymized may also prevent reviewers from discriminating against junior researchers or those from traditionally marginalized groups, or from being overly timid in their comments to very senior researchers.

Vouching for identity is perhaps the most important – and most neglected – role of the Editor in single- or double-anonymized peer review: the authors do not know who the reviewers are, so the Editor must vouch for their ability to provide useful feedback on the manuscript. This role manifests in a common starting sentence for decision letters “Your article has been assessed by three expert reviewers, and their comments appear below…”

Vouching for identity is perhaps the most important – and most neglected – role of the Editor in single- or double-anonymized peer review

Haseeb Irfanullah: When we talk about identity in peer review, authors and their peer reviewers quickly come to our mind. But if we think a bit deeper, editors’ identity also matters, so is the persona of journals and their publishers, which all are connected.

Authors, peer reviewers, and editors of a journal belong to a common large group as they all are researchers of a specific or interconnected disciplines but are playing different roles in different times in different contexts. I see peer review as “good karma” — I am voluntarily doing it for some researchers even without knowing their identity, believing that someone else will be doing the same favor to me, throughout my career, where I am an author.

That’s why, even if we are not familiar with a journal, if the publisher is reputable or at least seems authentic, we would accept the invitation to review its manuscripts. But, journals and their editors who don’t make a minimum effort to address us by our names, instead calling us just “colleague” or “sir/madam” or fail to leave a human name as they sign off an email, indicate that they are randomly casting a net to “catch” our opinions on a manuscript; if we don’t respond, “there are plenty of fish in the sea”.

Besides expecting due respect, we would also prefer to spend our time, energy, and expertise on something worth mentioning. Because, by associating ourselves with a journal / publisher as peer reviewers, we are also defining our individual identities. We are used to mentioning such voluntary peer-review engagements in our resumés and online profiles, but now initiatives, like Publons, help us to build that persona more visibly, authentically, and comprehensively.

Our research article itself — the way we write it, pose our arguments, and the language we use — represents our identity as authors.

Our identity is not only expressed by our names, affiliations, or whom we thank in the acknowledgements section of a journal article. Our research article itself — the way we write it, pose our arguments, and the language we use — represents our identity as authors. Similarly, our unnamed reviewers’ persona would also be defined by their comments — focusing more on authors’ language skills or attacking authors’ opinions or just saying “all is good.” Or, by their comments showing compassion and honestly guiding authors to improve a piece. I was amazed (some may say “lucky”) to meet several such excellent, nameless peer reviewers during the past year or so as my colleagues and I got four scholarly articles published or accepted.

I finish with a story I heard a few years back from a young researcher, who submitted a single-author manuscript to a society journal of his country. A very senior colleague of his department, a well-known expert of his discipline, received the manuscript to review (double-anonymized peer review). Being a very busy person, the senior researcher gave that manuscript to a junior colleague to write reviewer’s comments on his behalf. And that junior researcher happened to be the author of that manuscript! What happened next, I leave it to your imagination. But this story tells us, despite being huge, our scholarly publishing system is made up of small villages of expertise. Do we really know the neighbors of our village?

Alice Meadows: Identity influences many aspects of scholarly communications – and indeed life – but it presents some unique challenges and opportunities in the peer review process. Much current peer review is partly or fully anonymized — the identities of one or more of the parties involved are unknown to one or more of the other parties. The intention is to ensure a more neutral and, therefore, fairer process, minimizing the risk of bias. Ironically, however, anonymity also makes it extremely difficult to determine how much of an impact identity does or doesn’t have on peer review, because it means we aren’t routinely able to collect the information needed to do so.

Ironically, however, anonymity also makes it extremely difficult to determine how much of an impact identity does or doesn’t have on peer review, because it means we aren’t routinely able to collect the information needed to do so.

There are also legitimate questions to be asked about how anonymous peer review is in reality.  For example, even when an author’s identity is hidden, the reviewer may be able to easily identify them and vice versa — especially in smaller fields or when the author and reviewer are known to each other. This study found that about 13% of reviewers knew or suspected they knew the author(s). And even if authors and/or reviewers don’t know each other’s identities, in almost all cases the editor knows everyone involved in the process. There’s not much research that I know of into editorial bias from the perspective of peer review (if you know of any, please share it in the comments) but, based on research into bias in editorial board composition, it’s very likely to exist.

And what about those times when it’s helpful to know who an author and/or reviewer is during the peer review process, for example, when their identity is a core aspect of their research expertise?

So, what’s the answer? For me, there are some real benefits to open peer review as a way of both increasing transparency (of reviews and reviewers) and enabling recognition for reviewers. Knowing that your review is going to be publicly available is also a good incentive to keep it courteous and constructive! Because, while anonymity can certainly enable bad behavior, it can equally – perhaps more so – encourage good behavior.

…there are some real benefits to open peer review as a way of both increasing transparency (of reviews and reviewers) and enabling recognition for reviewers.

Jasmine Wallace: Identity has slammed into peer review like a Mack truck, forcefully making its way into processes and procedures, and overturning decades-old established policies!

Peer review has always been about who the community of researchers is and how to make the process or review work best. Be it by understanding whether an overburdened reviewer has enough time to give a robust, fair, and timely review or by verifying that an editor has no conflicts of interest with an author prior to manuscript assessment. But never has peer review demanded such a deep dive into the whole person paradigm of its community members. No longer is it just enough to go back to the corresponding author for byline changes after a revision, rather ensuring that all the authors are aware of the addition, or, removal of another co-author is necessary.

But never has peer review demanded such a deep dive into the whole person paradigm of its community members.

We must also make certain that each author is given fair credit for their contribution; and, guarantee that if their personal identity yields a new professional identity, we make it easy to update the scientific record. If personal identifiers were not enough to shake up peer review prior to the pandemic, the social justice movements took transparency and trust to the next level. Things like Black Lives Matter brought identity to the forefront of organizational strategic initiatives, which expedited diversity, equity, and inclusion efforts (belonging + accountability + access, and the list continues to grow). Publishers have had to quickly and intentionally make room in their plans to account for whether or not they themselves reflect their DEI efforts. For many societies, this has resulted in an update to their organizational identity. It has meant being conscious of whether their membership is a reflection of their stated identity; including the vendors and external stakeholders they do business with.

While identity has created new challenges for the publisher to consider about its constituents, it has also created new opportunities. As gatekeepers, publishers are very experienced in controlling access. Due to that historical role, publishers have prepared and are able to discover, remove and destroy barriers created by their business models. They can help establish and develop the space(s) where we see people as “whole people” — body, heart, mind, and spirit — and manage and lead accordingly, and in so doing recognize that identity is a critical part of sustainability. Valuing identity has begun to transform our community into a space where the people we are, and those we serve, want to not just stay and grow, but are also given room to be the best version of themselves.

Valuing identity has begun to transform our community into a space where the people we are, and those we serve, want to not just stay and grow but are also given room to be the best version of themselves.

Robert Harington: For me this is all about implicit bias. We are all subject to this kind of bias, and we necessarily need to recognize how entwined implicit bias is with identity. We also need to recognize that inclusiveness and identity are not one dimensional. We have to act positively to ensure representation from underrepresented groups in all aspects of society governance, our journal editorial boards, our book authors, and of course our internal employee base and operations.

Ensuring that we act to be inclusive is necessary, and when we consider identity, we are talking about the many ways that individuals may identify. We also need to understand that our biases extend to regions of the world, whether you work at an R1, or R2 institution, or a community college, or the way your name is spelt, etc. Essentially, we must recognize our own potential for unconscious bias and act positively to be inclusive. For me this means working on specific projects, such as deploying double-anonymized peer review, a policy recently approved by the American Mathematical Society for forthcoming implementation, or a project enabling author name changes in the digital record.

Essentially, we must recognize our own potential for unconscious bias and act positively to be inclusive.

Judy Luther: Perception and identity are closely intertwined, so both came to mind when I saw this topic. Identity is how we see ourselves in relation to others, whereas perception is the lens through which we experience the world, and it frames how we relate to each other. Being aware of both is the key to discovery in terms of our respective roles within our organizations and communities. Both are relevant to Peer Review which is fraught with critiques, which at their best are constructive and insightful, and at their worst are insulting and damaging. We are challenged today as we confront decades of ‘norms’ that fall short of how we might envision a fairer world.

So, what is our role at this point in time?

We are challenged today as we confront decades of ‘norms’ that fall short of how we might envision a fairer world.

Becoming aware of our blind spots is a good place to start. It is a learning curve. We can begin with the assumption that others will see things that we do not. So, I listen for the gap – what others contribute that is new or less familiar. And then approach that point with curiosity and seek to understand the context that could produce that result. This approach was useful when an editor Bcc’ed me as one of the reviewers when replying to the author. It felt as though I was in the room with someone who had a different perspective based on different experiences

Some societies embrace their role of preparing the next generation with a program for mentoring young scholars in peer review. This can be incredibly useful if done well. The Society for Scholarly Publishing (SSP) has a mentoring program that matches more experienced members with those newer to scholarly publishing. I could see value in a series of conversations facilitated by an experienced moderator who uses a construct with diverse individuals trained in providing feedback in thought provoking way. Many people who avoid confrontation miss the opportunity for discovery as they have yet to develop the skills that enable them to be honest without being disagreeable or aggressive. Thoughtful actions are better than inaction and moving forward can advance the conversation for the benefit of all involved.

__________________________________________________________

Now it’s your turn. How do you feel identity influences peer review?

…and don’t forget to check out the activities that have been lined up for Peer Review Week next week!