Welcome to Peer Review Week 2018! This year’s theme is “Diversity in Peer Review.” There are a lot of perspectives and resources on peer review addressing aspects of diversity, and many of them are highlighted in this week’s full slate of posts.
But first we’d like to step back and take a look at the big picture. The benefits of diversity — in peer review as in most activities — are well documented (over 600,000 results on Google Scholar!). What, though, are the environmental requirements for a diverse peer review ecosystem? Because the metaphor of the ecosystem is so regularly employed to describe scholarly communications, an interconnected and interdependent set of organizational types, perhaps we can learn from another environmental science concept: biodiversity. Biodiversity recognizes the ways that diversity is essential to the interconnectedness and interdependence that characterizes environmental health.
Likewise, what are the connections and interdependencies around peer review in the scholarly ecosystem? And what does that ecosystem need, both structurally and culturally, in order to facilitate inclusive and diverse peer review?
It’s important to note that, although peer review has the common goals of providing feedback on research in advance of publication, and guidance to editors in making decisions about publication, it is itself a variable activity within the incredible diversity of scholarly publishing contributions. From commercial science journal publishing to university press humanities monograph publishing, and much more beyond the spectrum those two poles suggest, the range of “peer review” expectations and delivery can be substantially different.
Given this variability, and granted the incredible number of (creative and significant) efforts to improve peer review in different ways, the single most important requirement for diversity in peer review has to be transparency. At the 30,000 foot level, transparency makes clear the connections and dependencies within peer review — and scholarly communications more broadly — enabling not just a more diverse and inclusive ecosystem, but also one that is ethical. At the ground level, transparency provides clarity around roles and responsibilities, and allows those roles to be recognized.
Because transparency was the theme of last year’s Peer Review Week we have already explored this issue from a variety of perspectives. Transparency is often invoked in the context of closed versus open review, where“closed” review usually refers to review in which one or both sides of the review process are anonymous; and “open” usually refers to processes in which authors and reviews are known to one another, but also may refer to online pre-publication reviews that are subsequently released to all readers of the publications. In a post previewing a panel on “Transparency in Peer Review” at the 2017 Peer Review Congress, the speakers considered various versions of “open” review, and the value of sharing information as a means of cultivating trust. Interestingly,,according to a new survey from Publons, the “Publons Global Reviewer Survey”, we may be witnessing a shift toward support for open peer review by early career researchers, 40% of whom are likely/highly likely to review for journals that make author and reviewer identities, as well as review reports, openly available. This compares with just 22.3% of older respondents. This survey, about which we’ll be hearing more from the Publons team later this week, captured responses from more than 11,000 reviewers and offers important and timely insights.
But in this post, we’re thinking about transparency as playing a broader, environmental role in peer review processes of all types — open and closed; in journal or book publications, grant applications, conference submissions, hiring and promotion committees, and more.
Openness is indeed a form of transparency, but it’s not the only one. In our post, “Peer Review in a World of Alternative Facts,” we argued that revealing the processes behind scholarship, such as peer review, is essential to knowledge creation and to democratic values. There is a significant analogue to how and why transparency is the key to achieving diversity in peer review: it is the responsible representation of process to the community and constituency. As David Crotty noted in a comment on one of last year’s Peer Review Week posts, variation in terminology is occluding peer review; clarifying terms is a contribution to transparency.
Transparency — like diversity and inclusion — is an ethical, as well as a pragmatic, position. It acknowledges the biases that are inherent in so much of institutional culture, history, and practice and, therefore enables us to find ways of identifying and addressing those biases in the peer review process. It encourages organizational introspection, and the alignment of values with practices, providing us with the information we need to ensure that we are practicing what we preach. It empowers all participants by making the process — and each step in the process — accessible and available, minimizing the risk of bias and also making it easier to spot and root out.
Transparency insists on clarity of roles and responsibilities. This includes training — whether for early career researchers or for editors and other publishing staff involved in peer reviewer selection and review mediation. No one can do a good job without a clear understanding of what, why, and how they’re meant to be doing it. Ensuring that training is available to all, irrespective of career stage, geography, discipline, organization type, gender, and more is critical to ensuring diversity in peer review. And clearly defined roles and responsibilities help authors and readers too — they have a right to know how reviewers are selected and what, exactly, they are being asked to evaluate. Transparency also allows for a fuller understanding of the labor involved in peer review — on the part of the reviewer, the editor, and also the other staff who manage the process. Whether one agrees with Tim Vines that there has been “A Curious Blindness” in peer review experiments and initiatives, it’s not hard to see that a full appreciation of the way that peer review is managed benefits everyone involved.
Transparency allows for full credit to the participants in peer review — both individually and collectively. At the individual level, this does not mean the revelation of each reviewer for each paper or essay reviewed, but rather a system whereby reviewers’ labors are openly recognized by the publication and can be claimed by the reviewer. More than a credential awarded for participation as a review, transparency allows reviewers to accrue a vital benefit: participation in a community of researchers. Collectively, it is also critical that organizations understand — and support — the amount of time their researchers spend on peer review activities of every sort. The Publons survey estimates that 13.7 million reviews were provided in 2017 for journal articles alone. Add to that the many other types of review contributions made by researchers, and you can understand why the survey also found that 84.8% of respondents believe that these contributions should be both required and recognized by their institutions; and that 83% believe that greater recognition and career incentives for peer review would have a positive effect.
It’s also clear that the opportunity to participate in the peer review process — and gain this recognition — must be equally, and transparently, available to all. This is not the case at present. Per the Publons survey, reviewers are more likely to be from established regions (68.1% of reviewers, 96.1% of editors!) and (probably) male (45% versus 22% female — the remainder are unknown), giving this demographic an unfair advantage. However, it is also resulting in researchers from established regions taking longer to complete their review (16.4 days compared with 15.1 for reviewers from emerging regions) — and being less likely to complete the review at all (49.5% compared with 56.6% completed). Improved clarity at every level, with a goal of diversity and inclusion, would benefit all — increasing the reviewer pool, and spreading the burden as well as creating a level playing field.
Increasing the “biodiversity” of the peer review ecosystem is essential to ensuring its future health. Achieving this requires shining a light on the lack of diversity and inclusion in peer review today, and using all the tools at our disposal — data gathering and analysis, training and other forms of institutional support, improved opportunities for credit and recognition — to identify and address bias, and to create the inclusive, diverse peer review we want and need. Transparency in peer review, as in so many aspects of scholarly communications, is the key.
