Editor’s Note: Just as each year the pumpkin spice lattés seem to appear earlier and earlier, so too the beloved autumnal tradition of Peer Review Week. The Scholarly Kitchen has, over the years, been host to a great deal of Peer Review Week content, and interest has only grown over time, to the point where we can’t fit it all into one week. And so, we will be getting a jump on things — perhaps it’s time to consider a change to “Peer Review Fortnight”. Our thanks go out to Jennifer Regala, Michael Groth, and Michael Casp for putting this post together.

This year’s Peer Review Week, September 19 – 23, 2022, focuses on Research Integrity: Creating and Supporting Trust in Research. As we look forward to an important week of celebrating peer reviewers in all disciplines across scholarly publishing, we must also consider the values and principles that are the foundation of peer review. Is research integrity woven so tightly with peer review that one cannot exist without the other? Is peer review required as a validation to prove that research has integrity? Is it possible to trust research without the peer review process? Are there solutions to ensure research integrity without the premise of peer review?

In announcing this year’s Peer Review Week (PRW) themed, Danielle Padula, Head of Marketing and Community Development at Scholastica and co-chair of the 2022 PRW steering committee noted:

The topic of research integrity has become more pressing than ever. With the proliferation of digital media outlets, we’re all tuned into constant content streams. At the same time, academia is seeing an acceleration of information sharing via preprints and expedited review processes. The upside of this is more widespread and rapid dissemination of information. But it’s also introducing challenges, particularly in terms of differentiating research that has been peer reviewed from unvetted findings and ensuring rigorous review on tighter publication timelines. This PRW is an opportunity to share innovations around promoting research integrity and look ahead to remaining challenges.

Considering the challenge of research integrity to the future of scholarly publishing and understanding the need to collaborate and innovate to reach solutions that work across disciplines, we did what we often do: We reached out to the Society for Scholarly Publishing (SSP) community directly for answers. We posed this question: “Is research integrity possible without peer review?” 

As with all “big” questions like this one, we understand that the answer is not an easy, one-size-fits-all solution. What we do know, though, is that our community is built on a willingness to learn from one another. We thank our colleagues from across the industry for their thoughtful responses to this challenging question. And we are glad for a forum where this question and the evolution of what research integrity means in the context of scholarly publishing can continue to evolve in a meaningful discussion.

Alexa Colella, Product Manager, Professional Services, Research Square Company

Okay — so, the answer to this question depends greatly on if you mean “review as we know it” or some method of critical review meant to endorse or disagree with the findings, ethics, or methodologies. Because ultimately, no, I do not think peer review (meaning how we conduct this critical review) as we know it is necessary for research integrity. I do, however, believe that research integrity is partially dependent on requiring unrelated parties with relevant expertise to advise and comment on the quality of the research so we can stand behind it as a community and learn from it going forward.  

Dana Compton, Managing Director and Publisher, American Society of Civil Engineers

When Jennifer texted me to ask, “hey, want to write a quick response to the question ‘Is research integrity possible without peer review’? it’ll be featured in a Scholarly Kitchen post next week…”, my first thought was, sure, that’s easy enough. Then I asked how long my response is supposed to be and she helpfully told me either “super short” or “longer” and just “answer off the top of my head.” Hmmm, okay. Off the top of my head?

“I guess anything is possible, but that doesn’t make it probable.”

Too flippant? 

I kept pulling the thread at this for another 24 hours, and it turns out I still mostly agree with my first answer, with a wishy-washy side of “It depends.”

If the question is, “Is research integrity possible without peer review in the form it exists today,” I’d say of course it is. Hey, I love peer review as much as the next girl, but there are countless ways it can be improved, from new methods to minimize bias to better technologies and automation. Hopefully that will come with time and money. And in the current environment, I do believe scholarly publishers are best positioned to handle the facilitation of peer review, but perhaps it won’t always be that way.

On the other hand, if the question is “Is research integrity possible without scientific peers reviewing and assessing the work of their colleagues to ensure the best possible research is put forward for publication and dissemination,” I’d say no, certainly not. To be clear, I’m not interpreting “research integrity” as limited to “free from misconduct.” If we consider the most positive results of peer review — uncovering unintended flaws or limitations in research methodology, spotting calculation or conversion errors, and so forth — I can’t think of a better check and balance than other experts in the field. Yes, sometimes flaws and errors are missed in peer review and (hopefully) caught post-publication via any number of other channels. Peer review isn’t perfect, but I’d argue that it’s an essential piece of a broader, collaborative ecosystem that serves to maintain the integrity of the research record.

I suppose I’ve been at ASCE long enough at this point that an engineering metaphor springs to mind: I can’t imagine arguing to do away with the various safety regulations and inspections for critical infrastructure such as bridges or skyscrapers. I can’t imagine suggesting that such assessments and approvals should be performed by the same individuals who created the design or managed the construction. I can’t imagine engaging government officials or business owners to conduct these reviews instead of skilled experts. Yes, occasionally flaws are missed. But for the most part (and of course with some notable exceptions), these are non-catastrophic. The process isn’t perfect. But we are all safer for it. 

Katie Duffy, Senior Director of Publications, American Society for Bone and Mineral Research

In my experience, editorial teams want to do everything they possibly can to raise the level of integrity and trust in academic publications, including trying new initiatives like screening technology or oversight positions. However, peer review is still vital to keeping this trust. We also need to recognize that the pressure on our authors and referees to work quickly and produce more and more can sometimes invite shortcuts or lapses in judgement. Peer reviewer education and strong support from editorial teams is critical to ensuring that right balance between quality and speed.  

Cindy Gruwell, Scholarly Communications Librarian, University of West Florida

While not perfect, peer review gives students and scholars some level of trust in research. 

Harrison W. Inefuku, Iowa State University, Scholarly Publishing Services Librarian

My research and expertise in scholarly publishing largely focuses on systemic racism in publishing, so my response is from that perspective. An important aspect of research integrity must be ensuring that published research isn’t grounded in racism, causing harm to communities of color. While Institutional Review Boards (IRBs) may prevent some harmful research, their scope and purpose are narrow. Peer review has not upheld research integrity in this regard — we continue to see research published that supports, promotes, and/or advocates racism and other forms of oppression. When harmful research is published, a question many ask is, “How did this pass peer review?” With the opacity of most peer review processes, this question is left unanswered. Publishers and editors, meanwhile, use the peer review process as justification for publishing the harmful research.  

Peer review has failed to prevent the publication of racist, homophobic, transphobic, ableist, and otherwise oppressive “research.” So, in this context, research integrity is possible without peer review because peer review doesn’t address this well. This does not mean that peer review has no utility in preventing publication harm to marginalized communities. Embedding anti-racist and anti-oppressive practices can make peer review, and the publishing process in general, more hospitable to marginalized communities, making it a more useful tool in upholding “research integrity.” Some first steps can include:

•       Learning about how systemic racism operates in scholarly publishing and its impact on communities and scholars of color;
•       Increasing diversity and representation within editorial boards and peer reviewer pools; and
•       Ensuring individuals who hold marginalized identities are participating in the review of research about their communities.

Finally, I acknowledge that the prompt and my response apply only to research generated in knowledge ecosystems that have been developed and shaped by whiteness. There are forms of knowledge that exist outside of these spheres, such as indigenous knowledge and oral knowledge, that do not require peer review for research integrity.  

Shaina Lange, Manager, Publishing Integrity Office, American Chemical Society

Some aspects of research integrity can be ensured by other parts of the publication process, such as clear ethical guidelines, plagiarism detection tools, and persistent identifiers to name a few; however, facilitated peer review is essential to assessing the validity and improving the quality, reproducibility, and communication of research. It enables a robust expert evaluation — ideally by a diverse set of reviewers — and collectively upholds a field’s standards while reducing the potential for a biased decision. Without peer review, we run the risk of not detecting critical issues or missing improvements that can augment the quality of a manuscript before its publication. Research integrity goes hand-in-hand with engendering public trust, and it’s hard for me to imagine achieving that same level of trust without peer review.

Avriel Licciardi, Editor Outreach Manager, Frontiers

Research integrity requires all stakeholders — authors, reviewers, editors, publishers and so on — to act ethically, fulfill expectations of their role, and meet the underlying assumptions of trust between everyone involved. As a researcher and industry professional, I view peer review as integral for upholding the credibility and integrity of the scientific record but at the same time as an incredibly fragile system; mismanagement in one individual’s court directly impacts others. Research integrity depends on ethical, fair and rigorous peer review, and effective peer review depends on academic integrity. For these reasons, looking forward, I think being open to new review models as a community and promoting educational efforts regarding peer review are incredibly important. Certain ideas may work, and others may not, but working collectively to minimize problems and concerns regarding peer review is a priority as well as a necessity for creating and supporting trust in research.

Erin McMullan, Director of Editorial & Peer Review Services, KnowledgeWorks Global Ltd.

Pre-publication peer review is an integral part of a multi-pronged approach to ensuring research integrity. While there is room for improvement in the quality of peer review, and this should be a continued goal, peer review allows qualified and impartial referees to flag concerns or raise important questions prior to acceptance and publication. Research integrity is further supported through a variety of means, for example, institutional vetting, funders’ assessments, sound ethical policies and editorial practices (e.g., COI statements, data sharing, humane treatment requirements, etc.), and post-publication review by peers and media. None of the supports listed can be seen as a singular solution to failures of research integrity; however, peer review is one of the last lines of defense. The literature would not be well served through the removal of this important checkpoint.

Lee M. Pachter, DO, Editor-in-Chief, Journal of Developmental & Behavioral Pediatrics

Integrity is possible without peer review, if you define having integrity as research that is honest, accurate, valid, and reliable. Peer review is a tool to help address integrity, but peer review in and of itself does not provide a “seal of integrity”. We all know that there’s a subjective component to peer review — editors and reviewers can bring their implicit or explicit bias to the process.  

Jennifer Pesanelli, Executive Officer, Biophysical Society

I would love to say, “Yes!” Yes, research integrity is possible without peer review because we can rely on all researchers to pursue and practice scientific excellence and to be honest, objective, and transparent in all the work that they do. But I must say, “Unfortunately, no.” Researchers are flawed human beings, like all of us, and whether it is the pressure to perform, get ahead, get that publication, get the grant, etc., or a different reason, there are those who will select a path other than the one that values integrity above all else. Having experts in the field assess a researcher’s work helps ensure not only quality and validity of the research, but peer review also increases accountability which helps support a culture of research integrity.

Donald Samulack, PhD, Head, Global Stakeholder Engagement, Cactus Communications

Heck, ensuring research integrity is a struggle, even with peer preview. Peer review is one of the last safeguards in the publication process – but it is only one of them. It may not be perfect, but the “real” responsibility for research integrity rests with the researchers themselves. Unfortunately, time and again we are shown that a subset of researchers choose not to abide by the ethical norms that the rest strive to develop and adhere to. Hence, research integrity is more about the actions of the researcher, than any safeguards we try to put into place. Bad players need to be called-out with greater severity and frequency in order to curb unethical publication practices.

John W. Warren, Director and Associate Professor, MPS in Publishing Program, George Washington University

Let’s start by acknowledging that, indubitably, entire dissertations have been framed around this question. I’ll also take an expansive view of what “research integrity” and “possible” signify. In my view, peer review is an important component of the research process, designed to improve research and research integrity; peer review, however, is not a panacea nor is it infallible. I can think of excellent examples of popular trade books such as I Contain Multitudes by Ed Yong, or The Invention of Nature by Andrea Wulf, which are, at least in part, research-based and did not entail a peer review process. Peer review, however, is an essential part of the process for university presses as they seek to fulfill their mission of advancing and disseminating scholarship. One can, of course, cite numerous examples of both trade and scholarly books and articles that are lacking in research integrity. Research integrity may be possible without peer review, but it would be problematic to depend on a long-term, sustainable, and reliable program of research and publication that lacked a robust process of peer review. Peer review takes many forms; it can and must be improved; publishers of books and articles are experimenting with different forms of peer review, hopeful that they might improve research integrity and inclusion. One recalls Winston Churchill’s 1947 quote about democracy: “Indeed it has been said that democracy is the worst form of Government except for all those other forms that have been tried from time to time…”

Gwen Weerts, Journals Manager and Editor-in-Chief of Photonics Focus, SPIE

Peer review has historically taken different forms. It began centuries ago as letters between scientists, before moving onto a system of scientific gatekeeping by a journal editor. It wasn’t until fairly recently that peer review evolved into the more formal systems of double- and single-anonymized review that we recognize today. Whatever the form, the purpose of peer review is, and has always been, to be held accountable to objectivity, facts, and scientific method. I don’t see a way to do that without a system of feedback from the scientific community.

Susan Willner, Publications Manager, American Society of Nephrology

I would argue that publishing integrity does not exist without peer review.  Good peer reviewers are subject matter experts that can validate the quality of the research and the presentation of the information. Without such substantiation, then substandard research seeps into the public discourse.

Laura Wilson, Head of Research Integrity & Ethics, Taylor & Francis

Research integrity means conducting research in such a way that allows others to have confidence and trust in the methods and the findings of the research. Certainly, when it comes to publishing the outcomes of the research, peer review forms one key component of this. In our view this is the cornerstone of academic publishing, because it fosters trust by crucially having independent reviewers assess the validity of an article. As a result, poor quality or fraudulent work is more likely to be filtered out and their reviews play a core function in helping to improve an article. This is the true in all guises of peer review, be it double-anonymous to fully open.

Paige Wooden, Senior Program Manager, American Geophysical Union

Is research integrity possible without peer review?

No. Research integrity — what’s right and wrong for scientific advancement — is not possible without social and cultural norms. And we cannot have a shared understanding of social and cultural norms without others — peers and figures of authority. They teach integrity by expressing expectations and critiquing actions. What’s right and wrong in scientific research — “research integrity” — is taught to young scientists not only through the scientific method but also through reprimanding errant behavior and rewarding desirable behavior and outcomes. As scientists advance along their professional paths, they continue to learn what is expected of them to succeed, and they integrate these expectations into their actions. Review, critique, praise, and requirements from peers, mentors, instructors, and institutions shape the way they do their science and share their results. For article publication, co-authors and non-co-author peers and mentors also play a major part in shaping formal output ahead of submission to a journal. Formal peer review by two-ish anonymous reviewers on behalf of a publisher immediately ahead of publication is only one piece of, and not always necessary for, the development of the scientific record with integrity.

Jennifer Workman, Senior Manager, J&J Editorial

Research integrity and peer review have a symbiotic relationship. It is difficult to imagine one without the other. The results of research are primarily communicated via peer reviewed journals. There is an expectation that research published in peer reviewed journals has undergone the analysis of experts in the research community who scrutinize the objectives, the methods, the results, the novelty, and impact of research prior to publication. Because researchers are expected to conduct responsible and ethical research, peer review is an important method by which research integrity is upheld and kept in check. This check and balance system allows peer reviewed literature to directly influence the progress of science in immediate and important ways. On a basic level, scientific work that has undergone rigorous peer review, directly advances scientific fields. It affects new developments and decisions regarding everything from public health and clinical treatments to machine learning and new technology. When new research is in the planning stages, a review of existing literature from peer reviewed sources is examined and taken into consideration as researchers seek to make additional advancements. In this regard, research and peer review share a connection that keeps the communication of science moving forward. Is the peer review process perfect and without bias or ethical concerns? Not at all! There is always room for progress and the need for improvements to be made. Organizations such as COPE, CSE, SSP, and ICMJE are a few examples of important resources for providing guidelines and best practice recommendations that help safeguard the peer review process. Such organizations also work to address issues with bias, DEIA advancements, and other ethical considerations. In other words, the peer review process and research integrity are inherently linked, and both share a scientific community that is advancing and continually evolving! 

Suzanne Cholwek, Peer Review Coordinator, American Society for Pharmacology and Experimental Therapeutics

Gosh! Thanks for asking. I have many thoughts about this, but my gut feeling is that with science, it takes a village. I know a lot of folks think it’s cumbersome, but I think it’s absolutly imperative to have expert eyes on one’s research to help ensure it’s as robust and and solid as it can be. I realize experts aren’t infallible, but when multiple experts review a paper, in my opinion, their comments offer authors an opportunity for improvement, which benefits them and all other stakeholders. There’s so much misinformation out there these days, we all benefit from having a discerning eye on research. Again… it takes a village. Integrity is vital!