Editor’s Note: Continuing our runup to Peer Review Week, today’s post has Jennifer Regala bringing in the thoughts of D. Robert Siemens to respond to the questions we posed to both the Chefs and the scholarly communications community. Jennifer is the Director of Publications/Executive Editor at the American Urological Association. Dr. Siemens is is Professor and former Chair of the Department of Urology at Queen’s University and is Director of the Center for Applied Urological Research.
One of the best parts of my career is working with brilliant editorial leaders in widely ranging areas of expertise, from the humanities to the life sciences to medicine and more, and assisting them in shaping policies and impact of content. Another favorite part of my work is volunteering in our community. Working with the smartest people in scholarly publishing is a privilege and a joy. Bonus points if I can find a way to connect those two passions!
Enter Dr. Robert Siemens, Editor-in-Chief of The Journal of UrologyⓇ (JU), the flagship journal of the American Urological Association. Selected for this role in fall 2020, Dr. SIemens spent a year training with our outgoing Editor, Dr. Joseph Smith, before he assumed the keys to the JU car on January 1, 2022.
In that short time, Dr. Siemens has worked with Dr. Smith to adopt open peer review for JU. As of November 1, 2021, all peer review reports for accepted manuscripts appear alongside the published articles. We also spend a lot of time considering peer review recognition and compensation.
I asked Dr. Siemens to respond to the question we asked the community and the Chefs: Is Research Integrity Possible without Peer Review? When I received his response, it was clear to me that it needed to stand alone as a unique perspective we always value from where we sit in our community: What does the RESEARCHER think? Thank you to the Chefs for allowing this glimpse into JU and our editorial team, where we’ve been, and where we hope to go. And thank you especially for giving Dr. SIemens, already a world-renowned medical leader, a chance to become “Scholarly Kitchen Famous”. His response is below:
Thank you for asking me to relay my thoughts about this question, “Is research integrity possible without peer review?”. At first glance, it seems like a fairly straightforward answer — of course not!
Peer review is a pillar of our academic life. It’s a requirement for evaluating everything from grant applications, scholarly publication, and tenure or promotion decisions. If a colleague in a different specialty at our university is promoted to full professor, one could be reassured they were ultimately qualified because they went through the same process of external peer review as everyone else. We are certain the decisions were not spurious or based on favor by a friendly Department Head or Dean. Peer review would hopefully guarantee that any explicit or implicit bias is negated when assessing that tenure application. As our ability to produce new knowledge explodes, it is impossible for a handful of editors alone to make informed decisions about its legitimacy. We need multiple assessments by content experts on manuscripts to search out major flaws and look for honest mistakes as well as more deceitful or misleading behavior. Further, as a clinician researcher, if we experience a series of bad outcomes, it would be common for our practice to be reviewed by peers to assess potential gaps in knowledge or skills. Given its ubiquity, how could peer review not be a central tenet of academic integrity?
But is it necessarily true that there is no research integrity without our contemporary understanding of peer review? Peer review, as we have come to understand it, is a relatively recent adjunct to the 350-year history of scientific publication. As you know, several preeminent journals did not rely on a systematic process of external refereeing until well into the 1970s. Has this recent innovation of requesting the opinion of other peers (read: other researchers that have recently published on the subject at hand) assured accuracy of reporting and reduced academic dishonesty? We have all read grant review feedback from external readers that have completely missed the point of your novel research idea as it seems to fly in the face of what others have published in the past.
As I think about the question, I keep coming back to how we would assess such a question in health services research. For example, does a certain surgical process or outcome (e.g., blood transfusion rates) have validity as a quality indicator of that surgical care? Can we rely on it as a tool to measure and benchmark individual surgeons or hospitals? I think peer review has excellent face validity. On the surface, peer review is likely seen reflexively, within and outside of the research community, as sacrosanct. But just because a paper is peer reviewed does not mean that it is correct or without flaws. Peer review tends to have generally low levels of inter-rater reliability, making it, statistically speaking at least, relatively capricious. If peer review was highly sensitive to error or fraud, Retraction Watch would not exist. Poor quality and unprofessional peer review is a constant headache with little ability to provide quality improvement. Although uncommon, examples of more nefarious bias and deceit by conflicted peer reviewers have highlighted its limitations as a reliable tool. Therefore, I doubt we could be confident that peer review has exceptional content or construct validity.
Nevertheless, peer review has afforded a level of trust towards the scientific community. Both the funders of research and the general public rely on the peer-reviewed “stamp” as a measure of quality and integrity of the observations reported and the research teams involved. Although there are times that today’s peer review is simply conditional, and maybe even unnecessary, it is undoubtedly synonymous with the concept of research integrity in most people’s minds. For those reasons alone, it will be an ongoing requirement for our research endeavors.
1 Thought on "Ask an Editor-in-Chief/Surgeon/Researcher/Author/Reviewer: Is Research Integrity Possible without Peer Review?"
I just wanted to leave a note about automated peer review, or at least something between peer review and editorial checks.
I completely agree with this sentiment “Peer review tends to have generally low levels of inter-rater reliability, making it, statistically speaking at least, relatively capricious.” I also believe, along with several of my colleagues, that automated tools can provide some level of consistency here.
Our recent paper “Is the future of peer review automated?” (PMID:35690782) does make some clear points about the use of AI and other tools to increase the consistency of peer review.