After the successful its successful debut last year, Peer Review Week is back — and it’s next week!
In the spirit of participation, we’ve asked the Chefs: What is the future of peer review?
Kent Anderson: If you take mainstream scientists and editors seriously, you could reasonably believe that the future of peer review will manifest as incremental improvements on current and past peer review approaches. If you take critics of peer review seriously, you could reasonably believe that the future of peer review will be a radical revision of current and past peer review approaches. If you take the gloomy forecasts of peer review workload seriously, you could reasonably believe that peer review — either incrementally improved or radically revised — is doomed to fail in the mid-term. So, what’s the future of peer review? I think it will represent all three forces proportionally — a vast and resilient center based on current practices enhanced by incremental improvements; a small and loud set of critics driving introspection and some embroidery around the edges; and a continued search for qualified reviewers and for reward systems that keep reviewers engaged.
Alice Meadows: It’s hard to imagine scholarly communications without some form of peer review. Getting feedback from one’s peers is and should always be integral to the research process. Whatever our line of work, we all benefit from constructive criticism and guidance, and this is even more the case in research, which typically employs an iterative approach. So first and foremost I strongly believe that peer review does have a future.
But whether it will look the same as it does now is another matter. There has been a rapid expansion in approaches to peer review in the past decade or so — blind/double blind review is still popular, but open peer review has been enthusiastically adopted by some individuals and communities. Ditto portable peer review — whether done before or after manuscript submission. Prepub site arXiv is an example of a different approach to peer review for publications in physics and maths, and similar sites for biology (bioRxiv) and social sciences (SocArXiv) have launched recently. It’s encouraging to see peer review continuing to evolve in this way to meet the discipline-specific needs of reviewers, the organizations for which they are reviewing, and the wider scholarly community.
However, although I believe that peer review in various forms (some yet to be developed) will continue to underpin scholarly communications, we do need to address — quickly — the challenge of finding and training the next generation of reviewers. This is especially important given that researchers aren’t just reviewing publications but also conference abstracts, promotion and tenure applications, funding applications, and more. Typically one organization doesn’t have any oversight of the work their researchers are doing for other organizations. So what may seem like a peer review challenge for one sector could become a genuine peer review crisis across all sectors if we aren’t careful. Acting soon to provide researchers with better training, support, and recognition for their peer review contributions — whatever form they take — is critical to averting such a crisis.
Rick Anderson: At the risk of falling victim to the “Bears Always Sound Smarter” trap, I’m going to go out on a limb and say that the future of peer review is going to look very much like the past of peer review. It seems to me that the concept of peer review – putting multiple sets of expert eyes on a paper in order to maximize the likelihood that important flaws will get caught and corrected before it joins the public record of science – is just too fundamentally sound to be replaced by something that’s radically different.
That’s not to say that peer review always functions as it should, or that it has no downsides in addition to its obvious upsides, or that we won’t find new and better ways of managing it. And of course there have been some high-profile experiments with post-publication peer review, which have had success but about which I have some pretty serious concerns. Overall, though, my suspicion is that pre-publication peer review, as we’ve known it for decades and as we currently understand it, is here to stay.
David Smith: One can frame this question a number of ways; mechanistically what is the future? Philosophically, what is the future? Practically? Scholars care greatly about peer review as a principle component of the scientific method. And they understand its role and its limitations in this context:
“The scientist has a lot of experience with ignorance and doubt and uncertainty, and this experience is of very great importance, I think. When a scientist doesn’t know the answer to a problem, he is ignorant. When he has a hunch as to what the result is, he is uncertain. And when he is pretty darned sure of what the result is going to be, he is in some doubt. We have found it of paramount importance that in order to progress we must recognize the ignorance and leave room for doubt. Scientific knowledge is a body of statements of varying degrees of certainty — some most unsure, some nearly sure, none absolutely certain”
So much is written about the flaws and limitations of peer review, yet change does not come. I think the statement above explains why. Peer review is a satisfactory philosophy of consultation, no more, no less. The successful scholar understands that, and the implications thereof. Mechanistically, aside from some assistive tools, I think there will be little change. Philosophically, there will be no change. Practically? Well we seem to be heading into a post factual world… There lie the threats to peer review, because there lie the threats to the scientific method.
Alison Mudditt: Peer review varies dramatically across disciplines and content formats, so I’m going to focus my response primarily on long-form content in the humanities and social sciences. While Kathleen Fitzpatrick argued for a system of open, direct peer review on public texts in Planned Obsolescence some ten years ago (and practiced it with that book), few have followed her lead. Changing peer review requires a significant cultural shift for the humanities where authorship is still very much a lone activity — the open dialog that Fitzpatrick and others recommend requires a radical shift. And of course, the other challenge is whether new models would be taken seriously by the institutions that rely upon peer review for accreditation.
So I think it is pretty unlikely that we will see double-blind review unseated as the dominant methodology any time soon. (And it’s interesting to note that AAUP’s new handbook of Best Practices for Peer Review , while acknowledging disciplinary differences and perhaps a different process for digital projects, focuses almost exclusively on the double-blind model.) That said, the growing dissatisfaction with the shortfalls of this model (see Martin Eve, for example) are likely to lead with increasing experimentation with alternatives such as open commenting and metric-based post-review.
As the director of a university press, I’m also interested in the future of a critical but frequently overlooked piece of monograph peer review, and that’s the curatorial role played by acquisitions editors. Editors not only undertake substantive editorial work on manuscripts, but this cultivation continues through the peer review process as editors select scholars who can best help to shape the manuscript, connecting it to other scholarship and to the right audience. This work is highly valued by authors, yet it is also expensive and resists efforts to drive down the cost of monograph publishing (as the recent Ithaka report confirmed). How can it be preserved as monograph models evolve?
Judy Luther: Peer review is an essential part of the scholarly research dialog and as such is crucial to the ongoing cycle of research. Whether feedback is public or private, prepublication or post publication, free or paid, and the reviewers recognized or unknown, it will continue as a function. What’s not yet clear is how the form will change with the influence of a networked environment and technology affecting workflow as there are a growing number of new tools that enable collaborative creation and commenting by the reader, creating a continuous flow.
Peer review is core to the editorial process of both books and journals producing a better work that serves as part of the scholarly record. At a time when there is an increasing volume of research available globally and competition for the readers’ attention, the published article serves as a milestone and one that warrants accuracy and quality.
I did not always feel this way. Enamored with the potential for a more complete picture of the research cycle and recognizing that emerging components could be easily linked, I could envision a future where the peer review process was diminished or no longer needed. However there are only so many hours in a day for consuming content, and there is value for the reader to know that ‘this article’ is distinguished by having a thorough review as part of the production process. It is far from perfect with all the frailties that humans brings to a process, however, it cannot be easily replaced and we are better with it than without.
Angela Cochran: I am a big fan of peer review and the editorial process. It’s one of the reasons I made the career switch from journals production to journals editorial.
What peer review is and how it should be implemented is certainly in flux right now. I believe that we are moving toward a more transparent and inclusive process. That said, I’m not sure we are settled on the degrees of open and inclusive.
There are still very real fears from junior reviewers that some form of retaliation could come their way for critical reviews. There are also real fears from editors that they will lose reviewers and even associate editors if all names are disclosed. There are no known solutions to these problems.
Openness of review is going to need to be championed and established by each scholarly community. I worry that forcing the issue of open review (author names and reviewer names known and published) will further reduce participation and lead to even more insulated feedback loops than we already have.
So, I really want to talk about a more transparent and inclusive process. There can be transparency without naming names and reviewers should be given a choice to sign their reviews. Inclusion is, in my mind, more important. US and European journals need to do a better job of getting reviews from outside their territories. I suspect we collectively could use more peer review participation from practitioners. It is time for journals to review their databases and see who, exactly, their reviewers are and perhaps initiate that discussion with their editorial boards. Reviewers hold enormous power and I don’t feel like this gets analyzed enough.
There is very little value in throwing things online and hoping for the best. Peer reviewed journals aren’t exactly hurting for submissions and readership continues to grow. It seems clear to me that despite any flaws, this process of peer reviewing scholarly content is valuable to authors and readers.
Phill Jones: The interesting thing about this question is that it makes the (completely valid) assumption that peer review will change from its current form, and by extension that it needs to. The problem is that there doesn’t appear to be much of a consensus as to what, if anything, is wrong with it. Some say that it’s unfair and overly conservative, while others disagree and say that it should be conservative. Some say that reviewers aren’t incentivized adequately, but what kind of incentive would be sufficient to divert more of a researcher’s attention from doing their own research? It seems to me that reviewers are happy to do a certain amount of reviewing, in order to support their fields, but that there are limits, in terms of how much time scholars can take away from their primary obligations.
What we do know is that peer review is under stress. It seems to be getting more and more difficult for editors and editorial assistants to find reviewers and get those reviews back quickly. The exponential growth in the number of submissions, is putting an increasing load on a limited pool of experts. But if there are more researchers writing, then surely there should be more peers available to do the reviewing. So what’s gone wrong? I think that the issue lies in a lack of tools to identify new potential reviewers. I’m assuming here that workflows haven’t changed all that much since my brief stint as ‘support to an editorial assistant’ many years ago. Back then the database of reviewers was manually populated from people that editors knew or were familiar with. That sort of workflow is likely to lead to a short list of favorite reviewers who will inevitably, over time, feel put. So if you’re asking me, which you are, the next big change in peer review workflows will be better ways to identify potential reviewers.
Ann Michael: Based on all of the evolution and disruption examples I’ve ever seen or read, future paths are often discovered when you address the root purpose of a process, product, strategy, etc., evaluate the root purpose for current relevance or appropriateness, and rebuild from there. In thinking about the future of peer review, perhaps that core question is: Why do we peer review?
If we assume the purpose of peer review is fundamentally to ensure the accuracy or “reasonableness” of structure (of an experiment or exploration — NOT of a manuscript), methods, conclusions, and even underlying data (currently growing in importance), is that purpose relevant or appropriate? I would venture to say that it is. (If it isn’t then we can start another thread on why or why not that is the case!)
So, how can we best accomplish that purpose? Could we use something like IBM Watson or Brainspace to first “read” all of the research done in a particular area or areas (for interdisciplinary publications) and ask it to rate the research for accuracy, method, logical path/soundness of conclusions and itemize any potential issues or shortcomings? Maybe something like this wouldn’t be the only step in “peer review,” but it could be input into the process, that might diminish the demands on a peer reviewer while also ensuring that a fair, consistent, and thorough review has been done?
I’m not advocating the above as THE answer or even AN answer, what I do advocate is inquisitively, creatively, and openly considering any process and how it might be improved. I also advocate learning from what might be considered the “crazy ideas” at the fringes. We can and should brainstorm, hypothesize, and test ideas (hmmm…does that process sound familiar!).
Another area of caution, one which I think all of my fellow Chefs would agree is critical, is being careful not to paint all of scholarly publishing with one brush. The peer review process may very well be completely redefined or reimagined within some disciplines and barely touched within others. That may not be because the areas of little change are populated by luddites or laggards, it may just be because the current process works for them to a large extent.
Obviously, none of us know the future. However, we can all talk about trends, behaviors, and observations and think about what the future might hold. What do you think?
What might be the future peer review?