Business has only two functions – marketing and innovation. — Milan Kundera
The fundamental question dogging the concept of post-publication peer-review as imagined by alt-metrics and open access (OA) advocates stems from this Milan Kundera quote — that is, is it an idea being marketed or a form of innovation?
To be a form of innovation, it would need to be superior in some way — more expeditious, more reliable, more likely to be used, or more convenient. If it isn’t any of these things, then it may be more heat than light.
A recent column in Wired by Dan Cohen argues that OA can’t be effective if post-publication peer-review isn’t made as robust as possible, and seems to dismiss pre-publcation peer-review as undesirable because it is “closed.” He argues that creating a system of awards — as opposed to rewards — is one possible way to do this.
We need a sensible shift towards an acceptable form of post-publication, rather than traditional pre-publication peer review. . . . Yet we already have an obvious form of post-publication peer review in wide use: awards.
Cohen and many others fail to appreciate that pre-publication peer-review doesn’t preclude post-publication peer-review. Everyone acknowledges that filters are important to differentiate quality and relevance, but the pressure to choose between pre-publication peer-review and post-publication peer-review is a false choice. The two are not mutually exclusive.
Nor is pre-publication peer-review anything more than a tool, one of many tools an editor or a journal can put against content to make it more likely to bring forward high-quality content that is relevant to its audience. Science is a process which includes publication, and there is no magical step that ensures perfection. Peer-review is part of the process, not the entire solution.
So to argue that you need to replace pre-publication peer-review with post-publication peer-review doesn’t make sense. OA needs pre-publication peer-review as much as any other business model around scientific content. Cohen realizes that validation is key to acceptance, but believes that awards can provide validation:
Without this validation, an open access system can never work on the demand side.
The distinction between reward and awards turns out to be a crucial one which Cohen misses. And that’s unfortunate. Because if he had addressed it, the entire premise of his essay would have changed.
Awards are usually meaningless in the long run. Win Teach of the Year? Nice, but if you’re teaching a course that’s doomed, it probably won’t save your job. Win an Oscar? Commendable, but it won’t make you into a box office champ like Samuel L. Jackson. Win a Grammy? Sure, it can help if you also have commercial appeal, but many Grammy winners fade into oblivion.
Rewards, on the other hand, combine both power and viability, and therefore beat awards hands down. Scientists seek grants, tenure advancement, higher positions in the academic hierarchy — in other words, the rewards of academic life, which emanate from other rewards like published studies, successful research grants, and robust and productive labs.
Rewards incentivize science, which is the ultimate form of post-publication peer-review. Science is, in fact, completely predicated on the notion of post-publication peer-review. The two are synonymous. Scientist A publishes his results. Scientist B reviews those post-publication, and sees a flaw. Scientist B can’t reproduce Scientist A’s results, but gets more interesting, different, and better results. Scientist B publishes her results, referring to Scientist A’s paper, and everyone following the field realizes what has occurred. And Scientist B gets some nice rewards — a better publication, more citations, wide recognition, and likely some leverage in grant applications and within her institution.
Some people continue to fail to appreciate what exists and seek to invent something that is less practical and less likely to succeed. While we need to test our system of rewards to ensure they align with what we want to achieve, a system of awards to divert scientists’ attention into a weak form of post-publication chatter and out of a powerful, rewards-based system of science seems unwise.
8 Thoughts on "Post-Publication Peer-Review Already Exists, Already Has Incentives, and Is Already Robust"
Was just thinking something very similar, I.e. Alt-metrics measuring attention = functionally a marketing measure. I could be wrong of course. Agree that scientists need to focus on good science, as do publishers on good publishing.
Ian: I agree but would add publishers need to concentrate publishing good science.
I think those outside of the STM peer review world are those same folks who believe that everyone’s opinion is equal. They are the my opinion is as good as yours crowd. This may have some validity in life but not in the rigorous world of science.
In the same vein these opinion crowd folks are the same ones who repeat whether believing them or not conspiracy theories. Thus, giving them some sense of validity!
I can see some OA journals touting the fact that there are 3,000 opinions or post reviews given on X article and therefore it is a great article!
In the editorial system used by university presses for reviewing monographs, pre-publication peer review is only one of the methods used to make decisions. Also key is the role of faculty editorial boards, which have final veto power over what gets accepted. And post-publication review of monographs is even more visible than it is for journal articles, since books, unlike articles, actually get formally reviewed. Book awards are nice as icing on the cake, but they are not available for every type of monograph in every field (the American Philosophical Association, for instance, offers just one book award, and it is restricted to authors 40 years old or younger). Book reviews, however, are critical and play a major role in promotion and tenure decisions.
Not the same point, but aren’t the following widespread forms of post publication peer review that already exist?
• Letters to the editor
• Solicited commentaries
• New articles that rebut or explain earlier articles
• Review articles
When I reject an article, I am not saying it can never be published, only that it cannot be published in our journal. Our readers want and expect selectivity, and that’s where our peer-review process comes in.
Quality assurance does not stop with publication. Post-publication discussions and replies have been part of our journal for almost half a century. I consider them an integral part of the quality process.
The European Geosciences Union has for many years relied principally on post-publication peer review; see http://www.egu.eu/publications/. All manuscripts are posted in a “discussion journal” and after sufficient comment, are either rejected or moved to one of the 14 OA journals. Copernicus Publications manages the process. In discussions with some of their publishing staff, it seems that many if not most of the manuscripts get no comments, which is the greatest weakness of a post-publishing review system. There is no one asking you to review a manuscript, no one following up and no good reason to take time away from your other tasks to sift through a sea of manuscripts to find one you might be interested in looking at. Another weakness in the EGU system is that rejected manuscripts are not removed from the discussion journal. When the manuscript is submitted to another publisher, plagiarism tools will find the paper and flag it. Even if the explanation of being on the EGU site is accepted as a reason for ignoring the flag, many publishers will refuse to proceed under rules of prior publication.
I think everybody agrees that we need both pre- and post publishing reviewing procedures.
Question is, what should these procedure look like.
In general I believe the pre-publishing peer review procedure works quite well, although there’s always room for improvement of course.
The current post-publishing review procedure, in my opinion, has something of a prehistoric feel to it though. To make myself clear, I don’t oppose to sending official Letters or even writing critical (review) articles challenging what’s been published in the past. It’s just that we miss out on a lot of important feedback.
Ask yourself, how often do you feel the urge to provide feedback on an article you’re reading and how often do you actually take the time and effort to do it?…not to often I guess…
This is because we all have limited time and want to spent our efforts as effective and efficient as possible. So we’re not going to ‘waste’ our time on details we think nobody will care about. Even though these details might turn out to be of great importance to others.
Personally I therefore believe that adding an open access post-publication reviewing process will add great value to articles, allowing experts to easily show (minor) flaws, draw (slightly) different conclusions or make valuable suggestions for follow up experiments.
In an attempt to get the discussion started I’ve built a life science site, http://www.linkedpapers.com, which is based on the PubMed database and allows for open access reviewing.
I’m convinced that this is the way forward, but would greatly value to hear your thoughts on this.