Host Jon Stewart in the studio of The Daily Sh...
Host Jon Stewart in the studio of The Daily Show in 2004 (Photo credit: Wikipedia)

An editorial written Tuesday by Virginia Barbour and Kasturi Haldar at PLoS regarding a specific retraction and a general stance relative to retractions generated a great deal of discussion this week. The sentence that spurred the controversy was:

If a paper’s major conclusions are shown to be wrong we will retract the paper.

Retraction Watch was quick to cover the posting, of course, and a good deal of the discussion took place on that blog, including what nowadays we nicely call a “walk back” by Barbour yesterday afternoon.

Comments within the controversy followed a common thread — science is necessarily a hunt-and-peck adventure, with many dead ends; those dead ends may often fall into the category of “wrong conclusions,” and would by this new logic require retraction; retracting findings that are just misinterpreted or currently viewed as uninteresting doesn’t seem helpful, appropriate, or scientific; barring clear problems in the underlying data, this seems likely to create big holes in the literature for no good reason; therefore, this new policy creates some real concerns about PLoS’ retraction practices going forward.

Having read the editorial and then Barbour’s clarification, I felt myself going into Jon Stewart mode, partly because her retreat from the editorial begins with a little bit of an accusation of reader error:

The phrase “If a paper’s major conclusions are shown to be wrong we will retract the paper” has been pulled out of the blog and over/misinterpreted.

This immediately struck me as disingenuous. If the catalyzing sentence Barbour notes were a toss-off line in the editorial, and not the punch line to a long and involved argument, Barbour might have a point. But here is the entire context of the statement she accuses readers of various stripes of “over/misinterpret[ing]”:

At PLOS our mission is to accelerate progress in science and medicine by leading a transformation in research communication. We firmly believe that acceleration also requires being open about correcting the literature as needed so that research can be built on a solid foundation. Hence as editors and as a publisher we encourage the publication of studies that replicate or refute work we have previously published. We work with authors (through communication with the corresponding author) to publish corrections if we find parts of articles to be inaccurate. If a paper’s major conclusions are shown to be wrong we will retract the paper. By doing so, and by being open about our motives, we hope to clarify once and for all that there is no shame in correcting the literature.

Barbour and her co-author are, in essence, arguing that because PLoS wants to accelerate the publication of research findings, they’ve effectively raised the stakes, making retraction more necessary and something that, in this new environment, we shouldn’t be ashamed of any longer. The logic is bolstered by strong writing, declarative sentences, and punchy punctuation. Did we overinterpret? Or did they overwrite?

Within 48 hours, Barbour was apparently so ashamed of this line of argument that she retreated from it in a significant way. In her note on Retraction Watch, she even refers back to COPE guidelines to illustrate what she portrays as their “intention,” which was apparently to comport more with the status quo notion as expressed in the COPE guideline she quotes:

Journal editors should consider retracting a publication if: they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error).

Barbour is right to regroup. There is a wide gulf between “wrong conclusions” and “unreliable findings.” The former is mostly a logical error — we once thought that ulcers were caused by stress, because we believed that the highly acidic stomach environment prevented bacterial growth. This was a “wrong conclusion,” not an unreliable finding. The stomach lining is highly acidic. We just hadn’t discovered H. pylori in the ulcers because we hadn’t thought that bacteria might survive, which kept us from applying Koch’s postulate. Once applied, our view changed, but none of the papers measuring stomach acidity should have been retracted just because this new finding revealed some of them reached a “wrong conclusion.” Unreliable findings are, as the statement notes, the result of data errors, not logical errors of interpretation. As one commenter on the PLoS editorial wrote:

No, no, no, no, no. This is a terrible policy. To use Feynman’s term, this is “cargo-cult” editing. Going through the motions of checking the scientific literature but without the intellectual integrity to acknowledge the limits of what is known. The conclusions of a scientific paper are the least important part of it. There is no reason to retract papers with wrong conclusions. It would be a bad thing to retract all papers with wrong conclusions. I come across many papers with wrong conclusions all the time. I often don’t even look at the conclusions, just at the data and then make my own conclusions.

In any event, perhaps the most heartening aspect of this hiccup is that the scientific community showed that when it comes to “wrong conclusions” about how to apply retractions, it can quickly generate enough concern for a de facto retraction to occur. And that seems profoundly appropriate.

Enhanced by Zemanta
Kent Anderson

Kent Anderson

Kent Anderson is the CEO of RedLink and RedLink Network, a past-President of SSP, and the founder of the Scholarly Kitchen. He has worked as Publisher at AAAS/Science, CEO/Publisher of JBJS, Inc., a publishing executive at the Massachusetts Medical Society, Publishing Director of the New England Journal of Medicine, and Director of Medical Journals at the American Academy of Pediatrics. Opinions on social media or blogs are his own.


16 Thoughts on "Retractions Retraction — Did We "Overinterpret" or Did PLoS Editors "Overwrite"?"

Gurr. Yes, thank you. It was late, and I was writing quickly. It’s been fixed. Thanks again!

Yes, that was odd. My experience with retractions is that there’s usually a bit of back and forth with authors and/or their institutions before the trigger is pulled, so that at least everyone has heard the dogs of retraction baying for a while.

The most interesting part of this news story to me is:

Several other journals with a higher impact factor than PLoS ONE had agreed to publish the paper, but on the condition that the authors retract the 2006 study—which the team refused unanimously[.]

Apparently, the editors (or reviewers, per the comments) of at least a couple of other journals also think that retracting a paper that has been refuted is appropriate. We’ll probably never find out who they are due to peer review process confidentiality.

It makes me wonder if the core of the problem people have with this incident is the PLOS One editors’ unilateral action.

I find this story interesting for two reasons:

1) Is it a signal that retraction is an analogue function struggling to transition to a digital age? Or is it just one of those things – not quite thought through enough. I sense a tension. Like Kent, I was wondering whether this was a response to some wider issues to do with the volume of material coming through the journal. It’s the use of the word ‘wrong’ here that is so interesting.

2) This is a publish then filter problem isn’t it? More papers, more calibrating the cost benefits of getting in first and then maximising the article output volume for a given discovery (or grant award…). To what extent does one need to put a formal marker that a piece of research is wrong? Up until now retraction has been a stigma. This is or was, a look at changing the terms of reference on the matter.

To me, despite the opprobium that has been heaped on the editors, they’ve done something important here. They’ve started a debate on how signals about correct and incorrect research get propagated through the information and to what extent those markers need to be formalised. If we are debating what the best economic models are for scholarly dissemination, a debate on the role of the quality markers goes hand in hand. Interesting that an OA journal has tentitively suggested that there’s a different way of thinking about it.

There’s a couple of great TSK articles by Phil that are worth reading alongside this one:

To me, the policy takes the messy epistemological process of science and reduces it to simple categories: Right Science versus Wrong Science. This wouldn’t bother me so much but PLoS has made a similar false dichotomy before over what constitutes a “sound methodology”.

The PLoS editors made a fundamental error in their logic, and yet, the discussion that took place in reaction to that editorial was constructive and thought-provoking and got to the heart of how science operates. Should PLoS retract that editorial? Using Barbour’s own categorical reasoning, some argued that it should, but that falls trap to the same fallacious reasoning.

Barbour’s editorial was “wrong,” but considering the discussion that it generated, it was anything but useless.

Indeed Phil. In fact on Popper’s conjectures and refutations model being wrong is the source of scientific progress. Should we retract Euclidean geometry given general relativity?

Well, I think the policy is highly flawed and I am concerned about this interpretation of COPE guidelines. I am also concerned that Barbour is the chair of COPE and it should make for an interesting COPE meeting in October. I for one grew weary of PLoS talking down to traditional journal editors and publishers a long time ago. They are right, we are wrong. Phooey! Perhaps this shows a kink in the PloSOne peer review policy of…well, no peer review. Of course this article appears in PLoS Pathogens and I don’t know what their peer review looks like but making this a policy across the board gives them an out when they accept uninteresting or flawed papers.

The fact that they did not bother to wait for an author response is troublesome and seems to show the true intent of the policy, regardless of the backstepping going on.

How about a boycott on PLoS like it was done on Elsevier? Nobody is perfect…

On another note, does PLoS not use the “letter to the editor” concept. That has been the way other journals raise differing opinions, often publishing both the comment that something is fishy, as well as the response from the authors; then it’s up to the reader to choose who to believe. For example of such scheme, check the debate on the global warming hockey stick.

But. . .they do have a “comment directly on the article web page” concept. This hasn’t had much uptake (for a variety of reasons), but I think it is safe to say that you can raise differing opinions there (and I have, in one or two cases).

We obviously need a “limbo” for all the articles (and scientific monographs?) that someone considers to be wrong for “good reasons.” All the unequivocally right research can stay published, and all the unequivocally bad research can be retracted. So where do we put “The Origin of Species,” Watson and Crick’s original crystallography results, or Steven Hawking’s papers on black holes? Queue the “Limbo” music…how low can you go?

Unfortunately I have yet to see a formal statement from PLoS that this asinine policy has been retracted. As noted by many above and elsewhere, such a policy reflects a fundamental misunderstanding of the nature of scientific discourse. ‘Major conclusions’ of scientific papers are often discussed, revised and replaced by subsequent work and deeper understanding; in fact an entire journal (Medical Hypotheses) is built upon the editorial policy that “… it is sometimes (but not always) better to be interestingly wrong than boringly right… “ (Charlton, B.G., Medical Hypotheses 70: 905-9; 2008). Who will retract the retractions when an editorial retraction is subsequently shown to have been in error by yet newer work? Editorial rewriting of history was famously described by George Orwell in his classic novel 1984, where the ‘Ministry of Truth’ was responsible for periodic elimination of inconvenient documents. Scientific editors should beware of consigning published papers to Orwellian memory holes by over-zealous use of the blunt axe of retraction.

Comments are closed.