A paper from March 2011 just surfaced for me, and it’s a pretty interesting, if small, analysis of citation patterns around rebuttals to original research. Entitled, “Do rebuttals affect future science?” it’s from the School of Aquatic and Fishery Science at the University of Washington, and deals with only seven papers in that domain. However, all seven papers were published in either Nature or Science, so these were not obscure publications.
The question the authors sought to address was simple — If a paper is rebutted, does the rebuttal itself get a lot of citations, or drive down or change citations to the original?
If science is indeed self- correcting, you would expect rebuttals to blunt citations to the original, garner citations themselves, or both.
The authors categorized each citations of a rebuttal according to its level of agreement with the original — from outright refutation of the rebuttal to neutral “controversies exist” to agreement with the rebuttal. Surprisingly, they also had to create an unexpected category for citations that implied the rebuttal agreed with the original article, a distortion variant.
Results were discouraging:
- Original articles were cited 17 times more often than rebuttals
- Articles that cited rebuttals were generally neutral about the original article
- Original articles were uncritically accepted when rebuttals were not cited
- Original papers were cited just as frequently after rebuttals were published
- Only 5% of nearly 3,000 citations were critical of the original article
As the authors write (and you almost feel like talking them off the ledge here):
For those convinced that science is self-correcting, and progresses in a forward direction over time, we offer only discouragement. We had anticipated that as time passed, citations of the original articles would become more negative, and these articles would be less cited than other articles published in the same journal and year. In fact, support for the original articles remained undiminished over time and perhaps even increased, and we found no evidence of a decline in citations for any of the original articles following publication of the rebuttals. . . . [A major news report] trumpeting “Fish stocks eaten to extinction by 2050” (Leake 2010), based on a highly contentious projection . . . [fails] to mention any of the 11 rebuttals that question this projection, but it misses the later consensus paper by the same author and many of his critics that reverses the earlier projection of collapse, and instead expects rebuilding to occur in 5 of 10 well studied ecosystems.
One observation the authors make may have some significance — that is, rebuttals did better if they were in lower-impact journals. This is probably for two reasons. First, the rebuttals published in Science or Nature were much shorter on average (1.7 pages) than those published elsewhere (7.2 pages). Second, lower-tier journals are more niche, so are more likely to reach the working scientists in the area and affect their thinking. It makes sense that shorter and less targeted rebuttals would leave less of an impression.
One possiblity the authors didn’t explore might be that the brands of the two journals involved — Science and Nature — were themselves responsible for trumping rebuttals. After all, if it was published in a big name journal . . . well, need I say more?
It’s a small study, but in intriguing one. And it provides more evidence that, for various reasons, citations and scientific discourse may not be up to the tasks we’ve assigned for them.
Discussion
11 Thoughts on "Is Something Fishy Going On? Citations Suggest Correction Comes Slowly, If It Comes At All"
Try googling “Fish stocks eaten to extinction by 2050” and see how Google supports or rebuts Kent’s conclusion.
I found lots of big name (eg daily telegraph) citations of the original – which makes an apparently reasonable point that we are depleting fish stock at an every increasing rate, and therefore can be projected to run out. And I found a rebuttal by the fish industry that said we are not, apparently because farmed fish will replace free fish, but not rebuttal for the point that the feeder fish for farm fish are being depleted too.
So what really happened? May be a scientific drew a valid conclusion and self-interest groups seeded a bunch of rebuttals.
I cant find coverage of author withdrawing the report.
To quote from the paper:
Not only does the article get the year wrong (2048 not 2050) and fail to mention any of the 11 rebuttals that question this projection, but it misses the later consensus paper by the same author and many of his critics that reverses the earlier projection of collapse, and instead expects rebuilding to occur in 5 of 10 well studied ecosystems (Worm et al. 2009).
Industry shills aren’t the problem. Rebuttals in the scientific literature are not being recognized or affecting later citation.
You shed an interesting light on Google as a source of further problems. Since Google is based on citations to a large extent, the uphill battle rebuttals face is only exacerbated by how Google bakes in the advantages the original paper has.
This seems to me to be a factor around the fact that we generally still think about these things in the old print publishing mode. If we start to think about how publishing should work from an online-centric view then it should be possible to improve this situation significantly by ensuring that rebuttals are published alongside (and linked closely to) the original article so that anyone viewing the original article will also immediately see the rebuttals. We should also publish versioning of articles as some are starting to do so that authors can update and correct errors when they are spotted or picked up in rebuttals subsequent to publication.
Then we just have to make sure people actually attempt to read the article (and hence go to the site to see the original paper, and hence also see the rebuttals and/or any new versions) before they decide to cite the article – but that is a different issue!
Rebecca’s right. In a DOI-rich CrossReffed digital publishing world, there’s no technical obstacle to fixing this problem (at least as far is the visibility of the discussion is concerned). The obstacle is cultural: we need to evolve an ethos that takes for granted that the original paper and its subsequent discussion (if not informal discussion, at least published rebuttals and formal comments on both) should be _expected_ to be linked and visible. I know there are practical reasons why this doesn’t happen; my point is that we need to evolve an intellectual culture that expects this and demands this. (Note to Google: you could help here, you know. This wouldn’t be a huge stretch for you.) This reminds me of the comment I made in a SK discussion a week or so ago, where I said we need to develop a culture in which “no experiment goes unpublished.” That prompted some thoughtful, well-informed comments saying why that can’t, and even shouldn’t, necessarily happen. I understand that. Maybe I’m being overly optimistic (it wouldn’t be the first time), but I still think we are moving toward a state where the culture will expect _all_ results of investigations to be published (funders could easily mandate, and enable, this) and where _all_ subsequent formal discussion/critique of an article is considered integral to the publication of that article. (Note that these things are linked: one of the objections to publishing negative results was that the experiment might have been flawed. Critiques or rebuttals could point that out.) Cultural changes take a long time, but they do happen. This is one I think will.
I’d guess that the expectations would be different in the humanities and social sciences. There have been hundreds of rebuttals of Plato and Kant, but who would expect the originals to be any less cited after rebuttals than before?
The analysis is interesting, but might be incomplete. In my field when a paper is discredited by a well-founded rebuttal, one just ignores the original paper and doesn’t cite it any more. It follows that there is no reason to cite the rebuttal, both of which should sink to oblivion in an ideal world. Clearly from the above the world is not ideal, but one wonders how many relevant papers in fisheries science could have cited the original papers and/or the rebuttals, but completely ignored both.
In Nature (and the Nature journals), correction statements are short and to the point, ie state what was wrong and to what extent the conclusions of the paper are affected. When necessary, these are peer-reviewed. But more to the point, these statements are bidirectionally linked to the original paper, and appended to it, both on the journal website and in web searches, indexes (eg PubMed) and so on. So anyone coming across the paper afresh will certainly see that it has been corrected. Many of the Nature journals, and soon all of them will, replace the PDF on their websites with the corrected version, complete with the statement of what was corrected and why. What more could a journal usefully do?
Given the lack of effect noted in this study, your question is the right one. Most journals are passive about corrections — you may have downloaded the paper, and I may know this, but when a correction is issued, do I email you? As you note, even the PDFs aren’t routinely changed, just the HTML. And this paper was talking more about rebuttals than corrections, so these were more like editorial comments or disputational analyses, not corrections. At least that’s how I took it, looking at the reference list.
Related story in Boston Globe today: http://tinyurl.com/3pczjok
If I understand correctly, Science was willing to publish the original (erroneous) paper, but not the corrected version!
