A paper from March 2011 just surfaced for me, and it’s a pretty interesting, if small, analysis of citation patterns around rebuttals to original research. Entitled, “Do rebuttals affect future science?” it’s from the School of Aquatic and Fishery Science at the University of Washington, and deals with only seven papers in that domain. However, all seven papers were published in either Nature or Science, so these were not obscure publications.

The question the authors sought to address was simple — If a paper is rebutted, does the rebuttal itself get a lot of citations, or drive down or change citations to the original?

If science is indeed self- correcting, you would expect rebuttals to blunt citations to the original, garner citations themselves, or both.

The authors categorized each citations of a rebuttal according to its level of agreement with the original — from outright refutation of the rebuttal to neutral “controversies exist” to agreement with the rebuttal. Surprisingly, they also had to create an unexpected category for citations that implied the rebuttal agreed with the original article, a distortion variant.

Results were discouraging:

• Original articles were cited 17 times more often than rebuttals
• Articles that cited rebuttals were generally neutral about the original article
• Original articles were uncritically accepted when rebuttals were not cited
• Original papers were cited just as frequently after rebuttals were published
• Only 5% of nearly 3,000 citations were critical of the original article

As the authors write (and you almost feel like talking them off the ledge here):

For those convinced that science is self-correcting, and progresses in a forward direction over time, we offer only discouragement. We had anticipated that as time passed, citations of the original articles would become more negative, and these articles would be less cited than other articles published in the same journal and year. In fact, support for the original articles remained undiminished over time and perhaps even increased, and we found no evidence of a decline in citations for any of the original articles following publication of the rebuttals. . . . [A major news report] trumpeting “Fish stocks eaten to extinction by 2050” (Leake 2010), based on a highly contentious projection . . . [fails] to mention any of the 11 rebuttals that question this projection, but it misses the later consensus paper by the same author and many of his critics that reverses the earlier projection of collapse, and instead expects rebuilding to occur in 5 of 10 well studied ecosystems.

One observation the authors make may have some significance — that is, rebuttals did better if they were in lower-impact journals. This is probably for two reasons. First, the rebuttals published in Science or Nature were much shorter on average (1.7 pages) than those published elsewhere (7.2 pages). Second, lower-tier journals are more niche, so are more likely to reach the working scientists in the area and affect their thinking. It makes sense that shorter and less targeted rebuttals would leave less of an impression.

One possiblity the authors didn’t explore might be that the brands of the two journals involved — Science and Nature — were themselves responsible for trumping rebuttals. After all, if it was published in a big name journal . . . well, need I say more?

It’s a small study, but in intriguing one. And it provides more evidence that, for various reasons, citations and scientific discourse may not be up to the tasks we’ve assigned for them.