Education, Metrics and Analytics, Research, Sociology

Reference List Length and Citations: A Spurious Relationship

The world is full of strange associations.  For instance, ice cream sales are highly correlated with the U.S. murder rate, but no one in their right mind would suggest that ice cream is responsible for violent behavior — that would give real meaning to “death by chocolate.”  The underlying cause that connects murder with ice cream sales is heat.  Heat makes people irritable and irrational, and given the right conditions, may lead to violent behavior.   There is no theoretical basis of ice cream leading to violent behavior — this relationship is merely spurious.

The length of a paper’s reference list is also associated with the number of citations that a paper receives, writes Zoe Corbyn in this week’s Nature News (“An easy way to boost citations,” Aug 13), reporting on a study of more than 50,000 papers published in the journal Science.  This association, writes Corbyn, provides evidence of widespread corruption in the citation process by involving a mutual “I’ll cite your paper if you cite mine” tactic designed to benefit both the citing and cited authors. She writes:

“There is a ridiculously strong relationship between the number of citations a paper receives and its number of references,” Gregory Webster, the psychologist at the University of Florida in Gainesville who conducted the research, told Nature. “If you want to get more cited, the answer could be to cite more people.”

Webster’s findings are based on statistical correlation — a simple measure of the association between two variables — and his plots look pretty impressive. But there is something odd about this relationship.  I’ve never consciously cited papers because they include long reference lists.  In fact, if you ask me about some of my favorite papers — the ones I cite over and over again — I’d be at a loss to give you even a ballpark estimate of their reference length.  Number of references never comes into my mind as a rationale  for citing an article.  So why the correlation?

In the next few paragraphs, and with the help of my own dataset of Science articles published in 2007, I will demonstrate that this relationship between references and citations is entirely spurious.  First I’ll replicate the association.  Then I’ll make it go backwards.  And for a final trick, I will make the association disappear completely.  (If you are impressed, you can hire me for your kid’s next birthday party).

Citations vs. References in Science Magazine (2007

Ladies and gentlemen, on your right is my scatter-plot — similar to Webster’s (p.10) —  showing the positive relationship between references and citations.  Even for articles aged two years, the relationship is statistically significant (p<0.001), although it can explain only 4% of the total variation in article citations.

Now, let’s add to this simple, predictive model another variable: length of article in pages, since longer papers tend to cite more references.

As you’ll see in my second plot (below), the act of adding the length of the article changes the direction of the relationship.  The relationship between references and citations is now negative.  More references now lead to fewer citations, and this relationship is highly significant (p=0.0077).

By simply adding the number of pages to the model, I have turned a significantly positive relationship into a significantly negative one, and in doing so, I can also explain another 12% of the variation in article citations.

References vs. Citations controlling for Page length

For my last trick, I’m going to make the relationship between reference length and citations completely disappear.  To do this, I need to build a more realistic model.  I’m going to add two more variables to the statistical model: the number of authors, and the subject section of the article.

Science is a multi-disciplinary journal, publishing articles in the biomedical sciences, physical sciences, and, to a lesser degree, the social sciences. By including the subject of the article, we can control for the differences in citation rates across the disciplines.  The section headings come straight from Science magazine — there is no easy field that can be used from ISI’s Web of Science.   Below are my abbreviated model results (Note: Ln stands for the natural logarithm of each variable).

Full model summary

Controlling now for pages, authors, and section, reference list length is no longer statistically significant.  In fact, it looks pretty much like a random variable (p=0.77) and adds no information to the rest of the regression model.  Page count and number of authors are still highly significant.  What you’ll note from the sum of squares is that the article’s section is, by far, the strongest predictor of article citations in our model. Adding section to our reference model allows us to explain a full 46% of citation variance.

So there you have it. The simple association between reference length and citations is spurious, confounded by such other simple explanations such as the length of the paper and the section in which it was published.  There may still be evidence of mutual citation gaming in science, although this study cannot  validate these claims.

Enhanced by Zemanta

About Phil Davis

I am an independent researcher and publishing consultant specializing in the statistical analysis of readership and citation data. I am a former postdoctoral researcher in science communication and former science librarian. http://phil-davis.org/

Discussion

10 thoughts on “Reference List Length and Citations: A Spurious Relationship

  1. Ha ha!

    Awesome. There’s been a lot of people trying to rationalize the more citations = more refs observation; far fewer saying it’s so much hogwash.

    Posted by rpg | Aug 18, 2010, 5:30 am
  2. Excellent takedown of the relationship. Just given that citation counts have a power law distribution, using an average of that makes little sense.

    Posted by Farhat | Aug 18, 2010, 6:58 am
    • “given…a power law distribution, using an average…makes little sense.”

      If only someone would shout that from the hills! Then we’d be spared many such articles in bibliometrics, economics and numerous scientific fields.

      Posted by Richard Sever | Aug 18, 2010, 9:40 am
  3. Although I cannot address the effects of section headings (which don’t go back to 1901) or number authors (which are not yet coded in my data), I can say that controlling for log number of pages has very little effect on the log-log references-citations relationship, which goes from a correlation of .70 to a partial correlation of .69.

    http://fun-research.netfirms.com/ispst/results.pdf

    Given that I am not an expert in this area, I would appreciate any constructive criticism interested readers of this blog can provide.

    Posted by Gregory D. Webster | Aug 18, 2010, 9:25 am
  4. How about this? The papers that have the most references are usually review articles, which also have longer page counts and tend to be cited more often. So to some extent the relationship could be real, but meaningless. This whole discussion is a great example of a very simple principle that I learned in the first year of grad school:

    A correlation, in and of itself, can NEVER prove cause and effect.

    Posted by Steve Hochhauser | Aug 23, 2010, 3:05 pm
    • For what it’s worth, Nature’s coverage of the article notes that the study took into account review articles versus other types of articles and found that wasn’t what was responsible:

      And — contrary to what people might predict — the relationship is not driven by review articles, which could be expected, on average, to be heavier on references and to garner more citations than standard papers…
      “By most metrics it is considered a pretty big effect,” says Webster. “There was a small difference with review articles but, in fact, it was in the wrong direction. On average, review articles actually showed less of a relationship than standard articles.”

      Posted by David Crotty | Aug 27, 2010, 5:01 pm
  5. Nice- I think a bigger problem with the number of references is the arbitrary limits that some journals (Nature comes to mind, I think Science does this as well) place on manuscripts. Authors are forced to remove citations to works taht they actually used. Or move them to the supplemental data, which is a problem because they are then not captured by web of science etc. (I think).

    Posted by Ivan Baxter | Aug 26, 2010, 2:22 pm
  6. The author of the paper in question has responded to the various comments here.

    Posted by David Crotty | Aug 31, 2010, 4:56 pm

Trackbacks/Pingbacks

  1. Pingback: Weekly roundup « Naturally Selected - Aug 18, 2010

  2. Pingback: Los científicos que citan más, reciben más citas, según un estudio de 50000 artículos publicados en Science « Francis (th)E mule Science's News - Aug 19, 2010

Side Dishes by Stewart Wills

Find Posts by Category

Find Posts by Date

August 2010
S M T W T F S
« Jul   Sep »
1234567
891011121314
15161718192021
22232425262728
293031  

The Scholarly Kitchen on Twitter

SSP_LOGO
The mission of the Society for Scholarly Publishing (SSP) is "[t]o advance scholarly publishing and communication, and the professional development of its members through education, collaboration, and networking." SSP established The Scholarly Kitchen blog in February 2008 to keep SSP members and interested parties aware of new developments in publishing.
......................................
The Scholarly Kitchen is a moderated and independent blog. Opinions on The Scholarly Kitchen are those of the authors. They are not necessarily those held by the Society for Scholarly Publishing nor by their respective employers.
Follow

Get every new post delivered to your Inbox.

Join 13,988 other followers

%d bloggers like this: