What constitutes a high number of article citations? It often depends on the field.
100 citations to a paper in cancer biology — a field dense with researchers, huge grants, short papers, and fast publication times — may not necessarily have the same impact as an economics paper with the same number of citations.
For this reason, comparing citation impact across disciplines is widely considered verboten.
In 2008, three Italian researchers tried to change that. In a paper published in PNAS, Filippo Radicchi, Santo Fortunato, and Claudio Castellano argued that it was possible to normalize citations across disparate disciplines so that the performance of papers published in different fields could be compared. Their technique was very simple — divide the number of citations to a paper by the average number of citations to all papers in its discipline for that year. As I reported in 2008, this simple transformation appeared to line citation distributions up like ducks in a row. With a universal citation metric, it would be far easier to evaluate the relative impact a paper, and its authors, had on science.
Such a discovery was big, bold, and beautiful. But like other novel and important claims made in science, it didn’t take long for other authors to attempt to validate the claim.
In a paper published in the January issue of the Journal of the American Society for Information Science and Technology, three researchers at the University of Leiden (Ludo Waltman, Nees Jan van Eck, and Anthony van Raan) dispute the universality of citations.
In their paper, titled “Universality of citation distributions revisited,” Waltman and others expanded their analysis from 20 disciplines, reported in the original Radicchi paper, to all 221 disciplines in the sciences and social sciences, as classified by Thomson Reuter’s Web of Science. After counting the citations to each paper after ten years, Waltman applied Radicchi’s normalization technique. Did the citation distributions for every field indeed line up? Not exactly. Waltman writes:
[M]any fields of science indeed seem to have fairly similar citation distributions. However, there are quite some exceptions as well. Especially fields with a relatively low average number of citations per publication, as can be found in the engineering sciences, the materials sciences, and the social sciences, seem to have nonuniversal citation distributions.
Apart from the limited sample in the original sample, Waltman describes two oddities with the Radicchi paper that may explain their divergent results:
First, Radicchi included research articles and letters in this analysis. Waltman finds this mix curious since research papers have a very different citation profile from letters.
Second, and more importantly, Radicchi excludes all uncited papers from his analysis. As many papers remain uncited over the years, Waltman questions why these were excluded from Radicchi’s analysis since they are valid observations. The Radicchi paper does not explain why uncited papers were excluded from his analysis, only that including them does not change their results — a claim that Waltman disputes by showing how the exclusion of uncited papers changes the distribution of citations, making the universality claim “more justifiable.”
Zero is a very important number in scholarly communication. For many indicators of scholarly impact (citations, comments, blogs, and tweets), zero is the most frequent number encountered. It forms the anchor of a long tailed, skewed distribution in which most attention is lavished on a coveted few. Remove zero, and you’ve changed the very nature of that distribution.
To quote Schoolhouse Rocks,”Zero is a wonderful thing. In fact, Zero is my hero.”