How much of the literature goes uncited? It seems like a simple-enough question that requires a straightfoward answer. In reality, this is one of the hardest question to answer, and the most appropriate response is “it depends.”

A citation is a directional link made from one paper to another. In order to count that event, that link must be observed. And while counting a citation confirms that an event took place, not observing a citation does not confirm that it didn’t.

To use an analogy, if I observe a tree falling in the forest, I can confirm that it indeed took place. However, trees fall all the time in different parts of the forest, and if I wasn’t there to view the event (or the aftermath), I cannot confirm that they fell. When I report on how many fallen trees I observed, the context is just as important is the observation.

In a 1991 piece in the journal Science, science journalist David Hamilton reported on a study undertaken at ISI that quantified uncitedness in their citation indexes. Ninety-eight percent of Arts & Humanities papers remained uncited, according to their data, with the Social Sciences not far behind at 74.7%. The intended message was that much of the sciences don’t matter, and humanities barely mattered at all. The article attracted stern responses by Eugene Garfield and David Pendlebury (both from ISI) that Hamilton’s piece lacked the context of the analysis—the percent of uncited articles included all types of articles indexed by ISI, not just research articles. Remove letters, editorials, obituaries, corrections, news, and other ephemera, and the percentage of uncited articles drops like a stone. Citations were counted just four years after publication, and extracted from the ISI index, which at that time, indexed just 10% of the scientific journal literature.

If the researchers waited 10 years instead of four, based their analysis on a more comprehensive index, focused just on research articles, and included citations from books—the source of many citations in the humanities—we would imagine that the results would come out very differently.

Garfield also argued that uncited literature is not necessarily a bad thing. Much of science progresses by making incremental improvements and advances over older studies. Reference lists are not supposed to be historical chronologies of everything that preceded it, but a carefully selected list of relevant sources.

There are other explanations for uncitedness.

Citation errors. Authors misspell journal names or include errors in the volume or page numbers. While DOIs and disambiguation software at the indexing stage can help correct well-intentioned mistakes, they still happen. Errors essentially prevent a directional link to be made from the citing article to the cited article, which means that it cannot be counted. Or stated more directly, counting assumes good metadata.

Citation takes time. An article may wait years for its first citation. Referred to as “Sleeping Beauties,” some papers go unnoticed for decades until they are awakened by a citation event (the kiss from a handsome prince), after which they attract a lot of attention. Generally speaking, the longer a paper waits to be cited, the less likely it is to be cited. Conversely, the earlier a paper is cited, more more likely it is to be cited. Citations beget citations.

Citation is limited by the universe of indexed papers. Thomson Reuters’ datasets follow a policy to index the “core” literature, meaning a smaller collection of elite journals. Scopus, in comparison, is based on a much broader selection of journals. And Google Scholar, as mentioned in a recent post, has a much broader definition of a scholarly document. All three indexes provide different citation counts.

How much of the literature goes uncited? Answering such a question ultimately depends on providing context to the observation period and the universe of observation.

Fortunately, such an approach can also be extended to questions such as, “How much of a library’s collection never circulates?” and “What percentage of manuscripts are never published?”

The answer to all of these is, “It depends.”