Like a modern-day Protagoras, Elsevier has offered us CiteScore, a new metric for a community that likes nothing better than to add two numbers together. Some might wonder if we need a new metric, if we are not in fact already awash in metrics. I am assuming that most readers of the Kitchen are familiar with CiteScore, but if you are looking to, ahem, take the measure of this new service, please see Richard Van Noorden’s piece in Nature, Carl Straumsheim’s article in Inside Higher Ed, and the incisive post by the Kitchen’s own Phil Davis. Among the things you will learn is that Elsevier is not proposing that CiteScore is the metric to end all metrics but that it is one of a “basket” of metrics to be used by researchers and librarians to assess the value of a journal. This is an interesting point of view in that one merit of metrics is their role in discovery, but adding to the already long list of metrics actually makes discovery more complicated. To a scholarly community searching for a needle, Elsevier offers a larger haystack.
Before I go any further, let me say that the legion of Elsevier-bashers will get no comfort from this post. I admire Elsevier; it is not only the largest STM publisher but it is arguably the finest. It publishes some of the world’s most important (and impactful) literature and has created or acquired a suite of valuable services. It is also the principal agent in the reduction (yes, you read that right) of costs to libraries, as measured by the cost per article in its large aggregations. At the heart of Elsevier is the instinct to study the marketplace very carefully, giving the company a very clear picture of user demand. If you want to accuse Elsevier of anything, accuse them of listening to their customers.
CiteScore, however, has already proved to be controversial. Let’s list some of the issues:
- Is CiteScore a good or better metric? How does it compare to the competition?
- Is there a conflict of interest in a publisher providing a service that evaluates those publications?
- How do we assess CiteScore as a business strategy? What is Elsevier hoping to achieve vis-a-vis the competition?
- Finally, what about the Hawthorne Effect? Will publishers alter their editorial programs in order to rank higher with CiteScore?
One problem with evaluating CiteScore is that you end up going all meta; you are evaluating a tool of evaluation. But that tends to obscure the more fundamental issue: What are metrics for? CiteScore inevitably is compared with Journal Impact Factor (JIF), which has been embattled for years. JIF is a useful, if not definitive, way to determine whether an editor is doing a good job — or at least a consistent job — and by extension it says something about the journal itself. It also gives us a rough idea about an individual article by telling us something about the company it keeps. But this kind of extrapolation can be taken too far, and it often is. JIF should not be used as a strict (not to mention exclusive) method of certification, and not only for a journal but also for its constituent articles and for those articles’ authors. JIF is a starting point; it is not a proxy for quality. Putting too much emphasis on JIF is a bit like the tendency among young people to consider SAT scores to be a measure of intelligence, something that would be more reasonable if we actually knew what intelligence was rather than engaging in circularity (“intelligence is getting multiple-choice SAT questions right”).
Which brings us to the meaning of JIF in the world today, which is a paradox: its value is not in what it measures but in the mistaken, aggrandized notion of what it measures and means. In creating a rival for JIF, Elsevier is participating in that same mistake. What does CiteScore tell us that we don’t already know that is worth knowing? You can add, subtract, multiply, and divide any numbers you like, but that doesn’t get you any closer to truth or quality or honor. It’s not that CiteScore is much ado about very little; it is that the whole bibliometrics game is much ado about very little.
On the other hand, perhaps CiteScore brings to us some special insights or important innovations. Have the best creative minds at Elsevier come up with something of unique value, something unlike anything we have seen before? Is CiteScore, in other words, a true contribution to scholarly communications and not simply something that serves Elsevier’s own interests? Well, let’s see. CiteScore covers 22,000 journals to JIF’s 11,000. I suppose some would call that a great creative leap. Another innovation: JIF works with two years of data, CiteScore with three. It is difficult to suppress sarcasm in thinking about this as a great creative effort. Is this the kind of thing that prompted Van Gogh to cut off his ear?
But perhaps creativity and innovation are not the point. Companies typically innovate in one of two ways. First, they try come up with something that is truly new. This for the most part is the province of start-ups, as a start-up without a truly new idea will find the competition from incumbents to be overwhelming. Established companies, on the other hand already have a market position in place and seek to entrench themselves even further and perhaps to extend that market position a little. If you are sitting with a market share in the vicinity of 25%, as Elsevier is reputed to, then entrench and extend may be an inevitable strategy. We should not be surprised that CiteScore does not open up new vistas; we would be more surprised if it did.
Like all people and organizations in a position of power and influence, Elsevier is at pains to point out that the defining quality of its service is not self-interest but the moral improvement it brings to the community. Thus the matter of transparency. People frustrated by JIF’s transparency or its lack presumably welcome a tool that says precisely what it counts, and CiteScore does this. But is transparency even relevant here? Whether or not the exaggerated importance of citation counts is plain to see, the importance is still exaggerated. Invoking transparency is simply the misapplication of a metric. We could as well evaluate a basketball by its smell. The questions that CiteScore does not answer are the important ones: What does any metric tell us about quality? Why do we think that quality can be quantified? Does toting up the number of citations tell us anything more than the total number of citations? CiteScore cannot answer these questions because they are unanswerable.
But we will not stop making these measurements any more than we can ever forget our SAT scores. This raises a second question: Is it appropriate for a publisher of content to create metrics to evaluate that content? I think the answer to that is no, and it does not matter whether CiteScore is good or bad. We don’t expect the defendant to appoint the judge. Here my heart is with the people of Elsevier. It must be very, very hard for a company to accept that it has grown as large within its ecological niche as it can get. To get bigger it will have to change the niche. To be the judge means you will change the law.
As a competitive strategy, though, CiteScore is simply brilliant. Giving the scores away for free must have gotten the attention of librarians wondering where to cut expenditures. CiteScore further reinforces Scopus, upon which it is built, and its presence in library collections, reducing libraries’ discretion in which services to keep, which to toss out. For some libraries facing difficult budgetary decisions, Scopus with CiteScore might be “good enough” to justify cancelling a Clarivate subscription to the Journal Citation Reports if not Web of Knowledge itself.
But perhaps the more immediate implications of CiteScore is that it radically changes downward the measurements of some very prestigious publications. It does this by counting all parts of the editorial program (editorials, commentaries, perspectives) as articles (this is described very well in Phil Davis’s piece cited above). And here we have yet another misapplication of a metric: CiteScore is a measure of research articles, but now it is being used to assess non-research aspects of many journals. It will come as no surprise that Elsevier does not control some of these major publications including Nature and Science, but even one of its own (Lancet) gets downgraded under CiteScore. If CiteScore becomes widely adopted, it will serve to restructure scholarly communications, as authors rethink where they want to send their best work and university administrators evaluate the portfolios of their faculty. Lancet appears to be collateral damage in an attempt to align the scholarly community with the kinds of publications that sit at the heart of Elsevier’s large and distinguished offerings.
Which gets us to the most important item of all: Won’t editors teach to the test? Won’t they determine how CiteScore generates its numbers and begin to change their editorial programs? The answer to that is yes. If CiteScore becomes more established, which seems probable inasmuch as it is bundled with the widely-distributed Scopus, editors will attempt to steer their publications in the direction that yields a higher score from CiteScore, so out with the editorials and the news features and in with more research. Something valuable will be lost in this process, but Elsevier’s own portfolio, which is anchored in research articles, will prosper. CiteScore, in other words, will make other publications look more and more like Elsevier’s. This is what metrics do: they normalize. And since Elsevier is the dominant player in this market, Elsevier will reap the principal benefit of this realignment.
So we have a new service that is beautifully executed, but adds little value to scholarly communications overall. It puts Elsevier into a conflict of interest (not for the last time, I fear). It increases the pressure on the competition. And it will foster an implicit philosophy of editorial design, one that serves the company’s economic interests. It’s a nice day’s work. I wish I had thought of it.