Démocrite et Protagoras
Démocrite et Protagoras, Salvator Rosa, c. 1664.

Like a modern-day Protagoras, Elsevier has offered us CiteScore, a new metric for a community that likes nothing better than to add two numbers together. Some might wonder if we need a new metric, if we are not in fact already awash in metrics. I am assuming that most readers of the Kitchen are familiar with CiteScore, but if you are looking to, ahem, take the measure of this new service, please see Richard Van Noorden’s piece in Nature, Carl Straumsheim’s article in Inside Higher Ed, and the incisive post by the Kitchen’s own Phil Davis.  Among the things you will learn is that Elsevier is not proposing that CiteScore is the metric to end all metrics but that it is one of a “basket” of metrics to be used by researchers and librarians to assess the value of a journal. This is an interesting point of view in that one merit of metrics is their role in discovery, but adding to the already long list of metrics actually makes discovery more complicated. To a scholarly community searching for a needle, Elsevier offers a larger haystack.

Before I go any further, let me say that the legion of Elsevier-bashers will get no comfort from this post. I admire Elsevier; it is not only the largest STM publisher but it is arguably the finest. It publishes some of the world’s most important (and impactful) literature and has created or acquired a suite of valuable services. It is also the principal agent in the reduction (yes, you read that right) of costs to libraries, as measured by the cost per article in its large aggregations. At the heart of Elsevier is the instinct to study the marketplace very carefully, giving the company a very clear picture of user demand. If you want to accuse Elsevier of anything, accuse them of listening to their customers.

CiteScore, however, has already proved to be controversial. Let’s list some of the issues:

  • Is CiteScore a good or better metric? How does it compare to the competition?
  • Is there a conflict of interest in a publisher providing a service that evaluates those publications?
  • How do we assess CiteScore as a business strategy? What is Elsevier hoping to achieve vis-a-vis the competition?
  • Finally, what about the Hawthorne Effect? Will publishers alter their editorial programs in order to rank higher with CiteScore?

One problem with evaluating CiteScore is that you end up going all meta; you are evaluating a tool of evaluation. But that tends to obscure the more fundamental issue: What are metrics for? CiteScore inevitably is compared with Journal Impact Factor (JIF), which has been embattled for years. JIF is a useful, if not definitive, way to determine whether an editor is doing a good job — or at least a consistent job — and by extension it says something about the journal itself. It also gives us a rough idea about an individual article by telling us something about the company it keeps. But this kind of extrapolation can be taken too far, and it often is. JIF should not be used as a strict (not to mention exclusive) method of certification, and not only for a journal but also for its constituent articles and for those articles’ authors. JIF is a starting point; it is not a proxy for quality. Putting too much emphasis on JIF is a bit like the tendency among young people to consider SAT scores to be a measure of intelligence, something that would be more reasonable if we actually knew what intelligence was rather than engaging in circularity (“intelligence is getting multiple-choice SAT questions right”).

Which brings us to the meaning of JIF in the world today, which is a paradox: its value is not in what it measures but in the mistaken, aggrandized notion of what it measures and means. In creating a rival for JIF, Elsevier is participating in that same mistake. What does CiteScore tell us that we don’t already know that is worth knowing? You can add, subtract, multiply, and divide any numbers you like, but that doesn’t get you any closer to truth or quality or honor. It’s not that CiteScore is much ado about very little; it is that the whole bibliometrics game is much ado about very little.

On the other hand, perhaps CiteScore brings to us some special insights or important innovations. Have the best creative minds at Elsevier come up with something of unique value, something unlike anything we have seen before?  Is CiteScore, in other words, a true contribution to scholarly communications and not simply something that serves Elsevier’s own interests? Well, let’s see. CiteScore covers 22,000 journals to JIF’s 11,000. I suppose some would call that a great creative leap. Another innovation: JIF works with two years of data, CiteScore with three. It is difficult to suppress sarcasm in thinking about this as a great creative effort. Is this the kind of thing that prompted Van Gogh to cut off his ear?

But perhaps creativity and innovation are not the point. Companies typically innovate in one of two ways. First, they try come up with something that is truly new. This for the most part is the province of start-ups, as a start-up without a truly new idea will find the competition from incumbents to be overwhelming. Established companies, on the other hand already have a market position in place and seek to entrench themselves even further and perhaps to extend that market position a little. If you are sitting with a market share in the vicinity of 25%, as Elsevier is reputed to, then entrench and extend may be an inevitable strategy. We should not be surprised that CiteScore does not open up new vistas; we would be more surprised if it did.

Like all people and organizations in a position of power and influence, Elsevier is at pains to point out that the defining quality of its service is not self-interest but the moral improvement it brings to the community. Thus the matter of transparency. People frustrated by JIF’s transparency or its lack presumably welcome a tool that says precisely what it counts, and CiteScore does this. But is transparency even relevant here? Whether or not the exaggerated importance of citation counts is plain to see, the importance is still exaggerated. Invoking transparency is simply the misapplication of a metric. We could as well evaluate a basketball by its smell. The questions that CiteScore does not answer are the important ones: What does any metric tell us about quality? Why do we think that quality can be quantified? Does toting up the number of citations tell us anything more than the total number of citations? CiteScore cannot answer these questions because they are unanswerable.

But we will not stop making these measurements any more than we can ever forget our SAT scores. This raises a second question: Is it appropriate for a publisher of content to create metrics to evaluate that content? I think the answer to that is no, and it does not matter whether CiteScore is good or bad. We don’t expect the defendant to appoint the judge. Here my heart is with the people of Elsevier. It must be very, very hard for a company to accept that it has grown as large within its ecological niche as it can get. To get bigger it will have to change the niche. To be the judge means you will change the law.

As a competitive strategy, though, CiteScore is simply brilliant. Giving the scores away for free must have gotten the attention of librarians wondering where to cut expenditures. CiteScore further reinforces  Scopus, upon which it is built, and its presence in library collections, reducing libraries’ discretion in which services to keep, which to toss out. For some libraries facing difficult budgetary decisions, Scopus with CiteScore might be “good enough” to justify cancelling a Clarivate subscription to the Journal Citation Reports if not Web of Knowledge itself.

But perhaps the more immediate implications of CiteScore is that it radically changes downward the measurements of some very prestigious publications. It does this by counting all parts of the editorial program (editorials, commentaries, perspectives) as articles (this is described very well in Phil Davis’s piece cited above). And here we have yet another misapplication of a metric: CiteScore is a measure of research articles, but now it is being used to assess non-research aspects of many journals. It will come as no surprise that Elsevier does not control some of these major publications including Nature and Science, but even one of its own (Lancet) gets downgraded under CiteScore. If CiteScore becomes widely adopted, it will serve to restructure scholarly communications, as authors rethink where they want to send their best work and university administrators evaluate the portfolios of their faculty.  Lancet appears to be collateral damage in an attempt to align the scholarly community with the kinds of publications that sit at the heart of Elsevier’s large and distinguished offerings.

Which gets us to the most important item of all: Won’t editors teach to the test? Won’t they determine how CiteScore generates its numbers and begin to change their editorial programs? The answer to that is yes. If CiteScore becomes more established, which seems probable inasmuch as it is bundled with the widely-distributed Scopus, editors will attempt to steer their publications in the direction that yields a higher score from CiteScore, so out with the editorials and the news features and in with more research. Something valuable will be lost in this process, but Elsevier’s own portfolio, which is anchored in research articles, will prosper. CiteScore, in other words, will make other publications look more and more like Elsevier’s. This is what metrics do: they normalize. And since Elsevier is the dominant player in this market, Elsevier will reap the principal benefit of this realignment.

So we have a new service that is beautifully executed, but adds little value to scholarly communications overall. It puts Elsevier into a conflict of interest (not for the last time, I fear). It increases the pressure on the competition. And it will foster an implicit philosophy of editorial design, one that serves the company’s economic interests. It’s a nice day’s work. I wish I had thought of it.

Joseph Esposito

Joseph Esposito

Joe Esposito is a management consultant for the publishing and digital services industries. Joe focuses on organizational strategy and new business development. He is active in both the for-profit and not-for-profit areas.


15 Thoughts on "The Measure of All Things: Some Notes on CiteScore"

A modest proposal: Report *all* the metrics, not just the ones that look good. Knowing a particular magazine rating of an automobile is useless with regard to quality, but having a list of *all* automotive reviews for a particular vehicle is sufficient information to make a decision. Unless you like endless discussions of “quality” (a word that simply means, “I like it”). Without *some* numbers, quality is too amorphous to matter. 3 letters from external judges are just that, N=3.

You could call it CompareTheMetric.com (a reference for UK readers perhaps?)

Question: does having more flawed metrics offering poor approximations for the qualitative information one is seeking add clarity or just further obscure things?

Don’t know about these metrics, but this is pretty well established in areas of statistics: bagging, boosting, random forests etc all employ “poorer” individuals models than the best single model you might fit to the data, but when combined or built in particular ways the resulting ensemble can do better than the best single model when predicting.

At the very least, if all the metrics are reported it will aid comparison across journals (otherwise publishers will just pick the suite of metrics that makes their journals look the best).

I can only presume that this article refers to another Elsevier that I am not aware of. If Elsevier is ‘listening to their customers’, then why are they abandoning them, not only on an individual and institutional scale, but more recently on a national scale (German universities having recently announced their plan to abandon them)? This does not mean that CiteScore is not a worthwhile or valuable indicator, but it is hard to get excited, or even interested in anything emanating from a company whose fundamental premise is based upon share price advancement and profit margin maximisation rather than educational advancement.

I think that “listening to the customer” is not the same thing as doing what customers say they want. There is a much-repeated adage in the publishing world, originating with Clark Kerr, as I recall, that with librarians, “watch what they do, not what they say.” Library practices have given rise to the shape of the publishing community. So, for example, the growth of consortia works in the interests of the largest publishers. Elsevier watches what libraries do better than anyone else.

My understanding of the situation in Germany is that this is a stalled negotiation. The German schools in question actually want to buy more services and products from Elsevier, not abandon them, and they are doing their best to drive a hard bargain. Some seem to be characterizing this as a boycott, when it appears to be just the opposite, moving from a “Big Deal” to a “Bigger Deal”.

My understanding of this, and all similar negotiations is that Elsevier’s prices are designed to support its excessive massive profit model…… ‘DEAL, released a statement saying that Elsevier’s initial offer did not meet its requirements for open access and that the company was asking too much money’. In comparing their negotiations with those in The Netherlands, it was stated that “The Netherlands model doesn’t go far enough for us,” says Horst Hippler, president of the German Rectors’ Conference in Bonn and lead spokesperson for Project DEAL. The Dutch arrangement allows the publisher to “double dip,” he says: “Researchers end up paying twice for many publications: once for publishing them and again for reading them.” DEAL’s goal, he says, is to move away from subscription fees and toward a model in which the institutes pay a flat fee for “article processing charges,” so that all publications become fully open access.

This does not sound like a ‘negotiation’ to me, but an essential difference in principle as regards publishing practice.

I won’t try to defend Elsevier’s prices or margins. But this is a consortia that is negotiating a new deal with Elsevier, and the public statements are all part of the negotiation tactics. What they are asking for is a bigger deal than they’ve had in the past — they want subscription access to all of Elsevier’s journals and they want their authors’ papers to be published in an Open Access manner. There is an underlying hope that the rest of the world will make similar moves and the subscription business model will eventually fade away, but given that outside of Europe, very few seem interested in making the additional investments to make this happen (and in the US, the fragmentary nature of the university system makes it virtually impossible), it is likely that this will end up costing the Germans more than their current arrangement. They are trying to drive a better price for their use of Elsevier’s product, to be sure, but this is in no way an abandonment of those products.

References to a ‘boycott’ are indeed incorrect. The reality is that individual institutions in Germany chose to not renew their contracts based on their assumption there was going to be a national license instead, which is being developed but will take more time to complete. We’ve agreed with our negotiation partners to continue our conversations in 2017.

In terms of differences, might be worth adding that Scopus is far larger than Web of Science, with less stringent acceptance criteria, but also with a larger proportion of foreign-language content and SSH content. Presumably the pool of citations is larger, but also that there is a longer ‘tail’ of poorly-cited content.

Scopus coverage is not all that much larger than Web of Science (Web of Knowledge) – but CiteScore serves up metrics on a larger number of the titles. JCR has traditionally published metrics for only the Science Citation Index-Expanded and the Social Sciences Citation Index, not the Arts & Humanities Citation Index, and not Conference Proceedings Citation Index (Science or Soc Sci and Humanities). A&HCI and CPCI do throw their cited references INTO the JCR metrics, however. Biosis Citation Index, Book Citation Indexes (Sci and Soc Sci-Humanities) are also part of WoK source coverage, but are not in JCR as either contributing to metrics or receiving metrics.

I’m not arguing for or against any of this – just doing a little fact-checking.


My interest, actually, is this:
The “long tail” in these datasets is pretty important in that it contains a quite different message and quite different meaning from the great clot of citations at the top-end. Often (not always…) these tails contain sub-populations of just such foreign-language and humanities titles that are less-visible to/in the traditional core of evaluative metrics.

Personally, I am not interested in what any of the metrics say about the top-drawer titles. I don’t need JIF or CiteScore to tell me that NEJM and the Lancet- or JAMA-suite of titles are important, prestigious medical journals. I need to use metrics to examine how the scholarly community is evolving in less-served topics and regions – and if I am LOCATED in or publishing in one of those long-tail topics/regions, I need to see where and how my works and my journals are being seen or used BY peer-group titles in the topic-region.

Citation metrics are published by aggregators, but “created” by scholars who write and reference – and I would aver that the overwhelming majority (but not all…) of that citation is intended to indicate scholarly relevance.

It’s worth noting also that the newer ESCI (Emerging Sources Citation Index) also changes the total number of publications indexed in WoS. They don’t get impact factors, to be sure, but the criteria for inclusion appears to be significantly less stringent than for Scopus.

Comments are closed.