“Many of the things you can count, don’t count. Many of the things you can’t count really count.” — Albert Einstein
Measuring the value of science is difficult. A recent report in the Chronicle of Higher Education outlined the difficulty policy-makers and scientists are having calculating the return on investment (ROI) for science funding — and the potential pitfalls in making the measurements, including the cost and labor involved, the huge opportunities for inaccuracies, and the classic problem of creating incentives for the wrong things. Another recent essay on Inside Higher Ed critiqued CourseSmart’s analytics, which purport to estimate student preparedness by measuring their engagement with an e-textbook. The essayist, an experienced teacher, knows he has a better way of evaluating preparedness — grades for work completed.
CourseSmart’s motivation is a classic marketing attitude, as their SVP of marketing says:
The big buzz in higher ed is analytics. Based on what we had and what issues there are with institutions around improving the return they’re getting on their investment in course materials, we realized we had a valuable data set that we could package up.
John Warner, the author of the essay, has this response:
CourseSmart is peddling a product for which there is “buzz,” but no actual need. This strikes me as a particularly 21st Century mindset.
Still, science has a strong quantitative bent, and we’ll always be trying to measure whatever we can. But are the important things available for reliable measurement? And can we accept that they may not be?
Thinking about the goals of altmetrics — identifying content that’s more relevant, more interesting, novel, or important, and doing so as quickly as possible after publication or, better yet, helping authors to find the best match for their works — made me wonder if we’re missing some obvious alternatives to metrics, ones based on words rather than numbers. Not alternative metrics (or altmetrics), but alternatives to metrics (which I’ll call alt2metrics).
In my experience with authors and editors, metrics are rarely definitive, but can add a little bit of information to an already information-rich landscape. Most authors know where they want to publish, and these preferences can be based on all sorts of factors, from the journal’s reputation to the editor’s reputation to the perceived audience the journal reaches to publication policies (speed, review process, perceived rigor). More often than not, academics and researchers are dismissive of metrics, as they’ve seen how, once you poke at them, they end up being relatively blunt measures with little nuance or depth.
I recently came across a blog post from 2011 along these lines, which has more of an organizational behavior perspective, but contains a lot of truth:
Metrics are one dimensional, human beings are not. . . . Prioritizing things that can be measured over these kinds of things has been very, very costly to business. We have, in the name of metrics, hollowed out our organizations, our organizational cultures and the employee-employer relationship. Assuming that we have got to be able to measure something to acknowledge its existence seems reckless to me and leads us down a very inauthentic and unproductive road. It is a false constraint. It is a false constraint supported by antiquated archetypes of the organization, of management, and the value creation process . . . and those that sell us metrics. We need not to struggle for measurement of things that cannot be measured, but help our organizations better understand the intangibles that are so valuable today, and that we can still pay attention to and even prioritize things that cannot be directly measured
It’s worth unpacking some parts of this quote.
First, there is a fetishization going on with metrics, and one that’s not necessarily helpful. The belief in metrics fosters a feeling that if you can’t quantify something, it’s illegitimate or lesser in some way. However, if you can express a dimension as a whole number, or better yet, with decimals, it’s somehow more real or considered. We may own powerful brands, but if our PageRank is only a 6, well, it’s nights crying in the pillow. At the same time, the criticism of the impact factor is that it’s too simple, too widely embraced, and an illegitimate substitute for a qualitative evaluation of a scholar’s or researcher’s work, which drives the festishization to another level of dubious — the next number will be better, we just know it!
Second, those who are pushing metrics are usually those wanting to sell us metrics, either directly or indirectly — that is, they’re salespeople at some level. Whether they’re in it for the money, the academic novelty, or a bit of both, metricians have invested interests in making everyone uneasy about not having metrics for things, even those things for which metrics are essentially misleading or inadequate, and likely always will be. Putting metrics into perspective — as useful adjuncts in specific circumstances — helps you to not overinvest in systems and tools that can look amazing, but usually end up with an inadequate return on investment.
Third, intangibles matter, and may be more valuable than things we can measure. Even those who embrace altmetrics know this, and operate accordingly. After all, they have no metrics about how effective their altmetrics might be — they are basing their enthusiasm on intangibles, like hope, ambition, novelty, and rebelliousness. These are all important intangibles we should applaud, but they are not themselves measurable motivators. Authors and readers in scientific publishing possess some of the same motivations, to which we can add fear, resentment, bitterness, hubris, pride, vanity, ego, passion, curiosity, commitment, professionalism, civic duty, and many other unmeasurable attributes to what propels science forward.
Perhaps key to all this is the fact that metrics take time to assemble — they are delayed, and secondary to activity. Non-measurement-dependent signals are more important, anticipatory, and upstream from metrics. And they are what scientists rely on every day to guide them and their searches for information. It would be a shame to spend all our time on secondary, derivative measurements while primary, original signals of value are ignored or downplayed inappropriately.
What are some of the primary, root signals of value that scientists pay attention to?
Brand: A journal’s brand is a major signal of various qualities and aspects — the focus, the reputation, the likelihood of the information inside being important, the editorial process, the longevity, the culture, the position in the overall market. To measure all these things numerically would be impossible. Yet, they have incredible value as guides to both authors and readers. Brands have predictive value.
Authorship. Authors also have predictive value for fellow professionals, and most scientists read within their spheres. This is something no metric in the world can change — for instance, does it matter to an engineer that a cancer review journal has a high Eigenfactor? Relevance and interest are trump cards for readers, and when the right author publishes an article on an interesting topic, it will grab more attention. With the implementation of ORCID, author names will become more like data, but not quantitative data. Rather, ORCID turns author names into structured data. The value is increased by reliability and disambiguation, but no numbers are needed.
Results. Studies also provide strong signals that aren’t quantifiable. Usually, in research communities, there are large-scale trials underway that people know about — the funding is significant, the labs involved are numerous, and the results are highly anticipated. Once they’re published, word spreads like wildfire, and downloads spike. Studies with catchy names can do even better, and this non-metric trend of coming up with acronyms has gripped science for almost two decades now.
Sponsorship. Society, funding, and academic affiliations also send signals readers and researchers can use to assess relevance and quality. The orbit into which a study falls is an important differentiator. Is it a mechanism study? A therapeutic study? Preliminary? Clinical? Who funded it? NIH? Or Pfizer? Did it come from a major physics center? Or a community college?
Altmetrics may be all the rage, and it will be interesting to see if its advocates are able to come up with a few measures that are meaningful, robust, and durable. Currently, there are plenty of non-quantitative signals at work, ones that useful, enduring, rich, addressable, human-readable, and well-understood — they just aren’t measurable.
Publishers and editors know how to use these signals — build stronger brands; get better authors; grab the best studies; put them into the most appropriate outlets; get them out as soon as feasible; and ensure discoverability. Follow this non-quantitative formula, and the metrics can measure the particulates in the air while you leave them in the dust.