Metadata are scholarly publishing’s favorite vexation, our equivalent of the underperforming local sports team or ne’er-do-well relative – easy to criticize, easy to fix in the abstract (“stakeholder X just needs to do Y”), but stubbornly hard to improve in reality. 

Part of the problem is that some metadata are well-established (e.g., articles being associated with their publishing journal), some well-established but error-prone (e.g., authors’ institutional affiliations), and others patchy and incomplete (e.g., funding information for individual articles). Our expectations for high quality metadata extend out to other fields – giving the sense that most metadata are wrong and that metadata in general are in a state of perpetual crisis. 

I argue here that the way we view metadata assertions – as a binary attribute that is either ‘right’ or ‘wrong’ – does not reflect the underlying reality. Moreover, this view inhibits the adoption of approaches that would fill in desperately needed metadata for many other fields (such as funder information for public datasets). 

We need to move to a probabilistic framework for metadata. In this framework, metadata are not ‘right’ or ‘wrong’. Instead, each assertion is accompanied by a statistic expressing our confidence that the assertion is correct. High quality metadata have very high levels of confidence. Metadata from less reliable sources or that have been inferred from circumstantial evidence have lower levels of confidence.

Let’s first walk through an (imaginary) example of how metadata are inherently probabilistic. 

We have in hand a single-authored article that credits ’NSF’ for funding. We could at this point assert that the US agency the National Science Foundation is the funder – after all, the NSF is a massive funder of research and ~ 90% of the articles listing ‘NSF’ are indeed funded by the National Science Foundation. In the absence of any other information, we’d be right about the National Science Foundation being the funder nine times out of ten. To put it another way, we’d have 90% confidence that the National Science Foundation funded this piece of research. 

Let’s add in more metadata: the author of this article is affiliated with a Norwegian institution. Although funding by the US National Science Foundation is still possible, it’s now much less likely – only 0.7% of articles with a Norwegian affiliation were funded by the US NSF in 2023. The ‘N’ in ‘NSF’ is now more likely to stand for ‘Norwegian’ or ‘Norges’. 

Another piece of metadata – the article title – bears this out, as it’s a study of head injuries in Norwegian skiers. The funder NSF in this case is most probably the Norwegian Ski Foundation or Norges Skiforbund. A search of their grants database reveals that an individual with the same name and affiliation as the author received a grant from the Norwegian Ski Foundation several years ago. 

The first take-home is that the intersection of three different metadata fields (the initials NSF, the author affiliation, and the entry in the grants database) led us to the right funder for the article. Each of those fields individually is not very informative, but the probability of a study listing ‘NSF’ funding that is a) about skiing injuries, b) is authored by Norwegian researchers, and c) by researchers who have a Norwegian Ski Federation grant not being funded by the Norwegian Ski Federation is very low. 

The second take-home is that we could still be wrong. Errors arise in many ways.

For example, this author might actually have intended to write ‘MSF’, or neglected to provide their current (perhaps US-based) affiliation. Alternatively, we could have been misled by the entry in the Ski Foundation grants database, as that grant is actually to someone else with the same name at that institution. The chance of each individual error is often small, but in combination these errors can significantly undermine our certainty in the assertion. Forcing the assertion into a yes/no binary state brushes the uncertainty under the rug, where it could trip up some unfortunate future user of our metadata.

To move away from its current binary state, scholarly metadata should instead come with a provenance (the name of the entity that made the metadata assertion) and a confidence term (a measure of that entity’s confidence in the assertion). This structure has a number of benefits:

1. Many organizations (particularly those using AI) could contribute new metadata about scholarly outputs. For example, DataSeer detects many instances of articles either generating or re-using online datasets. Since human curation is impossible at scale, these assertions about article-dataset connections have AI-related errors. We do not currently pass these assertions to e.g., DataCite, because we know that some are incorrect. A probabilistic metadata framework would allow us to signal our level of confidence in each assertion & enable us to make a more useful contribution to scholarly metadata.
   Contributions from a wider range of organizations (particularly those using AI) will be crucial because these organizations will expand metadata into new territories and fill in major gaps – the fact that some of the assertions are lower confidence is outweighed by our ability to quantify and monitor new aspects of research.
2. Attaching confidence terms to metadata also allows (expert) users to adjust their tolerance for errors according to their needs. For example, a survey aiming to capture as many articles re-using datasets as possible would tolerate more false positives than a survey quantifying a researcher’s output for a particular time period.
3. Confidence terms would also enable contributor organizations or even researchers themselves to identify parts of the academic research graph where adding other metadata fields would increase confidence. In the (imaginary) example above, metadata from the grants database boosted our confidence that the ‘NSF’ mentioned in the acknowledgments was indeed the Norwegian Ski Foundation. In the absence of confidence terms, it is impossible to locate these areas of scholarly metadata. A good example is described here.

Major metadata organizations such as DataCite already have confidence terms for some assertions, but these are hidden within their internal systems. Bringing them into public view would raise broader awareness that scholarly metadata are inherently probabilistic and could – maybe – usher in a new way to connect scholarly outputs.