During last fall’s Charleston Library Conference, I joined a panel discussion on how we are measuring the societal benefits of open-access (OA) publishing. Librarians, content providers, and service vendors face growing demand to measure how scholarly publishing (in particular OA publishing) enhances people’s lives – by reaching underserved populations, upgrading manufacturing standards, influencing better policies, or improving healthcare practices, etc. However, we lack best practices for leveraging usage data to assess social impacts, as our existing tools were designed to calculate returns on investments, such as universities and other organizations funding research projects and libraries subscribing to journals and databases.
While societal impact is not the only meaningful metric of scholarly publishing, a fundamental reason we conduct (and fund) research is to improve people’s lives. The key to any sort of impact is getting that research seen, yet how we measure the connections between visibility and impact is insufficient. Usage reports provided to funders rarely express social value or public benefits or presume that popularity metrics, such as hits or views, are an adequate proxy for such impacts. And yet at the same time, libraries and other research funders are increasingly expected to measure value and demonstrate broader impacts of scholarly communications. A 2023 Clarivate survey demonstrated that research impact is the second highest priority for Research Offices and researchers around the world (after sourcing funding) and that societal benefit is expected to become the most important impact metric in the very near future.
It is time to rethink how we leverage usage data and web analytics to tell the story of how publishing impacts our global communities. Our sector needs a new approach to visualizing, contextualizing, and exploring trends in how and why readers are using OA publications, and what those patterns mean to society. In this post, I aim to share a vision for the future of storytelling with OA publication data and, together, to overcome the logistical, technical, and procedural obstacles that stand in our way.
Community needs & challenges
In this Charleston session, we discussed the expectation that librarians and content providers can leverage usage data to express beneficial impacts of various kinds and the obstacles they encounter in meeting those expectations. Traditional usage reports do not adequately capture the broader context necessary to assess the contributions to public welfare and progress delivered by research publications (let alone the academic or financial benefits). Prevailing analytical tools such as Excel are not designed to address the questions facing our sector and often require a higher level of data literacy to effectively express human patterns within thousands of rows of tabular data. Notably, the difficulty of harnessing sophisticated data visualization tools, like Tableau, to unearth the meaning within usage metrics was echoed in other sessions across the Charleston conference.
Despite decades of work to standardize usage reports, libraries and other funders today are not equipped to measure the broader impacts of the OA content they finance via cooperative and transactional models. And content providers struggle to measure the impacts of publications presented in various formats and versions, hosted by multiple platforms. Even when traditional metrics can be effectively aggregated and contextualized, COUNTER was never designed to measure impact in this way. One cannot assume downstream social benefits from metrics such as requests or downloads, as these are devoid of contextual variables about who is requesting or downloading and why. A pageview or article hit means something happened, but we need metrics about who and where and in what subject area, plus some clue about why, such as the type of organization and their area of specialization.
Kicking off our Charleston panel, Elliot Hibbler from Boston College discussed the importance of ensuring their faculties’ research reaches beyond paywalls and benefits our wider society. However, he acknowledged that this requires libraries to track usage not just within their own institutions, but also globally, in order to calculate the costs and benefits of making research fully open. Hibbler said libraries need to aggregate metrics that represent who is using the research, how frequently, and for what purposes; clearly, these are very difficult to do with basic web traffic or download metrics. Hibbler also noted the privacy concerns that arise when tracking individual user behavior and the importance of collaborating with publishers to establish privacy safeguards.
Kesley Mrjoian from the University of Michigan Press (UMP) echoed these challenges, particularly when measuring the impact of OA monographs and their benefits to both local and global communities. While usage data can indicate higher engagement due to the lack of paywalls, Mrjoian noted that it is not enough to determine the true impacts on the public at large or specific populations. Mrjoian also shared how UMP has brought qualitative metrics to bear on usage data, to provide context and a richer understanding of publishing impacts, such as this author and reader study. In light of library concerns about user privacy, Mrjoian discussed how UMP is working to balance sharing valuable impact data while protecting personally identifiable attributes of its readers.
Storytelling with data
“Stories are data with a soul.” ~ Brene Brown, researcher and author
Stakeholders in scholarly communication are hungry for stories of how research publications influence the lives of people around the world. Yet, our existing data and analytical tools are not fit for purpose in today’s publishing and research environments. Industry-standard usage metrics are two dimensional, designed for closed-access models with basic ROI calculations, and are therefore insufficient to express how scholarly communications impact the world at large.
Instead, these generic counts of downloads or logins could be enriched with both quantitative and qualitative metrics to add context and meaning. Context-rich analytics triangulates the right metrics into a three-dimensional story about the impacts of OA publishing programs. For example, if we can observe strong readership of a new open journal on infectious diseases within a particular research team in Africa, then we can begin tracking its potential benefits to African healthcare practices and patient outcomes.
Storytelling with data means we shift from showing data to narrating with data. Communicating effectively with data is both a matter of good visual design and good data architecture. At LibLynx, we are tackling both as we rethink usage metrics to address the challenges facing libraries and publishers. We believe that, in order to tell a fuller story, we must build multidimensional metrics and intuitive data visualization tools that triangulate the data that empower libraries and content providers to tell their stories.
Attendees of this panel at last year’s Charleston conference spoke of struggling to effectively explore and unearth patterns in usage analytics, even if they were investing in fancy software and/or data scientists. The time is right to break free from these limitations and develop new methods for collecting and sharing contextualized data. Our sector is eager to use data to tell the human stories locked within usage reports, which can be enriched and contextualized to build a more valuable narrative.
A vision for the future
Where we used to rely on bibliometrics to assess impact, stakeholders in our community need new approaches to express the impact of research on global citizens. It is time for our community to meet this moment and elevate our ability to narrate the value of scholarly publishing with multi-dimensional, context-rich data and analytics tools. These tools should be delightful and easy to use. They should be accessible, adaptable, extensible, interoperable, and portable. They should allow us to playfully explore trends while also presenting illogical mashups.
We need new metrics that better tell the stories within OA usage reports. Initiatives such as the Open Science Monitoring Initiative and the Open Science Impact Indicator Handbook from PathOS (Horizon Europe) are important steps in the right direction. The new metric developed by OAPEN in 2023, the Transparent Open Access Normalized Index (TOANI) score, is evidence that our community is getting creative in addressing this gap. Efforts like TOANI demand a unified effort to inspire uptake and achieve the scale required to make a difference. Our community is overdue in developing ways to leverage usage reports and other assets to quantify the benefits and impacts of OA programs. If we do not meet this moment, we risk the sustainability and future successes of OA publishing.
A few actionable recommendations come to mind to better equip our community with better metrics of social impact:
- Publishers should engage with key OA funders to understand their needs in terms of social value and begin to benchmark discrete impacts on specific communities.
- Publishers should explore collaborations with funders, from libraries to private grantors, in mixed-methods experiments, bringing qualitative insights to bear on usage reports to paint a fuller picture of engagement with content.
- Publishers and service providers should enhance traditional reports with richer metadata and additional analysis from platforms and distribution channels to triangulate data points for a richer contextual view of content usage.
- Analytical user experiences should incorporate easy, friendly data-visualization tools that do not require an advanced degree and provide easy access for publishers and libraries to explore and narrate stories found within their data.
What other ways can we collaborate to more effectively measure the social impacts of OA publishing? Please add your thoughts in the comments below and further this important conversation.
Discussion
9 Thoughts on "Telling the Story of How Open Access Benefits Society: A Vision for the Future"
One aspect to bear in mind is the way in which the Open Access model has been used by some for-profit, volume-oriented publishers with little regard for quality or integrity. The term “predatory publisher” has been used, but even legacy publishers are being drawn in by the enticing financial mark-ups (for example, Wiley bought Hindawi, which anybody would have told them was suspect, and ended up having to vacate that stable of journals, at great cost to Wiley’s reputation). I am not sure how helpful this comment is because it makes the business of producing useful metrics that bit harder, but in the end they will be more reliable, valid, and insightful if they take this into account.
Thank you for the post and the Charleston panel, Lettie. I’m interested in how time factors into this important work. The human species is notoriously short-sighted. I don’t want research, especially basic research, to be left behind as we work to tell stories of how research artefacts are used and benefit society. What time frames should we consider when assessing the investments society makes in research? This is not an entirely rhetorical question.
In case you missed this on Tuesday, I talk about the disparities between funding timelines (1 to 5 years) and the real impact of a lot of research (decades): https://scholarlykitchen.sspnet.org/2025/02/04/revisiting-measuring-societal-impact-or-meet-the-new-metric-same-as-the-old-metric/
Thank you, David. Yes, I did skip it. I will head back there this afternoon.
Thank you, Robin, for your comment — and you raise an important point (as does David’s post from earlier this week) re: the long tail of impacts and influences research can have on future knowledge. This is why metrics like hits or views alone cannot capture the full picture of how publications benefit the world. We need new/additional metrics to contextualize usage data and those may or may not be the same across fields of study (e.g., tracing connections between journal articles and patents could be of value for some areas, such as robotics or medical tech, but would fall short for others, say linguistics or math). We must begin experimenting with these new metrics and meet this important moment with innovative solutions!
Coincidentally, an LSE post today addresses this topic as well: https://blogs.lse.ac.uk/impactofsocialsciences/2025/02/05/if-impact-is-created-but-never-shared-does-it-make-a-difference/
Because I’m on my way to retirement I’ve been taking a break for reading “the Scholarly Kitchen”. But the title caught my eye. Thanks, Lettie, for revealing the current concerns re OA. My comments are four. They are short and meant to be positive. First is the thinking you describe appears to be focused on the ideal of universal access to research as distinct from the mishmash (and possibly the limited potential) of OA practices. Second, it is arguable that (perhaps regulated) commercial activity could do a better job of spreading knowledge than mandated OA. Look at what “Nature” and “Nature Briefings” are doing to spread knowledge (and ensure their long-term survival). Third, roughly speaking, if the words “open access” were replaced with “research” e.g., “How RESEARCH benefits society” not much would change in your narrative. Fourth, machine-based measures will never tell the whole story. In my research on usage, I identified data patterns and suggested interpretations that made sense to any socially aware, thinking human mind. In ending, bearing in mind the various “Publishers should” statements, how about this? Librarians and OA enthusiasts should celebrate the immense value added by publishers to the dissemination of knowledge in society and work with them to achieve even more.
Thank you, Dr. Lorimer, for your comments — and your research and publications on this topic!
On your third point, yes, I agree. The reason I’m calling out OA publications here is the expectation (from funders) that publishers demonstrate the social benefits of their OA programs in ways they are often not expected to do with traditional/closed programs. I am very interested in reading more about the data patterns and interpretations you’ve developed in your research. Thank you!
I am curious, just who the wider audience is. Surely not me or most folks I know, including scientists who are not involved in the research being discussed in an article.
Considering the practice of science involves incremental steps there should not be an expectation of immediate benefits to society. In my opinion, the explosion of published material is probably more a hinderance than a benefit.
