Editors’ note: Today’s post is by Michelle Urberg, Head of Data at LibLynx.

Libraries and publishers face an increasingly complex ecosystem of robotic activity on scholarly content platforms. Some bots are actively malicious or waste valuable platform capacity for no benefit and should be blocked (“bad bots’\”), but many provide valuable services to knowledge platforms (“good bots”). They help researchers and students engage with scholarly content, or they help archive content for preservation, as with the LOCKSS archival bots.

However, good bots come in all shapes and sizes. Depending on organizational policy, some merit a higher level of access to drive discovery or inference, while others can be relegated to lower tiers of service. Those agents acting on behalf of human researchers should be included in usage reports.

This post was inspired by a virtual panel at the Charleston Conference, earlier this month, which featured a conversation with David Bietila at the University of Chicago Library, Jeremy Morse at Michigan Publishing, and Rikki Gracia at IEEE. In the solutions-oriented spirit of this year’s conference, we call on publishers and libraries to take better control of the bots invading the scholarly communications ecosystem.

Decorative image: Robot holding key card standing at open doorway

What Is a Bot Taxonomy? Do We Need One?

Until now, the nuances of robotic behavior have been largely overlooked because the bots weren’t so bad that they overloaded content platforms (as they are now for universities and publishers alike). Until recently, bots were a necessary cost of doing business, especially for open access platforms with content open to everyone on the internet. For all types of platforms, if a publisher didn’t want to deal with bots, they could simply be blocked without deeply affecting end user experience. Currently, however, bot blocking is having cascading effects for every piece of a platform ecosystem: Time to access is slower; users can be incorrectly being rejected from platforms; platform providers continue to be hammered.

Our community has now reached a tipping point where we need to define these robotic actors who are seeking information on content platforms of all kinds. We need new tools to manage bots differently than we did in the past — for example, specialized security filters to allow, block, or throttle bot activity, or standardized metrics for recording that activity. But, in order for these to work, a vocabulary describing bots is needed.

We know bots engage in a number of behaviors with content platforms, including:

  • interacting with content at a server level, repeatedly authenticating against a platform in the attempt to access content
  • reading content for the purposes of discovery and inference
  • scraping content for the purposes of archiving or training on a set of content.

Some bot behaviors result in a high number of repeated authentication attempts against a platform. Others result in a large number of requests for content. Publishers and libraries need better control over these interactions. In this case, information is power. If we can categorize and catalog these bots, we will be better equipped to handle authentication and recording the usage generated against publisher and library platforms.

At present, a number of lists exist that catalog bots, including the Cloudflare Bot Directory and the COUNTER list of robots. The COUNTER list is fairly brief, identifying just under 200 bots. The Cloudflare list included just 750 bots until recently, when it doubled to more than 1,500 and seems to be growing — but it is still recording only a fraction of all the bots on the internet.

These catalogs are also functionally siloed and built for specific purposes. The broader need for industry-specific bot information will only grow over time. We are proposing that a taxonomy of bots can help knowledge platforms take back control over content and understand who (or what) is accessing that content.

What We’re Learning

At LibLynx, we are exploring these ideas via interviews with thought leaders and conversations like the one held recently at the Charleston Conference. The four of us came together to discuss where we are as an industry and how we might solve the problem of defining who these bots are.

What concerns you about the bots landing at your platforms?

In general, we are seeing similar concerns across these three stakeholders. Volume and nefarious robotic behavior is endemic. Keeping costs down for managing robotic traffic is of paramount importance.

IEEE, a publisher with primarily controlled-access content, is concerned with violations of copyright and bots behaving with “smash and grab” scraping behavior. For the University of Chicago Library and for Michigan Publishing, whose policies dictate having a publicly available catalog and a significant amount of repository content open, managing the rate of the robotic activity is crucial. The staff and server costs are hard to control in the current environment. Usage statistics are not very reliable when different kinds of bots cannot be disambiguated from each other or from human-generated usage. For all three organizations, the bots need to be let in, but in a controlled way that aligns with policy and mission.

What is the most important issue for your organization?

The most significant issue for these organizations is telling: bots need to be given greater direction about where to go and whether they are welcome. IEEE would like to keep bot traffic relegated to the right access point. Chicago wants to prevent one agent from tying up the entire platform and to protect discovery for all users. Michigan would like adequate tooling to differentiate between the bots and agents behaving on behalf of humans and those bots running automated processes. While usage is important for stakeholders across scholarly communication, at this point, understanding who the bots are and their known behaviors is still more important than accurately tracking usage metrics.

Where should we go next?

The purpose of this panel was to find common ground with issues surrounding bot management and to propose some ideas about how bots can be categorized. It is still early days for this kind of taxonomic project, but the group generated a preliminary set of information about robots, which we invite the community to consider.

We agreed on a few items:

  • Who is this bot and what do they call themselves?
  • How does the bot identify itself? (and what identifiers does the bot use to identify itself?)
  • What content is sought by the bot? (e.g., only abstracts? only full content? both?)
  • What is the bot’s stated purpose, based on known behaviors (e.g., training, shopping, indexing, archival)
  • What is our level of confidence about the bot’s legitimacy? (and can we trust this bot to be who it says it is?)
  • Does the bot know how to evade turnstile mechanisms?

And a few things need to be sorted:

  • Intention is very, very hard to measure with usage metrics. Other information needs to stand in as a proxy for intention (e.g. such as a bot’s stated purpose). What should this information be?
  • Other tagging specific for the scholarly communications ecosystem needs to be selected and standardized. Tags available in the Cloudflare Bot Directory are not fit for purpose and the table currently hosted by COUNTER best meets COUNTER’s needs, as opposed to the broader need of knowledge platforms in our industry. What other metadata do we need to identify and catalog the bots?

If a bot taxonomy is doing its job, what would bot traffic management look like?

Meaningful bot management brought the panel back to usage and content management. There is hope here! If bots are managed effectively, content access will be better controlled. Bots generate tons of data points that can be distinguished from each other with the right metadata and analysis tools. With better bot handling, usage metrics will inherently become cleaner and measure what we want to measure.

Conclusions

The shared concerns and needs of IEEE, University of Chicago Library, and Michigan Publishing indicate that there is common ground to be found across different kinds of stakeholders working with knowledge platforms. Even with different platform designs, policies, content, and end users, knowledge sharing about bots will be possible.

By the end of our conversation, the excitement was palpable about tackling bot management with a taxonomy. David Bietila (University of Chicago Library) raised important and immediate questions to address in constructing a taxonomy:

  1. Who is the expected audience for a bot taxonomy? Is it something that individual libraries can feasibly make use of, or is it more helpful for those at the level of engineering bot management solutions?
  2. Is it a concern that bad actors may see the taxonomy? Could that enable them to find exploits and circumvent categorization?
  3. How would a taxonomy be future-proofed? As new categories of legitimate and illegitimate bot use emerge, how would it be updated? Can we design it to be extensible or anticipate some of the future categories of bots?

Some of these questions can be addressed with robust policies for managing the taxonomy. But others, such as the question about extensibility of the taxonomy, call into question who should take responsibility for managing and maintaining it. This is a big project, but an important one.

All panelists agreed, a community-led bot taxonomy is one part of the solution to bot management. If you have informed opinions about the future of bot management and would like to contribute to this collaborative Bot Taxonomy project, please add your thoughts below or contact Michelle at michelle@liblynx.com.

Michelle Urberg

Michelle Urberg

Michelle Urberg is the Head of Data at LibLynx, where she manages COUNTER integrations, analytics reporting, data workflows, and client data needs. She has previously worked for EBSCO Information Services and as a metadata librarian in the ExLibris business unit of Clarivate. Michelle holds a PhD in Music History and an MS in Library and Information Science. She is an active volunteer in SSP, NISO, and ALA. She co-chairs the COUNTER Advisory Committee and is keen on making the world a better place one piece of metadata at a time.

Discussion

Leave a Comment