In a guest post here last fall (“Exploring the ‘Hopes and Fears’ About Generative Artificial Intelligence in Web Scale Discovery”), I extended an invitation to Scholarly Kitchen readers to participate in a NISO Open Discovery Initiative (ODI) survey about the positive and negative expectations of generative AI in discovery tools. ODI has completed its analysis and published a white paper (Generative Artificial Intelligence and Web-Scale Discovery).
Many Scholarly Kitchen readers took part in this survey, so I wanted to share the key findings with this community and keep you posted on how the ODI is responding.
The survey garnered 224 completed responses from individuals who identified themselves as belonging to publishing, library, and vendor organizations. While we did not ask respondents to identify their country of residence, we did ask for their institutional affiliation. Of those who provided an affiliation, most could be identified as being located in the United States, with small numbers of respondents in other countries in Europe, East Asia, and South America. There are five key findings of our survey.
Hopes were consistent
Library and Content Provider respondents share common hopes about Generative AI in discovery. Their mutual top three choices were:
- Improved visibility of content. Respondents were generally hopeful that AI tools would help surface content they created or licensed. This perhaps calls out shortcomings in keyword-based search, not finding relevant materials when searched for using broader or narrower terms than actually appeared in the text, as well as the challenges of the “long tail” of search results, when, as experience shows, most users do not go past the first page of results.
- More accurate content recommendations. Respondents were generally hopeful that AI tools would provide information seekers with improved guidance for related or similar articles, presumably for the same reasons they felt content would be more visible.
- Saving staff time. Library respondents thought that AI tools would save the time of researchers who use these systems, while content providers thought they would save their own time. While respondents were not directly asked how time would be saved, free-text responses later in the survey suggest that AI tools could speed up the discovery of literature for researchers. Content providers saw Generative AI as having the potential to accelerate important but tedious tasks in content classification and description by providing starting points for human expertise.
Fears were more varied
There was less uniformity among fears, with each cohort responding in ways that were more closely aligned with their interests. Interestingly, both cohorts ranked “lack of transparency about inputs” in their top three, though presumably for different reasons.
- Library respondents’ top three fears were Misrepresentation or fabrication of content, Lack of transparency about inputs, and Quality of AI-generated summaries. These interrelated responses highlight one of the key characteristics of Generative AI systems: that internal decision-making and pattern-matching are opaque to even AI experts. It can be hard for individuals to evaluate the outputs without already being deeply knowledgeable about the subject matter, which may counteract the benefits of an AI-generated summary in an area the user is not already deeply familiar with.
- Content Provider respondents’ top three fears were similar: Copyright infringement, Misrepresentation or fabrication of content, and Lack of transparency about inputs. Of interest here is the diverging copyright infringement concern as the top concern, which makes sense when the product of the service is what is being reproduced or summarized.
Librarians see Generative AI enhancing web-scale discovery
Library respondents generally felt that Generative AI tools would enhance the value of web-scale discovery for users and staff, though not overwhelmingly so. In this sense, respondents seemed to view the balance of hopes and fears on the optimistic side, that users would, on the whole, benefit from the powers of Generative AI. Finding ways to reduce, or at least provide transparency around, the characteristics described as fears might help alleviate these concerns.
Content Providers are mixed on access and visibility
Content Provider respondents expressed hope that Generative AI would, in most cases, improve the visibility of their own content, while being less positive about the effect on access to it. This somewhat paradoxical combination of improved exposure with less visibility (the implication is that credit for the content may be lacking, while the content is used or shown) can be partly explained by the lack of direct and/or accurate references to the works that some Generative AI tools use to generate their summaries.
The environment matters
Library respondents, in particular, expressed concern about the potential environmental risks posed by the data centers needed to create Generative AI models and process queries. Many users supplied this response in a free-text box, reflecting concerns about the more global effects of potential improvements to traditional search.
Next steps
Based on this survey, the ODI Standing Committee recognizes the need for more transparency in Generative AI-enhanced discovery. While some concerns we learned about are broader than what any specific tool does, the overall trend indicates a certain lack of trust in black boxes. (This recalls the reason the ODI was originally created: to foster transparency among stakeholders in what were then-groundbreaking web-scale discovery tools.)
To that end, ODI plans several efforts to extend our decade-long work to enhance mechanisms for transparency in web-scale discovery to the use of Generative AI:
- Present these findings to the community through conference presentations and a NISO webinar, to invite community feedback on these findings.
- Develop one or more guides to foster transparency among libraries, content providers, and discovery providers around the use of Generative AI in web-scale discovery.
- Continue to engage with stakeholders to explore the overlapping hopes and concerns of content providers and libraries, and to further explore areas of mutual interest.
If you have any questions about the survey, white paper, or ODI’s next steps, please contact us at odi@niso.org — or add a comment below.
