Like many of us, I’ve been thinking a lot about transformative agreements lately. One of the thoughts that has been occurring to me is this one: If my organization were to become party to a transformative agreement, how would we evaluate its success or effectiveness? Increasingly intrigued by the question, I decided to go to the source for answers. Twelve sources actually.

I started by referring to the ESAC transformative agreement registry, which tracks transformative agreements globally. I identified the six largest deals (in terms of annual publication volume) that represented unique pairings of publishers and customers, thus ensuring that I’d end up with twelve unique respondents. The resulting cohort was:

• Wiley and Projekt DEAL (Germany)
• Springer Nature and Jisc (UK)
• Elsevier and VSNU-UKB (Netherlands)
• Taylor & Francis and Bibsam (Sweden)
• IOP and TIB (Germany)
• Cambridge University Press and U. California

I then posed representatives of each publisher and of each customer organization the same question:

Five years from now, how will you know whether your transformative deal has been a success?

I have collated the responses I received and posted them, in their entirety in USpace, (the University of Utah’s institutional repository) so that all who want to read and interpret them for themselves may do so freely. Here are my own thoughts about them:

First: It was interesting that some responses came immediately, and others took weeks or months. Some respondents got back to me more or less immediately with answers that showed they had considered the end from the beginning; others’ responses suggested that they and their organizations were still trying to figure out how success would be measured. (Still others apparently thought the question was rather silly, and pointed to the execution of the agreements themselves as the success measure of those agreements; one or two others responded with suspicion of my motives in asking, apparently perceiving this as some kind of “gotcha” question — which I thought was also kind of interesting.)

Second: The length of responses ranged hugely. The briefest response (at 18 words) came from IOP, and it was concise to the point of tautology: for IOP, the implementation of the deal represents the success of the deal, because the terms of the deal represent what IOP hoped to accomplish with it*. Womp. The longest response (at 545 words) was from Taylor & Francis; they outlined four criteria against which the future success of their deal will be measured, explained how they think about those criteria, and laid out some other factors that will likely be taken into account in future analyses. Several other responses were similarly detailed.

Third: The definition of “success” varied significantly — sometimes even between parties to the same agreement. For example, Wiley and Projekt DEAL both agreed that a significant success criterion for their arrangement would be the degree to which it “’move(s) the needle’ on Open Access” (in Wiley’s words), and DEAL opened by saying that it sees “the success of (its) agreement already at its entry point” (à la IOP). The DEAL representative also explained at some length the systemic changes that it hopes will be catalyzed by the arrangement — perhaps summarized best in this statement, which was also echoed by others: “Five years from now, if transformative agreements were no longer needed, that would be the ultimate success.” This is a case in which the two parties to a single agreement defined success in very similar terms. However, in the case of the Springer Nature/Jisc agreement, the Springer Nature side defined its terms of success almost entirely in terms of effects on Jisc members (who will be able to publish more open access [OA], be more compliant with funder requirements, save time and money, etc.), while Jisc defined success not only in terms of the characteristics of this particular deal, but also in broader and somewhat more outward-facing terms: a “move away from legacy pricing to a new framework” and towards additional agreements with other publishers that will enable those publishers to “offer national agreements that remove friction to authors and institutions.” An even starker difference can be seen between Elsevier’s summary of success in its deal with VSNU-UKB (“Success for us is meeting our customer’s needs in a sustainable way”) and that of VSNU-UKB — which basically declined to say anything about how it will evaluate its deal with Elsevier, instead focusing on its larger OA goal (which is “to reach new agreements that are increasingly transformative, aiming for 100% Open Access in the Netherlands as well as in other countries”).

I want to be quick to point out that where there are differences in goals between one party and the other in a particular transformative deal, this is by no means necessarily problematic. It’s in the nature of virtually all contractual agreements that the two parties have different goals. Seeing what those different goals are can be not only interesting in an abstract sense, but also instructive as we try to anticipate the future impacts of these deals.

“Five years from now, if transformative agreements were no longer needed, that would be the ultimate success”

Below, I’ve pulled extracts from each of the responses, focusing on the points that I think are particularly interesting or instructive. Again, though, I would encourage all readers to read the responses in their entirety and see what additional (and perhaps very different) insights they can derive from them.

Wiley/Projekt DEAL

Wiley: “A far greater number of total papers from Germany will be published OA in both gold and hybrid journals… we will (have demonstrated) how a highly selective OA journal developed in partnership with DEAL can thrive.”

Projekt DEAL: Talked about “success” in two layers: local (“I see the success of any agreement already at its entry point”); and global (“Real success will come when these agreements are no longer needed, because scholarly communication will have new modes of operating based on open access”). DEAL sees these arrangements as “(characterizing) a transition, as they build a bridge to the future… Five years from now, if transformative agreements were no longer needed, that would be the ultimate success.”

Springer Nature/Jisc

Springer Nature: Defined success in terms of scope of uptake and the deal’s sustainability, which will be determined by “ongoing constructive and transparent dialogue between publisher, consortia, and members” and by its support for “the transformation to OA on an institutional and regional level.”

Jisc: Outlined several discrete and concrete goals for the program, notably their aims to “contain the costs of publication and subscription access,” to “reduce the cost and administration barriers to hybrid OA publishing,” and “support compliance with UK funder policies.” They also eventually hope to include the other hybrid titles published by Springer Nature within Compact.”

Elsevier/VSNU-UKB

Elsevier: Entire response: “Success for us is meeting our customers’ needs in a sustainable way. In the Netherlands this means supporting them to reach 100% open access, alongside the co-development of open science infrastructure services designed to achieve our collective goal of improving research performance.”

VSNU-UKB: This answer constituted the only genuine non-response: “We don’t evaluate contracts in isolation… the result of all deals combined should be very clear to all who are somehow involved with Open Access.”

Taylor & Francis/Bibsam

Taylor & Francis: Outlined four primary success criteria: sustainability; increase in volume/proportion of OA content; real-world impact of the OA content; development of technology/tools to support OA workflows. Noted that already under the current agreement, OA publication rates by Bibsam authors had risen from 18% (2017) to 94% (2019). Also, in the future, T&F will examine conversion rates of titles from hybrid to full OA, and author satisfaction.

Bibsam: As this agreement nears its end, they will ask themselves “How well did the… agreement pave the way for (subsequent) agreements, and will they lead to a transition from reading fees to publishing fees?” In Sweden, there is a national goal of a complete flip to OA by 2026, so “in 2025/2026 the current transformative agreements should no longer be relevant.”

IOP/TIB

IOP: Entire response: “We would define success as an agreement that enables full open access publishing by authors at participating institutions.” (In other words, success lies in the implementation of the deal itself, and has therefore already been achieved.)*

TIB: Sees things a bit differently. For TIB, there are definite outcomes that would constitute success, and these are largely determined by the fact that their deal “is not really a transformative agreement but rather an offsetting agreement,” one that is “based on subscription costs and not on publication output.” TIB’s success criteria are more focused on what the current deal will allow them to do for their next deal: will it position them to negotiate a subsequent deal in which “costs are divided into a transparent publication and reading fee per institution” that helps to balance out the cost differentials between high-research institutions and others, or will it position them to negotiate a “subscribe-to-open agreement,”one under which “90-100%” of participants’ articles can be published OA “without further cost increase”?

Cambridge UP/University of California

Cambridge UP: Success will be defined primarily in terms of convincing authors to adopt OA publishing: or, in other words, “year on year increases for author take-up of open access publishing, to the point where the deal has achieved fully open publishing from University of California authors, across the Cambridge journals portfolio.” Interestingly (though maybe not surprisingly), O’Rourke very clearly assumes that successful communication of “the opportunity and benefit of publishing open access with Cambridge” will necessarily result in author uptake. (The potential for authors both to understand the value proposition and yet still not agree that it’s the best road for them does not seem to be contemplated.)

University of California: UC is “in the process of presenting assessment plans for our OA agreements to a variety of committees for input and feedback,” so it’s not yet clear, “what the final shape of those plans will look like or how they will specifically define or measure success.” However, they are also looking at criteria such as near-universality of uptake on the part of UC authors; compliance with funding guidelines (i.e., authors contribute from grant funds as called for in the model); affordability and sustainability; workflow efficiency/effectiveness; other libraries/consortia following UC’s example. UC was also the only respondent to cite impacts in the global community, specifically in the global South, as a potential indicator of success.

*IOP Publishing reached out with a clarifying comment: “It’s important to note that the agreement is just the start. In the next five years, we would like to see all relevant articles published OA with every German author under this agreement, and only then would we consider that it had been truly successful.”