Throughout Peer Review Week 2020 we have emphasized the importance of trust in a complex and dispersed process of peer review. We have also emphasized how trust and review are linked aspects of ethical practice throughout scholarly communications. I wanted a perspective on how the concepts and the practices of trust and review function outside of a context specifically associated with scholarship but still within the ecosystem, so I asked fellow Chef Roger Schonfeld to sit down across a zoom screen to talk about Ithaka S+R.

Ithaka S+R is the research and strategy arm of ITHAKA, “a not-for-profit with a mission to improve access to knowledge and education for people around the world.” Many in my home discipline of history, as well as others, may know Ithaka most regularly through JSTOR, their aggregator for scholarly content. Many of us may know Ithaka S+R , which “partners with foundations, consortia, and individual institutions to seek solutions and inform policies in a rapidly changing environment,” through their surveys and reports. I had been thinking a lot, for example, about the “Student Experiences During the Pandemic Pivot” report that Ithaka S+R issued in June, and how that project likely relied on trust among partners and a review process, and it’s a key reason I wanted to hear more about the wider context of trust and review for Ithaka’s work.

Could you tell us more about Ithaka S+R’s public-facing work?

We are probably most well-known for our research work, because that’s what we publish. This includes a dozen or more research reports in a given year, plus issue briefs and other kinds of publications. I think many Scholarly Kitchen readers will be familiar with our work on scholarly research practices – for example our faculty survey and studies of research practices by field – as well as our work on scholarly publishing and academic libraries. In addition, S+R has done work in depth on art museums, access to higher education, and student success, and this year we are conducting a number of projects looking at the university research enterprise, some areas that might not be on the radar of many Kitchen readers.

I think one of the things for me that has been essential to our work is clarity about our audience, which is a community of mostly practitioners. They’re folks involved in scholarly and professional publishing or academic librarians or museum professionals or they are provosts or policy makers or senior research officers. We are thinking about their needs, whether for the academic community or the museum community, and how we can help them keep pace with those needs as the context in which we do our work keeps changing. So, that’s where the research component comes in – helping them lead their organizations and accomplish their objectives.

Does a declining sense of public trust in science and scholarship – even expertise generally— affect the work you’re undertaking?

If you’re working with a community and an audience that’s primarily academic, you can’t help but feel caught in this changing environment around expertise and trust and confidence. We work closely with scientific researchers and senior research officers and scholarly publishers who are grappling with these issues, and who are also working to have more of a public role in the midst of all of this.

There’s also the politicization of research and certainly we’ve addressed some controversial and sensitive issues in our research, including diversity and equity issues. Obviously on some level, you know, higher education is politicized and so all of our work could be, on some level. But it certainly hasn’t caused us to avoid issues that we think are important or that are important to the community. That’s a huge part of maintaining that trust relationship we have with the community.

How important is it for Ithaka S+R’s work to be trusted, and how do you cultivate trust?

We have a community that we’re trying to serve. Our work is not about, here’s a report, we’ll send it out over the transom and we’ll see later on what happens. Every time we do something, we have a chance to do a little better than we did before, and to have a bit more impact. It can’t have impact if it’s not trusted. And we can’t do our best work if we’re not trusted in the community.

[An example is the June report on “COVID-19 and the Student Experience”]  So COVID arrives and our team made a decision that although we’ve got all the work in the world in front of us, we thought the thing we needed to do is to use the tools, the capabilities that we have, the expertise that we have, to respond to the moment. Two or three people sat down and drafted that survey instrument. Then we said, well, in order to run that survey, we need to partner with colleges and universities. So we had to knock on the virtual door of several dozen institutions and say, hey, we’ve got this survey, would you like to run it? There were all these things that happened within weeks. And then we fielded a survey that had to, you know, get all sorts of institutional approvals.

We couldn’t have done that if we were not a trusted party, no one would have partnered with us to do this. The whole thing was based on the fact that we were trusted and known. And we promised that we were going to share the data back to them. And then we had to figure out, well, how will they trust us to aggregate their data, so that we can publish the findings and share them more broadly? So I guess what I’m saying is there’s a whole set of trust things that are not happening only within the publications, but that are happening in the backend of the relationships that we’ve built, that we can be trusted when we say, well, we’re going to take data from your students and aggregate it anonymously and appropriately and ethically in a way that can help not only locally but also more broadly.

So there’s the institutional connections and relationships. Also, we’ve built trust at an individual human level with an awful lot of people. Whether it be my colleague, Christine Wolff-Eisenberg working with community colleges and the basic needs community or my colleagues Kurtis Tanaka and Danielle Cooper, building relationships with prison education programs and departments of corrections, everyone is kind of thinking about “who am I trying to engage with?” We want them to get to know us. It’s a trust relationship at that kind of level. And look, sometimes, sometimes people won’t like us or won’t trust us. But the hope is that we’re both doing the right things with the work, but also reaching individuals within the community of practice.

Does your work get reviewed, and is that part of how the work gains trust in the community you serve?

There are really two answers to that. We try to be transparent about methods of data collection, and we deposit our data whenever possible with a trusted archive. So there’s transparency on the outputs.

There’s also transparency on the inputs. Most of our projects have a set of advisors associated with them and we take their advice really seriously. As an example, we do our faculty survey every three years. And we’re very intentional about like, okay, we had this set of advisors last time. How do we refresh that group and think about making sure we’re getting a different set of perspectives, different set of institutions, different set of individuals. And we have protocols to test our research instruments before rolling them out broadly – for example, we conduct what are called “cognitive interviews” to test our survey questionnaires and adjust them as necessary before fielding them.

Then we have internal processes for every report. Everything that we publish is read by a colleague or two internally. In addition, we have a colleague who serves as our editor who is also herself a subject matter expert. She’ll cheerlead the things that need to get cheerleaded through, and she will screech to a halt the things that aren’t ready. And at some point in the process, almost everything we publish will be read by outside readers as well – often the same project advisors who contributed earlier in the process, but sometimes a different group. Now it’s not an anonymous double-blind review, but in my experience, the richness and candor of the feedback in our process can be as valuable, and sometimes more so, than in a peer review process.

There’s another thing to note, which is that because a lot of our work is sponsored by grants or by organizations in the space, we are transparent about that, which I think is really, really important. But we are also clear in every agreement we make that we’re the ones with editorial control and review at every level and not the sponsors.

Also, all of our work is about collaboration. For me, the distinction between collaboration and review is not a clear line sometimes. Even something that is single-authored is a part of a collaboration in one way or another and that’s the whole ethos of our work.

One more thing. When you’re writing to have an impact in a community of practice, collaboration and review is essential. You don’t want to put something out there that, if you’d done this or that differently it would have been more useful, had more impact. I think there’s a very pragmatic way to think about the value of review and trust.