I have always liked to think of myself as a good listener. Whether you are in therapy (or should be), conversing with colleagues, working with customers, embarking on strategic planning, or collaborating on a task, a dose of emotional intelligence – that is, embracing patience and the willingness to listen — is essential.
At the American Mathematical Society, we recently embarked on ambitious strategic planning effort across the organization. On the publishing side we have a number of electronic products, pushing us to consider how we position these products for the next generation of mathematician. We quickly realized that it is easy to be complacent. In our case we have a rich history online, and yet – have we really moved with the times? Does a young mathematician need our products?
We came to a sobering and rather exciting realization: In fact, we do not have a clear idea how mathematicians use online resources to do their research, teaching, hiring, and job hunting. We of course have opinions, but these are not informed by anything other than anecdotal evidence from conversations here and there.
To gain a sense of how mathematicians are using online resources, we embarked on an effort to gather more systematic intelligence embracing a qualitative approach to the research – ethhnography. The concept of ethnographic qualitative research was a new one to me – and it felt right. I quickly felt like I was back in school and a graduate student in ethnography, reading the literature, and thinking through with colleagues how we might apply qualitative research methods to understanding mathematicians’ behavior. It is worth taking a look at two excellent books: Just Enough Research by Erika Hall, and Practical Ethnography: A Guide to Doing Ethnography in the Private Sector by Sam Ladner.
What do we mean by ethnographic research? In essence we are talking about a rich, multi-factorial descriptive approach. While quantitative research uses pre-existing categories in its analysis, qualitative research is open to new ways of categorizing data – in this case, mathematicians’ behavior in using information. The idea is that one observes the subject (“key informant” in technical jargon) in their natural habitat. Imagine you are David Attenborough, exploring an “absolutely marvelous” new species – the mathematician – as they operate in the field. The concept is really quite simple. You just want to understand what your key informants are doing, and preferably why they are doing it. One has to do it in a setting that allows for them to behave naturally – this really requires an interview with one person not a group (because group members may influence each other’s actions).
Perhaps the hardest part is the interview itself. If you are anything like me, you will go charging in saying something along the lines of “look at these great things we are doing. What do you think? Great right?” Well, of course this is plain wrong. While you have a goal going in, perhaps to see how an individual is behaving with respect to a specific product, your questions need to be agnostic in flavor. The idea is to have the key informant do what they normally do, not just say what they think they do – the two things may be quite different. The questions need to be carefully crafted so as not to lead, but to enable gentle probing and discussion as the interview progresses. It is a good idea to record the interview – both in audio form, and ideally with screen capture technology such as Camtasia. When I was involved with this I went out and bought a good, but inexpensive audio recorder.
We decided that rather than approach mathematicians directly, we should work with the library at an academic institution. Libraries are our customers. The remarkable thing about academic libraries is that ethnography is becoming part of the service they provide to their stakeholders at many institutions. We actually began with a remarkable librarian, based at Rice University – Debra Kolah. She is the head of the user experience office at the Fondren Library of Rice University in Texas. She also happens to be the physics, math and statistics librarian at Rice. Debra is remarkable, and has become an expert in ethnographic study of academic user experience. She has multiple projects underway at Rice, working with a range of stakeholders, aiming to foster the activity of the library in the academic community she directly serves. She is a picture of enthusiasm when it comes to serving her community and to gaining insights into the cultural patterns of academic user behavior. Debra was our key to understanding how important it is to work with the library to reach the mathematical community at an institution. The relationship is trusted and symbiotic. This triangle of an institution’s library, academic, and outside entity, such as a society, or publisher, may represent the future of the library.
So the interviews are done – then what? Analysis. You have to try to make sense of all of this material you’ve gathered. First, transcribing audio interviews is no easy task. You have a range of voices and much technical jargon. The best bet is to get one of the many services out there to take the files and do a first pass transcription. They will get most of it right. Perhaps they will write “archive instead of arXiv, but that can be dealt with later. Once you have all this interview text, you need to group it into meaningful categories – what’s called “coding”. The idea is that you try to look at the material with a fresh, unbiased eye, to see what themes emerge from the data. Once these themes are coded, you can then start to think about patterns in the data. Interestingly, qualitative researchers have developed a host of software programs to aid the researcher in doing this. We settled for a relatively simple, web based solution – Dedoose.
With some 62 interviews under our belt, we are beginning to see patterns emerge in the ways that mathematicians behave online. I am not going to reveal our preliminary findings here – I must save that up for when the full results are in – but I am confident that the results will show a number of consistent threads that will help us think through how to better serve our community.
In summary, this experience has been a fascinating one – a new world for me. I have been trained as a scientist. As a scientist, I have ideas about what scientific method is, and what evidence is. I now understand the value of the qualitative approach – hard for a scientist to say. Qualitative research opens a window to descriptive data and analysis. As our markets change, understanding who constitutes our market, and how users behave is more important than ever.
Carry on listening!
16 Thoughts on "Ethnography: A Scientist Discovers the Value of the Social Sciences"
In days of yore great editors wore out the soles of their shoes and not the seat of their pants. Most learned their trade as college sales reps who were called college travelers. All that came to an end in about 1989 when companies said just write a bunch of letters and see if anyone is writing anything.
We used to call ethnographic research making cold sales or editorial calls!
I applaud what you are doing, although I might call it issue analysis rather than ethnography. In either case you are right not to begin with an imposed category system. The issues need to emerge from the interviews and there may not even be names for them. I call this meeting the beast.
In this context a mathematical point might be useful. A transcript is a linear structure, with one sentence after another. But the structure of the thoughts expressed is tree-like. See my http://scholarlykitchen.sspnet.org/2013/07/10/the-issue-tree-structure-of-expressed-thought/. This means there is a lot of jumping around among different lines of thought in an interview, because you cannot map a tree of thoughts into a linear array of sentences without putting closely related pieces relatively far apart. Moreover, different interviews may contribute different lines of thought to the underlying issue tree. So you need to find the scattered piece and put them together, in order to properly frame the issues. Good luck with that beast.
Excellent! Can’t wait to see some of the conclusions you reach for the future of your organization. And, I hope you did not look any of the mathematicians directly in the eye while observing them in their natural habitat!
I went to a fascinating presentation at this year’s UKSG conference about ethnography (one of the other attendees, Paula Cuccurullo, wrote a great blog about the session here: http://uksglive.blogspot.co.uk/2015/03/breakout-session-group-through.html). I have to admit the field was new to me, but it chimes very closely with the kinds of “user observation” and “behavioural research” that I was commonly undertaking under the guise of market research. Yours is a great example of the exciting trend in which previously disparate activities are coming together as publishers and societies partner more closely to understand how people actually engage with content. Thanks very much for sharing some of the details of your experiences!
The classic book in this genre is Laboratory Life (1986) by Bruno Latour and Stephen Woolgar: http://press.princeton.edu/titles/2417.html. You might want to taker a look at it.
as a publisher of ethnography books, I’m delighted to see this. There is a whole literature on the subject and disciplines, like anthropology, that have been refining these techniques for over a century. Good that mathematicians have finally discovered that you can learn things by talking to humans.
i have given lots of talks comparing the role of an academic book publisher to that of an ethnographer. Studying academic tribes to learn their culture is essential to figure out who and what to publish based upon an understanding of what the tribe considers important.
I highly recommend Danny L. Jorgensen’s (1989) Participant Observation: a methodology for human studies. Rather than approach the sociocultural milieu under investigation as an outsider, the value of Jorgensen’s methodology lies in observation from the perspective of a participating insider.
Looking forward to seeing your full findings! You should know that there is a small but lively collection of ethnographic researchers already producing detailed studies of mathematicians themselves. In addition to my own articles (preprints on mbarany.com) there’s an excellent book by Claude Rosental, a number of important papers by Christian Greiffenhagen, and a major ongoing research project in Scotland and England led by Ursula Martin, to give a few examples of recent work available in English.