The headline, “Scientists reading fewer papers for first time in 35 years” was published online in the news section of Nature by the astute science journalist, Richard van Noorden. This bold claim referred to a new, but unpublished paper, by Carol Tenopir and others who reported on a 2012 survey of reading habits of US-based academics. There are very few longitudinal studies of how scientists read and interact with the scholarly literature and this periodic survey is fundamentally important in understanding how changes in scholarly publishing have changed reading behavior.

Tenopir’s findings–the first time reporting a drop in reading–was indeed novel and newsworthy. Since the 1970s, scientists in each subsequent survey have reported reading more articles and these are the results we expect. When scientists have easier access to the scholarly literature, we expect them to consume more of it, not less. Even if scientists have given up walking to the library and perusing the shelves long ago, we’d imagine that recent changes in journal publishing since the mid-2000’s (open access publishing, broader access to papers from non-publishers’ websites and repositories like PubMed Central, article sharing services like Mendeley, easier remote login for scientists away from campus, among others) would continue to push reading up.

Of course, ‘reading’–if we define it by the attention of human eyeballs does not expand. It has a limit.

The figure in Nature that drew so much notice showed that reading was declining for the first time in 35 years, and when faced with data like these, everyone wants to explain it. Were scientists, like the rest of general public, becoming distracted in this age of mobile devices and social media and turning to Twitter and blogs? Just as the Superbowl turns many into a Monday morning quarterback, these results turned everyone into an armchair scholarly communication analyst.

Before we continue with a statistical analysis, let us look at this figure again. Average numbers of reported readings were lower in 2012 than they were in 2005, but if we put less emphasis on 2005 results, we could simply be witnessing the extension of an ongoing linear trend. The headline, “Scientists Continue to Read More Articles, Survey Finds,” doesn’t seem very novel or newsworthy anymore, at least for Nature magazine.

According to the final manuscript, which was accepted for publication in an upcoming issue of Learned Publishing, faculty respondents were asked to report how many articles they read in the previous month. The researchers multiplied this response by 12 (to arrive at an annual estimate) and calculated the average response for each sample. This is what is reported in the figure above. Even a cursory glance shows that the average response in 2012 was lower than in 2005. Or was it?

If the researchers were able to get responses from each and every faculty member in the United States, then such a comparison from year to year would not require much statistical analysis–a comparison of averages would be sufficient. However, the researchers were comparing two different samples of faculty responses taken from two different populations of potential respondents. The 2005 survey received 1,304 responses (28% response rate), compared to 837 responses (8% response rate) in 2012. Both surveys were sent to faculty by their librarians and only one of the five institutions between the two samples was the same.

One could question whether the study suffers from sampling bias and response bias. For the sake of argument, I’m going to assume that these two samples are similar enough and therefore can be used for comparison purposes.

What alerted me that these two samples (2005 and 2012) may not be so different was a few lines from their results. The variation in reported reading responses was very high. In 2005, some faculty reported reading as few as zero articles per month to as high as 255–whether you can trust this self-report or not, this represents an estimated 3060 articles read per year for that one respondent.

But if you accept, again for argument sake, that these extremes are based on actual experience, Tenopir is dealing with a very skewed distribution that would be highly sensitive to extreme values. In cases like this, reporting the median would give a much better estimate of what most scientists are doing than the mean. For the same reason that economists report median family income–to avoid biasing the results by the incomes of a few super rich–I wondered whether reporting average reading behaviors was really the right way to present the data. If Tenopir’s data were reanalyzed, were scientists really reading less for the first time in 35 years?

Responding to my queries, Tenopir did reanalyze her data using medians and confidence intervals, and discovered that there was no statistical difference between 2005 and 2012. Faculty are reading about the same amount in each year. A correction was posted in Nature and I admire the integrity and professionalism of both the researchers and the journalist for correcting the record. I am also happy to be part of the self-correcting mechanism of science.

The headline for the Nature piece now reads “Scientists may be reaching a peak in reading habits.” And the first paragraph reads as follows:

A 35-year trend of researchers reading ever more scholarly papers seems to be levelling off. In 2012, US scientists and social scientists estimated that they read, on average, 22 scholarly articles per month (or 264 per year). That is, statistically, not different from what they reported in an identical survey last conducted in 2005. It is the first time since the reading-habit questionnaire began in 1977 that manuscript consumption has not increased.

But is this replacement narrative–that reading habits have reached a plateau–now accurate? In my view, it is not. While we know that 2012 responses are no different than 2005 responses, this view ignores a general linear trend in the rest of the data (figure above). We won’t know if the trend is “leveling off” until another sample is taken in another 5-7 years. Now, one could get wrapped up in the semantics of how the word “may” was used in the title and paragraph, but I feel that the journalist replaced a novel–but inaccurate–narrative with an equally novel–but inaccurate–one. Without seeing the data and relying on just the figure, my headline would have read, “Scientists still reading more each year. Digital publishing cited as factor,” but that is neither novel nor newsworthy. It does, however, fit in what what most of us know and expect from the data, newsworthy or not.

What do you think? Does the new data support a new narrative that reading has leveled off? Does anyone have data to support (or contradict) this theory? And for the editors and science journalists among you, how would you have dealt with this situation?