The Economist recently suggested that the cure for reproducibility problems for scientific studies may lie in dispensing with peer review, “in favour of post-publication evaluation in the form of appended comments.” But correcting the record, and public perception after the fact is not an easy task.
Despite nearly constant refutations, disbarments and retractions, we still have a significant population that believes in a connection between vaccines and autism. And the New York Times has recently offered a feature on one of the great misunderstood legal cases in recent history:
More than 20 years ago, 79-year-old Stella Liebeck ordered coffee at a McDonald’s drive-through in Albuquerque, N.M. She spilled the coffee, was burned, and one year later, sued McDonald’s. The jury awarded her $2.9 million. Her story became a media sensation and fodder for talk-show hosts, late-night comedians, sitcom writers and even political pundits. But cleverness may have come at the expense of context.
The video below shows how a relatively clear cut case of corporate negligence was (and remains) twisted to suit different agendas. Something to consider when thinking about a “publish everything, sort it out later” approach to the literature.
Discussion
5 Thoughts on "Correcting the Record Is No Easy Task: The McDonald's Coffee Spill Case"
What a lovely, retraction-free world of traditional peer review you must live in, David.
But on a more serious note: would you include in your assessment here mixed forms of peer review, with some pre and some posted, but all solicited by and published in the journal alongside the article?
Retractions suffer from the same sorts of problems as seen here. Phil Davis has written about this a few times in the past, notably here:
http://scholarlykitchen.sspnet.org/2012/08/10/the-secret-life-of-retracted-articles/
And here:
http://scholarlykitchen.sspnet.org/2012/02/29/can-article-retractions-correct-the-scientific-record/
Which both point out the importance of doing our best to get it right the first time.
From my point of view, the more filters we can provide, the better. Post-publication peer review (if it’s possible to implement well) adds another layer, which is always welcome in an age where most readers struggle to keep up with the literature. I like to think of it as a “more please” situation rather than an “either or”.
“Retractions suffer from the same sorts of problems as seen here.”
This was my point. I was wondering why you brought this problem up with specific reference to the idea of post-publication peer review, with the implicit message being that the result of pre-publication peer review is somehow free of this kind of problem.
The other implicit message in the post is that the Economist article actually recommends “either or”, so I’m glad you spelled out the distinction between that and “more please”. My read of the suggestions made in the article — though I think they could have been stated more clearly, but you know, word counts and all that — is that an overall better system of scientific knowledge sharing through publication would involve two peer review stages (so, to be clear, very much in the “more please” camp): the first at the study design stage (“Ideally, research protocols should be registered in advance and monitored in virtual notebooks.”) and the second post-publication. If a researcher designs a study well enough to be accepted at the first stage, then they are not disincentivized to publish a negative result if that’s what they get, and (the author argues) this will benefit the research community more than the strong incentive under the current system to publish a positive result even if you didn’t get one.
Far from being a suggestion that “the cure for reproducibility problems for scientific studies may lie in dispensing with peer review”, the article actually makes a suggestion that you might agree with. How about commenting on that instead of essentially propagandizing the issue by pointing to the McDonald’s case as if it straightforwardly settles the matter?
I read that Economist article and all I could hope is their other news is better researched. It mentioned none of the pitfalls of post-publication peer review. Thank you for pointing out some of the nuances of the issue.
Where’s the beef? A misunderstood tort case about over-boiled flavourless coffes should not be confused with post-publication peer review of serious scientfic investigation. The two are totally unrelated.
