You probably remember reading about it — the experiment allegedly demonstrating human precognition, something akin to telepathy. It was published in a reputable journal, the results were statistically significant, and the media coverage was plentiful, if a little sporadic. There was something a little hesitant about it, as if even the reporters were having trouble accepting the basic premise, but the New York Times, New Scientist, the Guardian, the Telegraph, and New York magazine all covered it.
But it was a positive result, it was statistically significant, so it was published.
The study used cards, some of which had erotic pictures on the back, to test whether subjects could look into the future and discern which card they would soon discover had a risque image on its reverse side. (Some wondered if the test was for telepathy or pornotelepathy, but that’s beside the point.)
Ben Goldacre, a prominent voice of skepticism in science, was, like most of us, nonplussed by the results, wondering inherently about the paper’s extraordinary claims, which needed to be backed up by extraordinary evidence.
And what are extraordinary claims? It seems some psychologists enjoy double-speak, judging from a recent article in Psychology Today about this same precognition paper. The author, Sathoshi Kanazawa (himself an evolutionary psychologist with a PhD), quibbles with Carl Sagan’s quote that “Extraordinary claims require extraordinary evidence,” parrying the Sagan quote with Elaine Boosler’s “Popcorn is magic if you don’t know how it happens.”
Aside from the inanity of attempting to repudiate Sagan with Boosler, Kanazawa is being disingenuous — we can experience popcorn together, watch it pop together, buy it readily, and eat it endlessly. Popcorn is nothing extraordinary — not making it, not buying it, not eating it. How it happens isn’t a mystery — water in the kernel boils when heated, and the expanding steam causes the kernel to explode. No strange mechanism of action, nothing extraordinary about the results, nothing hard to observe or experience. And pretty tasty.
Claiming to be able to predict the future is not an ordinary claim. Seeing into the future breaks the physical laws of the universe as we understand them, and is currently considered impossible. It would be absolutely amazing if someone could do it. Therefore, it requires a lot of evidence to show that someone can actually predict the future. Not only that, but you’d better have a theory you’re pursuing, a framework in which it might be conceivable.
Back to Ben Goldacre. Aside from likely having a deep problem with the underlying study, his problem in his recent post is with the aftermath of the media coverage around this paper, namely that when other scientists replicated the experiment and found no effect, those negative results didn’t receive nearly the amount of media attention the initial findings received.
Not that it didn’t receive any attention — the blogosphere tackled the paper immediately, with one psychologist who blogs noting that “I don’t believe a word of it because a) let’s face it, it’s about precognition and b) there’s simply no effort to propose a mechanism that might support such an outrageous claim.”
Goldacre’s narrow complaint is about the mainstream media and how it continues to miss opportunities to provide more than merely superficial coverage. With niche coverage emerging all around, this flaw is more and more apparent. He ends his piece with the following observation:
What’s interesting is that the information architectures of medicine, academia and popular culture are all broken in the exact same way.
The information architecture of popular culture has been broken for a while now, but may be healing thanks to blogs, search engines, smartphones, and the like — things that take control out of the hands of the few and put them in the hands of the many. We’ll see if the accumulated wisdom of niche experts and endless monitoring from users can compensate for superficial coverage from marginal journalists.
But the mainstream media problem may be the least of the concerns this study raises. While an idealist might think it was a good thing to have the precognition study published because this allowed for attempts of replication and ultimately refutation, there is perhaps a greater cost in science around misinformation — the costly distraction of doing needless experiments to disprove claims based on statistics sans theory. How empty of theory was this paper? Let the authors explain:
The term psi denotes anomalous processes of information or energy transfer that are currently unexplained in terms of known physical or biological mechanisms. The term is purely descriptive; it neither implies that such phenomena are paranormal nor connotes anything about their underlying mechanisms.
Psi . . .
Theory is important here — it’s the “so what?” part of the experiment. So what if there is a slightly less-than-random chance of someone guessing there’s a dirty picture on the back of a card? What’s the theory? What does it portend? That people’s minds can travel forward in time? What’s the proposed mechanism for this? Can we control it? Are the brains of those who performed better observably different?
If extraordinary claims are being made, it’s a waste of everyone’s time if there’s no theory supporting them and if the evidence supporting them isn’t extraordinary.
Many of the studies refuting the initial experiment will be published, and the instigating study will likely be debunked. The problem isn’t just getting psychologists to know the study didn’t hold up. The problem isn’t only that a distracted public spent a few minutes rolling its eyes over a convoluted precognition study in a scientific literature it already thinks is pretty unreliable. The more pragmatic problem is the time wasted by scientists having to replicate and refute the wild assertion that was statistically significant but not theoretically meaningful.
Things like this make me reflexively feel we should publish less, be more selective, and stop polluting the literature if something just doesn’t make sense or isn’t likely to make a whit of difference in anyone’s life.
Niche publications in the sciences break their bond with their audience, but not in the same way the mainstream media does. The mainstream media breaks it by abandoning the audience after titillating coverage. The journals world does it by publishing too much baseless research and thereby wasting scientists’ time.
Negative results will become known in the sciences, but the effort taken to produce them in the face of strange outcomes from pointless experiments is much more costly than a throwaway newspaper story.