Nicholas Carr, who expanded his “Is Google Making Us Stoopid?” routine through the publication of “The Shallows,” recently wrote a post about a PLoS ONE article entitled, “Microstructure Abnormalities in Adolescents with Internet Addiction Disorder,” in which Chinese scientists studied 18 adolescents with something they term “Internet addiction disorder” or IAD, and compare their brain scans to similar scans from “normal” subjects. The authors of the study claim that IAD led to structural changes in the white and gray matter of the brains of those suffering from it. Carr “cautiously” touts the findings — he concludes that the results need to be confirmed by further studies, but you can almost hear him cheering in the background.
There are a few big problems with the study. Let’s go through them.
While the authors claim to be able to detect changes in the brains of the IAD subjects, they only detect differences between those they diagnose with IAD and those they don’t, while also noting that IAD is “not yet officially codified within a psychopathological framework.” So, not only do they not have before-and-after evidence of change, but they are using a framework that is probably edging toward a garbage can diagnosis. Wondering this, I went to the seven references the authors cite to bolster their statement about IAD’s validity. Of these seven, only one resolved to an actual PubMed or Google Scholar citation. Now, maybe PLoS ONE is having trouble with CrossRef, PubMed, and Google Scholar linking, but it seems odd that all three wouldn’t work, especially for straightforward search engine queries.
Because there’s no before-and-after comparison, there’s no way to claim cause-and-effect. Are people with certain gray and white matter brain compositions and more depressive natures drawn to video games and the Internet as ways to cope? Here’s one sentence from the paper:
IAD resulted in impaired individual psychological well-being, academic failure and reduced work performance among adolescents.
Let’s rearrange that to reverse the assumption of causation:
Adolescents with impaired psychological well-being, academic failure, and reduced work performance retreated into the Internet.
Which one do you find more plausible?
Then there’s the whole bias toward the medicalization of differences — you spend a lot of time on the Internet, so you must be sick. If that were the case, I should be hospitalized immediately, along with nearly every other person who works in an office. The threshold for “sick” was 10+/- hours of Internet use per day for about six days each week. Please. I think there’s something wrong with the others at this point, the ones deemed “healthy,” adolescents who used the Internet for less than one hour per day and for less than two days per week. And I’ll wager that these researchers themselves probably qualify on a couple of the IAD criteria.
Carr also points gleefully to a Scientific American article covering the study and cherry picks quotes that support his world view. Let me cherry pick (and tell you I’m cherry picking) one that gives more than a little pause:
The reason why Internet addiction isn’t a widely recognized disorder is a lack of scientific evidence.
This is from neuroscientist Nora Volkow of the National Institute on Drug Abuse. We can also cherry pick the last paragraph of the article:
In the end all of the researchers interviewed by Scientific American emphasized significance only goes so far in making a case for IAD as a true disorder with discrete effects on the brain. “It’s very important that results are confirmed, rather than simply mining data for whatever can be found,” Goldin says.
Goldin is Rebecca Goldin, a mathematician at George Mason University.
So, is there something here? Maybe, maybe not. But we need to raise the bar for skepticism. Too many times, we see a p-value and think everything’s OK when the real problems may emanate from things p-values can’t measure — i.e., citing papers that have vanished, proposing causality when it was impossible to measure, basing illness on criteria that aren’t validated, and on top of all that medicalizing something that has become normal for most people.
And I’ll harp again on one of my major concerns about how the spigot for scientific papers continues to be opened — here we have a “methodologically sound” paper that has huge conceptual problems within it, published in a journal that publishes the majority of papers it receives, the combination of which creates the illusion that rigorous and interesting science has been conducted and made it through a tight filter.
Maybe Google is making us stoopid — but maybe the “us” is looking back from the mirror.