Is Science Being Distorted?

A recent article in PLoS Medicine crossed my threshold. It’s not a very good article, and normally I’d let it slip into oblivion on its own, but its arguments are so labored and tiresome that it got under my skin a little.

The authors’ basic assertion is that an oligopoly of top-tier journals is distorting science by hogging more than the proper share of the “economics” of information.

The article goes off the rails pretty quickly, comparing research reports to economic commodities. Let’s ponder this comparison a moment, just as an appetizer.

A commodity is something that, by definition, can’t be differentiated by quality. So, zinc is zinc, and oxygen is oxygen. They are commodities. Research reports do, however, vary in quality. Therefore, they are not commodities. Bang, off the rails, like I said. They even realize it, but can’t seem to stop writing:

Usually we do not know what information will be most useful (valuable) eventually.

If that’s the case, then it’s not a commodity. If all information were equal, any pound of it would do as well as any other. That’s clearly not the case.

Then the authors go over the bridge and plummet off the trestle, into the river below, in a twisted mass of tortured logic by asserting a “winner’s curse” phenomenon exists in this economic system. This is when someone who wins an auction finds that the actual value of the item is much lower than what they paid, precisely because they just won it and set a value that others discount as soon as it’s pegged. Where exactly “the auction” exists in all this is a bit vague. Is it the journals vying for papers? The authors deliberating where to submit? The readers investing mindshare in the results? Ultimately, the authors assert the latter in a long-winded manner: that consumers of scientific information are dazzled by the big journals’ brands, so reports in those journals cause them — patients, administrators, other researchers — to put too much stock in those reports when similar experiments are obscured and given less credence. Hence, the winner’s curse. To me, this looks like a smoldering trainwreck at the bottom of Logic Canyon.

Coming from PLoS Medicine, such a messy and simplistic set of assertions surprised me (along with the rambling, barely edited text of the article). Medical research differs from bench research. Often, large patient populations have to be recruited, randomized, monitored, and modeled for years on end. Novel findings are highly prized, and definitive studies that won’t be replicated for decades due to their very nature serve as landmarks. The reports that emanate from the best experiments often come from top-tier institutions, funded by the largest sources and led by cream of the crop researchers.

This is the framing error the authors make. They see journals as the cause of research emphasis (a more accurate and less loaded term than “distortion”), when journals are the result of research emphasis. Funding agencies, academic programs, and individual researchers largely set the agenda of scientific inquiry. Patients and their advocates have more voice in setting the research agenda than journals ever will. Want proof?

• Is AIDS research published more often because journals have set a research agenda? Or because the NIH and others have funded it more aggressively than, say, malaria research?
• Are autism-vaccine articles still being published in journals because researchers haven’t settled the question, or because sensationalism in the public sphere is driving the agenda?
• Did “alternative medicine” research begin appearing in journals because the editors believed the hypotheses were interesting, or because politicians funded research under pressure from political action committees?

The tiresome part of this is the obsession with journals that continues to emanate from PLoS. Yes, journals can make money because they attract an audience others also want to reach, or an audience willing to pay for the information. That’s economics. But the forces driving what is researched comes from elsewhere: funding agencies, scientists, private foundations, public politics, and governments. The research agenda is not driven by the economics of information commodities, as if anything more complex than the binary code of 1-0-0-1 could be an information commodity.

Science is distorted to some extent by the human enterprise, there is no doubt. It is, after all, a human endeavor, and suffers some of the foibles humans bring to anything (as well as the virtues). However, to have an obsession with journals to the exclusion of so many other factors in research — scarce funding, scarce talent, and political realities converging in a world of uncertainty offset by dedication, technique, and creative thinking — strikes me as the real source of distortion here.