Authority, Controversial Topics, Economics, Ethics, Metrics and Analytics, Peer Review, Research, Social Role

Reproducibility — An Attempt to Test the Psychology Literature Underscores a Growing Fault Line

Fault Line Running into Cliff near Point of Ly...

Fault Line Running into Cliff near Point of Lyregeo Perpendicular cliff face on left which runs into long deep clefts in foreground and background. (Photo credit: Wikipedia)

We’ve quickly come to understand that reproducibility is a problem within the scientific literature. Companies and scientists are having trouble replicating results reported in journals. This problem has the potential to truly upend the scholarly publishing world, no matter what business model is being used. If trust in peer-review, journals, and articles falls beyond a certain point, the emperor will have no clothes.

The mainstream media is picking up on this problem, as a recent story in the New York Times demonstrates. In a tale of woe from the infectious disease community, an editorial team came to the conclusion:

. . . that science had turned into a winner-take-all game with perverse incentives that lead scientists to cut corners and, in some cases, commit acts of misconduct.

Given the incentives being baked into the system — from payments for publication in high-impact journals, to h-index calculations being used as convenient substitutes for more thoughtful and qualitative evaluation techniques, to the reliance on soft money in academia — this situation is not surprising. The question is, How far is too far? Or, more pointedly, Is “too far” already in the rear-view mirror?

With retractions on the rise (see figure below, from the New York Times), and a perception that more is not better when it comes to published papers, some people are attempting to develop a diagnostic test, if not a potential remedy.

One group is taking the psychology literature to task, according to a recent Chronicle of Higher Education story:

A group of researchers have already begun what they’ve dubbed the Reproducibility Project, which aims to replicate every study from those three journals for that one year. The project is part of Open Science Framework, a group interested in scientific values, and its stated mission is to “estimate the reproducibility of a sample of studies from the scientific literature.” This is a more polite way of saying “We want to see how much of what gets published turns out to be bunk.”

The approach has two interesting aspect — whether the original study procedures can be reproduced and, if so, whether the claimed results can be achieved.

This isn’t the only initiative in the field of psychology. The other, PsychFileDrawer.org, has similar ambitions. While the Reproducibility Project is truly nascent, PsychFileDrawer.org has had contributors attempt to reproduce nine studies — so far, only 33% have been reproduced.

Replication isn’t something the current system has sufficient incentives to encourage — it’s time-consuming and generates papers unlikely to be published in high-profile journals. Even mega-journals aren’t likely to receive these papers for the first reason — they’re time-consuming to conduct. The opportunity cost is simply too large for scientists to do the work.

Replication is also nearly impossible for some trials. The psychology literature is available for this because many psychology experiments involve a relatively small population performing tasks in a manageable timeframe. Longitudinal, invasive, or extremely technical studies aren’t as available to replication. Some studies — especially medical studies involving thousands of patients over decades — can never be replicated.

Scientific claims should be validated and replicated whenever possible. New incentives for such seeminly mundane but ultimately important work should be developed. We may need fewer new theories and more validation steps.

But science is not one thing — it consists of many domains, each different, each particular. Some claims will be harder to replicate, some easier. Some domains will be amenable to open source replication initiatives, while many won’t be. Statistics, careful review, reader-focused thinking, and clear editing matter even more in these fields, where mistakes can have lasting effects — because doing it over simply isn’t an option.

Enhanced by Zemanta

About Kent Anderson

I am the CEO/Publisher of the STRIATUS/JBJS, Inc., the home of the Journal of Bone & Joint Surgery, JBJS Case Connector, JBJS Reviews, JBJS Essential Surgical Techniques, the JBJS Recertification Course, PRE-val, and SocialCite. Prior to this, I was an executive at the New England Journal of Medicine. I also was Director of Medical Journals at the American Academy of Pediatrics.

Discussion

6 thoughts on “Reproducibility — An Attempt to Test the Psychology Literature Underscores a Growing Fault Line

  1. And might not some experiments, like the one conducted by Stanley Milgram at Yale about obedience, become so well known that participants in future replications would not be likely to be fooled into believing they were actually torturing the “victims”?

    Posted by Sandy Thatcher | Apr 26, 2012, 8:26 am
  2. From http://www.lhup.edu/~DSIMANEK/cargocul.htm

    “I explained to her that it was necessary first to repeat in her
    laboratory the experiment of the other person–to do it under
    condition X to see if she could also get result A, and then change
    to Y and see if A changed. Then she would know that the real
    difference was the thing she thought she had under control.

    “She was very delighted with this new idea, and went to her
    professor. And his reply was, no, you cannot do that, because the
    experiment has already been done and you would be wasting time.
    This was in about 1947 or so, and it seems to have been the general
    policy then to not try to repeat psychological experiments, but
    only to change the conditions and see what happens.”

    Adapted from Richard Feynman’s Caltech commencement address, 1974

    Posted by Bill Everett | Apr 26, 2012, 1:50 pm

Trackbacks/Pingbacks

  1. Pingback: Reproducibility — An Attempt to Test the Psychology Literature … « About Psychology - Apr 26, 2012

  2. Pingback: Sort of Significant: Are Psychology Papers Just Nipping Past the p Value? « The Scholarly Kitchen - Aug 21, 2012

  3. Pingback: Are Scientists Themselves to Blame for Exaggerated Claims in Science Journalism? « The Scholarly Kitchen - Sep 17, 2012

Side Dishes by Stewart Wills

Find Posts by Category

Find Posts by Date

April 2012
S M T W T F S
« Mar   May »
1234567
891011121314
15161718192021
22232425262728
2930  

The Scholarly Kitchen on Twitter

SSP_LOGO
The mission of the Society for Scholarly Publishing (SSP) is "[t]o advance scholarly publishing and communication, and the professional development of its members through education, collaboration, and networking." SSP established The Scholarly Kitchen blog in February 2008 to keep SSP members and interested parties aware of new developments in publishing.
......................................
The Scholarly Kitchen is a moderated and independent blog. Opinions on The Scholarly Kitchen are those of the authors. They are not necessarily those held by the Society for Scholarly Publishing nor by their respective employers.
Follow

Get every new post delivered to your Inbox.

Join 13,056 other followers

%d bloggers like this: