Over the past few months, a major scandal has been unfolding in economics. It involves a paper by two Harvard economists who claimed that when the debt load of a country reaches 90% of that country’s gross domestic product (GDP), growth slows significantly.

The paper became a cudgel in the hands of political forces urging governments to adopt austerity measures in response to the global financial meltdown of 2008, leading to major cuts to outlays across Europe, where the ideas found their most centralized and immediate application, and to the budgetary stinginess of the US Republican party, led by their champion, Paul Ryan.

The problem is that the paper is very likely wrong, its conclusions based on selective data and mathematical errors. In fact, it may be exactly wrong — that is, when the math is corrected and all the data included, debt load doesn’t seem to slow economic growth. In fact, it may actually help economies grow while reducing debt.

Carmen Reinhart and Kenneth Rogoff (who is also a chess grandmaster) published their controversial paper in 2010 in the American Economic Review. It was not peer-reviewed.

While the paper had been used in numerous venues to drive austerity policy, it wasn’t until April 2013 when Thomas Herndon, a graduate student at the University of Massachusetts Amherst released a paper refuting the findings of Reinhart and Rogoff, stating that:

. . . coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period.

There are so many interesting angles to this story it’s hard to know where to begin. So, let’s try listing them.

1. The variability and role of peer-review. While it has been rightly pointed out that peer-review may not have caught mathematical errors or selective use of data, it may have caught the fact that all four references in the paper are to prior works from Reinhart and Rogoff. But there is a more subtle problem, which comes down to the fundamental branding of a journal as a “journal.” That is, a “journal” is widely interpreted as containing peer-reviewed content. American Economic Review is a journal (“a general-interest economics journal”). However, the paper in question was published in a section called “Papers and Proceedings,” which is not peer-reviewed. It is misleading to have variable levels of peer-review within a single journal, not to mention non-peer-reviewed content in an information form predicated on peer-review. We are now in a publishing environment where how a paper has been peer-reviewed is becoming increasingly important, as variable and experimental forms of peer-review are emerging all around us. We can no longer assume that publication in a journal equates to peer-review, or when peer-review occurs that the term means what we think it means. This is a problem.
2. The post-publication review process. As David Crotty stated in a wise comment yesterday, the stochastic process of post-publication review is no substitute for structured pre-publication peer-review. This is a case in point. Not only did the post-publication peer-review consist mainly of mindlessly accepting the paper and trying to score political points with it, but the problems were not revealed for years — years during which unemployment in Europe skyrocketed, US public policy became tied in knots over austerity politics, and fiscal policy in many other nations remained frozen in fear of debt.
3. The power of incentives. Scientists are busy. Social scientists are busy. If there is no incentive to do work, it’s unlikely the work will get done. In this case of this paper, post-publication peer-review only occurred when it was assigned to a graduate student. That is, it was assigned review. There was an incentive to get it done. There was no assigned pre-publication review, and organic post-publication review raised none of the problems later identified. It was assigned, incentivized post-publication review that caught the problems. If we continue to believe that post-publication review will occur without incentives, that’s our mistake.
4. The damage one paper can do. The entire vaccines and autism debate was sparked by a single fraudulent paper in the UK. We had a decade of fear-mongering and social upheaval (and, very likely, a number of unnecessary deaths from preventable diseases like measles and mumps when parents refused to get their children vaccinated). Now we have high unemployment and increasing debt in Europe because a paper seemed to indicate this was the direction to move. We have long-term unemployment and crumbling infrastructure in the US for the same reason. As we publish more papers, maybe the “single paper” event will diminish. Or, perhaps there will be more entry points for flawed papers that might misdirect us and cause significant damage. It’s hard to know, but the risk is real either way. One bad paper can do a lot of damage, yet we’re emphasizing convenience for authors, reduced levels of peer-review, and rapid publication.
5. The lack of skepticism and accountability in an era of polemics. There is a core and perhaps false dichotomy of opinion in economics, between Keynesian intervention and growth vs. Hayek’s free market and austerity approaches. Readers of their actual writings actually believe the two disagreed less often than is commonly portrayed, but in our age of polemics, every proponent needs an opponent, so Keynes and Hayek have become mutual foils, deservedly or not. This means there are now two ideologies in a battle, a state of affairs that is all too common. The paper by Rienhart and Rogoff tapped into the austerity proponents’ point of view, and was embraced. However, the polemical attitude didn’t allow for a response probing the facts, only affording a more emotional response. This happens far too often. Once a position is staked out, it is defended to the point of rhetorical exhaustion. Apparently, this also makes it less likely for the facts to be the focus of intellectual activity.

It is this last problem that seems to be the most important and pernicious. When criticized for their mathematical and data-inclusion errors, Reinhart and Rogoff have repeatedly responded emotionally, the first time on Reinhart’s blog, as if the only issue is that they are being attacked:

. . . it has been with deep disappointment that we have experienced your spectacularly uncivil behavior the past few weeks.  You have attacked us in very personal terms, virtually non-stop, in your New York Times column and blog posts.  Now you have doubled down in the New York Review of Books, adding the accusation we didn’t share our data.  Your characterization of our work and of our policy impact is selective and shallow.  It is deeply misleading about where we stand on the issues.  And we would respectfully submit, your logic and evidence on the policy substance is not nearly as compelling as you imply.

This is standard defensive behavior, but it seems to me that it’s unacceptable for scientists. If you’re right, you’re right; if you’re wrong, you’re wrong. Making math errors and cherry-picking data should be impossible for a scientist to defend.

I’ve seen this sort of socially pervasive attitude many times recently, in situations when a person’s arguments or facts (or both) are pointed out to be simply and purely wrong, yet they persist — not by defending their arguments or facts, but by claiming to now be attacked, misportrayed, or misunderstood.

We see this behavior far too often in the age of polemicism and ideology, which has seeped into the sciences in many ways — the attempt to fight off facts with emotion and fend off critics with cries of form over statements of substance. It brings to mind what Daniel Patrick Moynihan famously said:*

Everyone is entitled to his own opinion, but not to his own facts.

How we establish facts, validate them, communicate them, and interpret them remain fundamental challenges.If we succumb to pride, hubris, and vanity rather than to the humility of scientific proofs and processes, we will make troubling and multi-year mistakes again and again.

* Hat tip to JI for reminding me of this appropriate quotation.