Juries and peer-review emanate from a common wellspring — the notion that peer judgment can help validate arguments, sharpen judgment, and triage information. In the art world, there are peer juries, for instance. The phrase “a jury of one’s peers” informs the notion of peer-review. Yet, saying they’re related is one thing. Seeing how they’re related, and feeling the differences first-hand, is quite another. When you’re sitting in a jury box, you realize that courtroom peer-review is significantly tougher, more intimate, and more consequential than the kind we deal with in scientific publishing.

I was recently a juror in a superior court for a murder trial. It was my second stint as a juror, having served as a juror (and jury foreman) for an embezzlement case back in the early 1990s in another state. But a lot has changed for me over the past 23 years, and a murder case is vastly more intense than an embezzlement case.

Over the last 20+ years, I’ve learned a lot more about scholarly peer-review. I’ve completed reviews, and had my papers reviewed. As a publisher, I’ve seen what peer-review can’t handle, and the resulting subpoenas, retractions, expressions of concern, scandals, and lawsuits. I’ve worked with editors and lawyers on all of the above. But mostly, things go well, and sometimes, peer-review saves the day — I’ve seen papers with surprising results validated by peer-reviewers and published to great effect and acclaim.

I know peer-review is necessary but not sufficient. It’s like mortar — you need bricks, sealer, a skilled mason, and a sound architectural plan to finish a building. But if you throw out the mortar, the building falls down.

When you’re sworn onto a jury, there’s a lot on the line. Our reviewers make recommendations about the fate of a paper, which may or may not affect someone’s career trajectory. Jurors in a criminal case determine whether a defendant walks free or goes to prison.

Something akin to editorial review starts for a juror starts as the jury is filled. This review is all about ensuring that the jury is qualified — somewhat akin to the qualification differences between actual peer-reviewers and commentators. The courts want people who have a good chance of rendering a fair decision. First, prospective jurors are reviewed by the court in relation to the trial at hand. This evaluation includes potential conflicts of interest — acquaintance with anyone involved, predisposition to hold certain views — and other factors, such as suitability and availability. No cuts are made at this point, but the information is recorded. Then, the court narrows the jury pool down through a bit of the adversarial nature of the system — the defense attorney and the prosecuting attorney together review each juror (with the judge presiding), and must agree to allow a juror to be seated. This is voir dire (commonly translated as, “see them say”). Each side has a number of challenges they can use to disqualify a juror. These challenges require no explanation — the lawyer using one has no obligation to justify the challenge. If a challenge is invoked, the juror is sent back into the jury pool. Since the biggest cases get the biggest initial pools, there is a natural movement down the ladder for jurors at this point. Smaller cases get the leftovers. There is a juror cascade system. We went through more than 100 potential jurors to get 16 seated. It was about a 15% acceptance rate.

Once the jurors were reviewed and approved by the court, testimony began. This is when the jurors begin their role as a jury of peers — that is, peer-reviewers of both sides being argued, and fundamentally peers of the accused. In the end, one argument will be mostly accepted, and one will be mostly rejected. A hung jury is a rare event.

Our first task as a jury was a bit unexpected, however — we went on a view, a visit to the places related to the events in question, so that we could see how various houses, streets, and alleyways existed relative to one another and what the terrain was like. We were bundled into a shuttle bus, with two police cars both fore and aft, lights on, escorting us to three different sites. The attorneys were only allowed to draw our attention to various things — the rough ground, the relationship of one street to another, an important house bristling with surveillance cameras (because its owner was often away and a little eccentric, we would later learn). Imagine visiting the lab or hospital of the corresponding author to check out how the work was done. Newer video abstracts hint at the interest this can engender.

The adversarial nature of the court ensures argument. Nobody has to act as the devil’s advocate, taking a stance to ensure arguments occur. The argument is built in. This actually makes the jury process both more demanding and much clearer. Unlike peer-reviewing a paper, there is an alternative perspective and a clear critique being offered by both sides.

Within the testimony of the trial, witnesses were consistently validated through their educational backgrounds, certifications, qualifications, and experience so we could judge whether the person was speaking with some authority and knowledge. But these qualifications only went so far. Not every witness was impressive, despite their qualifications and experience. Some wilted under cross-examination. Some were absolutely amazing people, far beyond even the most impressive qualifications.

There was a background of peer-review to the proceedings, as well. For instance, the forensic evidence was stringently peer-reviewed before it reached us, with experts checking on experts before the labs could testify to a result. After all, failure can have vast consequences, as a recent set of retractions from a Massachusetts forensics lab has shown.

For the jury, confidentiality and independence were exquisitely important. Each day, we were instructed not to discuss the case, do any online research related to the case, read news accounts, or answer questions. We were only allowed to rely on the evidence presented in the courtroom, and had to form our own individual opinions prior to deliberations. Even our mini legal pads were left behind at the end of each day. We couldn’t mull our notes overnight. The jury box was a separate world for us.

As the case ended its first full week of testimony, my thoughts were very unsettled. I really wondered if the prosecutors could overcome the presumption of innocence and reasonable doubt. Yet, I had nobody to talk with. I had no outside information. I only had my recollection of the courtroom proceedings and my own untested thoughts. I was part of an impaneled jury, but we were each alone, prohibited from speaking with one another about the case, each forming an independent opinion as the trial continued, making innocuous small talk to fill the breaks and lunches. This took a toll on each of us, I’m sure. After the trial was over, fellow jurors spoke of difficulty sleeping, of spending most of their waking hours contemplating the case. And without anyone to talk with, the burden of knowledge and responsibility weighed heavily indeed. We all craved some clarity. Where will the evidence lead us when all is said and done?

Ultimately, the prosecution was playing what seems in retrospect to have been a game of Texas Hold ‘Em, a progressive poker format in which betting proceeds as cards are revealed. The prosecutors allowed the three-card flop to sit on the table the first few days, allowed the Turn to come near the end, and then, on the last day of testimony, the prosecution revealed a River card that changed everything. It wasn’t definitive forensic evidence. It wasn’t a surprise witness. It was just the right card at the right time. And it made the junk I seemed to have in my hand suddenly coalesce into a flush.

We moved into deliberations. But first, our numbers had to be whittled down. Only 12 are allowed to deliberate, but 16 had been seated, allowing for four alternates. This is a little bit of a reverse of scholarly peer-review, where if you need extra reviewers, you invite them after the first round ends in a draw, so to speak. Here, the extras were built in, and set aside at the last minute. Nobody knew who the alternates would be. They weren’t selected until just prior to deliberations. Once the foreperson was named by the judge, an open lottery was held, and the court clerk called out the juror numbers he drew as alternates. After sitting on the jury for a seven-day trial, nobody wanted to become an alternate. We were all too invested. But a small number had to be alternates and skip deliberations. When their number was called, alternates were sequestered in a separate small conference room until we were done, were not allowed to know what we decided, and could not discuss the case themselves. They didn’t know our verdict until it was announced in the courtroom. This must have been really frustrating.

Within the actual deliberations, it was immediately clear why it’s important to have a good number of perspectives. Some jurors wanted to focus on certain evidence again; others wanted to talk through something that had proven confusing; some brought different ways to interpret various evidence; some saw things others had missed; and a few had already reached a conclusion.

We also needed to know the relevant law — clear guidelines about what we were judging. This is also an important part of peer-review, and one I think gets lost sometimes. We had more than 70 pages of jury instructions, including definitions of each charge and what it took to find for that charge. Journals and books should articulate their standards with as much precision as possible, as well. Otherwise, what are the peer-reviewers looking for? What fulfills the criteria you’re looking for? We went through these very carefully, to ensure that we were dispensing technically accurate decisions.

The judge was also extremely important, and this reminded me of something I’ve written about before — the importance of editorial review and rejection. Without the judge to enforce the law, rule on objections, consult with lawyers at sidebar, elucidate the aspects of the law we were to use, and run an efficient process, we would have floundered. A presiding official seems indispensable to any peer-review system that functions effectively over the long-term.

Our deliberations were not easy. They were frequently intense and momentarily heated. But they resulted in a decision we were all comfortable with. Nobody took it personally. Everyone wanted to get it right.

Deliberations reminded me of an amped-up version the best peer-review meetings I’ve been to, which are invariably held in-person. Unfortunately, not many journals can do this anymore, but there is a huge difference between disembodied reviews wafting their way in over email, and presentations, questions, and arguments shaping the outcome of a decision. I don’t think we realize how important this can be to quality. More journals should do this, and if they do it now, fight to preserve it.

As we delivered the verdict, a gasp went up in the courtroom. I couldn’t tell if it was vindication, shock, or anger — probably a bit of all three. Some people wept. Everyone was emotional in some way, except perhaps the judge (another indication of his value). As jurors, we were straight-backed and resolute. We had to get through it, hold ourselves together. We’d just delivered our verdict.

But it wasn’t over. Even our decision was reviewed. In what we learned later was a rarity, the defense asked that each juror be polled about the verdict, which meant we each had to stand individually and affirm our verdict for each charge verbally before the entire court. We all completed this without a problem. I credit this to both the strength and resolve of my fellow jurors, but also to what the fires in the deliberation room forged — a comfort with the verdicts we’d reached, even if the consequences made us uncomfortable.

Only after we returned to the jury room — the case finished and the verdict delivered — did the emotions start to come. The stunned silence, the haggard looks, and the slumped shoulders revealed the toll seven days of a murder trial had taken on this group — as diverse and lovely a group of people as you could hope to ever meet. Even days later, I found myself contemplating what I’d seen, felt, and experienced.

Peer-review has many layers in a trial court. Yet even in the emotional, high-stakes, and decisive setting of a murder trial, peer-review plays a vital role in the rational, fair, and reasonable evaluation of jurors, evidence, facts, and testimony. It is only part of the judicial system, and just like in science, it is a vital part. It is the mortar that binds the community together in an effort to discern the truth.