Editor’s Note: Today’s post is by Karen Stanwood, Editorial Director, Health Care Books and Journals at SLACK Incorporated. Karen has been in the journals group at SLACK since 1995, having worked her way up from Assistant Editor to Editorial Director. In her current role, she’s the point person for workflow implementation, efficiency, and standardization focusing on policies/procedures and technical troubleshooting.

Changing peer review systems, while sometimes necessary, is a daunting prospect and not a decision made lightly. The peer review process is the foundation of many journals, upon which their reputation is built. A great deal of thought and work goes into ensuring a good experience for authors, reviewers, and editors, and the idea of ‘starting over’ with a new peer review management system can make you break out in a cold sweat. But maybe the journal has expanded beyond its home-grown solution, maybe editorial boards are clamoring for updated features or functionality, maybe you’re dissatisfied with the level of customer service/support you receive, or maybe it’s the price tag. Sometimes you need a new system.

The risks inherent in changing systems can be costly in terms of the time, energy, and effort required to accomplish a smooth transition, and unfortunately, there don’t seem to be any standards or guides on how to do so successfully. In an effort to remedy this (and perhaps head off unnecessary panic), I’m offering a detailed description of our experience in the hope that it will help others considering or navigating a similar change.

In the last quarter of 2016, we transitioned 10 journals (about 3,400 total manuscripts) from a peer review system we had used for 13 years to a new system. I led the team — which included management, peer review staff, and internal editors — in planning and shepherding the transition. In this post, I’ll walk you through our process and highlight some tips and lessons we learned.

Timeline and Training

We had been unhappy with our old peer review system for some time and had investigated other systems. Our pain points included subpar user experience, slow technological updates (e.g., mobile-friendly sites), and difficulty making changes as the journals evolved. We selected a system that would meet our needs and decided to make the switch at the end of 2016. We gave our old vendor 6-months’ notice in June.

Next, we had to determine our timeline for handling manuscripts during the transition. We knew we were officially losing access to the old system on December 31. To understand what we had to communicate with our authors and when we had to stop accepting new submissions in the old system, we looked at the average length of the peer review process from submission to decision (60 days). Our goal was to get to a good stopping point for every paper (i.e., a decision of accept, reject, or revise).

Once we had our timeline in mind, we reached out to our editors, boards, and reviewers. First, we highlighted the improved user experience and new features/functionality that prompted us to switch systems. Then, we outlined the transition timeline, notified them of the dates for closing the old and opening the new system, and explained that they would be working in two systems during the transition. We also had to alert the reviewers that we’d be unable to grant extensions on reviews during the transition period. Late reviews would delay manuscripts receiving a decision before the system closed. Finally, if the journal was revising its expertise terms (more on this later), we let reviewers know they would need to select new terms in the new system.

Since the new system offered new features and functionality, we conducted training via video call for every editor to introduce them to the new system. We had to make some workflow adjustments too, so we spent time walking the editors through the changes and making sure they were clear on their tasks. Our peer review staff also spent a great deal of time on the phone with editors and reviewers, answering questions and helping them adjust.

Our Method to Avoid Madness

At the 60-day point, we closed the old system to new submissions because they most likely wouldn’t have a decision before the new system was implemented. We posted messaging on the old site explaining this to authors and directing them to the new sites, which opened December 1.

For authors already in the system, it was important to add wording to our revision letters (major and minor) with very specific instructions and dates. From 60 to 30 days before close, we allowed authors to submit revisions in the old system, as this was a reasonable time frame for re-review. At 30 days out, we changed the revision letters to state that all revisions had to be submitted in the new system, with clear instructions for doing so. We asked authors to submit their revision letter with their manuscript in the new system and identify that it was from the old system. One of our journals charges a submission fee, so we had to build in a specific waiver request indicating that they had paid the fee in the old system. To verify, we maintained a list of manuscripts where the author had previously paid, and when in doubt we granted the waiver.

As for internal staff, our journal managers were responsible for all accepted manuscripts during the transition, making sure everything they needed was downloaded from the old system (e.g., manuscripts, supplemental material, forms). They also got involved with revisions as the timeline closed. After the old system was closed to revised submissions, they printed all revise letters and noted how each paper was to be handled at resubmission (e.g., editor review only, send back to one reviewer/both reviewers).

Our peer review staff were responsible for tracking all statistics and manuscript numbers for revisions. As the timeline closed, they noted all accept and reject numbers/percentages they’d need for future reports, and at the end of the timeline, they recorded the manuscript numbers of all papers out for revision, so they could match them with the new manuscript numbers.

During the final 2 weeks of the timeline, our team met every few days, checking in with the peer review staff to determine how many papers remained in the old system and their status so they could follow up with reviewers/editors as needed. At the very end, we did a final check-in with the journal managers to make sure all accepted manuscripts and revisions were ready for a clean break from the old system.

Tracking and Statistics

Very important in the new system was tracking those revisions that spanned the two systems. We added a submission flag in the new system to identify revisions of papers that had started the process in the old system. This alerted reviewers and editors to the manuscript’s previous history, which we added to the submission from the notes we had taken. Our peer review coordinators noted on the list of papers out for revision the manuscript numbers of the resubmissions and the final decision outcomes.

Our annual statistics (which we present at our editorial board meetings) that first year were definitely more challenging than usual. We used the submission flags to differentiate manuscripts submitted that year that had originated in the old system from those that were actually new. We also wound up running two sets of stats (old system and new system) and adding them together to comprise the full year. There was a lot of checking and double checking to make sure no paper was unaccounted for.

Tips

Based on our experience, I have some tips for those involved in such a transition. First, build in buffer time when you’ll have access to the old system even after you’re fully functional in the new system. There are a lot of moving parts in the peer review process, and accounting for all of them occupied much of our time during the transition. However, you should expect that some things will come up afterward, despite your careful planning. In our situation, we had the luxury of access to the old system for an additional 2 months (through February 2017), without extra cost, because the old system was shutting down completely. This may not be feasible if you have to pay for two systems simultaneously. If that’s the case, my advice is to build an extra 2-4 weeks into your timeline. For example, if the old system becomes inaccessible December 31, build your timeline based on November 30. Having this buffer time was helpful when we were missing the manuscript history of a paper or if the history was unusual or confusing. It also gave our peer review coordinators a sense of security during the most hectic part of the transition (not an insignificant consideration).

Second, take notes (printed or electronic), so you can easily track revisions. We hoped authors would submit the revision letter with their manuscript, as we’d asked, and note that the manuscript was from the old system, but of course, that didn’t always happen. In that case, our own notes were immensely valuable in keeping the process running smoothly for the reviewers and editors.

Third, get backup data from the old system, if possible. We did get data from the old system (via Dropbox and DVDs), identifying where every manuscript was in the system at the time the data was downloaded (very end of timeline). We considered this enormous amount of data a safety net above and beyond our own notes and what we had downloaded. We only went back to this a few times after the old system closed (e.g., to verify that a paper had been accepted), but when we needed it, we were glad we had it.

Lessons Learned

There were also some areas for improvement and things that are important to keep in mind.

• Download reviewer expertise terms. For journals that retained the same expertise/keyword terms for matching reviewers with papers, we were able to transfer that data to the new system. However, some editors took advantage of the opportunity to overhaul their terms (especially because the new system offered an almost unlimited number of terms as well as nesting of terms). New terms meant we couldn’t transfer the data and had to ask reviewers to select new terms when they activated their account in the new system. Of course, not all of them did, even after multiple follow ups, which hindered the review process as reviewers without terms wouldn’t come up in a search. We realized it would have been helpful to download reviewer–term matches for those journals that did an extensive overhaul so we could at least match them as best as possible with fewer follow-up attempts.
• Lost reviewer history. Reviewer history includes how many papers they have reviewed, what their recommendations were, and often, comments from the editor (or ratings) about the quality of their reviews. Unfortunately (but not surprisingly), peer review systems are not built to be compatible with one another. There are no standards that make it easier to move reviewer data from one system to another. Although we could have transferred reviewer history from the old system, this was an additional (and not insignificant) cost for us. We made the difficult decision, with our editors, to let it go. In general, this turned out fine, but it was an adjustment, particularly during the transition period (e.g., we didn’t know if we had just asked someone to review a revision in the old system and a new paper in the new system).
• Realize the process will be hectic and stressful, but it will be okay. At SLACK, our peer review process is very high touch, and our peer review coordinators had worked in the old system for 13 years. They were used to being able to answer every question from an author, a reviewer, or an editor, and they were uncomfortable when they couldn’t. While discussing configuring a new system is beyond the scope of this post, the new system required a lot of staff time to set up. So, while we were working hard to make a clean break from the old system, we were learning the new system and trying to configure it to function as closely as possible to the old one in terms of workflow. Doing so helped allay staff’s and editors’ concerns and kept the peer review process running smoothly.

Learning a new system is stressful; there’s no doubt about it. I’m happy to say that, in the end, there was much less drama (and trauma) than we had anticipated. The new system has an author-friendly interface, so authors, in general, transitioned fine. Many of our reviewers had worked in the new system for other journals, so they were also comfortable during the transition. The biggest change between systems was for our outside editors (e.g., different interface, new tasks), and our goal was to make it as stress-free and seamless as possible for them. For our editors, journal work isn’t their full-time job, so we do everything we can to help them do it well. This required time, thought, and vigilance on our part; some days all we did was focus on peer review.

It was definitely a juggling act for a few months, but in hindsight, I think our prior planning, extensive notes, and clear communication  – internal and external  – made it successful. With the scary part behind us, we can focus on what we gained: an easier submission process for authors, greater flexibility and speed in making configuration changes our editors want, more frequent technological updates, and more robust reporting capabilities. I hope sharing our experience sheds some light on the process and gives you a sense of what to expect. Remember: “The scariest moment is always just before you start. After that, things can only get better.” – Stephen King