The Road to Double-Anonymous Peer Review

Martha Keyes

doi:10.36591/SE-D-4404-127

MODERATOR:
Jennifer Regala
Director of Publications, American Urological Association
Linthicum, Maryland

SPEAKERS:
Christine Beaty
Director of Journal Operations, American Heart Association
Dallas, Texas

Christina Nelson
Director of Publishing and Editorial Operations, The Journal of Bone & Joint Surgery, Inc.
Needham, Massachusetts

Anna Jester
Vice President of Sales & Marketing
eJournalPress
Rockville, Maryland

REPORTER:
Martha Keyes
American Urological Association
Linthicum, Maryland

For this CSE Peer Review Week Webinar, moderator Jennifer Regala launched the discussion on double-anonymous (often referred to as double-blind) peer review (DAPR) by explaining that the genesis for this CSE webinar topic stemmed from a good-natured debate with a colleague over the ease with which a publication can implement a DAPR process. Participants Christine Beaty, Christina Nelson, and Anna Jester outlined both the motivation behind the implementation of DAPR and the necessary changes to the mechanics of the peer-review process that must be considered by those editorial staff and editors who are weighing the factors in their own possible shift to DAPR for their journal(s).

Let the Science Speak for Itself

At the time of the webinar, Beaty and the editorial team of Circulation: Heart Failure were on the cusp of implementing DAPR. She shared their motivation for implementation and the steps taken thus far to install the DAPR process in their editorial office.

The goal of fair and equitable reviews, especially for early career investigators and those authors from diverse ethnic and geographical backgrounds, was the major impetus for change. Implementation of DAPR, Beaty stressed, will break down barriers by helping those authors who do not belong to the top tier of published authors (or “super groups”) stand a better chance of having their work considered and accepted based on the merits of the work alone—despite no “big names” to back it up. The reputation of the journal itself can only be strengthened, it could be argued, because it is less susceptible to charges of bias in the acceptance process.

Delving into the mechanics of DAPR, Beaty explained that the first step is made by the authors at submission when they elect to choose DAPR. Expectations must be clearly communicated to authors in the journal instructions and submission site (e.g., uploading of correct title page). Once a manuscript is submitted and flagged as DARP, editorial staff must ensure that the necessary coding changes are made and that all double anonymous parameters are met—within the manuscript files themselves and in all system-generated correspondence to editors and reviewers.

Beaty stressed an important point here in that an editorial office must remember to maintain overall stability in the eyes of editors, reviewers, and authors during this transition process. Readers and participants in the review process must be reassured that the quality and mission of the journal(s) have not waivered. Specifically, the editorial staff must maintain a steady brand presence and a consistent look and feel across all communications for their suite of publications.

What Does Success Look Like?

How do you measure success in reports back to your editorial boards? Editorial staff must show a steady growth in DAPR selection by authors and demonstrate that diversity goals are being met across all categories—gender, ethnicity, geographic location, and early career. Beaty mentioned that data can be collected via custom reports from the submission software and also from surveys, which can provide more “holistic” feedback.

The editorial team must also demonstrate that the implementation of DAPR did not negatively impact the peer-review process for the journal(s). For example, there can be no delayed time to decision. Retention of reviewers is actually another indication of a successful transition. It is the perception of the reviewers, Beaty remarked, that is the most changed by the implementation of DAPR, and retaining a steady pool of reviewers for your DAPR journal(s) is an overall sign of success.

Delving Into the Details

From Beaty’s efforts to visualize and initiate the DAPR process for Circulation: Heart Failure, the webinar pivoted to Christina Nelson of The Journal of Bone & Joint Surgery, who has been implementing DAPR for 5 of their 6 journals for years. Nelson reiterated many of the goals of DAPR—to reduce bias toward authors, encourage honesty, and protect journals against author accusations of biased reviews (Figure).

**Figure.** Some pros and cons of double-anonymous peer review. Image reproduced with permission from Christina Nelson.

Nelson’s significant experience with DAPR also put her in a great position to share all the detailed steps required of staff to successfully anonymize submissions. “Giveaways” that must be redacted from manuscripts include CTR and institutional review board numbers, single institution databases, and graphics that include institution names, foreign languages, or even zip codes on the image files are all examples of information that may need to be redacted before the manuscript can be sent out for peer review. This redacting information must be retained behind the scenes, however, as it must be put back into the paper at acceptance. Nelson remarked that there are many “judgement calls” that must be made in the anonymization process, and that editorial staff must be properly trained. Practiced staff can review and redact a manuscript within 5 to 10 minutes after working through the learning curve.

Benefits Versus Efforts

Anna Jester of eJournalPress followed up on Nelson’s points regarding the many safeguards that must be implemented to ensure fully anonymized submissions for those authors who elect DAPR. In addition to the manuscript files themselves, all editorial software interfaces—including reviewer forms, submission questions, letter templates, and all system-generated notification emails—linked to a DAPR manuscript all must be configured to journal preferences to guarantee anonymity. Jester stressed that staff must invest the time up front to walk through the entire peer-review process on the journal test site, checking that all screens, letter templates, notifications, and configurations are anonymized per journal preferences before launch to avoid unintended consequences. In addition, it is always recommended to reiterate the DAPR process (“and why you believe in it”) to authors and reviewers alike via on-screen messaging and reviewer and author instructions.

The anonymity of reviewers is another factor to consider. Should peer reviewers of a given paper be anonymous to each other? Jester suggested the publication of a general list of reviewers to recognize their contributions without the possibility of readers being able to link specific reviewers to particular manuscripts. This type of reviewer anonymity is especially important in specialties with relatively small reviewer pools.

Jester also addressed the potential need for staff to collect diversity, equity, and inclusion demographics when authors elect to identify (e.g., ethnicity, gender, career stage, etc.). There are challenges in pulling together this information in a meaningful way to measure success of the DAPR efforts in board reports, marketing, and other communications. Jester also recommended asking your legal team to weigh in prior to collecting demographic information via your peer-review platform.

At the conclusion of the meeting, the following references were provided for publications staff to share with their editorial boards to assist in determining whether DAPR would be the right fit for their journal(s):

Lastly, Jester encouraged all attendees to get involved with the Council of Science Editors community, which provides great resources and great networking opportunities for all!