While there exists a multiplicity of peer review models, the open peer review model has been garnering attention lately. “Open peer review” refers to an external peer review model where author and reviewer identities are apparent to both parties. In some cases of open peer review, the review is visible alongside the published article. This model relies on transparency between the reviewer and the author. With anonymity removed as a factor, current reviewers’ opinions vary on this model and the benefits or drawbacks it may offer. 

To explore the attitudes of reviewers toward open peer review, an online survey was sent to  5,977 persons who acted as reviewers for Annals of Internal Medicine from 2019 to 2022. The response rate was 24% (1,421 persons). The results of this survey were briefly presented in an abstract at the Peer Review Congress in 2022, Jill Jackson and co-authors concluded that, “an open review model could adversely affect the willingness of current Annals peer reviewers to continue to review and could alter the nature of reviewer comments.”¹ The abstract consolidated these responses and sought to provide insight into how moving from a single anonymized peer review model to an open review model might affect an established reviewer base. 

The table presented in the original abstract included data collected from 3 questions in the survey that focused on how likely or unlikely the respondents would be to continue to review should their identity be disclosed to the authors or if their identity and/or review was published with the article (Table). The data presented indicated that over half of the polled reviewers would continue to review if an open review model were to be implemented. In addition, when asked how an open review model would impact their comments, 41% of respondents indicated their comments would not be affected. However, this question included the ability to check off more than 1 option, and 20% of the 1,421 respondents who answered this question selected the “other” option, which allowed them the opportunity to provide open ended comments. In this article, we take a deeper look at these comments as they provide insight into respondents’ views toward open review.

Table. Willingness of respondents to continue to review with an open review model.*

*Adapted from Jackson et al.¹

Most respondents who chose to include a statement expressed opinions in line with the “My comments would be less critical” category. Moreover, 51 of the 282 provided comments expressed that their reviews would be affected in some way, but they could not predict how. Many stated that whether they would be more or less detailed or critical would depend on the paper and author list, or that they may be a combination of the options provided.

While the survey results indicated that over half of respondents remain willing to review if the journal moved to open review, the fact that a sizable minority would not is concerning. In addition, the open-ended comments raised some compelling concerns that warrant consideration as well. Among those concerns was the amount of time an open peer review model would require of reviewers versus the current model. Eleven percent of respondents who provided comments mentioned they would need additional time to prepare their review, the reason being that because the reviews are published, the reviewers need to spend more time copyediting their work and checking their references for accuracy before submitting their reviews. In addition, they must pay careful attention to tone and readability for an audience beyond the authors of the paper. Many respondents were concerned about grammar, with a number of them mentioning the need to refine their English skills. This also poses the question of whether open review will further restrict the reviewer pool by excluding international participants. While reviewers whose first language is not English may feel comfortable enough with their English to provide comments in traditional review, they may feel disinclined to accept a review request knowing their comments will be shared publicly. In open peer review, reviewers split their focus among multiple audiences, the public, the editorial team, and the authors. Some respondents were concerned that this split focus would lead to less constructive reviews because they are filtered through these lenses for public consumption.  

Another major concern of respondents was the potential for retaliation with the open review model. Of the comments provided, 15% were wary of career consequences from open review, both in the form of retaliation and bias. If a reviewer was highly approving of a manuscript, there may be an expectation that the authors return the favor for the reviewers’ own paper in the future. Likewise, if they are highly critical, respondents feared they may see reprisal from the authors, or even the readers, when their comments are shared publicly. Some respondents stated they will be less likely to be critical when reviewing in highly politicized or highly specialized subjects. In a niche field where collaboration is paramount and their name would be shared with colleagues they may work with in the future, respondents showed apprehension. They used terms like “less candid,” “more cautious,” and “less direct” to denote that they would be less likely to engage directly or as critically with an author if anonymity is not provided.¹ Furthermore, there exists a concern for early career researchers and minority groups that may be dissuaded from participating in peer review out of fear of upsetting a senior colleague or well-known name in their field. One respondent commented that as an early career woman in academics, she would be concerned some of her comments may be misconstrued as a lack of knowledge, and this may negatively impact her career.¹

These 2 issues, the increased work of a review and the fear of reprisal, were the most common refrain in the survey when respondents were allowed to enter their own responses. They are serious issues, and not ones that large journals with a history of single-anonymized peer review systems, like Annals of Internal Medicine, could easily resolve. One solution that many of the survey-takers requested is to continue to allow a section for confidential comments to the editors in addition to the version that would be shared with the authors. This way, reviewers could be more candid in their recommendations for publication or rejection without fearing they would be shared alongside the final published paper. These results provide much to think about, should a journal be looking to modify their review model, and indicate the process will certainly require a delicate hand. 

That being said, with an established reviewer base experienced in anonymized review and the benefits that currently exist within that structure, it would be difficult for a journal such as Annals of Internal Medicine to convert to an entirely open review model. Medical professionals volunteer their limited free time to participate in this process, and not an insignificant amount of those polled expressed that a change to open review may be the thing that pushes them to use that time elsewhere. Like all things in medicine, there exists risk, and all journals must weigh the potential for failure against the potential for success. The findings of this survey raise concerns that an open review model would adversely affect the willingness of current reviewers to continue to review and could adversely alter the nature of reviewer comments.¹

References and Links

1. Jackson J, Laine C, Kostelnik J. A survey of reviewers’ perspectives on options for open and transparent peer review at Annals of Internal Medicine. In: Ninth International Congress on Peer Review and Scientific Publication, September 8–10, 2022, Chicago, IL. 

Julie Kostelnik and Ashley Pfeiffer are with Annals of Internal Medicine, American College of Physicians.

Opinions expressed are those of the authors and do not necessarily reflect the opinions or policies of their employers, the Council of Science Editors, or the Editorial Board of Science Editor.