MODERATOR:
Jonathan Schultz
Director of Journal Operations
American Heart Association
SPEAKERS:
Jonathan Schultz
Patricia Baskin
Executive Editor
Neurology Journals, American Academy of Neurology
H.E. Hinson, MD
Associate Professor at Oregon Health and Science University
Ashwini Sehgal, MD
Professor at Case Western Reserve University
REPORTER:
Tony Alves
Highwire Press
For the session “Calling all Experts: Broadening the Definition of Expert Peer Review,” moderator Jonathan Schultz kicked off the session by noting that, although the traditional model of peer review is that an editor will send a manuscript to 2 or 3 reviewers who are experts on the topic of the manuscript, there is a growing recognition that sometimes a more specialized reviewer may need to be involved. In a presentation entitled “Technical/Methods/Reproducibility Review,” different forms of technical review were discussed. Because science articles are more complex, the American Heart Association (AHA) contracted with a team of technical editors who look at manuscripts at the revision stage and focus on the technical aspects of the manuscript like the abstract, methods, figures, and data—all elements that you need to understand reproducibility of the research.
Three of the AHA’s technical reviewers answered questions posed by Schultz. They discussed that their role as a technical reviewer is to look at the content of the article to be sure that authors are providing necessary key components such as background and methods details. The best way to do that is to look at the reporting guidelines and to use standard reporting guideline checklists. It’s also important to look at the abstract, make sure there are clear objective statements, look at the methods, and to take a close look at the data. Key things in the methods section are to be sure the researchers followed the appropriate guidelines, such as the ARRIVE Guidelines1 and the AHA’s Major Resource Table.2 Other key items include a detailed description of biological materials, how the experiment was performed, the source of cell lines, characteristics of antibodies, animal baseline characteristics, as well as data collection methods including the equipment. When reviewing figures, the reviewer needs to examine the quality of the materials used to derive the figures, and when possible, look at unedited figures. Overall, a technical reviewer is tasked with a detailed examination to be sure all the information is there for reproducibility, and to be sure there is enough detail to reproduce the experiment in their own laboratory. Schultz also pointed out that the technical reviewers at the AHA often serve as advisors on questions about figures, or new policies that are being considered.
Patty Baskin was the second speaker with a presentation called “Peer Reviewing to Promote Equity, Diversity, and Inclusion at a Journal.” She first discussed an incident in which a personal-reflection piece published in American Academy of Neurology’s (AAN) flagship journal Neurology was perceived as culturally insensitive, prompting numerous complaints. The journal retracted the article and removed it from their online site. The incident resulted in the journal’s reevaluating their editorial processes. They took the following steps: appointed editors to review the “physician experiences” section, established a website on diversity topics, provided sensitivity training for editors and staff, updated their style guide to include nonbiased language, increased representation of women and international members on their editorial boards, and appointed Diversity Associate Editors to review for diversity in submitted articles.
The role of the journal’s Diversity Associate Editors is to review papers for appropriate perspectives when particular groups are being described, including papers that touched on sex and gender, race and ethnicity, and people from marginalized categories (Figure 1). They are asked to review titles and abstracts of all papers at the revision stage to be sure there is no biasing language and to review other material, such as podcasts, blogs, etc., as requested by editors or staff.
Baskin interviewed one of AAN’s Diversity Associate Editors, Dr H.E. Hinson, who talked about the logistics of the review process. The review happens at the revision stage. The editors screen the titles looking for human subjects research that examines different groups based on characteristics such as sex, gender, ethnicity, and race. The intent is to support authors who may not think about diversity, equity, and inclusion (DEI) rather than to censor their work.
Hinson noted that race and ethnicity and sex and gender are the most common and frequent issues. There are straightforward issues such as outdated terminology, antiquated terms, and improper capitalization of terms. However, there are also more conceptual issues such as using gender and sex interchangeably and ascribing racial differences to genetics. Sometimes authors don’t acknowledge limitations in their studies, such as unmeasured social determinants that might undergird racial differences, and the Diversity Editors help authors recognize these issues. Papers are seldom rejected for DEI issues; rather, the editors work with authors to correct biases and limitations, and authors are usually grateful for the help in improving their papers.
The final presentation, “Community Members as Reviewers of Medical Journal Manuscripts,” by Dr Ashwini Sehgal, discussed a National Institutes of Health-funded project that examined the use of community members and lay people as reviewers. Manuscripts are generally reviewed by subject matter experts like physicians, researchers, and scientists. However, it is thought that community members might provide useful perspectives as patients and caregivers. There was concern that community members often lack research knowledge, lack an understanding of the scientific process and statistical analysis, and might lack objectivity or have unrealistic expectations. The study examined if it was possible to train and guide community members to provide reviews that are complementary to those provided by scientists.
The objectives of the randomized controlled trial were to determine the usefulness of community reviews, to see what common themes arose, and whether the comments were integrated into the published articles. Annals of Internal Medicine and Annals of Family Medicine participated in the study from June 2018 to November 2021. The intervention group consisted of one trained community reviewer and multiple scientific reviewers, the control group was multiple scientific reviewers, and all reviews were used by the editors in the decision-making process. Training for 28 community reviewers consisted of a six 90-minute sessions covering study design, inclusion/exclusion criteria, subject recruitment, human subject protections, methods of data collection and analysis, parts of a manuscript, understanding of tables, figures, funding, conflict of interest, the roles of editors vs. reviewers, and writing effective reviews. The study coordinator provided feedback on the reviews—they did not rewrite reviews—and revised reviews were then submitted to the journal.
The study results showed that the usefulness of the community reviews were rated 3.1 by the editors on a scale of 1–5, with 5 being the most useful. This compares to scientific reviews that were rated 3.3 by the editors. Editors were asked open-ended questions on the usefulness of the reviews. Most helpful were community perspectives like consideration of socioeconomic factors, and less helpful was the lack of specificity of changes. Qualitative analyses looking for common themes were conducted on the reviews and four community themes stood out: diversity of study participants (i.e., why was there so little diversity); relevance to patients and communities (i.e., scheduling appointments as a barrier); cultural considerations and social context (i.e., understanding people’s social and environmental settings); and implementation of research by patients and communities (i.e., limited time and budgets in schools). There were 55 accepted articles with 138 community comments integrated by the authors in those published articles. Also measured were the appearance of the 4 community themes in the published article. In the intervention group, there were 2.8 themes present, compared with 1.7 themes found in the control group.
Overall, the study concluded that the community reviews were useful because they addressed topics that are relevant to patients and communities, and that community comments are integrated into the published articles (Figure 2). Of course, there are limits and challenges to adoption of this approach, such as training and supervision of community reviewers that require compensation that is generally not available to most publishers and journals.
