MODERATOR:
Ben Mudrak
Product Manager
American Chemical Society
Durham, North Carolina
SPEAKERS:
Robert Althoff
Associate Professor of Psychology
University of Vermont
Burlington, Vermont
Liza Karlin
Senior Staff Editor
Association of American Medical Colleges
Washington, DC
REPORTER:
Rachel Winfield
Taylor & Francis Group
Oxford, United Kingdom
Peer review is a crucial step in the academic publishing process, however training and resources for peer reviewers are not always readily available. This panel discussion shared examples of best practices and innovation in this area from 3 experienced journal editors.
Robert Althoff, Associate Editor of Journal of the American Academy for Child & Adolescent Psychiatry, opened the session by outlining the strategy he has built for teaching students about how to write an effective peer review. Althoff emphasized the importance of asking a series of fundamental questions before writing a review, including:
- What is the purpose of the study?
- What were the major findings?
- What questions are still unanswered?
He advised that a review which is helpful for journal editors will open with a paragraph summarizing the article’s contents, such as the research question and key findings, along with a comment from the reviewer about the overarching relevancy and importance of the research. The rest of the review should consist of the reviewer’s qualified opinion of the article.
An effective review is well-structured and clear for the editor reading it. As such, Althoff recommends that reviewers split their comments out into major and minor concerns. Reviewers could also consider numbering their comments, or structuring their reviews around each section of the manuscript. In particular, the best reviews make concrete, specific suggestions about which aspects of the manuscript need to be changed or developed, including challenging any ambiguous or unreferenced statements made by the author.
Ben Mudrak, Product Manager at the American Chemical Society (ACS), continued the session by sharing his insights on reviewer training and tools from the ACS Reviewer Lab.1 The ACS Reviewer Lab is a free online course which covers every step of the peer review process through 6 interactive modules. The course is not specific to chemistry, so it could be useful to reviewers working in other disciplines. The modules address the following areas:
- Introduction to peer review
- Ethics in peer review
- Preparing for review
- Assessing significance and technical quality
- Assessing presentation and readiness for publication
- Writing your review
The ACS Reviewer Lab has seen 9,000 enrollments and more than 3,000 reviewers have completed all six modules since its 2016 launch. Upon completion, program graduates have the option to indicate their interest in reviewing for ACS; interestingly, over two-thirds of graduates have become ACS reviewers. The program has been a great tool in expanding the ACS reviewer pool.
Originally, the training modules were only available in English, and so the majority of users (52%) were from the United States. However, more recently the ACS Reviewer Lab also became available in Chinese and Japanese, which has led to an increase in users from these regions (Figure). This highlights the importance of considering diversity and inclusion in any training resources which are produced for peer reviewers.
The third and final speaker at this session was Liza Karlin, Senior Staff Editor at Academic Medicine, who explored the numerous resources offered through the journal’s online Reviewer Resource Hub.2 The resources available from Academic Medicine began in 2001 with the publication of a guide to the reviewing process. In 2013, work began on building a more useful suite of resources which would assist peer reviewers at different stages and with different learning styles. These resources, which cover a range of formats and topics, include the following:
- AM Rounds: a series of informal blog posts sharing tips and advice, written by previous winners of the journal’s annual Excellence in Reviewing Award
- Reviewer Recommendation Guidelines: a quick reference document which defines the different recommendations a reviewer might make (accept, reject, or major or minor revisions) and provides example comments which might be included for authors and editors
- Review Criteria for Research Manuscripts, 2nd Edition: a dynamic and searchable document featuring a comprehensive checklist for both experienced and novice reviewers to use, written by scholars involved with editing and peer reviewing
- What Editors Want: An Overview for Reviewers: a training video designed to be consumed from start to finish, rather than as a quick reference guide
- Practice Review Exercise: intended for individual or group use as a training tool, this exercise shares a submitted manuscript and the peer review reports which were associated with it; trainee reviewers can compare their own reviews with the example given
- Advice from a Master Peer Reviewer: a podcast episode sharing insights from a seasoned peer reviewer for Academic Medicine
In 2014, the journal also started running interactive Reviewer Workshops. These typically start with a brief presentation exploring the basics of peer review, but then become more hands-on. Participants are given a manuscript and asked to write notes for a review, compare their comments with those received by the journal, and then discuss these notes as a group. These tailored workshops have a very flexible format, so the session could last any amount of time from an hour to a half-day.
Overall, the Reviewer Resource Hub has been well-received by users, although Karlin noted that it required a significant investment of time to develop. Even once new resources have been developed and published, it takes further investment to ensure they remain up to date—though the time commitment is much smaller to update or add to resources compared to developing them from scratch.
Following the 3 presentations, Mudrak opened the floor for questions and comments from the audience. One attendee mentioned that Publons offers free online training through the Publons Academy.3 Other attendees expressed an appetite for more resources designed as “refresher courses” for seasoned peer reviewers, rather than beginners, or even a structured rubric or marking scheme to which reviewers could refer.
Overall, this session at the 2019 CSE Annual Meeting was informative and interesting. The speakers offered attendees a range of exciting perspectives on how to better serve their peer reviewer community with the training, tools and resources they need at every stage of their careers.
Discussion