The plenary session, “Editorial and Peer-Review Process Innovations,” at the 2017 Peer Review Congress in Chicago, Illinois, in September 2017 presented research on innovation in peer review. The scholarly publishing community is experiencing increasing scrutiny of the idea, value, and implementation of peer review as a concept, from those both within and outside our industry. During this session, presenters shared their findings related to adapting the peer-review process in ways that speak to the questions about the validity and function of peer review. Each presentation demonstrated how flexibility in peer review can help the publication process respond to evolving needs in the scientific community.
Editorial Policy and Biomedical Research Reporting
Malcolm Macleod, representing the Nature Publication Quality Improvement Project (NPQIP) Collaborative Group, presented the first plenary abstract assessing whether a change in editorial policy could increase specific types of author reporting in manuscripts. The impetus for this study was the desire to increase author reporting of the measures they took to reduce the risk of bias in their study design, including randomization, blinding, sample size calculation, and exclusions. In 2013, Nature Publishing Group (NPG) began mandating that authors complete a 74-item checklist at revision submission indicating which of the four aforementioned criteria were included in their manuscript. (The current checklist used by Nature can be found online in the Life Sciences Reporting Guidelines section of the For Authors information page.) NPG went from zero manuscripts meeting all four criteria before the implementation of the checklist to 17.1% of manuscripts being compliant after. This change was compared to the proportion of similar manuscripts meeting the criteria published in non-Nature journals before and after 2013, and those journals experienced no change (0.6% meeting all four criteria before compared with 0.5% meeting all four criteria after).
This outcome suggests the checklist and accompanying editorial policy had a positive impact on the number of Nature articles meeting key reporting criteria. It is important to understand the methods used by researchers to reduce the risk of bias and increase reproducibility of studies. In this study, an editorial policy implemented during the peer-review process addressed a need to ensure manuscripts clearly present study methods to better assess bias while simultaneously improving study reproducibility.
Perspectives from the Audience
- The investment of staff time to enforce such a policy must be taken into consideration and balanced against the results.
- It would be helpful to consider a way to simplify the checklist to make it easier for authors to complete and editors to assess.
- Institutions and individual researchers must share in the responsibility to ensure researchers have taken measures to reduce bias in their study designs.
Signed Peer Reviews: Principle and Practice
The abstract presented by Elizabeth Seiver, Public Library of Science (PLoS) Researcher, and Helen Atkins, Director of Publishing Services at PLoS, described peer-review signing preferences on three PLoS medical journals: PLoS ONE, PLoS Computational Biology, and PLoS Medicine. Transparency in peer review was the topic of the concurrent 2017 Peer Review Week and was a ubiquitous topic at the 2017 Peer Review Congress. At the three PLoS journals assessed in this study, reviewers can choose to sign their comments to authors, thus revealing their identities, but neither reviews nor reviewer names are made publicly available. The authors of this abstract analyzed the rate at which reviewers had signed their review comments from mid-2013 to 2016. During this time period, 7.7% of reviews on three PLoS journals were signed. To obtain further information on author and reviewer preferences, Seiver and Atkins added survey links to existing emails generated by the submission system at the time of manuscript or review submission. From the results of these surveys completed by active reviewers and authors, PLOS found authors prefer signed reviews, which allow them to understand the experience and expertise of a reviewer commenting on their manuscript, as well as create opportunities for further open communication with the reviewer. However, reviewers indicated they prefer not to sign their reviews because they feel they can be more honest without fear of retribution for negative comments. Across the three journals in the survey sample, 47.5% of authors preferred to receive signed reviews, but only 15.8% of reviewers reported signing their reviews. Authors’ desire for signed peer reviews is at odds with reviewers’ reluctance to sign reviews.
For journals that want to increase the transparency of their peer-review process, the PLoS team recommends an easy first step: give reviewers the option to put their names on their reviews. Survey results revealed that many reviewers had never been given the opportunity to sign their reviews. Other considerations include the field of the journal community (reviewers in PLoS Medicine were more likely to sign their reviews than reviewers in the other two journals), provide incentives for signing reviewers, clearly articulate to reviewers the benefits of signing their reviews, and encourage reviewers to consider the author’s perspective in their decision to sign a review.
Perspectives from the Audience
- Publishers need to address the obvious dissonance between individuals’ desires when they are authors and their actions when they are reviewers.
- Allowing optional review signing will lead to situations where, on the same manuscript, one review may be signed and another may not. It will be necessary to assess the impact of optional review signing on author–reviewer interactions.
Role of Persistent Identifiers: Use of ORCID
Alice Meadows, Director of Community Engagement and Support at ORCID, discussed the integration of ORCIDs into the peer-review process and her findings on the uptake of linking peer-review activity through ORCID records since that feature’s launch in 2015. From October 2015 to May 2017, more than 135,000 review activities were added to more than 9800 ORCID records by nine organizations. Peer-review activities must be connected to ORCID through an organization, such as a publisher or Publons; individuals cannot make these connections themselves. ORCID data from three organizations that were early adopters of ORCID (Publons, the American Geophysical Union, and F1000) were analyzed to see the rate of review activity linking. Publons was by far the top user of ORCID to connect peer-review activity, with 6.89% of Publons users having connected peer review activities to their ORCID records. This represents an overwhelming 92.8% of all review activities in ORCID.
The low uptake of this functionality indicates more education on this tool is needed to describe its functionality and benefits. Publons is one of the most important intermediaries between review activity and ORCID, so it will be interesting to see how Publons’ purchase by Clarivate Analytics (owner of the ScholarOne manuscript submission platform) in June 2017 will impact uptake. The downstream implications of linking journal peer-review activity to a researcher’s ORCID are the ability of that researcher to collate peer-review activities across publishers and to the share with stakeholders outside the publishing stream, including institutions and funders.
Perspectives from the Audience
- Recording every review ever completed by a reviewer may be excessive and unnecessary. Perhaps not all scholarly activity should appear on an academic CV.
- Researchers may be more willing to sign their reviews or participate in more transparent review if reviews received DOIs.
- Researchers who are not scientists may feel left out of the conversations around ORCIDs.
Adding Patient Review Alongside Peer Review: A Mixed-Methods Study
The final abstract presentation was given by Fiona Godlee, Editor-in-Chief of BMJ, on the implementation and outcomes of a patient reviewer program as part of BMJ’s patient partnership strategy. For medical journals, patients are the ultimate beneficiary of scientific research. This premise fostered an ethical imperative within BMJ to invite patient voices and perspectives into the publication stream. BMJ has more than 600 patient peer reviewers, recruited through marketing efforts and physician contacts, and in 2016, 55% of research papers in BMJ sent to external peer review invited at least one patient reviewer to review. Interestingly, BMJ found patients agree and decline to review at rates similar to traditional reviewers. When editors were surveyed on the value of patient peer review in their experience, there were mixed results: five of seven responding research editors indicated that patient reviewers add “a little” value, and two of seven editors felt that patient reviewers added “a lot” of value to the peer-review process. Ultimately, four of seven editors felt that other journals should adopt patient review, and the other three were unsure.
Based on survey results from editors and patients (88% of responding patients believe more journals should have patient review), BMJ found that patient review is feasible despite its challenges and is desirable to most editors and patients. Challenges include recruiting patient reviewers, communicating with them during the peer-review process, and ensuring patient reviewers are not influenced by industry (e.g., in Europe, there are links between patient advocacy groups and industry).
Perspectives from the Audience
- Resources for training patients on the basics of peer review (e.g., guides on peer-review process, important considerations in peer review) may be helpful to patient reviewers and editors.
Each of the abstracts described above, and all abstracts from the 2017 Peer Review Congress, can be found online.
Brittany Swett is the Executive Director and Editorial Services Coordinator at J&J Editorial