Abstract

There is an increasing demand for more transparency in peer review due to the potential benefits that this could offer. However, open peer review is notoriously challenging for publishers to implement. IOP Publishing (IOPP) partnered with Publons (part of Clarivate) to develop upon the existing Publons Transparent Peer Review (TPR) service using application programming interfaces (APIs) in order to deliver an optimal solution for their TPR pilot. The API-based system ensured that minimal manual work was required on the publisher’s side. This pilot tested the demand for TPR from both authors and reviewers on 3 IOPP journals using ScholarOne. The collaboration on this highly efficient TPR system led to a successful pilot, with the uptake of TPR from authors and reviewers on the pilot journals relatively high. These positive results led to IOPP planning to roll out TPR to all of its open access journals.

Introduction

Opening Up Peer Review
Peer review is the process of subjecting an author’s scholarly work, research, or ideas to the scrutiny of other experts in the same field. Reviewers and editors invest valuable time and effort into ensuring that only high quality research is published. Traditionally, the peer-review process has been closed in scientific publishing.

Making this process visible to the community increases accountability and allows reviewers to be recognized more for their hard work. This may provide an incentive for them to contribute quality reviews in a timely manner, potentially improving research integrity and overall review quality. The review content may also be of educational value, particularly to trainee reviewers. There have been growing demands for the publication of peer-review reports (particularly in the biosciences¹). The Publons Global State of Peer Review report² found a growing interest in opening up peer review, with younger researchers, in particular, valuing increased transparency.

With transparent peer review (TPR), the peer-review content is published, but reviewers may remain anonymous if they prefer (as opposed to open peer review, which requires the naming of reviewers).

Challenges Setting Up TPR
The process of making peer review transparent is notoriously challenging for journals to implement due to the existence of diverse entwined systems used in different aspects of the publishing process, such as the submission of manuscripts, peer review, production, and online hosting of the articles.

IOPP Publishing (IOPP), like numerous other publishers, wanted to test the demand for TPR in their communities in the areas of physics and materials science, as well as biomedical engineering and environmental science. IOPP investigated the possibility of setting up an in-house transparent peer-review workflow in 2018. While doing so, they were faced with multiple challenges. IOPP’s workflows cross 3 separate systems: the submission and peer-review system (ScholarOne), the production system (Proton) into which accepted articles flow, and the system that hosts the published journal content online (IOPscience). In order to extract and host the relevant content, a substantial amount of development work would have been required across these systems.

Developing a minimum viable product which would have moved the decision letters (including reviewer reports) into the production system, and which subsequently would have been posted as a PDF file along with the published article, was technically challenging. The development work was likely to be rather time-consuming, especially the changes to IOPscience. In addition, the in-house solution likely would have required ongoing manual work (e.g., by production staff when making the review content live).

IOPP and Publons TPR pilot

Because of the challenges that IOPP faced in setting up TPR, in 2019 IOPP and Publons (part of Clarivate) partnered in a pilot of Transparent Peer Review, the industry’s first cross-publisher, scalable transparent peer-review workflow based on ScholarOne technology (also part of Clarivate; note that this specific TPR setup is currently only available to journals using ScholarOne). This service works within established systems to support TPR, greatly reducing the work required by the publisher.

In the first stage of this partnership, TPR would be rolled out across 3 IOPP journals: JPhys Materials (JPMat), Journal of Neural Engineering (JNE) and Environmental Research Letters (ERL), in a 1 year pilot program to test the demand for TPR from the authors and reviewers on these journals. The Publons TPR workflow provides readers with the possibility, when reading an article, to access a comprehensive peer-review history including reviewer reports, editor decision letters, and author responses. Each of these peer-review elements has an individual digital object identifier (DOI), which can be easily referenced and cited. This workflow complies with best-practice data privacy regulation, ensuring the individual preferences of authors, peer reviewers, and journals are maintained. Publons and IOPP worked together to develop and optimise this TPR system for their pilot.

Publons TPR system
Publons TPR is a configurable product on 3 different levels: author opt-in/-out, reviewer opt-in/-out, and reviewer reports signed/anonymous (Figure 1). Journals are able to decide if they want all their authors and reviewers to opt-in by default or have the choice to opt-out, and if desired they could require that all the reviewers sign with their name, sign as anonymous, or give them the choice.

The chosen TPR journal configuration is then set up on that journal’s ScholarOne submission page, where authors and reviewers will have to answer the TPR-related questions upon the submission of a manuscript or of a reviewer report, respectively.

• Submission form question: “This journal is participating in a trial of Transparent Peer Review. If you (and all of the reviewers) agree then the reviewer reports, your responses, and the editor’s decision letter will be linked from your published article, should your article be accepted. If you choose to decline, the peer-review content will not be published. Accept/Decline”
• Report form question: “This journal is participating in a trial of Transparent Peer Review. If you (and the other reviewer(s) and authors) agree then the reviewer reports will be linked from the published article, should the article be accepted. If you choose to decline, the reviewer reports will not be published. Accept/Decline”

Once provided with an accepted article from the journal, Publons retrieves the answers to the questions from ScholarOne for each accepted article and determines which ones are part of TPR. For the articles that are opted-in to TPR, Publons builds an article page where the peer-review history will be published and each TPR element (reviewer reports, decision letters, and author responses) will be assigned an individual DOI. On publication of any articles with TPR, a link is made from the journal article page to the Publons page containing the review history.

IOPP decided to have all 3 journals in the pilot with the same configuration. This is to allow both authors and reviewers to opt-in/-out of TPR, and with opted-in reviewers free to select whether or not they want their names revealed on their reports.

API Development for Accepted Articles
In order to target the production stage of IOPP’s publishing workflow, IOPP and Publons worked together to develop an API that enables Publons to easily and automatically receive an accepted article feed from Proton, IOPP’s production system. Such a development was desirable for IOPP because it removed the need for them to manually compile and send details of accepted articles to Publons on a regular basis, saving the publisher a significant amount of time. Other publishers may send production reports on a weekly basis in order to inform Publons which articles have been accepted in the last 7 days.

Publons Badge
Publons and IOPP also worked together to develop a way to seamlessly display the peer-review content available at Publons for any given article (Figure 2). Publons developed a JavaScript widget that uses the article’s DOI to do a single reference to the Publons API to verify what peer-review content is available in Publons. In addition to checking for the peer-review content for the article it also confirms the number of times an article has been cited on Web of Science (WoS), which is also part of Clarivate.

Publishers can embed the Publons badge on their article hosting platforms (IOPscience in IOPP’s case), and depending on the data that Publons has on that article, a badge will show 1 of 4 categories:

1. Transparent peer review: Publons hosts reviews that can be seen on the Publons article page— blue badge with a red counter indicating the number of reviewer reports for that article.
2. Claimed reviews: On Publons, one or more reviewers have claimed recognition for their review of that article—blue badge (but no review counter).
3. No peer review for the article on Publons: There is no peer-review content for the article on Publons, but there is peer-review content available on other articles published by the same journal in the last 12 months—gray badge.
4. No peer-review content for the article or any other articles from the same journal—no badge.

The appropriate badge automatically appears when an article is published in any of the 3 pilot journals. Hovering the mouse pointer over the badge shows a summary of the open data available at Publons (Figure 2). For TPR articles, this information includes how many times the article has been cited according to the WoS core collection, the number of peer-review revision rounds, the number of reviews and how many of them are anonymous, the number of decision letters, and the number of author responses, as well as some additional information on the journal that published the article. Clicking on the badge takes the reader to the peer-review content on the Publons article page, saving the publisher the task of manually building links to this content.

Publons TPR Article Pages
Publons hosts the peer-review content on the Publons article pages. The link format of a Publons article page is always the same: https://publons.com/publon/ + Article DOI (e.g., https://publons.com/publon/10.1088/1741-2552/AB5E08/). The Publons article pages allow the community to score the article with the Publons score, and to write community reviews discussing the article or the peer-review history (Figure 3).

The peer-review content appears in a descending chronological order (most recent, first). Visible is the each peer-review element and the assigned DOI. Every peer-review element can be individually endorsed by Publons users. Below the content of each peer-review element is the name of the person(s) who wrote it, the editor who signed the decision letter, the authors and author responses and, if the reviewer signed with their name, the name of the reviewer. If the reviewer signed the reviewer report and additionally claimed recognition on Publons, the name of the reviewer will link to the reviewer’s Publons profile, including showing their picture if they have one on their profile (Figure 4).

Preliminary Results of IOPP Pilot
The main aim of the pilot program was to test the demand for TPR from both authors and reviewers, all of whom have to opt-in for a paper to be published with the peer-review content with it. Note that if either the authors or a reviewer opts out, then the peer-review content is not published. The main measure of success was the uptake from authors and reviewers: We hoped that the majority of both would opt-in (i.e., “Agree”) to TPR. The results from the first 6 months of the pilot are shown in Table 1.

Table 1. The number and percentages of authors and reviewers agreeing/declining to have the peer review content published on the 3 IOP Publishing trial journals in the first 6 months of the pilot.

There is little variance between the 3 fields covered (neural engineering, environmental science, and materials science), with the majority of authors opting for TPR for each journal. The author agreement rate of nearly 60% is in line with that reported by Nature,³ and comfortably higher than the uptake of 39% recently reported by PLOS.⁴ The agreement rate for reviewers is a little lower, but still approaching 50% (as far as we know, this is the first time that the demand from reviewers has been measured in this way). Most papers are only reviewed by 2 reviewers, and the proportion of papers with all reviewers agreeing to TPR is just under 20% (given that the author must agree too, we find that a little under 10% of articles are published with the peer-review content). For reviewers who opted in for TPR, most (over 80%) chose to remain anonymous. We have received uniformly positive feedback from authors who opted for TPR and had their papers published with the review content.

We have seen no observable impact on the willingness of reviewers to report under the TPR model, and no change in the average time to first decision. We have seen a small increase in average review quality since implementing TPR on these journals, with a slightly higher proportion of reports receiving the maximum “3” rating in ScholarOne for reviewers who opted in for TPR, compared with those who declined. Finally, we have seen no change in editor behavior since implementing TPR (in fact, editors are not aware of whether authors/reviewers have opted for TPR until after the first decision stage). There is little extra work required by journal staff, other than for TPR articles where the author response has been submitted as an attached file. The Publons TPR system does not currently handle attachments.

Future Publons TPR Initiatives
Publons is currently working on 2 main initiatives to improve TPR. First, being able to retrieve from ScholarOne any reviewer reports, decision letters, and author responses submitted as attached files, and display them with the rest of the peer-review content published on the Publons article page. This will enable the complete publication of all submitted peer-review content without the need for any manual work. Second, both Publons and IOPP are exploring ways to show more useful content on the Publons badge, so that it can be seen directly from the journal article page. An example of this could be flagging the name of the reviewers if they have agreed to sign the reviewer reports with their name.

Conclusion

The outcome of the partnership between IOPP and Publons is a more efficient and elegant solution to offering TPR for journals that use ScholarOne as their manuscript submission and peer-review system. In this API-based system, there is minimal manual work required on the publisher’s side. For any publishers looking to publish the peer-review content in their journals, we would suggest focusing on keeping manual work for staff to a minimum in order to increase scalability. To this end, we would recommend using APIs for requesting/transferring data between systems. It is also important to carefully consider the wording of the TPR questions on the author submission form (and the reviewer report form, if applicable). Ensuring that this is clear (and linking to supporting information) may help to increase the author/reviewer opt-in rates.

Given the positive results of the pilot program to date, in particular the relatively high demand from authors, and the scalability of this TPR system, IOPP is planning to roll out the TPR option on all of its fully open access journals in the near future. This is a key part of IOPP’s Open Physics initiative.⁵

References and Links

1. https://asapbio.org/letter
2. https://publons.com/static/Publons-Global-State-Of-Peer-Review-2018.pdf#page=49
3. https://www.nature.com/articles/ncomms13626
4. https://theplosblog.plos.org/2020/06/published-peer-review-1-year-across-plos/
5. https://ioppublishing.org/open-physics

Marc Domingo (ORCID: 0000-0001-8537-8536) is Product Manager, Clarivate, London, UK. Simon Harris is Managing Editor at IOPP Publishing, Bristol, UK.