The traditional peer-review process, although widely considered essential for scientific publishing, has limitations. In recent years, therefore, some journals have experimented with nontraditional methods of peer review. Here are a few examples.

Atmospheric Chemistry and Physics (www.atmospheric-chemistry-and-physics.net/home.html), established in 2001, uses the following procedure: A paper is published on the Web site Atmospheric Chemistry and Physics Discussions after the editor determines that it meets basic format and structural requirements. It then undergoes what the journal calls “interactive public discussion”, in which the designated reviewers publish their comments (which can be anonymous or not, at reviewers’ discretion).¹ Other members of the scientific community can read the paper and comment as well, but their comments must be attributed. Authors can respond to the comments, and the whole discussion is permanently and publicly archived. Authors can revise their manuscripts to reflect reviewers’ comments before the editor decides to accept or reject the paper. Accepted papers then appear in the journal.

Thomas Koop and Ulrich Pöschl, executive editors of Atmospheric Chemistry and Physics, said in a 2006 article that 25% of papers in this journal receive comments from someone other than the chosen peer reviewers, whereas only 1% of papers in other journals receive comments from the scientific community at large.¹ A brief glance at recent issues, though, indicates that the percentage is no longer nearly as high—perhaps 5% of papers receive outside comments.

Biology Direct (www.biology-direct.com) allows authors to choose their own three reviewers from the editorial board in the relevant subject. Each chosen board member then reads the manuscript and has 72 hours to decide whether to review the manuscript, ignore the submission (essentially rejecting it), or nominate someone to review in his or her place. Once a board member (or proxy) has agreed to review a manuscript, he or she has 3 weeks to prepare comments for the authors and comments that will appear, with the reviewer’s name, with the published version of the article. If the reviewer declines to make public comments, the statement “This reviewer made no comments for publication” appears under the reviewer’s name. The authors can withdraw the manuscript if they do not want it published alongside reviewers’ comments (and they cannot resubmit it, even to other editorial board members) or they can write rebuttals, which will then also be published.

Essentially, once three board members (or their proxies) agree to review a manuscript, it is considered “accepted” by the journal.² The only exception is if a reviewer and the editor-in-chief agree that the manuscript is not a legitimate scientific work.

BMC Medicine (www.biomedcentral.com/bmcmed) uses “open peer review”, in which articles are published online with editors’ comments to authors, signed reports from peer reviewers, and the initial manuscripts and any intermediate versions. “There are people who will refuse to review under the open–peer-review model,” says Robin Cassady-Cain, editor of BMC Medicine. “However, in our experience, this has never proven to be a barrier to attracting good-quality reviewers for our submissions.”

Electronic Transactions on Artificial Intelligence (www.etaij.org) was active from 1997 to 2000. It had a two-step review process. First, an open online discussion of articles (all posts had to be signed) gave the authors comments and the opportunity to respond to any criticism. Second, articles underwent a more traditional peerreview process in which articles were either accepted or rejected for publication.³

At the beginning of 2009, The EMBO Journal (www.nature.com/emboj) implemented an open–peer-review procedure in which it publishes, in a supplement to the published article, editors’ letters to authors, referees’ comments, authors’ responses, and statistics, such as the length of time between submission and publication of the manuscript. Publication of the supplement is optional, but “in our experience only around 10% of authors opt out,” says Bernd Pulverer, chief editor of The EMBO Journal. “Furthermore, we have not noted a decline in either the willingness of referees to referee for The EMBO Journal or the quality of their reports.”

According to the EMBO Journal’s Web site, the response from authors and referees has been largely favorable. In fact, when this article went to press, the editors were planning to begin the same open–peer-review process for the other four EMBO scientific publications. “In our view, this enhancement to the journal demonstrates the quality of the editorial process and serves not only to add transparency but to train less experienced referees and authors,” Pulverer says.

Nature experimented with a novel method of peer review in summer 2006. Authors’ participation was voluntary, but although an earlier survey had indicated substantial interest, only 5% of authors who submitted articles to Nature during that period agreed to participate.⁴ Those authors’ manuscripts were posted online for public comment in addition to undergoing the standard peerreview process. A little more than half the manuscripts received comment despite “a healthy volume of online traffic”, and the editors did not find the posted comments very helpful.⁴

Neoplasia (www.Neoplasia.org) instructs authors to submit their manuscripts to an Internet site called JournalSoft, where those whom the journal has approved to referee in the subject can review them. Authors and referees can then engage in a “conversation” online by posting comments about the articles. “Our customer base is largely very satisfied with the process,” says Divya Khanna, production editor of Neoplasia.

The Journal of Interactive Media in Education, or JIME (www-jime.open.ac.uk), has more than a decade of experience with a nontraditional system of peer review. When editors at the journal receive a manuscript, they set an interactive review process into motion and solicit peer reviewers. If the reviewers agree to forgo anonymity (and most do), the editor introduces them to the authors. For the next 3 to 4 weeks, reviewers and authors discuss the article on a private Web site, and reviewers have the option to e-mail private comments to the editor. At the end of the process, the editor decides to accept or reject the manuscript. Reviewers seem to like the approach: “A few reviewers have even told us that reviewing articles for JIME is fun!” founding editors Simon Buckingham Shum and Tamara Sumner have said.⁵

An article accepted by JIME is posted online for the community at large to review. Authors and the original reviewers may participate in the discussion as well, and editors add their own comments, either publicly or privately to the authors. Authors then revise the article, the final version of which must be approved by the editor before publication. The editor edits the “review debate”, which is published with the final article. “Although it has always been the case that authors, reviewers, and editors may not agree”, Buckingham Shum and Sumner said, “this has been hidden to all but those involved, and sometimes the resulting debate is worth preserving.”⁵

Buckingham Shum and Sumner have noted prerequisites for successful use of their journal’s model. “Our conclusion is that to use the Web for productive peerreview discourse, it is critical to appoint reviewers (not just hope that someone out there will comment) and set dates and deadlines,” they have said.⁵ That conclusion seems to be supported by other journals’ experience. For example, Nature did not appoint people to post comments on articles; as a result, some articles had no comments.⁴ “Authors report gaining far more from engaging in critical discussion of their work than they get in conventional review,” Buckingham Shum and Sumner have observed. They do, however, add a caveat: “Only scholars who know their communities can judge whether interactive, open peer review could be deployed fruitfully.”⁵

References

1. Koop T, Pöschl U. An open, two-stage peer-review journal. [Internet] [cited 2010 Mar 28]. Available from: www.nature.com/nature/peerreview/debate/nature04988.html. doi: 10.1038/nature04988.
2. Koonin E, Landweber L, Lipman D, Dignon R. Reviving a culture of scientific debate. [Internet] [cited 2010 Mar 28]. Available from: www.nature.com/nature/peerreview/debate/nature05005.html. doi: 10.1038/nature05005.
3. Sandewall E. Opening up the process. [Internet] [cited 2010 Mar 28]. Available from: www.nature.com/nature/peerreview/debate/nature04994.html. doi: 10.1038/nature04994.
4. Greaves S, Scott J, Clarke M, Miller L, Hannay T, Thomas A, Campbell P. Nature’s peer review trial. [Internet] [cited 2010 Mar 28]. Available from: www.nature.com/nature/peerreview/debate/nature05535.html. doi: 10.1038/nature05535.
5. Buckingham Shum S, Sumner T. JIME: an interactive journal for interactive media. Learned Publishing 2001;14(4):273-285.

Christina Sumners, a graduate student in science and technology journalism at Texas A&M University, wrote this article while a Science Editor intern.