Most experiments done by a researcher fail. It is the endless repetition and constant tweaking of the methods that leads to the occasional useful result. Failure of an experiment in itself is not informative; rather, it is the understanding of why something didn’t work that can be fruitful. Yet when it comes to innovative ideas in science communication, it is common to view failures as mathematical proof that a given idea can never succeed. The following is a look at postpublication discussion, preprints in biology, and crowdsourced protocol repositories—three brilliant ideas that initially flopped despite their greatness.

Postpublication Review and Discussion

Over a decade ago, BioMed Central (BMC) recognized the importance of postpublication discussion. Prepublication review can improve papers and catch errors, but only time and subsequent work of other scientists can truly show which results in a publication are robust and valid. Unlike a print journal (or print as a medium, in general), the Internet permits the readers to comment on published papers over time. So in 2002 BMC developed and enabled commenting on every one of its articles across its suite of journals. Not only does this allow for postpublication review, but it enables readers to easily ask authors and other readers a question, with public responses enriching the original manuscript, clarifying, and helping to improve the comprehension of the work.

This is a terrific idea, but it didn’t really catch on. When analyzed in 2008 by Euan Adie, after five years of commenting, only 2% of the 37,916 BMC papers had a comment.¹ Another innovative publisher, PLOS, enabled commenting a few years later, with 18% of the papers receiving comments, 40% of which were from the papers’ authors.² This commenting is still useful, even if rare, but seems to be far below the expectations of BMC and PLOS. Skeptics concluded from this experiment that scientists don’t have the interest or time to comment on other people’s papers—a reasonable but (I believe) wrong conclusion.

One barrier to postpublication commenting on journal websites is the reluctance of scientists to create accounts with individual publishers. Time is precious for researchers, and while they may have a minute to ask a question about a paper, they don’t have the 10 minutes it may take to open an account on some publishers’ websites. While registering at PLOS is relatively quick and painless, any registration button is a deterrent. However, if scientists can comment without creating an account, they do so happily; hence, there is vibrant and active discussion of research articles on Twitter.

Another big hurdle to postpublication discussion is the fear of repercussions for commenters. Neither PLOS nor the PubMedCommons commenting platforms permit anonymous posts. Figure 1 shows what happened when the online journal club PubPeer began to allow true anonymity.

Preprints

Remarkably, despite the creation of arXiv for physicists in 1990 and despite the enthusiastic embrace of preprints by the physics community, it has been assumed this is impossible for biology. The common argument is that biologists are different from physicists and the arXiv success is not informative. What many did find telling is the death of the 2007 preprint initiative from the Nature Publishing Group (NPG). NPG tried preprints with Nature Precedings, but adoption was low and in 2012 NPG pulled the plug on the experiment.³ This triggered some skepticism about the prospects of the bioRxiv preprint effort from Cold Spring Harbor Lab (CSHL) Press.⁴ Critics told the director of CSHL Press, John Inglis, that a preprint for biologists simply couldn’t work.⁵

Once again, we must ask the cause of the Nature Precedings failure. Did NPG kill it because biologists wouldn’t behave in the same way as physicists? We know that isn’t the case. Preprints in biology are all the rage today, with recent articles in the New York Times⁶ and the Economist.⁷ They speed up science communication and ensure an open-access version of the paper is available—scientists tend to love them. Compared to the 20 years of preprint use in physics, it’s still the early phase for bio-preprints, but deposits in bioRxiv are growing rapidly (Figure 2); in addition to bioRxiv, there are also thousands of biology preprints at The PeerJ, F1000 Research, and figshare.

I don’t have insider knowledge why Nature Precedings shut down. My guess is it was one or more of the following: lack of a clear monetization plan (Cold Spring Harbor Lab Press, which publishes bioRxiv, is a nonprofit and is supporting bioRxiv as a community resource); realization that preprint adoption requires a major culture shift and would take promotion and time; a hesitation of biologists to support a platform with content hosted by a commercial publisher (again, as part of the CSHL nonprofit, bioRxiv is in a good position to be the independent cross-publisher preprint resource and advocate).

Crowdsourced Community Protocols

In the winter of 2012, Alexei Stoliartchouk and I came up with the idea for protocols.io—a central place where scientists can share and discover science methods. We wanted to create a site where corrections and the constant tweaking of science methods could be shared, even after publication in a journal.

Before we launched our protocols.io journey, my PhD advisor Jasper Rine connected me to a former postdoctoral scholar of his, Chris Yoo. A few years before I came to Berkeley, Chris left Jasper’s lab to cofound bioprotocols.com.⁸ With a million dollars of venture capital, he set out to create exactly the protocol repository that Alexei and I were proposing to build, a decade later. Bioprotocols.com did not work out, and I suspect strongly that Jasper predicted my meeting with Chris would arm me with a long list of reasons why that company failed. Indeed, I got that list, but I also got an enthusiastic promise from Chris that he would do everything in his power to help us create protocols.io and make sure that this take two at his dream would be successful.

Few people know about bioprotocols.com, but many know about OpenWetWare (OWW) and Nature Protocol Exchange—both open-access community resources for sharing protocols. Both have been mentioned to me countless times as evidence that protocols.io wouldn’t work. As with preprints, the problems that OWW and Protocol Exchange faced seemed to be proof that biologists would not share details of their methods on such a platform. As with bioRxiv, we are in the early days of protocols.io, but judging from the growth in the figure below, it’s hard to argue that biologists don’t need this or that they won’t take the time to publicly share their methods.

Every researcher who fails to reproduce someone’s published result has to ask, “Is the published result wrong? Did I screw up? Or is it the difference in the cell line, strain, or some such detail in my application?” Knowing why a good experiment fails is critical and informative; assuming the wrong reason for an idea’s failure is a serious barrier to innovation in science communication.

References

1. Adie E. Who comments on scientific papers—and why? Nascent, July 22, 2008. http://blogs.nature.com/nascent/2008/07/who_leaves_comments_on_scienti_1.html.
2. Adie E. Commenting on scientific articles (PLoS edition). Nascent, February 11, 2009. http://blogs.nature.com/nascent/2009/02/commenting_on_scientific_artic.html.
3. Oransky I. Tracking retractions as a window into the scientific process: Nature Precedings to stop accepting submissions next week after finding model “unsustainable.” Retraction Watch, March 30, 2012. http://retractionwatch.com/2012/03/30/nature-precedings-to-stop-accepting-submissions-next-week-after-finding-model-unsustainable/.
4. Kaiser J. New preprint server aims to be biologists’ answer to physicists’ arXiv. Science, November 12, 2013. http://www.sciencemag.org/news/2013/11/new-preprint-server-aims-be-biologists-answer-physicists-arxiv.
5. Kaiser J. BioRxiv at 1 year: A promising start. Science, November 11, 2014. http://www.sciencemag.org/news/2014/11/biorxiv-1-year-promising-start.
6. Harmon A. Handful of biologists went rogue and published directly to Internet. New York Times, March 15, 2016. http://www.nytimes.com/2016/03/16/science/asap-bio-biologists-published-to-the-internet.html?_r=2.
7. Economist. Taking the online medicine. Economist, March 19, 2016. http://www.economist.com/news/science-and-technology/21694990-old-fashioned-ways-reporting-new-discoveries-are-holding-back-medical-research.
8. DeFrancesco L. BioProtocol, a web-based collection of protocols, is launched. Bioprocess Online, May 4, 2000. http://www.bioprocessonline.com/doc/bioprotocol-a-web-based-collection-of-protoco-0001.

Lenny Teytelman is the founder of www.protocols.io.

Save