SPEAKER:
Brian Nosek
Executive Director
Center for Open Science
Charlottesville, Virginia

REPORTER:
Peter J Olson
Senior Copyediting Coordinator
KnowledgeWorks Global Ltd.
Waterbury, Vermont

“I am delighted to not be with you all today!”

The smile on Brian Nosek’s face revealed the irony behind his opening quip. Speaking from his home office in Charlottesville, Dr Nosek was about to deliver the Keynote Address for the first-ever virtual annual meeting in CSE’s history, which was commencing at the outset of the COVID-19 pandemic. The inference of his remark, of course, was that his delight was inextricably entwined with his support of CSE’s effort to stem the spread of coronavirus by adapting to the “new normal” of virtual gatherings.

The COVID-19 pandemic has forced professionals, organizations, and institutions of every sphere not only to envision new ways of accomplishing their goals, but also to reexamine the origin and even the integrity of those goals. In that sense, Nosek’s address had a fortuitous sense of pertinence about it. As Executive Director of the Center for Open Science, one of his primary missions is to close the gap between the altruistic tenets of the scientific process and the real-world research practices that are in disharmony with those tenets. Not unlike the voids created by the COVID-19 crisis, Nosek sees this gap between ideals and reality as an opportunity to institute behavioral changes within the scientific research culture by reinforcing the core values that the scientific community purports to uphold. The question, he said, is: “How [can we] get researchers to live closer to the values that they possess—or that we collectively possess—for how science operates?”

To seek the answer, we must first understand what those values are. While acknowledging that there are several ways to define them, Nosek cited the Mertonian norms as a foundation. In the mid-twentieth century, world-renowned sociologist Robert Merton identified 4 key principles that are unique to the scientific enterprise: communality (the open sharing of information), universalism (merit-based evaluation of research), disinterestedness (selfless motivation), and organized skepticism (acceptance of critical scrutiny). Yet several “counternorms” that have taken hold in the real world are at odds with these ideals. A survey published by Anderson et al¹ in 2007 revealed that while a substantial majority of scientists in the United States say they both endorse and abide by principles that emulate the Mertonian norms, an equally substantial majority perceive that the scientific research community as a whole routinely abandons those principles in favor of the counternorms. Nosek offered a sobering explanation for this paradoxical result: The incentives for success in science have become rooted in the pursuit of publication rather than the pursuit of knowledge.

It makes sense, sociologically speaking. When he first embarked upon a career in the sciences, Nosek learned early on that certain behaviors are rewarded more than others, and that publication is the ultimate reward. Studies that yield negative results, exceptions, and inexplicable outcomes are less likely to be published than those that report neatly packaged, novel results that fulfill the study objectives. What is more, publication leads to the next job, the next grant, and the next step in career advancement; yet Nosek asserted that these incentives often lead to science that is not as accurate as it ought to be, regardless of the researchers’ intent. That said, he acknowledged that researchers generally have good intentions, and that those intentions can be safeguarded by transforming the current incentives into ones that promote transparency, reproducibility, and the credibility of evidence.

Returning to the Mertonian norms requires a substantial shift in the overall culture of scientific research, and changing the incentives is just one of several means toward such an ambitious end. According to Nosek, normative change and policy change are also critical parts of the formula, particularly in the “highly decentralized” world of scientific research. Yet, in order to effect such changes, researchers need to be provided with tools that can be integrated seamlessly into their daily workflows to minimize inconvenience and promote compliance. Nosek cited three such tools, one for each category of change, that are being implemented throughout the scientific community.

Observing that “the easiest kind of intervention is [one] that tries to promote visibility,” Nosek noted that several journals now reward authors with badges simply for complying with requests for transparency (see e.g., http://cos.io/badges).²  These are not actual badges, of course—they’re printed icons stamped on journal articles. Nonetheless, this is an example of a normative change that has a fundamental appeal and signals a desirable behavior to the scientific community, thus increasing the chances that it will become commonplace. In the category of incentive change, the registered reports model—in which peer review occurs after the study design phase in exchange for guaranteed publication, regardless of outcome—shifts authors’ incentives from publishing exciting results to asking exciting questions by establishing a culture in which robust, sound study designs are valued over positive outcomes (for more, see http://cos.io/rr/).³ In this model, the incentives also change for reviewers, who can jettison their concerns about a study’s outcome and instead ask: “Do we need to know the answer to this question, and is this method a good way of asking it?” Finally, policy changes can drive the other 2 categories of change. The Transparency and Openness Promotion (TOP) guidelines (http://cos.io/top/),^4,5 a set of stakeholder criteria promoting reproducibility and transparency within the scientific community, provide a policy framework for journals, funders, and institutions to guide the behaviors of their authors, grant recipients, and employees, respectively. In addition, these guidelines incorporate a system of rating journals (http://topfactor.org/)⁶ rather than ranking them, ultimately rewarding journals for the integrity of their processes rather than the outcomes they report.

These endeavors are taking root within the scientific community, and some are already having a discernable impact. In 2014, the journal Psychological Science adopted the use of badges, and now 80% of Psychological Science articles bear badges indicating the open sharing of data. Registered reports are being cited frequently, despite the fact that they publish more negative results than articles that have been submitted via traditional workflows.^7,8 Finally, more than 1000 journals have adopted the TOP guidelines, and all of the major publishers have expressed support for them as a viable set of guiding principles for the scientific process. It appears that the counternorms are being countered by new norms.

Despite these encouraging signs, Nosek conceded that the task at hand remains a daunting and unfinished one. “The challenge that we face for ultimate change of the research culture,” he said, “is that each researcher is embedded in an ecosystem of different incentives.” However, he also opined that “the best solution we have is sunlight.” And although his reference to the sun was steeped in the idea of transparency, this imagery had additional, richly woven layers: Not only did it evoke an essential, alimentary component of the aforementioned ecosystem, it also conveyed Nosek’s optimism about a bright future for the scientific research process—and indeed, the very integrity of the scientific enterprise.

References and Links

1. Anderson MS, Martinson BC, De Vries R. Normative dissonance in science: results from a national survey of U.S. scientists. J Empir Res Hum Res Ethics. 2007;2(4):3–14. https://doi.org/10.1525/jer.2007.2.4.3
2. http://cos.io/badges
3. http://cos.io/rr/
4. http://cos.io/top/
5. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, Buck S, Chambers CD, Chin G, G Christiansen, et al. Promoting an open research culture. Science. 2015;348(6242):1422–1425. https://doi.org/10.1126/science.aab2374
6. Center for Open Science. TOP standards [accessed September 28, 2020]. https://topfactor.org/
7. Allen C, Mehler DMA. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 2019;17(5)e3000246. https://doi.org/10.1371/journal.pbio.3000246
8. Hummer LT, Singleton Thorn F, Nosek BA, Errington TM. Evaluating registered reports: a naturalistic comparative study of article impact [preprint created December 4, 2017; last edited July 31, 2019; accessed September 28, 2020]. OSF preprints. https://osf.io/5y8w7/