For many decades, researchers and librarians have been evaluating a journal’s impact by counting citations. However, simply counting citations has recently been deemed an insufficient metric to evaluate a journal’s total impact, and the publishing community has begun exploring alternatives.

Carissa Gilman introduced this session by explaining how the topic of journal impact became a special interest of hers. A few months ago, Cancer began displaying the Altmetrics score of articles on its website. She was interested in exploring how other journals were using these data, but this led her and some colleagues to engage in a philosophical discussion of sorts: namely, “What is the social responsibility of a scientific journal to measure the impact of the science it publishes?”

Kerry Kroffe was first to contribute to this discussion by sharing what PLoS ONE has been championing. He began with a popular quote to challenge the way we think about journal impact: “Not everything that can be counted counts, and not everything that counts can be counted.” With this in mind, he introduced the concept of “altmetrics,” a way of measuring a journal’s impact that does not aggregate a journal’s activity to one metric. Including approximately 20 data sources, PLoS ONE examines many sources of activity at the article level, including social media. The impact factor is slow to be released, and a benefit of these alternative metrics is that they can be updated daily. He then introduced a suite of online tools that PLoS makes available to allow anyone to investigate the impact of any article published in PLoS journals. Entitled “ALM Reports,” this tool has the potential to change the way contributors and end users are able to evaluate impact.

Next, Christine Casey described how Morbidity and Mortality Weekly Reports Serials has taken an entirely different approach to evaluating journal impact. Casey encouraged editors to think beyond bibliometrics. She began by reminding the audience that scholarly communication emerged from personal correspondence and that modern scientific publishing retains similar attributes and purposes. Specifically, editors have many roles and responsibilities, such as disseminating content, archiving literature, quality controlling the “best” information, and influencing the field. To assess the impact of its work, the Centers for Disease Control and Prevention has adapted an Institute of Medicine “narrative framework” that has five domains: disseminating science, creating awareness, catalyzing action, effecting change, and shaping the future. Using a case study about guidelines for field triage of injured patients, Casey highlighted how a journal can have more impact through encouraging repurposing of published content, working with stakeholders, and following up on the application of the research. Although the framework is well suited for scholarly content that is highly applied, Casey suggested it could be appropriate for a traditional researchoriented article as well.

Last, Christina Mills discussed the idea of an evaluation framework for social impact. Posing the question “Do we have a social responsibility and how do we measure it?” Mills described how MEDICC Review aims to contribute to health equity. She asked the audience to consider how journals can get beyond the high-level statements about impact and actually evaluate it. One suggestion was to select articles that a journal thinks might have social impact and follow up with the authors, helping connect them to appropriate decision makers. She admitted that it is easier for publishers to look up their bibliometrics but challenged the audience to think critically about what publishers can control: what they publish, who they publish, how they work with authors (especially of rejected submissions), how articles are ultimately published, and what a journal does after an article is published. She invited CSE members to collaborate in developing an evaluation framework.

Though more traditional methods of measuring journal impact certainly have utility, the common theme of this session was to think beyond those metrics to evaluate what we can do as a publishing community to maximize the impact our published articles have on our communities and scientific fields.