How do we know that a journal is “healthy” or “successful”? By looking at its subscription base? Financial performance? Journal impact factor? By counting the number of submissions? Or is success best measured by a combination of metrics? Where do subjective factors such as the quality of peer review figure into an overall assessment of a journal’s impact?

In an interesting session dealing with how to best monitor the health of an STM journal, moderator Ken Heideman, Director of Publications, American Meteorological Society, led a panel of experts at the recent CSE meeting in Philadelphia.

Heideman opened the discussion by noting that a journal’s health is assessed differently by each type of stakeholder, including authors, editors, publishers, and reviewers. “Assessing journal health by looking at the impact factor only is like assessing general personal health by looking only at your cholesterol level; it just doesn’t work that way.”

The panel, which consisted of Brooks Hanson, Director of Publications, American Geophysical Union; Margaret Perkins, Director of Manuscript Editing, New England Journal of Medicine (NEJM); and Julie Steffen, Director of Publishing, American Astronomical Society, discussed perspectives on journal metrics and the ways their organizations choose to determine success. All agreed that metrics evaluation, based on the community that each journal serves, largely affects a “success” measurement.

“Know your stakeholders and their needs” was the message emphasized by Hanson, who believes that reader feedback complements such measures as impact factor, number of submissions, and readership surveys in assessing the health of their portfolio.

“There is inertia in the status quo,” Hanson said. “You need to have current data to guide and assess status quo, opportunities, and experimentation.” He suggests collecting the following information to fully understand stakeholder needs: author surveys (after publication or rejection); competitor benchmarking (what are your competitors doing?); reader surveys; researcher surveys (is your journal contributing to their field?); and reviewer surveys (for suggestions on innovation in the peer review process). He also recommends analyzing meetings and conferences, and understanding the direction in which your field is heading.

Margaret Perkins, who posed the basic question “Why monitor?” agrees with Hanson but takes data collection for NEJM a bit further, incorporating into their analysis other metrics, such as acceptance rate, turnaround times, website traffic, news coverage, and social-media activity.

Author feedback is also very important to NEJM. “Recognizing that the road from submission to publication requires a good deal of effort on the part of our authors,” Perkins said. They write to the authors to solicit feedback on their submission, review, and production experience in order to best continue in their mission of advancing medical knowledge and improving care. Another survey area of interest involves understanding what factors help authors make publishing decisions—for example, reputation, quality, quality of peer review, turnaround time—and asking authors to rank these in order of importance.

The American Astronomical Society is a relatively small organization whose global journals moved to electronic publishing in 1995; Steffen faced the challenge of rendering detailed math content online, while streamlining production of their two journals, to enhance the value and utility of the society’s publications. “It was a huge change,” Steffen reflects on the mid 1990s innovation in their publication strategy. “Twenty years ago, we decided online would be the version of record and so we worked towards that.”

For the next phase of that transition, they appointed a task force, solicited community feedback, produced a white paper on what could be done next in electronic publishing, consolidated journal peer review operations, and developed a plan for rebranding. After a successful transition, the society’s journals remain leaders in the field.

All panelists agreed that regardless of size, stature, or standing of a scientific publication, the challenges in today’s publication environment are universal when it comes to innovating while maintaining the integrity of the science.