Abstract

Background: Anecdotal evidence indicates that supporting online materials (SOM) may present an increased editorial burden for hosting journals. This study tested the hypothesis that SOM present an increased editorial burden that may not be outweighed by reader-perceived value.

Methods: Separate online questionnaires regarding aspects of SOM were distributed to managing editors via e-mail and to readers of peer-reviewed life-science and medical journals via e-mail, social media, and online forums. Editors were surveyed with respect to their journals’ guidelines, the perceived impact of SOM on editorial burden and publication effectiveness, and methods by which readers can access SOM. Readers were asked how often and why SOM are accessed, their perceived value, and preferred methods of access. Survey results were analyzed by using SPSS, version 21.0.

Results: Seventy-nine editors were invited to participate. Of the 30 respondents, 26 (87%) worked for journals that publish SOM. Of the 26, editors who indicated that SOM have a somewhat or very important impact on editorial burden were 2.25 times more likely to work for journals that have published guidelines or limits for SOM than editors who did not so indicate. Similarly, editors at journals that typically edit or request changes in SOM were more likely to indicate a more important impact on editorial burden than editors at other journals. However, editors who rate SOM as somewhat or very important for editorial burden were less likely than other editors to rate SOM as having a somewhat or very important impact on their journals’ effectiveness as a publication. The reader survey received 105 responses; 65% were from academic employees. Industry employees were nearly 3 times more likely than academic employees to access SOM and 1.4 times more likely to rate SOM as highly valuable; government employees were half as likely as academic employees to access SOM or to rate them as highly valuable. Readers who access SOM frequently or always were significantly more likely to rate SOM as highly valuable than readers who access SOM occasionally. Finally, readers who frequently or always access SOM were more likely to access SOM through links embedded within PDFs than through other methods.

Conclusions: Journal editors perceive SOM as an additional editorial burden even when guidelines or limits are imposed. However, the burden appears to be merited according to reader-perceived value. The use of more stringent guidelines and publication of SOM on author-hosted sites are recommended for consideration by journals that publish such materials.

Introduction

Academic journals are increasingly publishing peer-reviewed articles online. Online publishing escapes the constraints of traditional print media in that more content can be accommodated. Although print articles are typically limited in numbers of pages or words and of tables or figures, content published only online—often referred to as supplemental or supporting online materials (SOM)—does not have to be limited in the same way, because the costs of publishing online are lower.¹

Rapid increases in the publication of SOM in recent years have led to some important observations.² In particular, the practices for publishing SOM vary widely. Some journals do not accept any SOM. Others have relegated key pieces of an article (such as the methods) to the online publication, and nearly all articles published by some journals have accompanying online-only content. Some journals that accept SOM have strict guidelines as to the types of content that can be included (for example, file-size limits and numbers of additional figures); others do not impose any limits on SOM. The variety of practices has catalyzed recent efforts to produce guidelines for publishers in accepting SOM.³

No data exist to describe the frequency of use of or the importance attached to SOM by journal readers. There are also questions about the rigorousness with which SOM are scrutinized by editors and peer reviewers. The incorporation of SOM into a journal’s publishing repertoire increases not only the content published and managed by the journal but probably the workload of journal staff—who must receive and process the materials ahead of publication, manage and maintain the electronic resources, and incorporate SOM into production workflows—and peer reviewers. These issues highlight possibilities for standardization in how SOM are accepted and incorporated into publications; they also expose gaps in the data needed to support the continued publication of these materials.

Our study sought to determine whether readers attach value to SOM that warrants the potential editorial burden and whether SOM present a perceived increase in work burden for editors of peer-reviewed academic journals.

Methods

Study population

A cross-sectional survey involved the electronic distribution of two separate online questionnaires. All responses were submitted anonymously, but Internet protocol (IP) addresses were recorded to determine countries of origin. One questionnaire was intended for readers of peer-reviewed academic journals in the life, medical, and physical sciences. A hyperlink to the online questionnaire was initially disseminated through e-mail and social media to science professionals, particularly in academic research. However, later “viral” sharing of the hyperlink through participants’ contact lists, social networks, and science-related message boards prevented determination of the response rate for the reader survey; 105 persons completed the questionnaire.

The second questionnaire was disseminated to managing editors of peer-reviewed academic research journals in the life, medical, and physical sciences. Seventynine managing editors were identified from journal directory listings and contacted via e-mail with a hyperlink to the questionnaire; 30 managing editors completed the questionnaire.

Surveys

The two questionnaires were designed and hosted using Survey Monkey (www.surveymonkey.com). Each questionnaire contained 10 or fewer multiple-choice questions that were designed to collect participants’ demographic information (position and sector) and perceptions regarding SOM without introducing bias.

The questionnaire for the reader survey comprised the following 10 questions:

1. Please describe your current career level (undergraduate student/graduate student/postdoctoral scholar/instructor/ researcher/assistant professor/associate professor/professor/other, please specify).
2. Please describe your place of employment (academic institution/industry/ government/other, please specify).
3. Please describe your field of expertise (biological sciences/materials sciences/ computer sciences/other, please specify).
4. On average, how many peer-reviewed research articles do you read per week (0 / 1–2 / 3–5 / 5–7 / 7+)?
5. On average, what percentage of these research articles do you access online only (ie, not from print editions) (not applicable/none/less than 50%/more than 50%/100%)?
6. If an article states that additional, supporting materials (eg, raw data, databases, code, videos, color photos) are available online, how often do you attempt to access these materials (never/occasionally/frequently/ always/other, please specify)?
7. If supporting online materials are available, what is your most likely reason for accessing them (not applicable/ material seems appealing in a visual or informational way/material was deemed important by article’s authors/ material may affect my conclusions about the article/material may have significance for me because it is directly relevant to my field of research)?
8. When attempting to access supporting online material, how often do you find that the links to the material are nonfunctional (not applicable/never/ occasionally/frequently/always)?
9. Thinking about the occasions when you have accessed supporting online materials, upon reflection, how valuable (not at all valuable/slightly valuable/somewhat valuable/highly valuable) would you say the materials were to the following: your overall assessment of the article; your conclusions about the article; the contributions of the article to its field; and the contribution of the article to your own research?
10. If supporting online materials are available for an article, what factors would make you more likely to access them; choose all that apply (links embedded in HTML version of article/links embedded in PDF article/centralized access point on journal homepage/other)?

The questionnaire for the editor survey comprised the following seven questions:

1. Are you an employee/editor of a peer reviewed academic journal (yes/no)?
2. If yes, does your journal publish and/or host supporting/supplemental materials for research articles online (yes/no)?
3. If yes, please estimate what proportion of the manuscripts accepted for publication by your journal (online or in print) include supporting materials to be published online (0–20%/20–50%/50–80%/80–100%).
4. For supporting materials to be published online, does your journal regularly: edit these materials (yes/no); request changes to these materials (yes/no); submit these materials for peer review (yes/no)?
5. Does your journal include guidelines and/or limits for authors on the: content of supporting materials (yes/no); quality of supporting materials (yes/no); quantity of supporting materials (yes/no)?
6. How would you assess the impact of supporting online materials on your journal’s: editorial burden; Web hosting burden; overall readership; overall effectiveness as a publication (not at all important/not very important/neutral/ somewhat important/very important)?
7. How are readers directed to access supporting online materials (choose all applicable answers: Web address listed within article/hyperlinks at beginning or end of article/hyperlinks embedded at relevant points within HTML text of article/hyperlinks embedded within text of PDF article/centralized access point on journal Web site/other, please specify)?

Statistical analysis

Responses from both questionnaires were independently compiled into a spreadsheet. Survey results were analyzed by using SPSS, version 21.0. Logistic regression was used to determine odds ratios (ORs) with 95% confidence intervals (95% CIs). Rate ratios were calculated by comparing the incidences of responses from the two groups.

Results

Readers were surveyed to determine the frequency with which they use SOM and the value that they attach to them.

The reader survey received
105 responses, of which 104 indicated that the field of expertise was in the life and medical sciences. Most of the 104 respondents identified themselves as “researchers” employed at academic institutions in North America (Table 1). Most indicated that they read an average of three to five peer-reviewed academic articles per week (Fig. 1). Nearly 82% of respondents reported reading at least 50% of those articles online.

When asked how frequently they access SOM that accompany an article of interest, most respondents indicated that they access them “frequently” or “occasionally” (Fig. 2A). The likelihood that a reader accessed SOM occasionally, frequently, or always was associated with the sector in which the reader was employed (Table 2).
Industry employees were more likely than academic employees to access SOM (OR 2.79, 95% CI 0.56–13.91), and government employees were less likely (OR 0.41, 95% CI 0.15–1.14).

Readers were asked how valuable (not at all, slightly, somewhat, or highly) they found SOM to be to contributions to their own research, to contributions to the field, to their conclusions about an article, and to their overall assessment of the article. Readers tended to rate SOM as highly valuable for contributing to their own research but only somewhat valuable for all the other categories (Fig. 2B). Respondents’ employment sector was associated with the likelihood of accessing SOM because of relevance to their field and with rating SOM as highly valuable (Table 3). Industry-based respondents were more likely than academic respondents to access SOM because of relevance to their own field (OR 1.38, 95% CI 0.37–5.16) and more likely to rate SOM as highly valuable (OR 1.37, 95% CI 0.37–5.14).

A rate-ratio method was used to assess whether the preferred method by which a reader accessed SOM was associated with the reader’s rating of the value of SOM (Table 4). Readers who indicated that they were more likely to access SOM if hyperlinks were embedded in the PDF version of an article were more likely to rate SOM as somewhat or highly valuable than readers who were more likely to access SOM through links embedded in the HTML document or through a centralized access point (such as a table of contents).

Managing editors were surveyed to determine their journals’ practices regarding SOM and their editors’ perceptions of SOM as related to their workloads. Of 79 editors invited to participate, 30 editors responded to the questionnaire (response rate, 38%). Of the 30, 26 (about 87%) indicated that their journals published SOM (Table 5). Later responses were analyzed only for editors who worked at journals that published SOM. Most editors indicated that their journals published SOM for fewer than 50% of articles.

Editors were asked later about their journals’ practices for handling SOM. Most (about 85%) indicated that their journals typically send SOM for peer review, but less than 50% indicated that authors are typically asked to make changes in SOM, and about 27% indicated that their journal staff edit SOM (Table 5). Few journals appeared to have guidelines or limits regarding the quantity of SOM that can be accepted, and about half the survey journals appeared to have guidelines or limits regarding the quantity and content of SOM.

Finally, editors were questioned about their perceptions of the importance of SOM with respect to aspects of their journals’ activities. About 40% of editors rated SOM as somewhat or very important for their journals’ effectiveness as publications and overall readership, and 42% rated SOM as somewhat or very important with respect to their editorial burden (Fig. 3). Logistic-regression analysis revealed that editors at journals that did not regularly edit SOM were less likely to find SOM somewhat or very important with respect to the editorial burden than editors at journals that regularly edit SOM (OR 0.7, 95% CI 0.9–5.4; Table 6). Editors at journals that regularly request changes in SOM were more likely to find SOM somewhat or very important with respect to editorial burden than editors at journals that did not request changes in SOM (OR 4.38, 95% CI 0.56–33.95). Editorial burden was perceived as somewhat or very important even when a journal had guidelines or limits regarding SOM (Table 6).

Discussion

Although our study was limited by small samples and requires cautious interpretation, the findings indicate that journal readers in the life sciences and the medical sciences place value on SOM particularly if an article is relevant to their field. Readers’ employment sector appears to influence the likelihood that they will access SOM and find them valuable. Thus, reader interest appears to warrant the publication of SOM by academic journals.

In contrast, editors of journals that publish SOM perceive them as having an important effect on their editorial burden. That perception of editorial burden was observed even for journals that have guidelines or limits regarding SOM. At the same time, editors perceive SOM as important to their journals’ readership and effectiveness. Therefore, although SOM appear to impose a higher editorial burden on journal staff, the burden may be merited by reader-perceived value of SOM.

The adoption of more stringent and uniform guidelines for accepting and publishing SOM, like those proposed by the National Information Standards Organization and the National Federation of Advanced Information Services,³ may alleviate the editorial burden by reducing the workload. More detailed studies of reader use of SOM may help to refine guidelines so as to tailor SOM to reader needs and interests. For example, journals may consider relegating some types of SOM to author-hosted sites, thereby reducing the quantity of SOM that editors must handle. Indeed, in 2010, The Journal of Neuroscience announced that it was ending its publication of SOM and would permit authors instead to insert a link to author-hosted (and nonreviewed) supplements.⁴ Citing exponential growth in the amount of supporting material submitted, the editor-in-chief described the adverse effect on editors’ workloads, the burden on peer reviewers, often-inadequate reviews, and rising costs of hosting SOM. A failure to implement strict guidelines may prompt other publications to exclude SOM from their repertoires.

Future studies should expand on the work presented here to develop a clearer picture of both reader value and editorial burden of SOM. For example, more detailed questionnaires may gather specific data on reader characteristics to determine the types of journals that they read (such as medical versus life science) and the specific guidelines that are in place for those journals. A broader survey of publication-related staff may identify specific features of the perceived burden of SOM—such as management of peer review, handling of files, production timelines, and financial costs—to help to refine and implement guidelines to have the largest possible effect.

References

1. Collins MP, Berge ZL. IPCT Journal: A case study of an electronic journal on Internet. Journal of the American Society for Information Science 1994;45:771–776.
2. Schwarzman A. Supplemental information: Who’s doing what and why. Council of Science Editors Annual Meeting. 2012 May 18–21; Seattle, WA.
3. National Information Standards Organization (US) and National Federation of Advanced Information Services (US). Recommended Practices for Online Supplemental Journal Article Materials. January 2013. Baltimore, MD. NISO RP-15-2013.
4. Maunsell J. The Journal of Neuroscience. Announcement regarding supplemental material. 2010;30(32):10599–10600.

SHEILA M CHERRY is president and senior editor at Fresh Eyes Editing, LLC, Reynoldsburg, Ohio.