Jason Roberts discussed problems in statistical review as part of the peer-review process (i.e., assessment of study design elements such as appropriate inclusion and exclusion criteria, appropriate significance testing, power analyses, and correct application of statistical technique). Incorrect statistical application leads to a lack of reproducibility and, in turn, slows down scientific progress, wastes research funding and editorial pages, and results in retractions. Flaws in statistical methods are caused mainly by unrealistic expectations or lack of self-awareness on the part of investigators. Reviewers do not defer to statistical experts when they should, and editors do not necessarily require statistical review on a regular basis. Next, Roberts turned to the implementation process for journal statistical review. Using the 2009 implementation model of the journal Headache as an example, he explained as follows. First, it is important to integrate statistical review into the review process. The policy for statistical review should be communicated to all stakeholders. Rollout should be accomplished via a multipronged communication process, which for Headache included two editorials,1,2 and simultaneous launch of new Instructions for Reviewers to outline the new standards. At Headache, statistical and methods reporting are now assessed for reproducibility and validation, and authors are required to upload relevant reporting guideline checklists. Each decision letter (for all manuscripts not rejected) includes information on statistical review as well as design and methods review. Statistical reviewers are instructed to “refine rather than reject” and be collaborative rather than adversarial. They are directed to recommend rejection only for those manuscripts with major problems, previous publication, or a study design that cannot achieve the desired goal.
Mary Beth Schaeffer of the Annals of Internal Medicine presented information about the Annals statistical review process. Reviewed manuscripts are discussed at weekly editorial meetings, and “approved papers” are assigned for statistical review (typically three to four per week). Statistical review assignments are to be completed within two weeks.
Finally, Eliseo Guallar, associate editor of Annals of Internal Medicine, spoke from the scientific editor’s point of view. He summarized what to look for in statistical review. In the Introduction, the author should highlight deficiencies in previous research, explain how the current study can advance the field, and then clearly state the hypothesis (i.e., What is the research question?). In the Methods section (however it may be formatted for the specific journal), investigators should define the time frame; consider a figure or flow chart for clarity; state sources of participants; and describe the sampling design, response rate, selection (inclusion/ exclusion criteria), compliance with ethical requirements, and potential for selection bias. They should also clearly describe data collection, main exposure/predictor/intervention variable(s), and mediators/modifiers/confounders, and outcome(s). Other potential statistical pitfalls include model selection, variable selection, the form(s) of variables, subgroup analysis, covariate screening model building, and measurement error (i.e., the overall effect of measurement error is not always to bias results toward null hypothesis).
Regarding other statistical issues, Guallar advised the following: Ensure that methods are appropriate for the study question and study design (study design should be stated in the Methods section and briefly in the abstract); avoid outdated/invalid methods even if pre-specified in the protocol; provide enough detail so that a knowledgeable expert can reproduce the analysis; and describe the software used for statistical analyses. For measurement of association and risk, check whether there is an association versus an effect and whether results are clinically meaningful. It is also important to report missing variables, missing outcomes, missing data, and between-groups differences. Statistical methods must be appropriate for the design; avoid methods that are now known to be invalid (e.g., last observation carried forward) or biased. Because validity and reproducibility are key, investigators must be willing to provide the following as appendix material when appropriate: study protocol; statistical codes; appropriate references for methods; and study data. Last, it is important to always check that the conclusions in the abstract and the end of the manuscript are supported by the data presented in the results. Guallar ended with a sage statement about the important contribution of statistical review to quality control in the peer-review process.
References
- Houle TT, Penzien DB. Statistical Reviewing for Headache. Headache, 2009;49(2):159–161.
- Sheftell FD. Best Practices in Publishing and the American Headache Society. Headache, 2009;49(S2):S47–S48.