Original ArticleTrial sequential analysis reveals insufficient information size and potentially false positive results in many meta-analyses
Introduction
Meta-analyses aim to increase the power and precision of the estimated intervention effects [1], [2]. Meta-analyses are, however, criticized because the included trials are inevitably clinical diverse regarding patients, interventions, outcomes, etc. Hence, pooling the potentially heterogeneous trial results is sometimes inappropriate [3], [4]. Meta-analyses may also obtain false positive results (type I errors) or overestimate treatment effects due to systematic errors (bias) and random errors (play of chance). Bias may originate from publication bias [5], [6], [7], inclusion of trials with high-bias risk [8], [9], [10], [11], outcome measure bias [12], premature stopping of “positive” trials [13], and small trial bias [14]. Meta-analyses could also be data driven because they are retrospectively conducted. Random errors may arise due to repetitive testing as data accrue and testing of multiple outcome measures, which inevitably, sooner or later, lead to type I errors [15].
The required number of participants (information size) for a meta-analysis should be at least as large as an adequately powered single trial. Trial sequential analysis (TSA) is an approach that provides the required information size in meta-analyses [16]. To adjust for random error risk, meta-analyses not reaching the required sample size are analyzed with trial sequential monitoring boundaries analogous to interim monitoring boundaries in a single trial [16], [17], [18], [19], [20], [21]. Trial sequential monitoring boundaries adjust the P-value that is required for obtaining a statistical significance according to the number of participants and events in a meta-analysis. The fewer participants and events, the more restrictive the monitoring boundaries are and the lower P-value is required to obtain statistical significance.
The use of TSA in meta-analyses has been debated because the analysis ignores potential bias and heterogeneity [22], but adjustment for these factors seems possible [16]. We recently audited clinical guidelines taking Cochrane Neonatal Group reviews as basis for deciding which intervention to use [23]. Therefore, we have examined meta-analyses in these reviews with TSA with and without bias and heterogeneity adjustment to reassess the evidence they provide.
Section snippets
Material and bias definition
We identified all meta-analyses that included more than two trials reporting on a binary outcome measure from the 188 Cochrane Neonatal Group reviews in The Cochrane Library, Issue 4, 2004 [24]. From each review, whenever possible, we included three meta-analyses. We selected the meta-analyses on mortality outcomes and the first two eligible meta-analyses on clinical outcome measures according to the review authors' priority (or three, in case mortality was not meta-analyzed).
The meta-analyses
Eligible meta-analyses
We identified 188 Cochrane Neonatal Group systematic reviews in The Cochrane Library, Issue 4, 2004 [24]. Of these, we excluded 76 because the review included less than three randomized clinical trials, 29 reviews because they did not report a binary outcome measure, and 6 reviews because all trials had high-bias risk (i.e., had unclear or inadequate allocation concealment). From the remaining 77 reviews, we included a total of 174 eligible meta-analyses.
Characteristics of meta-analyses
The 174 meta-analyses included a median
Discussion
This study is the first to apply TSA on a large cohort of meta-analyses. Applying three different TSAs to Cochrane Neonatal Group meta-analyses revealed that many meta-analyses have insufficient information size and there are several potentially false positive results. The respective TSAs supported only the “traditional” significance (P < 0.05) in 61% (TSA30%), 33% (TSA15%), and 73% (TSALBHIS) of 79 significant meta-analyses. Applying TSA30%, TSA15%, and TSALBHIS on the 95 nonsignificant (P ≥ 0.05)
Conclusion
The interpretation of meta-analyses is complex. To adjust for random error risk in meta-analyses, we suggest applying TSA (e.g., TSA with a relevant prespecified intervention effect in combination with TSALBHIS) on meta-analyses. In this way, authors and readers of meta-analyses may reach a more balanced conclusion on the effect of interventions.
References (41)
- et al.
Putting clinical trials into context
Lancet
(2005) - et al.
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses
Lancet
(1998) - et al.
Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis
J Clin Epidemiol
(2008) - et al.
Overcoming the limitations of current meta-analysis of randomised controlled trials
Lancet
(1998) - et al.
Cumulating evidence from randomized trials: utilizing sequential monitoring boundaries for cumulative meta-analysis
Control Clin Trials
(1997) - et al.
Meta-analysis. Is moving the goal post the answer?
Lancet
(1998) - et al.
Meta-analysis in clinical trials
Control Clin Trials
(1986) - Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions 4.2.5. Available at...
- et al.
Discrepancies between meta-analyses and subsequent large randomized, controlled trials
N Engl J Med
(1997) Have you ever meta-analysis you didn't like?
Ann Intern Med
(1991)
Registering clinical trials
JAMA
Publication and related biases
Health Technol Assess
Contradicted and initially stronger effects in highly cited clinical research
JAMA
Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment in controlled trials
JAMA
Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses
Ann Intern Med
Methodological quality and treatment effects in randomised trials—a review of six empirical studies
Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles
JAMA
Randomised trials stopped early for benefit: a systematic review
JAMA
Are trial size and reported methodological quality associated with treatment effects? Observational study of 523 randomised trials
Why most published research findings are false
PLoS Med
Cited by (832)
Adjunctive cariprazine as a novel effective strategy for treating major depressive disorder: A systematic review and meta-analysis
2024, Journal of Psychiatric ResearchPrevalence of and factors associated with potentially redundant randomized controlled trials: a cross-sectional study
2024, Journal of Clinical Epidemiology