Abstract
Background: Inconsistent reporting in published meta-analyses compared to registered protocol are poorly understood. The aim of the study was to assess inconsistencies between registered protocols and published reports among oncology drug meta-analyses. Materials and Methods: A cross-sectional study was performed including oncology drug meta-analyses published between January 1st and November 14th 2016 with a published protocol. Two investigators extracted data on: selection criteria, outcome(s) and statistical plan in protocol and manuscript, plus self-acknowledgement of inconsistent reporting between protocol and publication. Results: Protocol registration was present in 19% (23/119) of all oncology drug meta-analyses. In meta-analyses with protocol (n=23), 70% (16/23) had issues with inconsistent reporting between protocol and published report concerning; inclusion criteria, comparator group, intervention, outcome (PICO) or statistical analysis. Self-acknowledgement of changes between protocol and publication was found in 50% (8/16). Conclusion: In meta-analyses with protocol, discrepancies between registered protocols and publications are frequent.
The medical community continuously strives to seek the latest and most beneficial treatment for its patients, while minimizing harm. In order to ensure the highest standards of care, the selection of treatment is preferably based on the aggregated scientific body of evidence available at that time point, rather than based on results from single trials (1). The usage of meta-analysis to summarize results from several studies has successively gained wide popularity. The increase of meta-analyses ranges from one meta-analysis published in 1977 to 16,362 meta-analyses indexed in PubMed 2016 (2).
Selective and inconsistent reporting of randomized controlled trials (RCTs) are widely reported and recognized. At the RCT level, discrepancies between registered protocol and published manuscript with regards to outcome measures can pertain to the changing or omitting of primary endpoints, known as inconsistent or selective reporting (3-7). A potential consequence of inconsistent reporting is the publication of a seemingly positive trial powered for another (possibly not mentioned) outcome. Exploratory analyses of outcomes might cause false-positive results and problems with trial reproducibility, and mislead the readers of the importance of findings. In a clinical context, the worst-case scenario of the changing of an outcome might lead to drugs being approved for medical use based on a single study with a changed primary outcome (8, 9). Meta-analysis of high-quality RCTs is considered the highest level of evidence according to the Grades of Recommendation, Assessment, Development and Evaluation Working Group (GRADE) scale (10). However, the external validity of meta-analyses might be jeopardized due to selective and inconsistent reporting (11), with changes to the published manuscript other than those originally planned for (12). Furthermore, the retrospective nature of most meta-analyses might introduce bias with regards to insufficient blinding for study outcomes.
Several measures have been taken to increase transparency in the reporting of RCTs. Firstly, trial protocol registration in public databases is mandatory by the International Committee of Medical Journal Editors (ICMJE) since 2005 (13). A published protocol increases transparency for deviations from the protocol and makes it possible for anyone to assess. Despite the fact that meta-analysis protocol registration is now an available option (14), data on the usage of this option and the incidence of inconsistent reporting of meta-analyses is scarce. Furthermore, scientific journals of today encourage, but do not demand meta-analysis protocol registration (15).
In a Cochrane review aiming to study selective reporting at the systematic review level (review/meta-analysis protocol versus published review/meta-analysis), selective reporting of outcomes was found in 38% of the included systematic reviews (12). These results were further supported by the studies by Page et al. (1) and Tricco et al. (16). However, previous studies have been limited to systematic reviews, which might limit the generalizability to meta-analyses. Further, we are unaware of any previous study investigating the role of inconsistent reporting with regards to the full PICO (patient, intervention, comparison, outcome) outline and statistical analysis in meta-analyses (17).
Due to the lack of mandatory protocol registration for meta-analyses in most scientific journals we hypothesized that meta-analysis protocol registration might be low and inconsistencies between protocol and publication might be high. The aim of this study was to assess meta-analysis protocol registration and inconsistencies between study protocol and publication.
Materials and Methods
Reporting guidelines. This study was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration (18). This study was not registered in Prospero since none of the outcomes were related to patient or clinical outcome (19).
Criteria for inclusion in the study
Eligibility criteria. We included published oncology drug meta-analyses with a previously published meta-analysis protocol. Oncology was defined as related to cancer treatment. Drug therapy was defined as systemic treatment related to cancer disease, including anti-cancer treatment, palliation or other drug therapy to improve symptoms related to cancer disease.
Data sources and search strategy. An electronic search was performed in PubMed. The search was restricted to PubMed in order to limit findings to established indexed journals. The search was conducted on Nov. 14th 2016 including controlled vocabulary for more precision. Searches were limited to dates between 2016/01/01 and 2016/11/14.
Inclusion and exclusion criteria. Oncology drug meta-analyses published in 2016 with a registered meta-analysis protocol were included. A meta-analysis protocol was defined as a protocol published in a publicly available database. Exclusion criteria at abstract and title level were: non-oncology meta-analysis, a systematic review without a meta-analysis, non-drug meta-analysis, non-meta-analysis, non-English text, review or commentary. Systematic reviews without meta-analysis were excluded since they do not require statistical analysis. Further, exclusion criteria at full-text screening level were lack of protocol registration.
Study selection. The screening of studies was performed by one investigator (blinded for review), with reasons for exclusion discussed with a second investigator (blinded for review). Reference to a meta-analysis protocol was sought for in published manuscripts and supplementary material.
Data extraction. Data-extraction from protocol and published manuscript was performed independently by two investigators, in studies meeting all inclusion criteria. The following data was extracted: reported study selection criteria, primary outcome, secondary outcome(s) and statistical plan in protocol and manuscript, plus self-acknowledgement of inconsistent reporting between protocol and publication. Extracted study co-variables from publications were impact factor (20), funding source, conflict of interest, sex of the first author and origin of publication. First author sex determination was performed as previously reported (21).
Assessment of inconsistent reporting between meta-analysis protocol and publication. Assessment of inconsistent reporting was further evaluated through comparison between extracted data from protocol and published meta-analysis for each of the four domains based on PICO and statistical analysis. The first domain included “patient/population=type of study design”, “intervention” and “comparator”. The second domain included primary outcome, the third domain included secondary outcome and the fourth domain included statistical analysis. Inconsistent reporting was defined as any deviation in the published manuscript compared to the meta-analysis protocol. Inconsistent reporting in one domain was defined as “minor” inconsistency. Inconsistent reporting in two or more domains was defined as “major” inconsistency. Self-acknowledgement of changes was sought for in studies attributed with inconsistent reporting after evaluation of protocol and publication.
Quality assessment. Adherence to PRISMA guidelines was used as a surrogate marker for quality assessment (18). PRISMA covers 27 different domains, and adherence to all of these gave a maximum of 27 points. This score was used since the guidelines strive to improve reporting and quality of meta-analyses.
Main outcome variables. The main outcome variables were the presentation of a meta-analysis protocol in the published manuscript or supplementary material, and the frequency of inconsistent reporting in oncology drug meta-analyses.
Secondary outcome variables. Secondary outcome variables were: inconsistent reporting according to PICO and statistical analysis. Self-acknowledgement of inconsistent reporting in the published meta-analysis compared to the registered protocol was also evaluated.
Data synthesis and analysis. Summary statistics were used to describe the proportion of meta-analysis protocol registration, the characteristics of the included studies, and the prevalence and type of inconsistent reporting. Summary statistics were presented with 95% confidence intervals.
Role of the funding source. This study received no funding.
Patient involvement. No patients were involved in the study.
Results
Search results. The electronic search identified 283 hits that were screened for possible inclusion in this study. A total of 164 studies was excluded with reasons after full-text evaluation, and 119 oncology drug meta-analyses were identified as described by the flow-chart in Figure 1. Meta-analysis protocol registration was present in 23 (19%, 95% CI, 13% to 27%) studies (22-44). Excluding Cochrane reviews yielded a registration rate of 7% (CI, 3% to 13%). Study characteristics and possible co-variables
Trial search and selection according to PRISMA. PRISMA 2009 Flow Diagram (Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097. For more information, visit www.prisma-statement.org).
A full description of included studies (n=23) is presented in Table I. The majority of meta-analyses, 18 studies (78%, CI, 58% to 90%), were published in intermediate (≥5, ≤20) impact journals. Further, most of the meta-analyses, 16 (70%, CI, 49% to 84%), were Cochrane reviews. No meta-analysis was funded by a for-profit organization and a female first author was perceived in 11 (48%, CI, 29% to 67%) studies. Risk of bias assessment revealed a mean adherence to PRISMA guidelines of 23 points.
Inconsistent reporting between meta-analysis protocol and publication. Inconsistent reporting between registered meta-analysis protocol and published meta-analysis was found in 16 (70%, CI, 49% to 84%) of 38 meta-analyses. Inconsistent reporting was then further categorized in four domains according to PICO and statistical analysis. Issues with domain one, including patient/population=study design (P), comparator (C) or intervention (I), were reported in 8 (35%, CI, 18% to 55%) studies and were most commonly related to the type of studies included in the meta-analysis (P). Issues related to domain two, the primary outcome (O1), were found in 5 (22%, CI, 10% to 42%) of 23 meta-analyses and were mainly related to inadequate definition of the primary aim. Inconsistent reporting in domain three, which is the secondary aim (O2), was present in 7 (30%, CI, 16% to 51%) studies and most often related to the addition of a secondary aim in the published manuscript compared to the registered protocol. Issues related to domain four, the statistical plan, were found in 12 (52%, CI, 33% to 71%) meta-analyses. A summary of meta-analysis protocol registration and inconsistent reporting in the meta-analyses is reported in Table II.
Characteristics of the included meta-analyses with a published protocol.
Meta-analysis protocol and inconsistent reporting in four domains according to PICO and statistical analysis. Inconsistent reporting between meta-analysis protocol and publication.
Discussion
Prospective registration of meta-analysis protocol might increase awareness of the scientific value of pre-specified analyses and study conclusions based on this. As of today, registration of meta-analysis protocol is not mandatory, but might be encouraged in certain journals (15). Our study shows that there is a low prevalence of prospective meta-analysis protocols among oncology drug meta-analyses. We found that significant alterations are common when comparing the registered protocol with the published manuscript. Inconsistent reporting related to the statistical analysis occurred in more than half of included studies. Furthermore, self-acknowledgement of changes was found in only 50% (8/16) of meta-analyses attributed with inconsistent reporting.
To the best of our knowledge, this is the first study to assess several aspects other than inconsistent reporting of the primary outcome between meta-analysis protocol and published meta-analysis (12). Based on the financial incentives associated with cancer drugs, we decided a priori to limit our study to oncology drug meta-analyses. The cost for these drugs is rapidly increasing (45, 46), though this increase is not related to more clinically-efficient drugs (47). Elevation of oncology drug price has been widely discussed in the medical community (19). We did not restrict our study only to anticancer treatment, but included all pharmacological systemic treatments related to cancer patient therapies. Furthermore, we decided to focus on meta-analyses since they combine statistical results as opposed to systematic reviews. Combining results might potentially lead to bias when adding, omitting or changing subgroups or sensitivity analyses. However, this study has certain limitations. Importantly, most non-Cochrane meta-analyses lack protocol, thus the frequency of inconsistent reporting in these might be underestimated. Limiting this cross-sectional study to oncology might lower generalizability to other therapeutic areas.
Systematic review registration was reported to 16% in February 2014 (1). In this study we report a prevalence of meta-analysis protocol registration of 19%. While these data suggest that prospective protocol registration is still at a low level, the true evolution of meta-analysis protocol registration is unknown due to lack of previously published similar studies with comparable data. The prevalence of inconsistent reporting of primary outcomes among systematic reviews has been reported between 22-47%. However, several studies have been limited to Cochrane reviews published between 2000-2009, before the publication of PRISMA guidelines (48-50), which is supported by a more recent publication restricted to systematic reviews with a Prospero protocol (16). In our study, we found inconsistent reporting of primary outcomes in 22% of included studies. Concerning inconsistent reporting of secondary outcomes, their prevalence is assessed at 17% of Cochrane reviews (49). We reported changes of secondary outcomes in 30% of the meta-analyses. In a recent study it was found that adverse outcomes were missing or partially reported in 86% of the reviews from the included randomized and non-randomized trials (51). PRISMA guidelines recommended that changes to systematic review after the review started should be reported. Further, Cochrane introduced a subheading entitled “Differences between protocol and review” in 2008, in order to clarify any deviations from the protocol (18). However, self-acknowledged changes to the published manuscript compared to the protocol was low.
A previous study has shown low reproducibility rate (11%) of the findings of “landmark” oncology studies (52). This has been further supported by a replication study assessing published oncology and cardiovascular findings (53). Problems with inadequate usage of statistical tests might also contribute to the low replication rates (54). Prospectively published meta-analysis protocols allow the public to compare it with the published meta-analysis and make its own decision on deviations from the protocol. It has previously been suggested that meta-analysis protocol registration could improve the reporting of meta-analysis (55).
Our study shows that the rate of prospective meta-analysis protocols is unsatisfactorily low. Registered protocols were found in only 7% of the meta-analyses when we excluded Cochrane reviews. Given the rapid increase of the use of meta-analyses in scientific evidence synthesis, we believe that active measures have to been taken in order to reduce reporting bias. We suggest that meta-analysis protocol registration should be demanded by the ICMJE as previously has been implemented with RCTs through the introduction of the CONSORT guidelines (56, 57).
We used the PRISMA guidelines adherence as a surrogate marker for quality (18), since PRISMA charts are often required at the stage of submission to many journals. PRISMA adherence should ideally be 100% in meta-analysis. We found a mean adherence to PRISMA of 89%. This fact could easily be improved by an automatic software screening tool, which could help authors and editors to improve reporting according to PRISMA guidelines (58, 59).
Bias related to issues with inconsistent reporting might occur at several levels. Bias might be introduced with patient inclusion, comparator, intervention, outcome or statistical analysis. Avoiding selective reporting at these levels might lead authors to less biased summary estimates, thus reducing misinterpretation of the available scientific evidence. We do acknowledge that a pre-specified outcome or subgroup analysis might not be possible to perform, due to the lack of data in included studies, and therefore not acknowledging this as inconsistent reporting in our study. Also, meta-analyses are retrospective by nature, and therefore an identical overlapping meta-analyses protocol with a published meta-analysis might imply that the study protocol in fact was performed after the study was conducted. In meta-analyses with poorly specified outcomes in the protocol, a more precise definition of the outcome could increase the transparency in order for the reader to assess the presence of inconsistent reporting. Clear definitions of study outcomes could be demanded by public meta-analysis registries, such as PROSPERO, before being made publicly available. Further, we suggest that the PRISMA guidelines should be updated with the addition of a subheading concerning self-acknowledgement of changes made from protocol.
Our results support that it is not sufficient only to encourage the registration of meta-analysis protocol before starting the meta-analysis. On the contrary, this work highlights the need for medical editors and scientific journals to demand meta-analysis protocol registration during the editorial process when reviewing the growing number of meta-analyses, in order to detect not acknowledged or unjustified changes between protocol and publication.
Conclusion
Meta-analysis protocol registration among oncology drug meta-analyses is uncommon. We found a high rate of discordance between meta-analyses protocol registries and published meta-analyses.
- Received June 2, 2017.
- Revision received July 15, 2017.
- Accepted July 19, 2017.
- Copyright© 2017, International Institute of Anticancer Research (Dr. George J. Delinasios), All rights reserved
