Abstract
Background/Aim: To assess the quantity and quality of systematic reviews of in vitro cancer studies. Materials and Methods: PubMed, MEDLINE, Embase, Web of Knowledge and PROSPERO databases were searched. Articles described as systematic reviews of in vitro studies, focused on or relevant to cancer and published in English were selected and appraised using an adapted version of AMSTAR 2 ‘critical domains’. Results: From 4,021 records, 41 reviews described as systematic and cancer-related were identified. Publication dates indicate increasing frequency of systematic review conduct. Mean number of databases searched was three (range=1-8). Thirty-six reviews (88%) reported search methods, 35 (85%) specified inclusion criteria, 26 (63%) reported study selection methods, and 21 (51%) used reporting guidelines. Only 13 reviews (32%) involved formal quality assessment. Conclusion: Detailed investigation of reviews of cancer-relevant in vitro studies indicates need for further development and use of robust search strategies, appropriate quality assessment tools, and researchers with relevant skills.
- Neoplasms
- systematic reviews
- critical appraisal
- in vitro studies
- in vitro techniques
- cell culture
- quality
- review
Systematic methods for collating research have been used to answer specific questions since James Lind published A Treatise on the Scurvy in 1753 (1). The rise of clinical epidemiology and evidence based medicine have encouraged and necessitated the application of research results into clinical practice, with Archie Cochrane’s seminal text Effectiveness and Efficiency (1972) having made a significant impact (2). To ensure practice based on ‘best evidence’, research was to be identified, selected and assessed using systematic methods that were transparent, objective and reproducible. The first set of systematic reviews focused on pregnancy and childbirth but the scope quickly expanded to include a wide range of health interventions. In most cases, systematic reviews collate, analyse and synthesise (either narratively or statistically, as in meta-analysis) the results of randomised controlled trials (RCTs) (3). The results of a series of RCTs that individually may not have provided conclusive results on whether particular interventions were effective could, when combined, achieve clear answers to such questions.
Alongside the increasing scope of systematic reviews (and meta-analyses), there has been a considerable development in techniques and tools to support their conduct. Thus, reporting tools such as PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) (4) were developed and are now widely used, while training programmes, checklists and criteria exist to support health professionals in reading and appraising research (5). Together with these initiatives, developments have taken place in statistical techniques to enable increasingly sophisticated approaches to combining the results of different trials (6).
While far less extensive than the developments in clinical/human research, similar initiatives have been established in other fields of research. In animal research, the CAMARADES initiative (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies) has facilitated international, collaborative approaches to collating data and a supportive framework for groups involved in such work (7). Reporting guidelines for in vivo studies have also been published (8) in toxicology, and the Evidence Based Toxicology Collaboration is currently undertaking a number of projects including advocating use of systematic methods in conducting chemical risk assessments (9).
It has previously been claimed that adherence to reporting guidelines in systematic reviews of in vitro studies is generally poor (10). Reporting assessed against the PRISMA tool resulted in a mean overall quality score of reported items of 68%. The authors scored the papers on whether or not specific items in the PRISMA checklist were reported but not on precisely what methods were used (i.e., they assessed whether databases and quality criteria were reported but not which criteria were used or which databases were searched). While criteria for assessment of the quality of studies is crucial for unbiased, reliable reviews of the research literature, assessment of relevance of the technique or method employed is also a key element. Increasing numbers of systematic reviews of in vitro studies over the previous 10 years were noted but reporting did not appear to improve over this period.
While these findings are important to the field, only systematic reviews published up to 2016 were included. In addition, a wide range of research areas was covered by the included systematic reviews, with at least 50% relating to dental or orthodontic research. There exists a need for detailed investigation of contemporary use of the systematic review approach in the cancer field due to specific challenges in representing cancer in in vitro models and in conducting human relevant in vitro cancer research. The current investigation addresses this gap. It is also crucial that systematic reviews are conducted effectively and appropriately to the field to reveal unnecessary replication and/or poor research in the field so that optimal use is made of future resources.
This review uses systematic methods to analyse the prevalence, focus, and methods used in publications presented as systematic reviews of in vitro studies in cancer or of particular relevance to the cancer field.
Materials and Methods
The current investigation involved a systematic analysis of current practice in reporting, evidence identification, and quality assessment in systematic reviews of in vitro studies relating to cancer.
Search and screening methods. Searches were carried out for systematic reviews involving in vitro methods in the field of cancer or with relevance to cancer. Databases searched included PubMed, MEDLINE, Embase, Web of Knowledge, and the PROSPERO International Prospective Register of Systematic Reviews. Searches were conducted between April and July 2018 and updated in December 2019. PubMed, MEDLINE and Embase were searched again in April 2021. Search terms are shown in Table I. Search results were downloaded into Excel and Abstrackr. Resulting titles and abstracts were screened for relevant articles with reference to inclusion/exclusion criteria. Screening of all records was conducted by at least two of three reviewers (KP, MB, and GP) working independently, with disagreements resolved through discussion or by consultation with a third reviewer. All study titles were screened for potential relevance to cancer research and potentially relevant articles checked for specific reference to cancer by searching the full-text. The purpose of this additional screen was to identify reviews of generic relevance to in vitro cancer research that were not explicit in title/abstract descriptions or indexing.
Inclusion/exclusion criteria. Articles were included that met all of the following criteria:
Title/abstract/article indicating a systematic review
Included studies using in vitro research methods and/or an article described as a systematic review of in vitro studies
Focus on cancer or relevant to cancer research (and including a discussion on relevance)
Published in English
Articles that were not described by the authors as a systematic review, included other research methods (e.g., review of in vivo plus in vitro research), had no obvious or direct relevance to the cancer field or were published in languages other than English were excluded.
Data extraction and appraisal. Once relevant studies were identified, information was extracted on to a template. This included: area of focus, aim, whether the PRISMA framework was used, other guidelines or quality assessment criteria used, whether these frameworks were adapted or amended for assessment of in vitro studies, databases searched, strategy and search terms reported, inclusion criteria, screening and selection process, data extraction process and method of synthesis. Data was extracted by one author (MB or KP), and checked for accuracy by a second author (KP or GP).
Appraisal of each systematic review was carried out using a set of criteria originally developed for clinical studies. Based on AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2 ‘critical domains’ (11), the following were assessed:
adequacy of the literature search;
justification for excluding individual studies by assessing if inclusion criteria were reported;
risk of bias from individual studies being included in the review;
effect on the interpretation of the results (by checking whether the quality of included studies had been assessed and if the result and impact of this assessment was referred to in the abstract);
We also recorded whether screening, selection and data extraction were appropriately done (including whether at least two researchers had been involved in each stage). Pre-registration of a protocol prior to commencing the review was not assessed, as there is currently no facility to register systematic reviews other than those involving human or animal studies (12). Publication bias was also not possible to assess.
Results
A total of 4,021 records were retrieved by the searches; after removal of duplicates (records in more than one database that related to the same paper), 3,825 remained, which were screened with reference to the inclusion/exclusion criteria. One hundred and eighty-three records were selected as possibly relevant. Two researchers (two of MB, KP and GP) independently assessed the full text of the papers. Of these, 41 were finally included (13-53). Figure 1 shows the screening process. Table II provides a summary of the basic characteristics. The assessment of the main methods and reporting of the reviews is presented in Table III.
Publication dates and journal of publication. The publication dates indicated an increasing number of systematic reviews of in vitro studies in recent years with 21 published since 2019, 12 between 2016 and 2018, 4 in the previous 3-year period and only 4 prior to 2013. Reviews were published in the following types of journals: cancer-specific (10 reviews); pharmacology/toxicology/pharmaceutics related (4 reviews); general journals with broad scientific and medical scope (5 reviews); with the remainder published in a wide range of specialist journals.
Areas of focus. Table II shows the areas of focus for included reviews. Most frequently, the reviews focused on multiple cancers or had generic relevance (16 reviews). Where the type of cancer was specified, breast (5 reviews) (18, 24, 26, 27, 49) and colorectal (4 reviews) (14, 32, 37, 47) were most frequent, followed by head and neck (3 reviews) (20, 41, 46), lung (2 reviews) (15, 53), skin (2 reviews) (45, 51) and thyroid cancers (2 reviews) (21, 36), with single reviews on remaining cancers. Based on the stated aims, a range of aims were described including analysing, summarising, investigating, identifying, categorising, evaluating, verifying and comparing research on the specific topic.
Use of reporting guidelines. Twenty of the reviews claimed to be reported according to PRISMA guidelines and a further nine used the PRISMA flowchart (or similar) to show the process of selecting studies for inclusion. The remaining 12 did not mention PRISMA although one mentioned the Strengthening the Reporting of Observational Studies in Epidemiology guidelines (STROBE) (65). Of the 21 reviews published since 2019, 16 referred to PRISMA while 5 did not.
Databases searched. A range of databases was used to identify relevant studies to include in the systematic reviews. Four reviews did not report databases searched (30, 39, 41, 53). The three databases most frequently utilised were PubMed and/or MEDLINE (n=37, 90%), followed by Embase (n=18, 44%) and Web of Science/Web of Knowledge (n=16, 39%). Other databases reported to have been used in more than one review also included Scopus (13), Science Direct (8), Proquest (4), EBSCO (3), Google Scholar (3), CINAHL (2) and LILACS (Latin American and Caribbean Health Sciences) database (2). Several reported resources are publishers’ repositories rather than true bibliographic databases (that is, they are platforms for accessing the collections of publications of specific publishers rather than the structured and indexed records from a wide range of publications found in bibliographic databases). The Cochrane Library/CENTRAL was used in 12 reviews, even though this database focuses primarily on systematic reviews of human clinical trials. The mean number of databases searched for all studies was three (range=1-8). Only two reviews involved a search of databases in a language other than English (Chinese) (13, 34) and two covered the Latin American and Caribbean literature (20, 46).
Search strategies. Thirty-six reviews (88%) provided some information on how they had searched for relevant studies, while the remaining five provided no information on this. Of the 36 providing some information, nine provided only a list of search terms with no details on how these were combined as a strategy while both search terms and strategy were reported in the remaining 27 reviews. The strategies ranged from a simple approach in which terms representing the two main concepts were combined, to highly sophisticated strategies making full use of indexing terms, Boolean logic and other search techniques. Few strategies incorporated search terms that were specific for in vitro studies. Of those used, the terms in vitro (including in vitro, in vitro mode, in vitro model, in vitro activity) and ‘cell line(s)’ were the most frequently applied. Relevant indexing terms used by the National Library of Medicine to index MEDLINE records, such as In Vitro Techniques, Culture Techniques or more specific terms such as Cell Culture Techniques were not used.
Assessment of methods used to produce review. The majority of reviews (35, 85%) reported the inclusion criteria used in selecting studies, with only six not reporting this information. One review only reported cell lines included (31). In 26 reviews, at least minimal information was provided on the process of screening search results while, in 13 reviews, the numbers of records retrieved by the searches and the reviewers carrying out the screening of results were both reported. In a further 9, only the numbers of records were reported and, in the remaining six, information on the screening and selection process was not presented. Thirteen reviews reported the process of data extraction and appraisal including the personnel involved and the data items extracted. Five reported two of the three items (people, process and data). Six reported only the data items extracted while one just reported cell lines. The remaining 16 reviews provided no details.
Quality assessment criteria. Only 13 reviews (32%) incorporated a formal, structured assessment of the quality of the studies included in the review. Twenty-one reviews made no reference to quality assessment and two stated specifically that quality was not assessed (36, 51). One review reported using quality criteria which were used as exclusion criteria (26) while one or more generic criteria was used to assess studies for inclusion in three reviews (27-29); for example, statements such as “assessed for potential bias based on funding or author affiliations”. Two reviews (35, 38) used criteria from previously published systematic reviews and one review (13) drew on criteria compiled from the results of a previously published scoping review on quality of toxicologically relevant studies (54). In two reviews, a customised risk of bias or quality tool was developed for use (37, 43).
Five reviews (17, 24, 34, 44, 46) indicated that the following tools were used without any reported adaptations by review authors: Cochrane risk of bias (55); Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) (59); ToxRTool (61); STROBE (65); Grading of Recommendations, Assessment, Development and Evaluations (GRADE) (58) respectively. Authors of three reviews (20, 32, 39) adapted or modified validated criteria for use with in vitro studies. These included GRADE (adapted for in vitro) (58); Newcastle-Ottawa quality assessment (adapted) (60), modified Research Triangle Institute (RTI) Item Bank for cross-sectional studies (64). In one case, the criteria were based on a combination of two sets of criteria: the National Toxicology Program’s Office of Health Assessment and Translation (OHAT) Risk of Bias Rating Tool for Human and Animal Studies (56) and criteria from a published paper suggesting standards for in vitro research (57).
Existing quality assurance criteria were adapted by authors with the intention of increasing their relevance to in vitro methods. Adaptation to the GRADE framework was described thus: ‘we have classified in vitro studies using comparable baselines, comparing them to the RCT studies published by Xiao et al. in 2013 using the GRADE method’ (20). The authors noted the lack of standardised available criteria for assessment of in vitro studies. Adaptation of the Newcastle-Ottawa scale included adding an assessment of in vitro data collection methods and if cancer cells were used as the in vitro model system (32).
Methods for synthesis of the results. A narrative or descriptive approach was used in all reviews to present the overall results. Tables summarising the included studies (or selected characteristics of these) supplemented this in 35 reviews (85%). Only six attempted a quantitative synthesis (meta-analysis).
Discussion
The aim of this investigation was to assess the quantity and quality of published reviews of in vitro studies described as systematic, and to investigate the quality assessment methods employed by review authors. We identified a relatively small number of completed systematic reviews relevant to cancer that focused on in vitro methods. This was consistent with expectations based on initial scoping searches.
The aims of the systematic reviews were very varied and, in some cases, it appeared that the main aim was simply to scope the field of research. This is not the main value of the systematic review approach, which has been designed to facilitate an objective, structured assessment of the research to answer a pre-specified question. The techniques that comprise a systematic review, however, such as comprehensive searching for research and clear presentation of the strengths and weaknesses of each study, may well be valuable in helping to collate research, reduce duplication and clearly demonstrate advances (and limitations) in techniques. Nevertheless, for systematic reviews of in vitro research to be effective, searching for relevant research must be comprehensive so that there is no bias in the set of studies included or omission of key studies.
The core AMSTAR 2 criteria include ‘Adequacy of the literature search’ but, while many reviews reported a search strategy, few were optimal or made use of the search facilities available. This suggests that more use could be made of specialist information staff and/or training for researchers in the field. While search strategies (use of AND/OR operators to combine terms, for example) were reported in 36 reviews (88%), most reported generic searches without describing strategies for each database and, therefore, it is not possible to assess if and how changes to the strategies may have affected the results. Given that the average number of databases searched in included reviews was three (range=1-8), this reporting standard was relevant to the majority of cases here considered.
While the majority of researchers carrying out reviews searched either PubMed or MEDLINE, there was limited use of other databases. This includes those with a specific focus on pre-clinical or basic biological research which may be relevant to reviews of studies using in vitro methods. At present, much reliance is placed on one source (the MEDLINE database either in this format or via the PubMed interface). Other databases may improve searches: in the clinical field, this is certainly true (66) but may also be the case for in vitro research. For example, Embase covers nearly 3,000 journals that are not covered by MEDLINE (and has been particularly focused on pharmacology and related literature). Further research may be necessary to quantify the effect that exclusion of this and other databases may have on evidence retrieval for in vitro reviews. Similarly, few review teams made use of specific index terms as search terms even though these have the potential to improve the efficiency of searching. This step is crucial for ensuring the reviews offer a truly systematic overview of available evidence, and minimise related risks (e.g., unnecessary replication, failure to identify key findings relevant to future study designs, selection bias in papers included in meta-analyses).
It is also recommended that systematic reviews justify including and excluding individual studies by reporting inclusion criteria. The majority (35) did report the inclusion criteria. The number of records retrieved and screened were also reported by many of the reviews and this increases transparency of the searching and selection processes, i.e., how effectively the review team identified the relevant studies. Fewer reviews (18) reported how the studies were treated; little information was provided on what data was extracted from each study and how this was done. The lack of relevant data may be due to limitations in reporting in the original studies. One systematic analysis specifically focused on the challenges of conducting systematic reviews of in vitro studies found that insufficient reporting of essential cell culture parameters prevented cross-study comparisons (67).
The most noticeable difference between these reviews and the key indicators of an effective systematic review was in quality assessment of included studies. A previous analysis of systematic reviews of in vitro research in a wide range of areas found that only 29.2% (19/65) reported data on risk of bias (quality) for each study (10). In the current analysis, less than half of the reviews carried out any assessment of quality and less than a third included a formal, structured assessment. The lack of widely accepted quality criteria was also very apparent as a wide range of different tools and techniques were used. In some cases, authors produced their own checklist while in others, criteria were adapted from existing checklists in the clinical field.
Criteria have previously been proposed that specifically focused on the in vitro methods comprising items including clear delineation of cell viability, cell type, culture medium and constituents (67). Other authors have found it necessary to add their own guiding examples to generic criteria, e.g., for the criterion ‘Clear description of research setting’, Rahman et al. added the following examples: ‘Culture set-up, name and type of insert membrane, temperature’ (68). Only a minority of reviews used published criteria for assessment of quality in included studies and 13 different sets of criteria were employed. Of particular note were the three reviews in which authors adapted quality assessment criteria developed for use in evaluation of clinical research. The justification given in these cases was that such criteria were not appropriate to in vitro studies in their original form. This indicates an area for further investigation with regard to the applicability of more general quality criteria within this field. Lastly, even when quality was assessed, there was minimal reference to this when presenting conclusions. This means that it is not possible to assess the effect on the interpretation of the results.
Limitations in reporting may be linked to the comparatively low use of established reporting criteria for systematic review in the included studies, as poor reporting of systematic reviews of in vitro studies has previously been identified (10). While the numbers of such publications had increased over a 10-year period, reporting quality was not significantly improved over the same period. This current analysis found that while there was increasing reference to the PRISMA reporting guidelines over recent years, the potential for significant improvement in reporting was also revealed.
Finally, several reviews were described as systematic even though they exhibited none of the characteristics of such a review. For example, no methods section was present nor were indications of how studies were identified, selected or assessed. It is important that the description systematic review is restricted to those reviews that are conducted according to generally agreed methods.
Recommendations from this review are the following: 1. The description ‘systematic review’ is restricted to those reviews that include methods for identifying, selecting, and assessing included studies; 2. Searches for studies are carried out on more than one database and make use of well-constructed search strategies; 3. Methods for screening, data extraction, and quality assessment are reported (and preferably involve more than one reviewer/researcher); 4. Criteria for the assessment of in vitro research are adopted; 5. Results of the above stages are reported appropriately; 6. Conclusions refer to quality of the included studies.
Further to this, reporting guidelines that are better suited to the in vitro field are required. These could take the form of a PRISMA extension, which has been used widely in other fields to adapt these general guidelines for specific contexts (e.g., PRISMA extension for Ecology and Evolutionary Biology), that includes criteria such as: Describe the type of replication unit (e.g. individuals, broods, study sites) (69).
Limitations in this review relate to the challenges in identifying studies using in vitro methods and in determining relevance to cancer. In order to focus on systematic reviews of in vitro studies, we excluded a large number of reviews that discussed in vitro studies alongside other study types and this resulted in a small number of reviews for this analysis. To extend the range of reviews assessed, we included those that made specific reference to relevance to cancer. However, this required examination of the full text of each article and further potentially relevant reviews may have been omitted. Lack of standards for systematic reviews of in vitro studies also impacted on assessment of methods, therefore, criteria were adapted from a tool widely used for clinical studies. Nevertheless, the results indicate that there is potential value in the systematic review approach to address a range of pertinent questions: this has already led to international efforts to develop appropriate techniques in the field of animal research and development of the approach now needs to receive a similar level of attention in the cancer in vitro field.
This review article indicates that concerns regarding consistency in application of reporting and quality assurance criteria in primary in vitro studies are also relevant to reviews of those studies relating to research in cancer. The present study makes several contributions. Firstly, it is, to our knowledge, the first study to provide a review of current practice and reporting standards in systematic reviews of cancer-related in vitro studies. It provided further support for findings of an initial exploratory search, that reporting standards for evidence retrieval and quality assessment are highly variable and that lack of common quality assessment criteria for assessment across studies is a limitation for reviewers. It also demonstrates that there is an interest in the use of the systematic review method in the field and that efforts to develop and adapt current approaches to the needs of the in vitro field would be of value. Further development of the systematic review approach is now required to optimise its use and value in the in vitro field and overcome the current limitations. This will require development and use of robust search strategies, appropriate quality assessment tools and researchers with the relevant skills.
Acknowledgements
Mike Bracher’s position was funded by a grant from Animal Free Research UK. The funding body had no role in the design of the study or collection, analysis, and interpretation of data or in writing the manuscript.
Footnotes
This article is freely accessible online.
Authors’ Contributions
Karen Pilkington conceived the study and designed the searches. All Authors were involved in the screening and selection process, and data extraction and analysis. Mike Bracher drafted the paper and all authors revised the paper and contributed to the final manuscript.
Conflicts of Interest
The Authors have no conflicts of interest to declare in relation to this study.
- Received September 1, 2021.
- Revision received September 23, 2021.
- Accepted September 28, 2021.
- Copyright © 2021 International Institute of Anticancer Research (Dr. George J. Delinasios), All rights reserved.