Systematic reviews, methods for combining data from several studies, and meta-analysis

PLEASE NOTE:

We are currently in the process of updating this chapter and we appreciate your patience whilst this is being completed.

 

A systematic review draws together the results of several primary research studies. They are used when there is an important clinical question, but many clinical studies, perhaps with conflicting results. A systematic review seeks to provide an overview of the findings of the individual studies, highlighting possible answers, as well as any remaining gaps in knowledge.

The Cochrane Collaboration defines a systematic review as 'a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research and to collect and analyse data from studies that are included in the review.'1

Systematic reviews synthesise the results of multiple primary investigations by using strategies that limit bias and random error. These include a comprehensive search of all potentially relevant articles.2 In the past, literature reviews were not as rigorous, and were often compiled by experts writing with a particular point of view, making dispassionate analysis difficult to achieve. It is thought, for example, that if original studies of the effects of anti-platelet therapies after myocardial infarction had been systematically reviewed, the benefits of therapy would have been apparent as early as the mid-1970s.3
 

The systematic review process

There is an established process recommended to minimise bias when selecting articles for review. Explicit, reproducible search strategies and eligibility criteria are used and every effort should be made to search a variety of sources for relevant articles, including grey and unpublished literature.

Davies and Crombie outline the steps in this process:3

  1. Defining an appropriate clinical question
  2. Searching the literature
  3. Assessing the studies for eligibility, quality and findings
  4. Combining the results to provide a ‘bottom line’
  5. Placing the findings in context
     

Data sources for a systematic review2

Greenhalgh provides a checklist of possible data sources that could be searched to provide articles to include in a systematic review.4 All of these should ideally be used and, if they have not been, the authors of the review should explain why.

  • Medline database
  • Cochrane controlled clinical trials register
  • Other medical and paramedical databases
  • Foreign language literature
  • Grey literature (academic theses, internal reports, non-peer reviewed journals, pharmaceutical industry files)
  • References (and references of eligible references, etc.) listed in primary sources
  • Other unpublished sources known to experts in the field (seek by personal communication)
  • Raw data from published trials (seek by personal communication)
     

Strengths of a systematic review

  • A well conducted systematic review provides a summary of multiple studies that is easily accessible to clinicians, health care providers and policy makers
  • By critically examining primary studies, systematic reviews can improve understanding of inconsistencies among diverse research evidence2
  • The explicit methods applied limit bias in identifying and rejecting studies2
  • Provide a more precise and reliable estimate of effect by combining studies to effectively increase the sample size2,4
  • Results from different studies can be systematically compared to establish generalisability of findings and consistency (homogeneity) of results2,4
  • Clinical and methodological heterogeneity can be identified and new hypotheses generated about specific subgroups2
  • Useful for decision making and evidence-based medicine
  • Helps define limits of what is known and unknown and helps to formulate hypotheses for further investigation
     

Limitations of systematic reviews

  • Knowledge of an average treatment effect may not apply to an individual patient
  • Like any research, a systematic review may be done badly – it is important to ensure the methods used were valid and reliable
  • Inappropriate aggregation of studies that differ in terms of intervention used or patients included can lead to the drowning of important effects3
  • The findings of systematic reviews may not be in harmony with the findings from large-scale clinical trials
     

Meta-analysis

A meta-analysis is a statistical technique used to combine and summarise the results of several independent studies that addressed the same hypothesis or clinical question in the same way.

As well as synthesising the numerical data to provide a single estimate of effect, the meta-analysis should tabulate relevant information on the inclusion criteria, sample size, baseline patient characteristics, withdrawal rate, and results of primary and secondary end points of all the studies included.2

Meta-analyses are commonly used to assess the clinical effectiveness of health interventions from two or more randomised controlled trials. They are viewed as providing a more effective and accurate method of estimating a treatment effect, drawing on data from all the studies included. In a meta-analysis the overall effect of an intervention is calculated using weighted averages of the results from multiple trials. The weighting given to individual studies is based on the inverse variance of the effect size, which itself is largely a function of the sample size. So larger studies tend to result in a smaller variance, and thus contribute more to the final meta-analysis than smaller studies with a larger variance. There are two broad types of meta-analysis models used: fixed effects and random effects. Fixed effects meta-analyses are used when each of the included studies is thought to be clinically and methodologically similar (i.e. they are relatively homogenous and are thus each measuring the same – or fixed – effect). Random-effects meta-analyses are used where there is heterogeneity between included studies, and these are more conservative – giving wider confidence intervals for the final pooled estimate and larger p values. The results of a meta-analysis are plotted on a forest plot. These show the effect estimates (such as an odds ratio or relative risk) from each individual study as a shaded square, where the size of the square is proportional to its weighting, along with its confidence interval. The pooled estimate is given at the bottom as a diamond, where the middle of the diamond represents the pooled effect size and the edges delineate the pooled confidence interval.

An example of a forest plot is shown below.(Reproduced from 5)

 

Insert Forest Plot diagram here:

 

 

 

 

 

Meta-analyses are used increasingly to establish clinical policy. The validity of the meta-analysis depends on both the quality of the original studies and the methods of systematic review used to identify them.

The statistical methods used to assess heterogeneity and publication bias are addressed in the Section 1B chapter “Comparison of survival rates, heterogeneity, funnel plots, the role of Bayes theorem”.
 

Strengths of a meta-analysis

  • A well conducted meta-analysis can provide an objective evaluation of available evidence.
  • Well conducted meta-analyses allow a more objective appraisal of the evidence than traditional narrative reviews, provide a more precise estimate of a treatment effect, and may explain heterogeneity between the results of individual studies2.
     

Limitations of a meta-analysis

  • Poorly conducted meta-analyses may be biased due to the exclusion of relevant studies or inclusion of inadequate studies.
  • May be subject to publication bias - studies with a negative effect may not get published and will therefore be excluded, while studies that that report a large treatment effect may be more likely to be published.
  • Bias may be introduced if all relevant studies are not included.
  • Study heterogeneity may limit the generalisability of the results of a meta-analysis.

 

References

  1. Higgin J, Green S. Cochrane handbook for systematic reviews of interventions. Wiley, 2011.
  2. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997 March 1;126(5):389-91
  3. Davies HTO, Crombie IK. What is a systematic review? 2001. http://vivrolfe.com/ProfDoc/Assets/Davis%20What%20is%20a%20systematic%20review.pdf  - Accessed 8/04/17
  4. Greenhalgh T, How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses). BMJ 1997;315:672-675.
  5. Roberts D, Dalziel SR. Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth. Cochrane Database of Systematic Reviews 2006;3:CD004454

 

 

© Helen Barratt, Maria Kirwan 2009, Saran Shantikumar 2018