Epidemiology: Systematic Reviews and Meta-Analysis
A systematic review draws together the results of several primary research studies. They are used when there is an important clinical question, but many clinical trials, perhaps with conflicting results. A systematic review seeks to provide an overview of the findings of the individual trials, highlighting possible answers, as well as any remaining gaps in knowledge.
The Cochrane Collaboration defines a systematic review as 'a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research and to collect and analyse data from studies that are included in the review.'1
Systematic reviews synthesize the results of multiple primary investigations by using strategies that limit bias and random error. These include a comprehensive search of all potentially relevant articles.2 In the past literature reviews were not as rigorous, and were often compiled by experts writing with a particular point of view, making dispassionate analysis difficult to achieve. It is thought, for example, that if original studies of the effects of anti-platelet therapies after myocardial infarction had been systematically reviewed the benefits of therapy would have been apparent as early as the mid-1970s.3
The process of systematic review
There is an established process recommended to minimise bias when selecting articles for review. Explicit, reproducible criteria are used and every effort should be made to search a variety of sources for relevant articles, including grey and unpublished literature.
Davies and Crombie outline the steps in this process:3
- Defining an appropriate clinical question
- Searching the literature
- Assessing the studies for eligibility, quality and findings
- Combining the results to provide a ‘bottom line’
- Placing the findings in context
Data sources for a systematic review2
Greenhalgh provides a checklist of possible data sources that could be searched to provide articles to include in a systematic review. All of these should ideally be used and, if they have not been, the authors of the review should explain why.
- Medline database
- Cochrane controlled clinical trials register
- Other medical and paramedical databases
- Foreign language literature
- Grey literature’ (academic theses, internal reports, non-peer reviewed journals, pharmaceutical industry files)
- References (and references of references, etc) listed in primary sources
- Other unpublished sources known to experts in the field (seek by personal communication)
- Raw data from published trials (seek by personal communication)
Strengths of a systematic review
- A well conducted systematic review provides a summary of multiple studies that is easily accessible to clinicians, health care providers and policy makers.
- By critically examining primary studies, systematic reviews can improve understanding of inconsistencies among diverse research evidence2
- The explicit methods applied limit bias in identifying and rejecting studies2
- Provide a more precise estimate of effect because of methods used2,4
- Results from different studies can be systematically compared to establish generalisability of findings and consistency (lack of heterogeneity) of results2,4
- Reasons for heterogeneity can be identified and new hypotheses generated about specific subgroups2
- Useful for decision making
- Helps define limits of what is known and unknown and helps to formulate hypotheses for further investigation
- Knowledge of an average treatment effect may not apply to an individual patient
- Like any research, a systematic review may be done badly – it is important to ensure the methods used were valid and reliable
- Inappropriate aggregation of studies that differ in terms of intervention used or patients included can lead to the drowning of important effects3
- The findings of systematic reviews may not be in harmony with the findings from large-scale clinical trials
- A well conducted meta-analysis can provide an objective evaluation of available evidence.
- Well conducted meta-analyses allow a more objective appraisal of the evidence than traditional narrative reviews, provides a more precise estimate of a treatment effect, and may explain heterogeneity between the results of individual studies2.
- Poorly conducted meta-analyses may be biased due to the exclusion of relevant studies or inclusion of inadequate studies.
- May be subject to publication bias - studies with a negative effect may not get published and therefore be excluded, while studies that that report a large treatment effect may be more likely to be published.
- Bias may be introduced if all relevant studies are not included.
- Heterogeneity is a major problem in meta-analysis, are all studies comparable.
- January 2005 – Paper I, Question 1
- January 2008 – Paper I, Question 2
Limitations of systematic reviews
A meta-analysis is a statistical technique used to combine and summarise the results of several independent studies that addressed the same hypothesis or clinical question in the same way.
As well as synthesising the numerical data, to provide a single estimate of effect, the meta-analysis should tabulate relevant information on the inclusion criteria, sample size, baseline patient characteristics, withdrawal rate, and results of primary and secondary end points of all the studies included.2
Meta-analyses are commonly used to assess the clinical effectiveness of health interventions from two or more randomised controlled trials. They are viewed as providing a more effective and accurate method of estimating a treatment effect, drawing on data from all the studies included. In a meta-analysis the overall effect of an intervention is calculated using weighted averages of the results from multiple trials. Meta-analyses are used increasingly to establish clinical policy.
The validity of the meta-analysis depends on both the quality of the original studies and the methods of systematic review used to identify them. The statistical methods used in meta-analysis, including forest plots and assessments of heterogeneity, are addressed further in the statistics section of the Part A syllabus.
Strengths of a meta-analysis
Limitations of a meta-analysis
Systematic reviews and meta-analysis are addressed by the following Part A questions:
- The Cochrane Collaboration, Cochrane handbook for systematic reviews of interventions.
- Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997 March 1;126(5):389-91
- Davies HTO, Crombie IK. What is a systematic review? 2001. http://www.medicine.ox.ac.uk - Accessed 21/01/09
- Greenhalgh T, How to read a paper: Papers that summarise other papers (systematic reviews and meta-analyses). BMJ 1997;315:672-675.
© Helen Barratt, Maria Kirwan 2009