Health Care Evaluation: Study design for assessing effectiveness, efficiency and acceptability of services including measures of structure, process, service quality, and outcome of health care

Health care evaluation is the critical assessment, through rigorous processes, of an aspect of healthcare to assess whether it fulfils its objectives. Aspects of healthcare which can be assessed include:

• Effectiveness – the benefits of healthcare measured by improvements in health
• Efficiency – relates the cost of healthcare to the outputs or benefits obtained
• Acceptability – the social, psychological and ethical acceptability regarding the way people are treated in relation to healthcare
• Equity - the fair distribution of healthcare amongst individuals or groups

It is usually impossible to maximise all these factors.

Healthcare evaluation can be carried out during a healthcare intervention, so that findings of the evaluation inform the ongoing programme (known as formative evaluation) or can be carried out at the end of a programme (known as summative evaluation).

Evaluation can be undertaken prospectively or retrospectively. Evaluating on a prospective basis has the advantage of ensuring that data collection can be adequately planned and hence be specific to the question posed (as opposed to retrospective data dredging for proxy indicators) as well as being more likely to be complete. Prospective evaluation processes can be built in as an intrinsic part of a service or project (usually ensuring that systems are designed to support the ongoing process of review).

There are several eponymous frameworks for undertaking healthcare evaluation. These are set out in detail in the Healthcare Evaluation frameworks section of this website and different frameworks are best used for evaluating differing aspects of healthcare as set out above. The steps involved in designing an evaluation are described below.

Steps in designing an evaluation

Firstly it is important to decide which dimensions are to be evaluated? – inputs, process, outputs, outcomes, efficiency etc. Objectives for the evaluation itself should be set (remember SMART) -

S- specific – effectiveness/efficiency/acceptability/equity
M- measurable
A- achievable – are objectives achievable
R– realistic (can objectives realistically be achieved within available resources?)
T– time- when do you want to achieve objectives by?

Having identified what the evaluation is attempting to achieve, the following 3 steps should be considered:

1. What study design should be used?

When considering study design, several factors must be taken into account:

• Will the approach be quantitative / qualitative / mixed?
• Level of data collection and analysis - will it be possible to collect what is needed?
• The design should seek to eliminate bias and confounding
• The strengths and weaknesses of each approach should be weighed up when finalising a design.

Study designs include:

a) Randomised methods

• through the random allocation of an intervention confounders are equally distributed. This is usually carried out prospectively. Randomised controlled trials can be expensive to undertake rigorously and are not always practical in the service setting.

b) Non randomised methods

• Cohort studies - involve the non-random allocation of an intervention, can be retrospective or prospective, but adjustment must be made for confounders
• Case-control studies – investigate rare outcomes, participants are defined on the basis of outcome rather than healthcare. There is a need to match controls however the control group selection itself is a major form of bias.

c) Ecological studies

• cheap and quick, cruder and less sensitive than individual level studies, can be useful for studying the impact of health policy

d) Descriptive studies

• used to generate hypotheses, help understand complexities of a situation and gain insight into processes eg case series.

e) Health technology assessment

• examines what technology can best deliver benefits to a particular patient or population group. It assesses the cost-effectiveness of treatments against current or next best treatments. See economic evaluation section of this website for more details.

2. What measures should be used?

The choice of measure will depend on the study design or indeed evaluation framework used as well as the objectives of the evaluation. For example, the Donabedian approach considers a programme or intervention in terms of inputs, process, outputs and outcomes.

• Inputs - (also known as structure) describes what has gone into an intervention to make it happen eg people, time, money
• Process - describes how it has happened eg strategy development, a patient pathway
• Outputs - describe what the intervention or programme has produced eg throughput of patients
• Outcomes - describes the actual benefits or disbenefits of that intervention or programme.

The table below gives some further examples of measures that can be used for each aspect of the evaluation. Such an evaluation could measure process against outcomes, inputs versus outputs or any combination.

3. How and when to collect data?

The choice of qualitative versus quantitative data collection will influence the timing of such collection, as will the choice of the evaluation being carried out prospectively or retrospectively. The amount of data that needs to be collected will also impact on timing, and sample-size calculations at the beginning of the evaluation will be an important part of planning.

For qualitative studies, the sample must be big enough that enlargement is unlikely to yield additional insights eg undertaking another interview with a member of staff is unlikely to identify any new themes. Most qualitative approaches, in real life, would ensure that all relevant staff groups were sampled.

For quantitative studies the following must be considered (using statistical software packages such as Stata):

• the size of the treatment effect that would be of clinical/social/public health significance
• the required power of the study
• acceptable level of statistical significance
• variability between individuals in the outcome measure of interest

Trends in health services evaluation

There is an increasing trend in healthcare evaluation to ensure that patients’ views and opinions are taken into account. This means that evaluation is carried out increasingly from the patients’ rather than healthcare providers’ perspective. Assessment of service user opinion can include results from surveys, external assessment (such as National Patient Surveys) as well as outcomes reported by patients themselves (patient reported outcome measures) which from April 2009 are a mandatory part of commissioners’ service contracts with provider organisations.

© Rosalind Blackwood 2009