Clinical tests have shown...
Everyday we encounter statements that try to influence our decisions and choices by claiming that research has demonstrated that something is useful or effective. Before you believe such claims you need to be sure that the study was not undertaken in such a way that it was likely to produce the result observed regardless of the truth. Conducting a study in a way that tends to lead to a particular conclusion, regardless of the truth, is known as bias. Bias can be defined as ‘the systematic deviation of the results of a study from the truth because of the way it has been conducted, analysed or reported'.
Sometimes you can get a real insight into how bias can creep into research if you think about how you could do a study to generate the result you want. Imagine for a moment that you are the maker of the beauty product ‘EverYoung' and you want to advertise it by citing research suggesting that it makes people look younger; for example, ‘Nine out of every ten woman we asked agreed that ‘EverYoung' makes their skin firmer and younger looking.'
Inventing a claim that is not based on a study should be avoided because this could backfire should it come to light.
Which of the following two designs would you choose if you wanted to maximise the probability of getting the result you want?
A. Ask women in shops who are buying ‘EverYoung' whether they agree that it makes their skin firmer and younger looking?
B. Ask a random sample of women to try ‘EverYoung' and then comment on whether they agree it made their skin firmer and younger looking?
Study A will tend to select women who are already likely to believe that the product works (otherwise they would not be parting with good money to buy it). This design thus increases the chance of a woman being surveyed agreeing with your statement. Such a study could find that nine out of ten women agreed with the statement even when study B shows that nine out of ten women who try the product do not believe it helps.
When critically appraising research, it is important to first look for biases in the study; that is, whether the findings of the study might be due to the way the study was designed and carried out, rather than reflecting the truth. It is also important to remember that no study is perfect and free from bias; it is therefore necessary to systematically check that the researchers have done all they can to minimise bias, and that any biases that might remain are not likely to be so large as to be able to account for the results observed. A study which is sufficiently free from bias is said to have internal validity.