A Concise Guide to Observational Studies in Healthcare

Preface

Research studies are required for developing effective public health policies and clinical practice. Observational studies are perhaps the most common type of research, and they are essential for describing the characteristics of a group of people or finding ways to understand, detect, prevent or treat disease, or avert early death.

The purpose of the book is to provide researchers and health professionals with a focussed and simplified account of the main features of observational studies. It is important to first understand the key concepts. Specifics about the calculations involved in analyses should come after and are covered in other textbooks. The book is aimed at those who conduct their own studies or participate in studies coordinated by others, or to help review a published report. No prior knowledge of design, analysis or conduct is required. Examples are based on clinical features of people, biomarkers, lifestyle habits and environmental exposures, and evaluating quality of care.

This book is a companion to the book A Concise Guide to Clinical Trials (Hackshaw A, BMJ Books/Wiley-Blackwell). An overview of the key design and analytical features are provided in Chapters 1–4; then each study type is discussed using published studies (Chapters 5–8), showing how they were conducted and interpreted. Chapter 9 introduces prognostic markers, a topic which is often misunderstood, while Chapter 10 covers systematic reviews and how to deal with inconsistent results. Chapter 11 summarises how to conduct and publish an observational study.

One of the important goals of the book is to show that study features such as the design of questionnaires and interpreting results are common to most study types, so these topics are repeated throughout the chapters. By having many examples, the reader can see how a variety of study designs and outcomes can be interpreted in a similar way, which will help to reinforce key aspects.

The content is based on over 23 years of experience teaching evidence-based medicine to undergraduates, postgraduates, and health professionals; writing over 130 published articles in books and medical journals; and designing, setting up and analysing research studies for a variety of disorders. This background has provided the experience to determine what researchers need to know and how to present the relevant ideas.

I am most grateful to Jan Mackie, whose thorough editing of the book was invaluable. Final thanks to Harald Bauer.

Professor Allan Hackshaw

Deputy Director Cancer Research UK & UCL Cancer Trials Centre

University College London

Chapter 1

Fundamental concepts

This chapter provides a summary background to observational studies, their main purposes, the common types of designs, and some key design features. Further details on design and analysis are illustrated using examples in later chapters, and from other textbooks [1–3].

1.1 Observational studies: purpose

Two distinct study designs are used in medical research: observational and experimental. Experimental studies, commonly called clinical trials, are specifically designed to intervene in some aspect of how the study participants live their life or how they are treated in order to evaluate a health-related outcome. A key feature of a clinical trial is that some or all participants receive an intervention that they would not normally be given. Observational studies, as the term implies, are not intentionally meant to intervene in the way individuals live or behave or how they are treated.¹ Participants are free to choose their lifestyle habits and, with their physician, decide which interventions they receive when considering preventing or treating a disorder. Box 1.1 shows the most common purposes of observational studies.

Box 1.1 Common purposes of observational studies

Examine the opinions of a single group of people on a health-related topic(s)

Describe the health-related characteristics (e.g. demographics, lifestyle habits, genes, biological measurement, or imaging marker) of a single group of people

Estimate the occurrence of a disorder at a given time, or trends over time

Examine features of a disorder (e.g. how it affects patient’s lives, how they are managed/treated, and short- or long-term consequences)

Find associations between the health-related characteristics among a single group of people or across two or more groups

Examine risk factors (including casual factors) for a disorder or early death

Examine prognostic factors (i.e. those that can predict the occurrence of a disorder or death from the disorder)

Evaluate a healthcare intervention for prevention or treatment

Find new scientific information

Plan the use of future resources

Change public health education, policy, or practice

Change clinical practice

Disease prevention, detection, or treatment

1.2 Specifying a clear research question: exposures and outcomes

The research question(s), which can also be referred to as objectives, purpose, aims, or hypotheses, should be clear, easy to read, and written in non-technical language where possible. They are usually developed to address a research issue that has not been examined before, to corroborate or refute previous evidence, or to examine a topic on which prior evidence has had shortcomings or been scientifically flawed.

There is a distinction between objectives and outcome measures (or endpoints). An outcome measure is the specific quantitative measure used to address the objective. For example, a study objective could be ‘to examine the smoking habits of adults’. Possible corresponding endpoints could be either ‘the proportion of all participants who report themselves as smokers’ or ‘the number of cigarettes smoked per day’, but they are quite different endpoints. Box 1.2 shows examples of objectives and outcome measures.

Box 1.2 Examples of objectives and outcome measures (endpoints)

It can be easy to specify the research question or objective for studies that involve simply describing the characteristics of a single group of people (e.g. demographics, or biological or physical measurements). For example:

What proportion of pregnant women give birth at home?

What is the distribution of blood pressure and serum cholesterol measurements among men and women aged over 50?

Are patients satisfied with the quality of care received in a cancer clinic?

Clinical trials often have a single primary objective, occasionally two or three at most, each associated with an endpoint. However, there can be more flexibility on this for observational studies unless they have been designed to change a specific aspect of public health policy. Many observational studies have several objectives, some of which may only arise during the study or at the end, and they can also be exploratory.

Examining the effect of an exposure on an outcome

While some researchers seek only to describe the characteristics of a single group of people (the simplest study type), it is common to look at associations between two factors. Many research studies, both observational studies and clinical trials, are designed to:

Examine the effect of an exposure on an outcome

Box 1.3 gives examples of these. To evaluate risk factors or causes of disease or early death, an outcome measure must be compared between two groups of people:

Exposed group

Unexposed group

Box 1.3 Examples of studies examining the effect of an exposure on an outcome

A&E, accident and emergency department.

Body weight is highlighted to show that a factor can be either an outcome or exposure, depending on the research question:

‘What is the effect of body weight on the risk of developing diabetes?’

‘What is the effect of a new diet on body weight’

*The risk of developing the specified disorder, except body weight which is a continuous measurement so there is no direct concept of risk.

An exposure is often thought to be a factor that can be avoided or removed from our lives, such as a lifestyle habit or something encountered at work or in the environment, but it can be any of the following:

Physical or clinical characteristic

Gene or genetic mutation

Biomarker (measured in blood, saliva, or tissue)

Imaging marker

Intervention for prevention or treatment

Also, a factor can be either an exposure or an outcome, depending on the research question (e.g. body weight in Box 1.3). Considering a research study in the context of examining the relationship between exposures and outcomes greatly helps to understand the design and analysis.

Make everything else the same: natural variation, confounding, and bias

An important consideration for all observational research studies is variability (natural variation). For example, smoking is a cause of lung cancer, but why do some people who have smoked 40 cigarettes a day for most of their adult lives not develop lung cancer, while others who have never smoked do? The answer is that people vary. They have different body characteristics (e.g. weight and blood pressure), different genetic make-up, and different lifestyles (e.g. diet, and exercise). People react to the same exposure in different ways.

When an association (risk or causal factor)² is evaluated, it is essential to consider if the observed responses are consistent with natural variation or whether there really is an effect. Allowance must be made for variability in order to judge how much of the association seen at the end of a study is due to natural variation (i.e. chance) and how much is due to the effect of the risk factor of interest. The more variability there is, the harder it is to detect an association. Highly controlled studies (such as laboratory experiments or randomised clinical trials) have relatively less variation because the researchers have control over how the study subjects (biological samples, animals, or human participants) are selected, managed, and assessed.

The best way to evaluate the effect of an exposure on an outcome is to ‘make everything the same’, in relation to the characteristics of the two (or more) groups being compared except the factor of interest. For example, to examine whether smoking is a cause of lung cancer, the risk of lung cancer between never-smokers and current smokers must be compared; to evaluate statin therapy for treating people with ischaemic heart disease, survival times between patients who did and did not receive statins are compared. Ideally, the exposed and unexposed groups should be identical in terms of demographics, physical and biological characteristics, and lifestyle habits, so that the only difference between the groups is that one is exposed to the factor of interest (smokes or receives statins) and the other is not exposed. [In reality, the two groups can never be identical; there will always be some random (chance) differences between them due to natural variability.] Consequently, if a clear difference is seen in the outcome measure (lung cancer risk or survival time), it should only be due to the exposure status, and not any other factor. This is a fundamental concept in medical research, and one that allows causal inferences to be made more reliably. An example is shown in Box 1.4.

Box 1.4 Illustration of how differences between exposed and unexposed groups influence the effect of an exposure on an outcome measure

Interest is only in examining the effect of smoking on the risk of a heart attack. The risk is twice as high among smokers than never-smokers, so we could conclude that smoking is associated with heart disease. But it is not possible to distinguish whether this difference (effect) could be due to:

The difference in smoking status

The difference in diet

A combination of the two

In a randomised clinical trial, the process of randomisation aims to ‘make everything the same’, except the intervention given. The researcher randomly allocates the interventions (exposures) leading to two similar groups. Any differences in the outcome measure should only be due to the intervention, which is why clinical trials (and systematic reviews of them) usually provide the best level of evidence in medical research, and a causal relationship can often be determined. Published reports of all randomised studies contain a table confirming that baseline characteristics are similar between the trial groups.

In observational studies, however, the exposure cannot be randomly allocated by the research team. The researchers can only observe, not intervene, and it is likely for several differences to exist between the groups to be compared. The more differences there are, the more difficult it will be to conclude a causal link. The two main sources of these differences are confounding and bias. Confounding and bias might still be present to some small extent in a randomised clinical trial, but the purpose of randomisation is to minimise their effect.

Confounding and bias can each distort the results and therefore the conclusions (Box 1.5).

Box 1.5 Confounding and bias

Confounding represents the natural relationships between physical and biochemical characteristics, genetic make-up, and lifestyle and habits, which may affect how an individual responds to an exposure.

It cannot be removed from a research study, but known confounding factors can be allowed for in a statistical analysis if they have been measured, or at the design stage (matched case–control studies).

Bias is usually a design feature of a study that affects how participants are selected, treated, managed, or assessed.

It often arises through the actions of the study participants and/or the research team.

The effect of bias could be minimised or prevented by careful study design and conduct, but human nature makes this difficult.

It is difficult, sometimes impossible, to allow for bias in a statistical analysis because it cannot be measured reliably.

The confounding and bias factors themselves are relatively unimportant. What matters more is whether they greatly influence the study results:

Make an effect appear spuriously, when in reality there is no association

Overestimate the magnitude of an effect

Underestimate the magnitude of an effect

Hide a real effect

Some researchers consider confounding as a type of bias, because both have similar effects on the results. However, a key difference is that it is usually possible to measure confounding factors and therefore allow for them in the statistical analysis, but a factor associated with bias is often difficult or impossible to measure, and therefore it cannot be adjusted for in the same way as confounding. Confounding and bias could work together, or in opposite directions. It may not be possible to separate their effects reliably.

Researchers try to remove or minimise the effect of bias at the design stage or when conducting the study. The effect of some confounding factors can also be minimised at this stage (matched case–control studies, see Chapter 6, page 114).

Confounding

A confounding factor is often another type of exposure, and to affect the study results, it must be associated with both the exposure and outcome of interest (Figure 1.1). The factor could be more common in either the exposed or unexposed groups.Figure 1.2 shows a hypothetical example of how confounding can distort the results of a study. The primary interest is in whether smoking is associated with death from liver cirrhosis. In Figure 1.2a, if the death rates are simply compared between smokers and non-smokers, they appear to be higher among smokers (15 vs. 9 per 1000). It could be concluded that smokers have a higher risk, and this could be used as supporting evidence that smoking is a risk factor for cirrhosis. However, from Figure 1.2a, it is clear that smokers are more likely to be alcohol drinkers (66 vs. 34%), and it is already known that alcohol increases the risk of liver cirrhosis. Because the exposed (current smokers) and unexposed (never-smokers) groups have different alcohol consumption habits, they are not ‘the same’, and the difference in death rates could be due to smoking status, the difference in alcohol consumption, or a combination of the two.

Figure 1.1 The effect of an exposure on an outcome, with a third factor, the confounder.

Figure 1.2 Hypothetical example of how a confounder can distort the results when examining the effect of an exposure on an outcome and how it can be allowed for.

Because drinking status has been measured for all participants, it is perhaps intuitive that to remove its confounding effect, the association between smoking and cirrhosis deaths can be examined separately for drinkers and non-drinkers. This is shown in Figure 1.2b. By comparing the death rates between smokers and never-smokers only among non-drinkers, alcohol cannot have any confounding effect, because the two exposure groups have been ‘made the same’ in terms of alcohol consumption. The death rates are found to be identical, and the conclusion is reached that smoking is not associated with cirrhosis in this group. A similar finding is made among drinkers only, where, although the death rates are higher than those in non-drinkers (as expected), they are identical between smokers and never-smokers. The effect of confounding has been to create an association when really there was none. Analysing the data in this way (called a stratified analysis) is the simplest way to allow or adjust for a confounding factor. In practice, there are more efficient and sophisticated statistical methods to adjust for confounders (regression analyses; Chapter 4). If there is uncertainty over the relationship between the confounder and either the exposure or the outcome, it is worth taking it into account as a potential confounding factor.

A factor should not be considered a confounder if it lies on the same biological (causal) pathway between the exposure and an outcome [2]. For example, if looking at the effect of a high-fat diet on the risk of heart disease, high cholesterol is a consequence of the diet, and it can also lead to heart disease. Therefore, cholesterol would not be a confounder because it must, by definition, be causally associated with both exposure and outcome, and its effect should not (or cannot) be removed.

Bias

A bias occurs where the actions of participants or researchers produce a value of the study outcome measure that is systematically under- or over-reported in one group compared with another (i.e. it works in one direction). Figure 1.3 is a simple illustration. In the middle figure, only people who smoke lie about (misreport) their smoking status, and the effect of this is to bias the study result (in this case the prevalence of smoking). If, however, the number of non-smokers who lie about their smoking status is similar to that in smokers, even though there are lots of people who misreport their habits, the study result itself is not biased. But non-smokers rarely report themselves as smokers. It is important to focus on the bias in the result rather than the factor creating the bias.

Figure 1.3 Illustration of bias, using an example in which the aim is to estimate the proportion of people who smoke, based on self-reported measures.

Unlike confounding (where in the example above it was simple to obtain the alcohol status of the study subjects and, therefore, allow for it when examining the effect of smoking on liver cirrhosis), it is difficult to measure bias, because it would require the participants to admit whether or not they are lying, which, of course, would not happen. Researchers attempt to minimise bias at the design stage. In the example in Figure 1.3, estimating smoking prevalence could be assessed using biochemical confirmation of smoking status using nicotine or cotinine in the saliva of the participants, where high levels are indicative of being a smoker. However, even this is not perfect, because a light smoker could have low concentrations that overlap with non-smokers, and non-smokers heavily exposed to environmental tobacco smoke could have levels that overlap with some smokers. Many other biases are similarly difficult or impossible to measure.

There are several types of biases (Box 1.6), and they can arise from something either the researcher or study participant has done [4]. To determine whether bias exists, the following questions should be considered:

Was there anything unusual in how the participants were selected for the study?

Were some participants managed, assessed, or examined differently from others?

Is it plausible that certain participants could misreport, or under- or over-report, their responses to a questionnaire and hence distort the results?

Box 1.6 Common types of potential biases

Selection bias: The participants chosen for the study are not representative of the population of interest. An example is the healthy worker effect, where disease rates are lower in the study group than in the general population.

Response/responder (or non-response) bias: People who agree to take part in a study have different characteristics from those who do not, and this distorts the results when making conclusions about the whole population.

Recall bias: People with disease are often better at remembering past details (including past exposures) about their life than people without disease.

Withdrawal bias: Participants who decide to discontinue with a study have different characteristics from those who continue, and this can distort the results because follow-up data (e.g. outcome measures) will be missing for some participants.

Assessment bias: Different groups of participants are managed using different assessments or at different times according to their characteristics, exposure status, or health outcome.

Measurement bias: Measuring exposures is performed differently for people with different health outcomes.

Observer or interviewer bias: If an interviewer is aware of the participant’s health (or other) status, this may influence the questions asked, or how they are asked, which consequently affects the response.

1.3 Types of observational studies

Studies are conducted among two different types of participants:

Population: Participants are approached from the general population. They may or may not have the disorder of interest. Researchers sometimes use the word healthy individual or control when describing some study participants, but this usually only means that the participants do not have the disorder of interest. They may have other disorders. Better terms could be affected and unaffected.

Patients: Only people who have already been diagnosed with a specific disorder are recruited to a study.

The study objectives are usually quite different for each of these two types. For studies of the population, interest is often in risk factors that lead to the occurrence of a disorder, but for patient studies interest could be in how an existing disorder develops including the management of it. Both can be used when describing characteristics of a group(s).

A variety of study designs can be used to examine associations, risk factors, and interventions.

A cross-sectional survey: face-to-face interviews with participants or collecting self-completed participant surveys.

A (retrospective) case–control study: people with and without a disorder of interest are identified and asked about their past habits, possibly also obtaining data from their medical records.

A prospective cohort study: people without the disorder of interest are identified, baseline characteristics are measured, and participants are followed up for a period of time (several months or years) during which specific data is collected regularly.

A retrospective cohort study is essentially a prospective cohort study that has already been conducted.

Longitudinal study: a prospective cohort study in which exposures and often outcomes are measured repeatedly during follow-up.

Studies based on routinely collected data: these could come from regional or national registries or databases (e.g. cancer or death notification systems) and contain a few key factors on each individual (e.g. age, sex, city of residence), as well as the disease status. Many such databases have adequate or good data quality processes in place, but a common limitation is that potential confounding factors are unavailable.

These terms for types of study designs should not be regarded as fixed. There may be occasions when one type could be used synonymously with another, a design is nested within another, or there are variations on a specific design. For example:

There are nested case–control studies, which involve selecting and only analysing cases and controls (individuals with and without a disorder of interest) from a cohort study.

Cases and controls could provide information about their current or past characteristics, but they might also be followed up for a certain length of time for other outcome measures, so these data are collected prospectively (similar to a prospective study).

Researchers simply need to be clear where the participants for a particular study have come from and how data are collected from or about them.

Large-scale studies could be preceded by a pilot study to examine the likely recruitment rate and how data are to be collected. Problems that arise can be dealt with before launching the full study. Pilot studies should have few participants and have a short duration.

An ecological study is one in which the unit of interest is a group of people, not an individual. For example, the relationship between income and risk of heart disease could be examined by using the average income from 20 countries and the corresponding rates of heart disease in each country, and then examine the correlation. However, such studies can often only provide a crude measure of association because potential unmeasured confounding factors could explain the effect (ecological fallacy); confounding is best dealt with at an individual level. The findings in ecological studies can therefore be inconsistent with those based on individuals.

A common but special type of observational study is a qualitative research study. This is usually based on relatively few participants (often < 50). Although a structured questionnaire could be used to ascertain some information about the participants, the main source of data is by face-to-face or telephone interviews, with largely open-ended questions to find out about their characteristics, lifestyle habits, opinions, or experiences (other study types almost always use structured questions). The interviews are usually recorded, allowing researchers to play back the recording later and code the responses in a way that can be interpreted and summarised. The findings are often descriptive, and the data produced cannot be readily quantified, and therefore not analysed using statistical methods covered in Chapter 4. For these reasons, they are not discussed in detail in this book, but they are well described elsewhere [5, 6]. A qualitative study can be used:

As a precursor to the study designs mentioned previously in order to better design the questionnaires for a larger and more structured study (i.e. how to measure factors, exposures and outcomes), or to obtain an initial understanding of the research question

To attempt clarification of some of the findings of studies, or a deeper understanding of them, especially if they are unexpected

Other types of observational studies include case or case series reports, which are based on unusual or sporadic occurrences found by a health professional, often during clinical practice [7]. They may provide interesting findings, but no firm conclusions should be made from these. They usually lead to better designed studies.

1.4 Strengths and limitations of the different types of study designs

There are ways of assessing the reliability of evidence from a particular study, such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE) [8]. The most reliable type of study is, in order (generally, but there are exceptions):

Systematic review of randomised clinical trials

Individual randomised clinical trial

Systematic review of observational studies

Individual prospective cohort study

Individual case–control study

Individual cross-sectional study

Hospital audits case reviews

When examining risk factors and causality, there are many situations where a randomised trial cannot be conducted. For example, the best way to determine that smoking is a cause of cancer is to randomly allocate never-smokers to either take up smoking regularly for several years or remain non-smokers, then follow them up and compare the proportions that develop cancer between the two groups. This would clearly be unethical, so the only way