Analysis of Variance - ANOVA: Eleisa Heron Eleisa Heron
Analysis of Variance - ANOVA: Eleisa Heron Eleisa Heron
Eleisa Heron
30/09/09
Introduction
• Analysis of variance (ANOVA) is a method for testing the hypothesis that there is no
difference between two or more population means (usually at least three)
• Often used for testing the hypothesis that there is no difference between a number of
treatments
30/09/09 ANOVA 2
Independent Two Sample t-test
• Recall the independent two sample t-test which is used to test the null hypothesis that
the population means of two groups are the same
• Let and be the sample means of the two groups, then the test statistic for the
independent t-test is given by:
with and
30/09/09 ANOVA 3
Why Not Use t-test Repeatedly?
• The t-test, which is based on the standard error of the difference between two means,
can only be used to test differences between two means
• With more et
Why Look at Variance When Interested in Means?
• Three groups tightly spread about their respective means, the variability within each
group is relatively small
• Easy to see that there is a difference between the means of the three groups
30/09/09 ANOVA 5
Why Look at Variance When Interested in Means?
• Three groups have the same means as in previous figure but the variability within each
group is much larger
• Not so easy to see that there is a difference between the means of the three groups
30/09/09 ANOVA 6
Why Look at Variance When Interested in Means?
• To distinguish between the groups, the variability between (or among) the groups must
be greater than the variability of, or within, the groups
• If the within-groups variability is large compared with the between-groups variability, any
difference between the groups is difficult to detect
• To determine whether or not the group means are significantly different, the variability
between groups and the variability within groups are compared
30/09/09 ANOVA 7
One-Way ANOVA and Assumptions
• One-Way ANOVA
- When there is only one qualitative variable which denotes the groups and only one
measurement variable (quantitative), a one-way ANOVA is carried out
- For a one-way ANOVA the observations are divided into mutually exclusive
categories, giving the one-way classification
• ASSUMPTIONS
- Each of the populations is Normally distributed with the same variance
(homogeneity of variance)
- The observations are sampled independently, the groups under consideration are
independent
30/09/09 ANOVA 8
Simulated Data Example
• 46 observations
• 15 AA observations
mean IQ for AA = 75.68
• 18 AG observations
mean IQ for AG = 69.8
• 13 GG observations
mean IQ for GG = 85.4
30/09/09 ANOVA 9
Introduction of Notation
• For the simulated verbal IQ and genotype data, , representing the three possible
genotypes at the particular locus of interest. Each person in this data set, as well as
having a genotype, also has a verbal IQ score
• Want to examine if the mean verbal IQ score is the same across the 3 genotype groups
- Null hypothesis is that the mean verbal IQ is the same in the three genotype groups
30/09/09 ANOVA 10
Within-Groups Variance
• Remember assumption that the population variances of the three groups is the same
• Under this assumption, the three variances of the three groups all estimate this common
value
- True population variance =
- Within-groups variance = within-groups mean square = error mean square =
• For groups with equal sample size this is given by the average of the variances of the
groups
• For unequal sample sizes, the variances are weighted by their degrees of freedom
30/09/09 ANOVA 11
Within-Groups Variance
• Since the population variances are assumed to be equal, the estimate of the population
variance, derived from the separate within-group estimates, is valid whether or not the
null hypothesis is true
30/09/09 ANOVA 12
Between-Groups Variance
• If the null hypothesis is true, the three groups can be considered as random samples
from the same population
(assumed equal variances, because the null hypothesis is true, then the population
means are equal)
• The three means are three observations from the same sampling distribution of the
mean
30/09/09 ANOVA 13
Between-Groups Variance
• For equal sample sizes, the between-groups variance is then given by:
30/09/09 ANOVA 14
Testing the Null Hypothesis, F-test
• If the null hypothesis is true then the between-groups variance and the within-groups
variance are both estimates of the population variance
• If the null hypothesis is not true, the population means are not all equal, then will be
greater then the population variance, it will be increased by the treatment
differences
• To test the null hypothesis we compare the ratio of and to 1 using an F-test
30/09/09 ANOVA 15
Testing the Null Hypothesis, F-test
30/09/09 ANOVA 16
F Distribution and F-test
• The F distribution is the continuous distribution of the ratio of two estimates of variance
• The F distribution has two parameters: degrees of freedom numerator (top) and degrees
of freedom denominator (bottom)
• The F-test is used to test the hypothesis that two variances are equal
• The validity of the F-test is based on the requirement that the populations from which the
variances were taken are Normal
• In the ANOVA, a one-sided F-test is used, why not two-sided in this case?
30/09/09 ANOVA 17
F Distribution
30/09/09 ANOVA 18
ANOVA Table
Slightly more
I-1 = 3-1 = 2, since complicated as the
3 genotype groups, sample sizes are
AA, AG, GG not all equal (15-1
+ 18-1+13 -1) = 43
Sum Sq / Df =
Mean Sq
30/09/09 ANOVA 19
Assumption Checking
- Needed since the denominator of the F-ratio is the within-group mean square,
which is the average of the group variances
- Rule of thumb: the ratio of the largest to the smallest group variance should be 3:1
or less, but be careful, the more unequal the sample sizes the smaller the
differences in variances which are acceptable
30/09/09 ANOVA 20
Assumption Checking
- Examine boxplots of the data by group, will highlight visually if there is a large
difference in variability between the groups
- Plot residuals versus fitted values and examine scatter around zero,
residuals = observations – group mean, group mean = fitted value
• Bartlett’s test: null hypothesis is that the variances in each group are the same
- For the simulated genotype, IQ data:
30/09/09 ANOVA 21
Assumption Checking
Normality Assumption
- The dependent variable (measurement, quantitative variable) should be Normally
distributed in each category of the independent variable (qualitative variable)
- Quantile-Quantile plots (QQ plots) of the residuals, which should give a 45-degree
line on a plot of observed versus expected values,
- Usual tests for Normality may not be adequate with small sample sizes, insufficient
power to detect deviations from Normality
30/09/09 ANOVA 22
Residuals for Simulated Genotype, IQ Data
30/09/09 ANOVA 23
Residuals vs Fitted for Simulated Genotype, IQ Data
30/09/09 ANOVA 24
Assumption Checking
• If data are not Normal or don’t have equal variances, can consider
- non-parametric alternatives
30/09/09 ANOVA 25
R Code for ANOVA
• In R, due to the way an ANOVA is carried out the data needs to be in a specific format
• If the factor is numeric, R will read the analysis as if it were carrying out a regression, and
not a comparison of means, (coerce the numeric into factor with as.factor())
30/09/09 ANOVA 26
R Code for ANOVA
# “anova” function computes the ANOVA table for the fitted model object (in this case g)
> anova(g)
Response: Genotype_IQ[, 2]
Df Sum Sq Mean Sq F value Pr(>F)
Genotype_IQ[, 1] 2 1372.6 686.3 3.6765 0.03358 *
Residuals 43 8026.8 186.7
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
30/09/09 ANOVA 27
Summary So Far
30/09/09 ANOVA 28
What to do with a Significant ANOVA Result (F-test)
• If the ANOVA is significant and the null hypothesis is rejected, the only valid inference
that can be made is that at least one population mean is different from at least one other
population mean
• The ANOVA does not reveal which population means differ from which others
- Only think about investigating differences between individual groups when the overall
comparison of groups (ANOVA) is significant, or that you had intended particular
comparisons at the outset
- There are many approaches for after the ANOVA, here are some
30/09/09 ANOVA 29
What to do with a Significant ANOVA Result (F-test)
• Modified t-test:
- To compare groups after the ANOVA, use a modified t-test
- Modified t-test is based on the pooled estimate of variance from all the groups
(within groups, residual variance in the ANOVA table), not just the pair being
considered
- Arrange the treatment means in order of magnitude and the difference between
any pair of means can be compared with the least significant difference
30/09/09 ANOVA 30
What to do with a Significant ANOVA Result (F-test)
- The test corrects for the multiple comparisons that are made
• Linear Trend
- When the groups are ordered we don’t want to compare each pair of groups, but
rather investigate if there is a trend across the groups (linear trend)
30/09/09 ANOVA 31
Two-Way ANOVA, Factorial Designs
• Advantage is that factorial designs can provide information on how the factors interact or
combine in the effect that they have on the dependent variable
Factor A
Factor B 1 2 3 …… c
1 X….X X….X X….X …… X….X
2 X….X X….X X….X …… X….X
3 X….X X….X X….X …… X….X
: : : ; :
: ; : ; ;
: : ; : :
r X….X X….X X….X …… X….X
30/09/09 ANOVA 32
Factorial ANOVAs
• Different types of effects are possible, the treatment effect: a difference in population
means, can be of two types:
- Main effect: a difference in population means for a factor collapsed over the levels
of all other factors in the design
- Interaction effect: when the effect on one factor is not the same at the levels of
another
• Interaction: whenever the effect of one factor depends on the levels of another factor
30/09/09 ANOVA 33
Interactions in ANOVA
Factor 1 Factor 1
Factor 2 AA AG GG Factor 2 AA AG GG
F 74 74 74 F 74 74 74
M 74 74 74 M 70 70 70
74 74 72 72
30/09/09 ANOVA 34
Interactions in ANOVA
Factor 1 Factor 1
Factor 2 AA AG GG Factor 2 AA AG GG
F 74 78 76 F 68 75 71.5
M 70 74 72 M 82 75 78.5
72 76 75 75
30/09/09 ANOVA 35
Interactions in ANOVA
Factor 1 Factor 1
Factor 2 AA AG GG Factor 2 AA AG GG
F 70 79 74.5 F 70 90 80
M 74 75 74.5 M 74 75 74.5
72 77 72 82.5
30/09/09 ANOVA 36
Repeated Measures ANOVA
• Repeated measures ANOVA is used when all members of a random sample are
measured under a number of different conditions
• A standard ANOVA would treat the data as independent and fail to model the correlation
between the repeated measures
Time
(mins)
Subject 0 10 20 …… 60
1 X X X …… X
2 X X X …… X
3 X X X …… X
: : : ; :
: ; : ; ;
: : ; : :
20 X X …… X
30/09/09 ANOVA 37
ANCOVA
- Most useful when the covariate is linearly related to the dependent variable and is
not related to the factors (independent qualitative variables)
30/09/09 ANOVA 38
Non Parametric ANOVA – Kruskal-Wallis Test
• When the null hypothesis is true, follows a Chi squared distribution degrees of
freedom
• Any variation among the groups will increase the test statistic (upper tail test)
- It’s based on ranking the data in each row of the table from low to high
- Each row is ranked separately
- The ranks are then summed in each column (group)
- The test is based on a Chi squared distribution
- Just like with the ANOVA the Friedman test will only indicate a difference but won’t
say where the difference lies
30/09/09 ANOVA 40
Literature Example
30/09/09 ANOVA 41
Literature Example
30/09/09 ANOVA 42
Further Reading
BOOKS:
30/09/09 ANOVA 43