Professional Documents
Culture Documents
STAT 509-001: Fall 2015
STAT 509-001: Fall 2015
Fall 2015
Lecture Notes
Dewei Wang
Department of Statistics
1
Contents
1 Introduction 4
1.1 What is Statistics? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Where to Use Statistics? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Deterministic and Statistical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Statistical Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Probability 7
2.1 Sample Spaces and Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Axioms of Probability and Addition Rule . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Conditional Probability and Multiplication Rule . . . . . . . . . . . . . . . . . . . . . 13
2.4 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2
4.8 Statistical Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.9 Testing hypotheses on the mean µ with known variance σ 2 . . . . . . . . . . . . . . . . 87
4.10 Testing hypotheses on the mean µ with unknown variance σ 2 . . . . . . . . . . . . . . 91
4.11 Testing hypotheses on the population proportion p . . . . . . . . . . . . . . . . . . . . 95
4.12 Testing hypotheses on the population variance σ 2 . . . . . . . . . . . . . . . . . . . . . 97
3
1 Introduction
1.1 What is Statistics?
• The field of Statistics deals with the collection, presentation, analysis, and use of data to make
decisions, solve problems, and design products and processes. (Montgomery, D. and Runger
G.)
• Statistics is the science of learning from data, and of measuring, controlling, and communicat-
ing uncertainty; and it thereby provides the navigation essential for controlling the course of
scientific and societal advances (Davidian, M. and Louis, T. A., 10.1126/science.1218685).
1. In a reliability (time to event) study, an engineer is interested in quantifying the time until
failure for a jet engine fan blade.
2. In an agricultural study in Iowa, researchers want to know which of four fertilizers (which vary
in their nitrogen contents) produces the highest corn yield.
3. In a clinical trial, physicians want to determine which of two drugs is more effective for treating
HIV in the early stages of the disease.
4. In a public health study, epidemiologists want to know whether smoking is linked to a particular
demographic class in high school students.
5. A food scientist is interested in determining how different feeding schedules (for pigs) could
affect the spread of salmonella during the slaughtering process.
6. A research dietician wants to determine if academic achievement is related to body mass index
(BMI) among African American students in the fourth grade.
Remark 1. Statisticians use their skills in mathematics and computing to formulate statistical
models and analyze data for a specific problem at hand. These models are then used to estimate
important quantities of interest (to the researcher), to test the validity of important conjectures, and
to predict future behavior. Being able to identify and model sources of variability is an important
part of this process.
4
1.3 Deterministic and Statistical Models
• A deterministic model is one that makes no attempt to explain variability. For example, in
circuit analysis, Ohm’s law states that
V = IR,
– In both of these models, the relationship among the variables is built from our underlying
knowledge of the basic physical mechanism. It is completely determined without any
ambiguity.
– In real life, this is rarely true for the obvious reason: there is natural variation that arises
in the measurement process. For example, a common electrical engineering experiment
involves setting up a simple circuit with a known resistance R. For a given current I,
different students will then calculate the voltage V .
∗ With a sample of n = 20 students, conducting the experiment in succession, we might
very well get 20 different measured voltages!
V = IR + ,
where is a random term that includes the effects of all unmodeled sources of variability that
affect this system.
• Descriptive statistics describe what is happening now (see Chapter 6 of the textbook).
• Inferential statistics, such as estimation and prediction, are based on a sample of the subjects
(only a portion of the population) to determine what is probably happening or what might
happen in the future.
Y = pull strength (a measure of the amount of force required to break the bond)
of a semiconductor. The population herein could be all the finished semiconductor. A sample of size
25 was collected and from each I measured the pull strength (Y ), the wire length (x1 ) and the die
height (x2 ). All 25 observations are plotted in Figure 1.4.1a.
5
Figure 1.4.1: (a). Three-dimensional plot of the wire bond pull strength data; (b). Plot of predicted
values of pull strength from the estimated model.
The goal here is to build a model that can quantify the relationship between pull strength and
the variables wire length and die height. A deterministic model would be
Y = f (x1 , x2 ),
for some unknown function f : [0, ∞)×[0, ∞) → [0, ∞). Perhaps a working model could be developed
as a statistical model of the form:
Y = β0 + β1 x1 + β2 x2 + ,
where is a random term that accounts for not only measurement error but also
(a) all of the other variables not accounted for (e.g., the quality of the wire and/or how all the
welding has been done, etc.) and
(b) the error induced by assuming a linear relationship between Y and {x1 , x2 } when, in fact,
it may not be.
In this example, with certain (probabilistic) assumptions on and a mathematically sensible way to
estimate the unknown β0 , β1 , and β2 (i.e., coefficients of the linear function), we can produce point
predictions of Y for any given {x1 , x2 }. Using the regression technique (Chapter 12) results in an
estimated model (plotted in Figure 1.4.1b)
• How accurate are the estimators of the coefficients or the prediction for a given {x1 , x2 }?
• How should samples be selected to provide good decisions with acceptable risks?
To answer these questions or to quantify the risks involved in statistical inference, it leads to the
study of probability models.
6
2 Probability
If we measure the current in a thin copper wire, we are conducting an experiment. However, day-
to-day repetitions of the measurement can differ slightly because of
• changes in ambient temperatures
• impurities in the chemical composition of the wire (if selecting different locations)
In some cases, the random variations are small enough, relative to our experimental goals, that
they can be ignored. However, no matter how carefully our experiment is designed and conducted,
the variation is almost always present, and its magnitude can be large enough that the important
conclusions from our experiment are not obvious. Hence, how to quantify the variability is a key
question, which can be answered by probability.
An experiment that can result in different outcomes, even though it is repeated in the same manner
every time, is called a random experiment.
The set of all possible outcomes of a random experiment is called the sample space of the experi-
ment. The sample space is denoted as S.
A sample space is discrete if it consists of a finite or countable infinite set of outcomes.
A sample space is continuous if it contains an interval (either finite or infinite) of real numbers.
Example 2.1.1. Let us find the sample space for each of the following random experiments and
identify whether it is discrete or continuous:
• Calls are repeated place to a busy phone line until a connection is achieved.
7
Figure 2.1.1: Tree diagram for three messages.
Example 2.1.2. (Tree diagram) Now let us consider a little bit more complex case. Each message
in a digital communication system is classified as to whether it is received on time or late. Describe
the sample space of the receive time of three messages.
S=
An event is a subset of the sample space of a random experiment. The following are three basic set
operations:
• The union of two events is the event that consists of all outcomes that are contained in either
of the two events. We denote the union as E1 ∪ E2 .
• The intersection of two events is the event that consists of all outcomes that are contained
in both of the two events. We denote the intersection as E1 ∩ E2 .
• The complement of an event in a sample space is the set of outcomes in the sample space
that are not in the event. We denote the complement of the event E as E 0 . The notation E c
is also used in other literature to denote the complement.
Example 2.1.3. Consider Example 2.1.2. Denote that E1 is the event that at least two messages
is received late. Then E1 = {100, 010, 001, 000}. Let E2 be the event that the second messages is
received later. Then E2 = {101, 100, 001, 000}. Now we have
E1 ∪ E2 =
E1 ∩ E2 =
E10 =
8
Example 2.1.4. As in Example 2.1.1, the sample space of the working time of an air conditioner
is S = (0, ∞). Let E1 be the event the working time is no less than 1 and less than 10; i.e.,
E1 = {x | 1 ≤ x < 10} = [1, 10), and E2 be the event the working time is between 5 and 15; i.e.,
E2 = {x | 5 < x < 15} = (5, 15). Then
E1 ∪ E2 = E1 ∩ E2 =
E10 =
E10 ∩ E2 =
One visualized way to interpret set operations is through Venn diagrams. For example
9
2.2 Axioms of Probability and Addition Rule
Probability is used to quantify the likelihood, or chance, that an outcome of a random experiment
will occur. The probability of an event E is denoted by P (E).
“My chance of getting an A in this course is 80%” could be a statement that quantifies your
feeling about the possibility of getting A. The likelihood of an outcome is quantified by assigning a
number from the interval [0, 1] to the outcome (or a percentage from 0 to 100%). Higher numbers
indicate that the outcome is more likely than lower numbers. A 0 indicates an outcome will not
occur. A probability of 1 indicates that an outcome will occur with certainty. The probability of
an outcome can be interpreted as our subjective probability, or degree of belief, that the outcome
will occur. Different individuals will no doubt assign different probabilities to the same outcomes.
Another interpretation of probability is based on the conceptual model of repeated replications
of the random experiment. The probability of an outcome is interpreted as the limiting value of the
proportion of times the outcome occurs in n repetitions of the random experiment as n increases
beyond all bounds. For example, we want to quantify the probability of the event that flipping a fair
coin gets a head. One way is to flip a fair coin n times, and record how many times you get a head.
Then
number of heads out of n flips 1
P (flipping a fair coin gets a head) = lim = .
n→∞ n 2
0.7
Limiting proportion
0.6
0.5
0.4
0.3
Number of flips
Equally Likely Outcomes: Whenever a sample space consists of N possible outcomes that are
equally likely, the probability of each outcome is 1/N .
For example, we want to detect the rate of defectiveness of products form a same product line.
The number of products could be million. It is time-consuming and expensive to exam every prod-
uct. People usually randomly select a certain number of product and count how many of them are
defective. We call the selected items as a random samples.
10
To select ramdonly implies that at each step of the sample, the remained items are equally likely
to be selected. .
It means that, suppose there are N items. When drawing the first sample, each item has the chance
of 1/N being selected. To select the second sample, each of the N − 1 remained items will be selected
with probability 1/(N − 1), so and so on.
Example 2.2.1. The following table provides an example of 400 parts classified by surface flaws
and as (functionally) defective.
Then
P (defective) = P (D) =
P (surface flaws) = P (F ) =
P (surface flaws and also defective) = P (D ∩ F ) =
P (surface flaws but not defective) = P (D0 ∩ F ) =
For a discrete sample space, P (E) equals the sum of the probabilities of the outcomes in E.
Example 2.2.2. A random experiment can result in one of the outcomes {a, b, c, d} with probabilities
0.1, 0.3, 0.5, and 0.1, respectively. Let A denote the event {a, b}, B the event {b, c, d} and C the event
{d}. Then
11
Axioms of Probability: Probability is a number that is assigned to each member of a collection
of events from a random experiment that satisfies the following properties: if S is the sample space
and E is any event in a random experiment,
1. P (S) = 1
2. 0 ≤ P (E) ≤ 1
These axioms imply the following results. The derivations are left as exercises at the end of this
section. Now,
P (∅) =
P (E1 ) P (E2 ).
Addition rule:
P (A ∪ B) =
Ei ∩ Ej = ∅.
P (E1 ∪ E2 ∪ · · · ∪ Ek ) =
Example 2.2.3. Let S = [0, ∞) be the sample space of working time of an air conditioner. Define
the events E1 = (2, 10), E2 = (5, 20), E3 = (5, 10), E4 = (0, 2]. Suppose P (E1 ) = .4, P (E2 ) = 0.7,
P (E3 ) = 0.2, P (E4 ) = .05. Then
12
2.3 Conditional Probability and Multiplication Rule
Sometimes probabilities need to be reevaluated as additional information becomes available. A useful
way to incorporate additional information into a probability model is to assume that the outcome
that will be generated is a member of a given event. This event, say A, defines the conditions that
the outcome is known to satisfy. Then probabilities can be revised to include this knowledge. The
probability of an event B under the knowledge that the outcome will be in event A is denoted as
Of the parts with surface flaws (40 parts), the number of defective ones is 10. Therefore,
P (D | F ) =
and of the parts without surface flaws (360 parts), the number of defective ones is 18. Therefore,
P (D | F 0 ) =
Practical Interpretation: The probability of being defective is five times greater for parts with
surface flaws. This calculation illustrates how probabilities are adjusted for additional information.
The result also suggests that there may be a link between surface flaws and functionally defective
parts, which should be investigated.
P (B | A) = P (A ∩ B)/P (A).
P (D | F ) =
P (D | F 0 ) =
13
Multiplication Rule:
Assume E1 , E2 , . . . , Ek are k mutually exclusive and exhaustive sets, then for any event B, we have
P (B) =P (B ∩ E1 ) + P (B ∩ E2 ) + · · · + P (B ∩ Ek )
=P (B | E1 )P (E1 ) + P (B | E2 )P (E2 ) + · · · + P (B | Ek )P (Ek ).
Example 2.3.2. Assume the following probabilities for product failure subject to levels of contam-
ination in manufacturing:
In a particular production run, 20% of the chips are subjected to high levels of contamination, 30%
to medium levels of contamination, and 50% to low levels of contamination. What is the probability
of the event F that a product using one of these chips fails?
Let
14
Then
P (F ) =
=
2.4 Independence
In some cases, the conditional probability of P (B | A) might equal P (B); i.e., the outcome of the
experiment is in event A does not affect the probability that the outcome is in event B.
Example 2.4.1. As in Example 2.2.1, surface flaws related to functionally defective parts since
P (D | F ) = 0.25 and P (D) = 0.07. Suppose now the situation is different as the following Table.
Then,
P (D | F ) = and P (D) = .
That is, the probability that the part is defective does not depend on whether it has surface flaws.
Also,
P (F | D) = and P (F ) =
so the probability of a surface flaw does not depend on whether the part is defective. Furthermore,
the definition of conditional probability implies that P (F ∩ D) = P (D | F )P (F ), but in the special
case of this problem,
P (F ∩ D) = P (D)P (F ).
Two events are independent if any one of the following equivalent statements is true:
1. P (A | B) = P (A)
2. P (B | A) = P (B)
3. P (A ∩ B) = P (A)P (B)
15
Noting that when A and B are independent events,
P (A0 ∩ B 0 ) =
=
=
Question: If A and B are mutually exclusive, and P (A) > 0, P (B) > 0, Are A and B independent?
Example 2.4.2. (Series Circuit) The following circuit operates only if there is a path of functional
devices from left to right. The probability that each device functions is shown on the graph. Assume
that devices fail independently. What is the probability that the circuit operates?
Example 2.4.3. (Parallel Circuit) The following circuit operates only if there is a path of func-
tional devices from left to right. The probability that each device functions is shown on the graph.
Assume that devices fail independently. What is the probability that the circuit operates?
16
If the events E1 , E2 , . . . , Ek are independent, then
Example 2.4.4. (Advanced Circuit) The following circuit operates only if there is a path of
functional devices from left to right. The probability that each device functions is shown on the
graph. Assume that devices fail independently. What is the probability that the circuit operates?
17
3 Random Variables and Probability Distributions
A random variable is a function that assigns a real number to each outcome in the sample space
of a random experiment.
A discrete random variable is a random variable with a finite (or countably infinite) range.
A continuous random variable is a random variable with an interval (either finite or infinite) of
real numbers for its range.
Notation: A random variable is denoted by an uppercase letter such as X and Y . After experiment
is conducted, the measured value of the random variable is denoted by a lowercase letter such as x
and y.
For example, let X be a random variable denoting the outcome of flipping a coin. The sample
space of this random experiment is {head, tail}. We can let X = 1 if it is a head; X = 0 otherwise.
When you are actually conduct this experiment, you may observe a head. Then the notation for
describing this observation is x = 1.
For a discrete random variable X with possible values x1 , x2 , . . . , xk , a probability mass function
(pmf ) is a function such that
(1) f (xi ) ≥ 0
Pk
(2) i=1 f (xi ) =1
(3) f (xi ) = P (X = xi )
18
Example 3.1.1. (Digital Channel) There is a chance that a bit transmitted through a digital
transmission channel is received in error. Let X equal the number of bits in error in the next four
bits transmitted. The possible values for X are {0, 1, 2, 3, 4}. Based on a model for the errors that
is presented in the following section, probabilities for these values will be determined.
P (X = 0) = 0.6561, P (X = 1) = 0.2916,
P (X = 2) = 0.0486, P (X = 3) = 0.0036,
P (X = 4) = 0.0001.
The probability distribution of X is specified by the possible values along with the probability of
each. The pmf of X is then
Once a probability mass function of X is presented, one should be able to calculate all types of
events in the sample space; i.e., P (X ≤ a), P (X < a), P (X ≥ a), P (X > a), P (a < X < b), P (a ≤
X < b), P (a < X ≤ b), P (a ≤ X ≤ b). For example, in the example above,
P (X < 1) =
P (X ≤ 1) =
P (X ≤ 3) − P (X ≤ 2) =
P (1 ≤ X < 3) =
P (1 < X < 3) =
P (1 < X ≤ 3) =
P (1 ≤ X ≤ 3) =
19
3.1.2 Cumulative Distribution Function
The cumulative distribution function (cdf ) of a discrete random variable X, denoted as F (X),
is
X
F (x) = P (X ≤ x) = f (xi ).
xi ≤x
2. 0 ≤ F (x) ≤ 1
cdf of X
1.0
● ● ●
●
0.8
●
0.6
F(x)
0.4
0.2
0.0
−1 0 1 2 3 4 5
20
And based on cdf F (x), you should be able to calculate the probabilities of the following types
P (X < 1) =
P (X ≤ 1) =
P (X ≤ 3) − P (X ≤ 2) =
P (1 ≤ X < 3) =
P (1 < X < 3) =
P (1 < X ≤ 3) =
P (1 ≤ X ≤ 3) =
Two numbers are often used to summarize a probability distribution for a random variable X. The
mean is a measure of the center or middle of the probability distribution, and the variance is a
measure of the dispersion, or variability in the distribution.
The mean or expected value of the discrete random variable X, denoted as µ or E(X), is
X
µ = E(X) = xf (x).
x
The expected value for a discrete random variable Y is simply a weighted average of the possible
values of X. Each value x is weighted by its probability f (x). In statistical applications, µ = E(Y )
is commonly called the population mean.
Example 3.1.2. The number of email messages received per hour has the following distribution:
Determine the mean and standard deviation of the number of messages received per hour.
µ=
Interpretation: Over the long run, if we observed many values of Y with this pmf, then the average
of these X observations would be close to
21
Let X be a discrete random variable with pmf f (x). Suppose that g, g1 , g2 , ..., gk are real-valued
functions, and let c be any real constant.
X
E[g(X)] = g(x)f (x).
all x
1. E(c) = c
2. E[cg(X)] = cE[g(X)]
E[g(X)] =
Example 3.1.3. In Example 3.1.2, suppose that each email message header reserves 15 kilobytes
of memory space for storage. Let the random variable Y denote the memory space reserved for all
message headers per hour (in kilobytes). Then
Y =
Thus
E(Y ) =
The expected reserved memory space for all message headers per hour is
√
The population standard deviation of X is σ = σ2.
1. σ 2 ≥ 0. σ 2 = 0 if and only if the random variable Y has a degenerate distribution; i.e., all
the probability mass is located at one support point.
2. The larger (smaller) σ 2 is, the more (less) spread in the possible values of X about the popu-
lation mean µ = E(X).
22
Let X be a discrete random variable with pmf f (x). Suppose g(x) = ax + b where a, b are constants,
we have
V [aX + b] =
V (Y ) =
The variance of reserved memory space for all message headers per hour is
The measures of mean and variance do not uniquely identify a probability distribution. That is,
two different distributions can have the same mean and variance. Still, these measures are simple,
useful summaries of the probability distribution of X.
23
3.2 Bernoulli Distribution and Binomial Distribution
Let us consider the following random experiments and random variables:
1. A worn machine tool produces 1% defective parts. Let X = number of defective parts in the
next 25 parts produced.
2. Each sample of air has a 10% chance of containing a particular rare molecule. Let X = the
number of air samples that contain the rare molecule in the next 18 samples analyzed.
3. Of all bits transmitted through a digital transmission channel, 40% are received in error. Let
X = the number of bits in error in the next five bits transmitted.
4. A multiple-choice test contains 10 questions, each with four choices, and for each question, the
chance of you gets right is 90%. Let X = the number of questions answered correctly.
Each of these random experiments can be thought of as consisting of a series of repeated, random
trials:
Each of the repeated trials consists of two possible outcomes: (generally speaking) success and
failure, and we want to know how many (generally speaking) successes occur in the a certain
number of trials. The terms success and failure are just labels. Sometime it can mislead you. For
example, in the 1st example, we are interested in the number of defective parts (herein, “success”
means defective).
To model a trial with two outcomes, we typically use Bernoulli Distribution. We say random
variable X follows a Bernoulli distribution, if it has the following probability mass function:
(
p if x = 1, represents success
f (x) =
1−p if x = 0, represents failure
µ = E[X] =
σ 2 = V [X] =
24
Now, let us get back to our original examples. What we are interested in is the number of
successes occurs in a certain number of identical trails, each trail has two possible outcomes (success
and failure) with certain probability of success. Thus, we are investigating the summation of a given
number of identical Bernoulli random variables.
1. In the first example: we investigate the random variable X is the summation of n = 25 identical
Bernoulli random variables, each of which has two possible outcomes (defective = “success,”
indefective=“failure”), with probability of success being p = 0.01
To model these quantities, one commonly used distribution is Binomial Distribution: Suppose
that n independent and identical Bernoulli trials are performed. Define
We say the X has a Binomial Distribution with number of trials n and success probability p.
Shorthand notation is X ∼ B(n, p). The probability mass function of X is given by
!
n
px (1 − p)n−x , x = 0, 1, 2, 3, . . . , n
f (x) = x
0, otherwise
where !
n n!
= , and r! = r × (r − 1) × · · · × 2 × 1 (note 0! = 1).
x x!(n − x)!
µ = E[X] =
σ 2 = V [X] =
25
There are three key elements for correctly identifying a Bernoulli distribution:
(2) Each trial results in only two possible outcomes, labeled as “success” and “failure.”
Let us see the 3rd example: Of all bits transmitted through a digital transmission channel, 40% are
received in error. Let X = the number of bits in error in the next five bits transmitted. Now we
calculate P (X = 2) by assuming all the transmitted bits are independent.
26
Thus, from above we can see that X is actually a Binomial random variable; i.e., X ∼ B(5, 0.4).
Now let us answer the following questions:
(a) What is the probability that at least one bits are received in error?
Now considering the first example, we have X ∼ , what is the probability when
X ≤ 10? Computing this probability “by hand” could be very time-consuming. We will use TI-84.
The codes are (in “DISTR”):
f (x) = P (X = x) F (x) = P (X ≤ x)
binompdf(n, p, x) binomcdf(n, p, x)
(a) What is the probability that there are exactly five defective parts?
(b) What is the probability that there are at least five defective parts?
(c) What is the probability that there are at most ten defective parts?
27
3.3 Geometric Distributions
The geometric distribution also arises in experiments involving Bernoulli trials:
However, instead of fixing a certain number of trials and then finding out how many of them are
successes, trials are conducted until a success is obtained.
We say that X has a geometric distribution with success probability p. Shorthand notation is
X ∼ Geom(p).
Example 3.3.1. The probability that a bit transmitted through a digital transmission channel is
received in error is 0.1. Assume that the transmissions are independent events, and let the random
variable X denote the number of bits transmitted until the first error. Calculate P (X = 5).
1
µ = E(X) =
p
1−p
σ 2 = V (X) = .
p2
28
Figure 3.3.1: Geometric distribution of selected values of the parameter p.
Back to Example 3.3.1, the probability mass function is plotted in the above figure with solid
dots. And
µ = E(X) =
σ 2 = V (X) =
Example 3.3.2. Biology students are checking the eye color of fruit flies. For each fly, the probability
of observing white eyes is p = 0.25. In this situation, we interpret the Bernoulli trials as
If the Bernoulli trial assumptions hold (independent flies, same probability of white eyes for each
29
fly), then
(a) What is the probability the first white-eyed fly is observed on the fifth fly checked?
(b) What is the probability the first white-eyed fly is observed before the fourth fly is examined?
30
3.4 Negative Binomial Distributions
The negative binomial distribution also arises in experiments involving Bernoulli trials:
However, instead of fixing a certain number of trials and then finding out how many of them are
successes, trials are conducted until the rth success is obtained.
We say that X has a negative binomial distribution with waiting parameter r and success probability
p. Shorthand notation is X ∼ NB(r, p).
Example 3.4.1. The probability that a bit transmitted through a digital transmission channel is
received in error is 0.1. Assume that the transmissions are independent events, and let the random
variable X denote the number of bits transmitted until the 3rd error. Calculate P (X = 5).
31
If X ∼ NB(r, p), then the probability mass function of X is given by
!
x−1
pr (1 − p)(x−r) , x = r, r + 1, r + 2, . . .
f (x) = r−1
0, otherwise.
r
µ = E(X) =
p
r(1 − p)
σ 2 = V (X) = .
p2
Note that the negative binomial distribution is a mere generalization of the geometric. If r = 1, then
the NB(r, p) distribution reduces to the Geom(p).
Back to Example 3.4.1, the probability mass function is plotted in the above figure with solid
dots. And
µ = E(X) =
σ 2 = V (X) =
To calculate f (x) = P (X = x) when X ∼ NB(r, p), one could use the TI-84:
Unfortunately, there is no simple TI-84 codes to calculate the cumulative distribution function of a
negative binomial distribution; i.e., F (x) = P (X ≤ x). The only way is
x
( x )
X X
F (x) = p × binompdf (k − 1, p, r − 1) = p × binompdf (k − 1, p, r − 1) .
k=r k=r
However, when you can use computer (like when doing homework but not in exams), you can always
use R to compute this type of probability.
32
Example 3.4.2. At an automotive paint plant, 25 percent of all batches sent to the lab for chemical
analysis do not conform to specifications. In this situation, we interpret
• batch = “trial.”
If the Bernoulli trial assumptions hold (independent batches, same probability of nonconforming for
each batch), then
(a) What is the probability the third nonconforming batch is observed on the tenth batch sent to
the lab?
(b) What is the probability that no more than two nonconforming batches will be observed among
the first 4 batches sent to the lab?
(c) What is the probability that no more than three nonconforming batches will be observed
among the first 30 batches sent to the lab?
33
3.5 Hypergeometric Distribution
Consider a population of N objects and suppose that each object belongs to one of two dichotomous
classes: Class 1 and Class 2. For example, the objects (classes) might be people (infected/not), parts
(defective/not), new born babies (boy/girl), etc.
Randomly select n objects from the population (objects are selected at random, random means
each remain object has the same chance of getting selected, and without replacement). Define
We say that X has a hypergeometric distribution and write X ∼ hyper(N, n, K). The probability
function of X ∼ hyper(N, n, K) is
K N −K
( x )( n−x )
, x ≤ K and n − x ≤ N − K
(Nn )
f (x) =
0, otherwise.
34
Example 3.5.1. A supplier ships parts to a company in lots of 100 parts. The company has an
acceptance sampling plan which adopts the following acceptance rule:
Suppose among the 100 parts there are 10 parts which are defective.
(b) What is the probability that at least 3 of the 5 parts sampled are defective?
In the previous example, we could compute the probabilities of interest, using R, as follows:
35
3.6 Poisson Distribution
The Poisson distribution is commonly used to model counts in an interval of time, an area, a volume
or other unit, such as
In general, we define
A Poisson distribution for X emerges if these “occurrences” obey the following rules:
(I) the number of occurrences in non-overlapping intervals (of time or space) are
independent random variables.
(II) the probability of an occurrence in a sufficiently short interval is proportional to the length of
the interval.
We say that X has a Poisson distribution and write X ∼ Poisson(λ). A process that produces
occurrences according to these rules is called a Poisson process.
λx e−λ
x! , x = 0, 1, 2, . . .
f (x) =
0, otherwise.
and
E(X) = λ
V (X) = λ.
Remark: In a Poisson process, suppose the mean of counts in one unit is λ, then the
mean of counts in 2 units is 2λ, in 3 unites is 3λ, ...
36
Example 3.6.1. Let X denote the number of times per month that a detectable amount of radioac-
tive gas is recorded at a nuclear power plant. Suppose that X follows a Poisson distribution with
mean λ = 2.5 times per month.
(a) What is the probability that there are exactly three times a detectable amount of gas is recorded
in a given month?
(b) What is the probability that there are no more than four times a detectable amount of gas is
recorded in a given month?
(c) What is the probability that there are exactly three times a detectable amount of gas is recorded
in two given month?
(d) Given the event that there are four times a detectable amount of gas is recorded in September,
what is the probability that there are exactly three times a detectable amount of gas is recorded
in October?
37
Example 3.6.2. Orders arrive at a Web site according to a Poisson process with a mean of 12 per
hour. Determine the following:
(c) Length of a time interval such that the probability of no orders in an interval of this length is
0.001.
38
3.7 General Continuous Distribution
Recall: A continuous random variable is a random variable with an interval (either finite or
infinite) of real numbers for its range.
• Contrast this with a discrete random variable whose values can be “counted.”
• For example, if X = time (measured in seconds), then the set of all possible values of X is
{x : x > 0}
If X = temperature (measured in degree o C), the set of all possible values of X (ignoring
absolute zero and physical upper bounds) might be described as
Assigning probabilities to events involving continuous random variables is different than in dis-
crete models. We do not assign positive probability to specific values (e.g., X = 3, etc.) like we did
with discrete random variables. Instead, we assign positive probability to events which are intervals
(e.g., 2 < X < 4, etc.).
Every continuous random variable we will discuss in this course has a probability density
function (pdf ), denoted by f (x). This function has the following characteristics:
In another way, f (x) can be view as the first derivative of F (x); i.e.,
f (x) = F 0 (x).
= F (x2 ) − F (x1 ).
39
5. For any specific value x0 , P (X = x0 ) = 0. In other words, in continuous probability models,
specific points are assigned zero probability (see #4 above and this will make perfect mathe-
matical sense). An immediate consequence of this is that if X is continuous,
This is not true if X has a discrete distribution because positive probability is assigned to
specific values of X. Evaluating a pdf at a specific value x0 , that is, computing f (x0 ), does not
give you a probability! This simply gives you the height of the pdf f (x) at x = x0 .
The population standard deviation of X is given by the positive square root of the variance:
√ p
σ= σ2 = V (X).
Let X be a continuous random variable with pdf f (x). Suppose that g, g1 , g2 , ..., gk are real-valued
functions, and let c be any real constant.
Z ∞
E[g(X)] = g(x)f (x)dx.
−∞
1. E(c) = c
2. E[cg(X)] = cE[g(X)]
40
Example 3.7.1. Suppose that X has the pdf
(
3x2 , 0 < x < 1
f (x) =
0, otherwise.
(g) If we define Y = 3X, find the cdf and pdf of Y . Further calculate the mean and variance of Y .
41
3.8 Exponential Distribution
The Exponential Distribution is commonly used to answer the following questions:
• How long will it take before a call center receives the next phone call?
All these questions concern the time we need to wait before a given event occurs. We often model
this waiting time by assuming it follows an exponential distribution.
A random variable X is said to have an exponential distribution with parameter λ > 0 if its pdf
is given by (
λe−λx , x > 0
f (x) =
0, otherwise.
F (x0 ) = P (X ≤ x0 ) =
(
=
Thus, for any specified time x0 (of course it is positive), the probability of the event happens no later
than x0 is
P (X ≤ x0 ) = F (x0 ) =
P (X > x0 ) = 1 − F (x0 ) =
Now, let us define Y = λX. What is the cdf and pdf of Y ? What are the mean and variance of Y ?
42
Using Y , we are able to calculate the mean and variable of X ∼ Exp(λ) as
µ = E(X) =
σ 2 = V (X) =
√
Consequently, the standard deviation of X is then σ = σ2 = .
Example 3.8.1. Assume that the length of a phone call in minutes is an exponential random variable
X with parameter λ = 1/10, (or the question may tell you the value of λ through the expectation;
i.e., this based the expected waiting time for a phone call is 10 minuets). If someone arrives at a
phone booth just before you arrive, find the probability that you will have to wait
43
Memoryless property: Suppose Z is a continuous random variable whose values are all non-
negative. We say Z is memoryless if for any r ≥ 0, s ≥ 0, we have
Interpretation: suppose Z represents the waiting time until something happens. This property
says that, conditioning on that you have waited at least t time, the probability of waiting for addi-
tionally at leat s time is the same with the probability of waiting for at least s time starting from
the beginning. In other words, In other words, the fact that Z has made it to time t has been
“forgotten.”
In the following, we show that any exponential random variable, X ∼ Exp(λ) has the memoryless
property.
Example 3.8.2. In previous example, what is probability that you need wait for more than 10
minuets given the fact you have waited for more than 3 minutes?
44
Poisson relationship: Suppose that we are observing “occurrences” over time according to a
Poisson distribution with rate λ. Define the random variable
Then,
W ∼ Exp(λ).
NOTE THAT it is also true that the time between any two consecutive occurrences in a Poisson
process follows this same exponential distribution (these are called “interarrival times”).
Example 3.8.3. Suppose that customers arrive at a check-out according to a Poisson process with
mean λ = 12 per hour. What is the probability that we will have to wait longer than 10 minutes to
see the first customer? (Note: 10 minutes is 1/6th of an hour.)
45
3.9 Gamma Distribution
We start this subsection with a very interesting function: the Gamma Function, defined by
Z ∞
Γ(α) = tα−1 e−t dt, where α > 0.
0
Notice that
∞
1 α−1 −t
Z
1= t e dt
0 Γ(α)
(change variable x = t/λ, for λ > 0)
Z ∞
1
= (λx)α−1 e−λx d(λx)
0 Γ(α)
Z ∞ α
λ
= xα−1 e−λx dx
0 Γ(α)
Z ∞
= f (x)dx (Thus f (x) is a valid pdf.)
−∞
where
λα α−1 −λx
(
Γ(α) x e , x>0
f (x) =
0, otherwise.
A random variable X is said to has a gamma distribution with parameters α > 0 and λ > 0 if its
pdf is given by ( α
λ α−1 e−λx , x > 0
Γ(α) x
f (x) =
0, otherwise.
α α
E(X) = , V (X) = 2 .
λ λ
• When α = 1, we have
λ1 1−1 −λx
(
Γ(1) x e = λe−λx , x > 0
f (x) =
0, otherwise.
Hence, exponential distribution Exp(λ) is a special case of Gamma distribution; i.e., Gamma(1, λ).
In other words, the gamma distribution is more flexible than the exponential distribution.
46
• Plot of pdf and cdf:
0 2 4 6 8 10 0 2 4 6 8 10
x x
0 2 4 6 8 10 0 2 4 6 8 10
x x
0 2 4 6 8 10 0 2 4 6 8 10
x x
0 2 4 6 8 10 0 2 4 6 8 10
x x
47
• Plot of pdf and cdf:
0 2 4 6 8 10 0 2 4 6 8 10
x x
0 2 4 6 8 10 0 2 4 6 8 10
x x
0 2 4 6 8 10 0 2 4 6 8 10
x x
0 2 4 6 8 10 0 2 4 6 8 10
x x
48
• Poisson relationship: Suppose that we are observing “occurrences” over time according to
a Poisson distribution with rate λ. Define the random variable
Then,
W ∼ Gamma(α, λ).
NOTE THAT it is also true that the time between any two occurrences (unlike last subsection,
these two occurrences does not need to be consecutive) in a Poisson process follows a gamma
distribution.
• The cdf of a gamma random variable does not exist in closed form. Therefore, probabilities
involving gamma random variables (when α 6= 1) must be computed numerically (e.g., using
R).
Type F (x) = P (X ≤ x)
X ∼ Exp(λ) pexp(x, λ) or pgamma(x, 1, λ)
X ∼ Gamma(α, λ) pgamma(x, α, λ)
Example 3.9.1. Calls to the help line of a large computer distributor follow a Poisson distribution
with a mean of 20 calls per minute. Determine the following:
(c) Probability that the time till the third call occur within 15 seconds
49
3.10 Normal Distribution
A random variable X is said to have a normal distribution if its pdf is given by
1 1 x−µ 2
f (x) = √ e− 2 ( σ ) , −∞ < x < ∞.
2πσ
Shorthand notation is X ∼ N (µ, σ 2 ). Another name for the normal distribution is the Gaussian
distribution.
E(X) = µ, V (X) = σ 2 .
Example 3.10.1. If X ∼ N (1, 4), find the mean, variance, and standard deviation of X.
0.4
0.4
0.3
0.3
f(x)
f(x)
0.2
0.2
0.1
0.1
0.0
0.0
x x
Figure 3.10.3: The left one presents the plot of pdf of N (−10, 1), N (−5, 1), N (0, 1), N (5, 1), N (10, 1)
(from left to right).
The right one presents the plot of pdf of N (0, 1), N (0, 22 ), N (0, 32 ), N (0, 42 ), N (0, 55 ), N (0, 88 )
(from top to down)
X−µ
Example 3.10.2. Denote Z = σ , identify the distribution of Z.
50
Standard normal distribution: when µ = 0 and σ 2 = 1, we say the normal distribution N (0, 1)
is the standard normal distribution. We denote a standard normal random variable by Z; i.e.,
Z ∼ N (0, 1).
If random variable X ∼ N (µ, σ 2 ), we can standardize X to get a standard normal random variable:
X −µ
= Z ∼ N (0, 1).
σ
Φ(z) = P (Z ≤ z).
However, the function Φ(z) does not exist in closed form. Actually, for any normal random variable,
its cdf does not exists in closed form.
51
For X ∼ N (µ, σ 2 ),
X −µ
= Z ∼ N (0, 1).
σ
In the other way, we can express
X = µ + σZ.
Thus, all the normal distribution shares a common thing which is the standard normal distribution. If
we know standard normal, we know every normal distribution. For example, in terms of calculating
probabilities, if X ∼ N (µ, σ 2 ), we can always standardize it to get the standard normal Z and
calculate the probabilities based on standard normal.
Similarly, we have
P (X > x1 ) = , P (X < x2 ) =
(a) P (µ − σ ≤ X ≤ µ + σ) =
(b) P (µ − 2σ ≤ X ≤ µ + 2σ) =
(c) P (µ − 3σ ≤ X ≤ µ + 3σ) =
52
Interpretation:
• Symmetric: The pdf of a normal distribution is always symmetric respect to its mean. Thus
For X ∼ N (µ, σ 2 ),
P (X − µ > x) = P (X − µ < x)
P (−x < X − µ < x) = 1 − 2P (X − µ > x) if x > 0.
• Find the inverse of the cdf of a normal distribution: We have already known how to
compute F (x) = P (X ≤ x) when X ∼ N (µ, σ 2 ). In the opposite way, suppose the question
tells you P (X ≤ x) = α, how to find x based on the value of α?
In the other way, if you need find the value of x such that P (X > x) = α, use
invNorm(1 − α, µ, σ).
53
Example 3.10.4. If X is normally distributed with a mean of 10 and a standard deviation of 2.
(b) Determine the value for x that solves each of the following:
Example 3.10.5. Suppose that the current measurements in a strip of wire are assumed to follow
a normal distribution with a mean of 10 milliamperes and a variance of σ 2 (milliamperes)2 , where
σ 2 is unknown.
(a) Suppose we know the probability that a measurement exceeds 12 milliamperes is 0.16, approxi-
mate σ 2 via the Empirical rule.
(b) Based on part (a), find the value x satisfies that the probability that a measurement exceeds x
milliamperes is 0.05.
(c) Ignoring the findings in (a-b), suppose we know the probability that a measurement exceeds
13.29 milliamperes is 0.05, find σ.
54
3.11 Weibull Distribution
Reliability analysis is important in engineering. It deals with failure time (i.e., lifetime, time-to-
event) data. For example,
We call T a lifetime random variable if it measures the time to an “event;” e.g., failure, death,
eradication of some infection/condition, etc. Engineers are often involved with reliability studies
in practice, because reliability is related to product quality. There are many well known lifetime
distributions, including
• exponential
• Weibull
• lognormal
A random variable T is said to have a Weibull distribution with parameter β > 0 and η > 0 if its
pdf is given by
β−1
β t β
e−(t/η) , t > 0
fT (t) = η η
0, otherwise.
• We call
β = shape parameter
η = scale parameter.
• By changing the values of β and η, the Weibull pdf can assume many shapes. The Weibull
distribution is very popular among engineers in reliability applications; e.g., here are the plots
55
of probability density function and cumulative distribution function of several Weibull distri-
butions.
1.0
0.15
0.8
0.10
0.6
pdf
cdf
0.4
β=2, η=5
0.05
β=2, η=10
0.2
β=3, η=10
0.00
0.0
0 5 10 15 20 25 30 0 5 10 15 20 25 30
t t
• Note that when β = 1, the Weibull pdf reduces to the exponential(λ = 1/η) pdf.
Example 3.11.1. Suppose that the lifetime of a rechargeable battery, denoted by T (measured in
hours), follows a Weibull distribution with parameters β = 2 and η = 10.
(b) What is the probability that a battery is still functional at time t = 20?
56
(c) What is the probability that a battery is still functional at time t = 20 given that the battery is
functional at time t = 10?
FT (t) = P (T ≤ t).
This can be interpreted as the proportion of units that have failed by time t.
This can be interpreted as the proportion of units that have not failed by time t; e.g., the unit
is still functioning, a warranty claim has not been made, etc.
d d
fT (t) = FT (t) = − ST (t).
dt dt
P (t ≤ T < t + |T ≥ t)
hT (t) = lim .
→0
The hazard function is not a probability; rather, it is a probability rate. Therefore, it is possible
that a hazard function may exceed one.
The hazard function (or hazard rate) is a very important characteristic of a lifetime distribution.
It indicates the way the risk of failure varies with time. Distributions with increasing hazard
functions are seen in units for whom some kind of aging or “wear out” takes place. Certain types of
57
units (e.g., electronic devices, etc.) may display a decreasing hazard function, at least in the early
stages of their lifetimes. It is insightful to note that
We can therefore describe the distribution of the continuous lifetime random variable T by using
either fT (t), FT (t), ST (t), or hT (t).
Example 3.12.1. In this example, we find the hazard function for T ∼ Weibull(β, η). Recall that
when t > 0, the pdf of T is
β t β−1 −(t/η)β
fT (t) = e
η η
The cdf and survivor function of T are, respectively,
β β
FT (t) = 1 − e−(t/η) and ST (t) = 1 − FT (t) = e−(t/η) .
Plots of Weibull hazard functions are given in Figure 3.12.4. It is easy to show
• hT (t) is increasing if β > 1 (wear out; population of units get weaker with aging)
• hT (t) is decreasing if β < 1 (infant mortality; population of units gets stronger with aging).
Figure 3.12.4: Weibull hazard functions with η = 1. Upper left: β = 3. Upper right: β = 1.5. Lower
left: β = 1. Lower right: β = 0.5.
58
4 One-Sample Statistical Inference
4.1 Populations and samples
Overview: This chapter is about statistical inference. This deals with making (probabilistic)
statements about a population of individuals based on information that is contained in a sample
taken from the population.
Example 4.1.1. Suppose that we wish to study the performance of lithium batteries used in a
certain calculator. The purpose of our study is to determine the mean lifetime of these batteries
so that we can place a limited warranty on them in the future. Since this type of battery has not
been used in this calculator before, no one (except the Oracle) can tell us the distribution of X, the
battery’s lifetime. In fact, not only is the distribution not known, but all parameters which index
this distribution aren’t known either.
A population refers to the entire group of “individuals” (e.g., parts, people, batteries, etc.) about
which we would like to make a statement (e.g., proportion defective, median weight, mean lifetime,
etc.).
• It is generally accepted that the entire population can not be measured. It is too large and/or
it would be too time consuming to do so.
• We will assume that the sample of individuals constitutes a random sample. Mathematically,
this means that all observations are independent and follow the same probability distribution.
Informally, this means that each sample (of the same size) has the same chance of being selected.
Our hope is that a random sample of individuals is “representative” of the entire population
of individuals.
X1 , X2 , ..., Xn .
That is, X1 is the value of X for the first individual in the sample, X2 is the value of X for the
second individual in the sample, and so on. The sample size tells us how many individuals are in the
sample and is denoted by n. Statisticians refer to the set of observations X1 , X2 , ..., Xn generically
as data. Lower case notation x1 , x2 , ..., xn is used when citing numerical values (or when referring
to realizations of the upper case versions).
59
15
10
Count
5
0
Example 4.1.2. BATTERY DATA: Consider the following random sample of n = 50 battery life-
times x1 , x2 , ..., x50 (measured in hours):
4285 2066 2584 1009 318 1429 981 1402 1137 414
564 604 14 4152 737 852 1560 1786 520 396
1278 209 349 478 3032 1461 701 1406 261 83
205 602 3770 726 3894 2662 497 35 2778 1379
3920 1379 99 510 582 308 3367 99 373 454
In Figure 4.1.1, we display a histogram of the battery lifetime data. We see that the (empirical)
distribution of the battery lifetimes is skewed towards the high side.
• Which continuous probability distribution seems to display the same type of pattern that we
see in histogram?
• An exponential Exp(λ) model seems reasonable here (based on the histogram shape). What is
λ?
60
• In general, (population) parameters which index probability distributions (like the exponential)
are unknown.
• All of the probability distributions that we discussed in Chapter 3 are meant to describe (model)
population behavior.
µ = population mean
σ 2 = population variance
p = population proportion.
All of the probability distributions that we talked about in Chapter 3 were indexed by population
(model) parameters. For example,
• the N (µ, σ 2 ) distribution is indexed by two parameters, the population mean µ and the popu-
lation variance σ 2 .
• the Weibull(β, η) distribution is indexed by two parameters, the shape parameter β and the
scale parameter η.
• the B(n, p) distribution is indexed by two parameters, the size n and the population proportion
of successes p.
Suppose that X1 , X2 , ..., Xn is a random sample from a population. The sample mean is
n
1X
X= Xi .
n
i=1
The sample standard deviation is the positive square root of the sample variance; i.e.,
v
√
u n
u 1 X
S= S2 = t (Xi − X)2 .
n−1
i=1
61
A statistic is a numerical quantity that can be calculated from a sample of data. Some very common
examples are:
X = sample mean
S 2 = sample variance
pb = sample proportion.
For example, with the battery lifetime data (a random sample of n = 50 lifetimes),
x = 1274.14 hours
s2 = 1505156 (hours)2
s ≈ 1226.85 hours.
In R, the following codes can help you calculate the sample mean, sample variance, and sample
standard deviation. In the following codes, the battery data is saved in the variable with the name
“battery.”
Summary: The table below succinctly summarizes the salient differences between a population and
a sample (a parameter and a statistic):
Statistical inference deals with making (probabilistic) statements about a population of individuals
based on information that is contained in a sample taken from the population. We do this by
(b) quantifying the uncertainty (variability) that arises in the estimation process.
These are both necessary to construct confidence intervals and to perform hypothesis tests, two
important exercises discussed in this chapter.
62
4.3 Point estimators and sampling distributions
NOTATION: To keep our discussion as general as possible (as the material in this subsection can be
applied to many situations), we will let θ denote a population parameter.
• For example, θ could denote a population mean, a population variance, a population proportion,
a Weibull or gamma model parameter, etc. It could also denote a parameter in a regression
context (Chapter 6-7).
• The data X1 , X2 , ..., Xn come from the sampling process; e.g., different random samples will
yield different data sets X1 , X2 , ..., Xn .
• In this light, because the sample values X1 , X2 , ..., Xn will vary from sample to sample, the
value of θb will too! It therefore makes sense to think about all possible values of θ;
b that is, the
distribution of θ. b
E(θ)
b = θ.
In other words, the mean of the sampling distribution of θb is equal to θ. Note that unbiasedness is
a characteristic describing the center of a sampling distribution. This deals with accuracy.
E(X) = µ
E(S 2 ) = σ 2 .
63
• Goal: Not only do we desire to use point estimators θb which are unbiased, but we would also
like for them to have small variability. In other words, when θb “misses” θ, we would like for it
to “not miss by much.” This deals with precision.
• Main point: Accuracy and precision are the two main mathematical characteristics that arise
when evaluating the quality of a point estimator θ.
b We desire point estimators θb which are
unbiased (perfectly accurate) and have small variance (highly precise).
In other words, the standard error is equal to the standard deviation of the sampling distribution
of θ.
b An estimator’s standard error measures the amount of variability in the point estimator θ. b
Therefore,
smaller se(θ)
b ⇐⇒ θb more precise.
σ2
X ∼ N µ, .
n
That is, the sample mean X is an unbiased estimator of the population mean µ.
• This result also shows that the standard error of X (as a point estimator) is
r
σ2 σ
q
se(X) = var(X) = =√ .
n n
We assume
X ∼ N (µ = 1.5, σ 2 = 0.16).
We call this the population distribution, because it describes the distribution of values of X for
all individuals in the population (here, in-traffic drivers).
64
Question: suppose that we take a random sample of n = 5 drivers with times X1 , X2 , ..., X5 . What
is the distribution of the sample mean X?
Solution: If the sample size is n = 5, then with µ = 1.5 and σ 2 = 0.16, we have
σ2
X∼N µ, =⇒ X ∼ N (1.5, 0.032).
n
This distribution describes the values of X we would expect to see in repeated sampling, that is, if
we repeatedly sampled n = 5 individuals from this population of in-traffic drivers and calculated the
sample mean X each time.
Question: Suppose that we take a random sample of n = 25 drivers with times X1 , X2 , ..., X25 .
What is the distribution of the sample mean X?
Solution: If the sample size is n = 25, then with µ = 1.5 and σ 2 = 0.16, we have
σ2
X ∼ N µ, =⇒ X ∼ N (1.5, 0.0064).
n
Central Limit Theorem: Suppose that X1 , X2 , ..., Xn is a random sample from a population
distribution (does not have to be normal distribution) with mean µ and variance σ 2 (not necessarily
a normal distribution). When the sample size n is large, we have
σ2
X ∼ AN µ, .
n
The symbol AN is read “approximately normal.” This result is called the Central Limit Theorem
(CLT).
• Sampling distribution of X from normal distribution guarantees that when the under-
lying population distribution is N (µ, σ 2 ), the sample mean
σ2
X ∼ N µ, .
n
• The Central Limit Theorem says that even if the population distribution is not normal
(Guassian), the sampling distribution of the sample mean X will be approximately norma
when the sample size is sufficiently large.
• The central limit theorem demonstrates that, for the population mean µ, the point estimator
√
X works consistently well in terms of that, se(X) = σ/ n converges to zero as sample size n
increases (which is reasonable, since a larger sample size means that it contain more information
about the population, thus the resulting estimator should be more accurate).
• Waring: The central limit theorem says no matter which distribution the samples are collected
from, the sample mean follows approximately normal when sample size is large. However,
this does not mean that when sample size is large, any distribution becomes normal.
65
0.4
0.3
Population distribution
Sample mean, n=5
Sample mean, n=25
PDF
0.2
0.1
0.0
0 5 10 15 20
Figure 4.4.2: Rat death times. Population distribution: X ∼ Exp(λ = 1/5). Also depicted are the
sampling distributions of X when n = 5 and n = 25.
Example 4.4.2. The time to death for rats injected with a toxic substance, denoted by X (measured
in days), follows an exponential distribution with λ = 1/5. That is,
X ∼ Exp(λ = 1/5).
This is the population distribution, that is, this distribution describes the time to death for all
rats in the population.
• In Figure 4.4.2, I have shown the Exp(1/5) population distribution (solid curve). I have also
depicted the theoretical sampling distributions of X when n = 5 and when n = 25.
• Main point: Notice how the sampling distribution of X begins to (albeit distantly) resemble
a normal distribution when n = 5. When n = 25, the sampling distribution of X looks very
much to be normal. This is precisely what is conferred by the CLT. The larger the sample size
n, the better a normal distribution approximates the true sampling distribution of X.
66
Example 4.4.3. When a batch of a certain chemical product is prepared, the amount of a particular
impurity in the batch (measured in grams) is a random variable X with the following population
parameters:
µ = 4.0g
σ 2 = (1.5g)2 .
Suppose that n = 50 batches are prepared (independently). What is the probability that the sample
mean impurity amount X is greater than 4.2 grams?
Solution: With n = 50, µ = 4, and σ 2 = (1.5)2 , the CLT says that
σ2
X ∼ AN µ, =⇒ X ∼ AN (4, 0.045).
n
Therefore,
√
P (X > 4.2) ≈ normalcdf(4.2,1099 ,4, 0.045) = 0.1728893.
Important: Note that in making this (approximate) probability calculation, we never made an
assumption about the underlying population distribution shape.
Quantiles of the standard normal distribution. Recall in Section 3.10, I have introduced that,
when X ∼ N (µ, σ 2 ), Find x such that P (X > x) = α can use the commend “invNorm(1 − α, µ, σ).”
Thus, for standard normal distribution; i.e., Z ∼ N (0, 1), we denote its upper α-th quantile as zα
which can be calculated as
zα = invNorm(1 − α, 0, 1).
Based on the symmetric of standard normal distribution (with respect to 0), we have
67
0.4
0.3
f(z)
0.2
0.1
0.0
−3 −2 −1 0 1 2 3
Figure 4.5.3: N (0, 1) pdf. The upper 0.025 and lower 0.025 areas have been shaded. The associated
quantiles are the upper 0.025-th quantile as z0.025 ≈ 1.96 and the lower 0.025-th quantile as −z0.025 ≈
−1.96, respectively.
For example, if α = 0.05 (see Figure 4.5.3), we know that the upper α/2-th quantile and the
lower α/2-th quantile are
It means that the probability of a standard random variable Z follows in the interval (−zα/2 , zα/2 )
is 1 − α. When α = 0.05, we have the probability of a standard random variable Z follows in the
interval (−1.96, 1.96) is 95%.
Example 4.5.1. When α = 0.01, find the upper and lower α/2-th quantiles.
68
4.5.1 Known population variance σ 2
To get things started, we will assume that X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) popu-
lation distribution. We will assume that
E(X) = µ.
However, reporting X alone does not acknowledge that there is variability attached to this estimator.
For example, in Example 4.4.3, for with the n = 50 measured amount of impurity, reporting
x ≈ 4.099 g
as an estimate of the population mean µ does not account for the fact that
• the 50 batches measured were drawn randomly from a population of all pipes, and
• different samples would give different sets of pipes (and different values of x).
In other words, using a point estimator only ignores important information; namely, how variable
the population of the amount of impurity in a batch is.
To avoid this problem (i.e., to account for the uncertainty in the sampling procedure), we therefore
pursue the topic of interval estimation (also known as confidence intervals). The main difference
between a point estimate and an interval estimate is that
• a point estimate is a “one-shot guess” at the value of the parameter; this ignores the variability
in the estimate.
We start our discussion by revisiting the sampling distribution of X from normal distribution in
the last subsection. Recall that if X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) distribution,
then the sampling distribution of X is
σ2
X ∼ N µ,
n
and therefore
X −µ
Z= √ ∼ N (0, 1).
σ/ n
69
Now we start building a confidence interval estimator of µ. In general, for any value of α, 0 < α < 1,
We call
σ σ
X − zα/2 √ , X + zα/2 √
n n
a 100 × (1 − α) percent confidence interval for the population mean µ. This is sometimes written
(more succinctly) as
σ
X ± zα/2 √ .
n
Many confidence intervals we will study follow this same general form.
“We are 100(1 − α) percent confident that the population mean µ is in this interval.”
• Unfortunately, the word “confident” does not mean “probability.” The term “confidence” in
confidence interval means that if we were able to sample from the population over and over
again, each time computing a 100(1 − α) percent confidence interval for µ, then 100(1 − α)
percent of the intervals we would compute would contain the population mean µ.
• That is, “confidence” refers to “long term behavior” of many intervals; not probability for a
single interval. Because of this, we call 100(1 − α) the confidence level. Typical confidence
levels are
70
• The length of the 100(1 − α) percent confidence interval
σ
X ± zα/2 √
n
is equal to
σ
2zα/2 √ .
n
Therefore,
– the larger the sample size n, the smaller the interval length.
– the larger the population variance σ 2 , the larger the interval length.
– the larger the confidence level 100(1 − α), the larger the interval length.
Clearly, shorter confidence intervals are preferred. They are more informative!
Example 4.5.2. Civil engineers have found that the ability to see and read a sign at night depends
in part on its “surround luminance;” i.e., the light intensity near the sign. The data below are
n = 30 measurements of the random variable Y , the surround luminance (in candela per m2 ). The
30 measurements constitute a random sample from all signs in a large metropolitan area.
10.9 1.7 9.5 2.9 9.1 3.2 9.1 7.4 13.3 13.1
6.6 13.7 1.5 6.3 7.4 9.9 13.6 17.3 3.6 4.9
13.1 7.8 10.3 10.3 9.6 5.7 2.6 15.1 2.9 16.2
Based on past experience, the engineers assume a normal population distribution (for the population
of all signs) with known population variance σ 2 = 20.
Question. Find a 90 percent confidence interval for µ, the mean surround luminance.
Solution. We first use TI-84 to calculate the sample mean x. Put numbers in a list L1 , then call
“1-Var Stats.” It gives us x = 8.62. For a 90 percent confidence level; i.e., with α = 0.10, we use
This can be determined from TI-84: invNorm(1-.05,0,1)≈ 1.645. With n = 30 and σ 2 = 20, a 90
percent confidence interval for the mean surround luminance µ is
√ !
σ 20
x ± zα/2 √ =⇒ 8.62 ± 1.645 √ =⇒ (7.28, 9.96) candela/m2 .
n 30
Interpretation (no credits if no interpretation): We are 90 percent confident that the mean
surround luminance µ for all signs in the population is between 7.28 and 9.96 candela/m2 .
71
One-sided confidence bounds: A 100(1 − α)% upper-confidence bound for µ is
√
µ ≤ x + zα σ/ n
Example 4.5.3. ASTM Standard E23 defines standard test methods for notched bar impact testing
of metallic materials. The Charpy V-notch (CVN) technique measures impact energy and is often
used to determine whether or not a material experiences a ductile-to-brittle transition with decreasing
temperature. Ten measurements of impact energy (J ) on specimens of A238 steel cut at 60o C are
as follows: 64.1, 64.7, 64.5, 64.6, 64.5, 64.3, 64.6, 64.8, 64.2, and 64.3. Assume that impact energy is
normally distributed with σ = 1J.
Question. Construct a lower, one-sided 95% confidence interval for the mean impact energy.
Solution. The interval is
√ 1
x − zα σ/ n ≤ µ ⇒ 64.46 − 1.64 √ ≤ µ ⇒ 63.94 ≤ µ.
10
Practical Interpretation: The lower limit for the two-sided 95% confidence interval in last example
was 63.84. Because zα < zα/2 , the lower limit of a one-sided interval is always greater than the lower
limit of a two-sided interval of equal confidence. The one-sided interval does not bound µ from above
so that it still achieves 95% confidence with a slightly larger lower limit. If our interest is only in the
lower limit for µ, then the one-sided interval is preferred because it provides equal confidence with a
greater limit. Similarly, a one-sided upper limit is always less than a two-sided upper limit of equal
confidence.
Note that, we assume that we know the population distribution is normal. And based on nor-
mality, we have all the confidence intervals derived. However, what if we do not know that? Is this
a serious cause for concern? Probably not. Recall that even if the population distribution (here, the
distribution of all light intensity measurements in the city) is not perfectly normal, we still have
σ2
X ∼ AN µ, ,
n
for n large, by the Central Limit Theorem. Therefore, our confidence interval is still approximately
valid. A sample of size n = 30 is “pretty large.” In other words, at n = 30, the CLT approximation
above is usually “kicking in” rather well unless the underlying population distribution is grossly
skewed (and I mean very grossly).
72
4.5.2 Sample size determination
Suppose that X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) population, where σ 2 is known. In
this (known σ 2 ) situation, recall that a 100(1 − α) percent confidence interval for µ is given by
σ
X ± zα/2 √ .
n
| {z }
=E, say
FORMULA: In the setting described above, it is possible to determine the sample size n necessary
once we specify these three pieces of information:
• the value of σ 2 (or an educated guess at its value; e.g., from past information, etc.)
Choice of Sample Size: If x is used as an estimate of µ, we can be 100(1 − α)% confident that
the error |x − µ| will not exceed a specified amount E when the sample size is
z
α/2 σ
2
n=
E
Note that n must be an integer. Herein, always round your calculation up.
Example 4.5.4. In a biomedical experiment, we would like to estimate the population mean re-
maining life µ of healthy rats that are given a certain dose of a toxic substance. Suppose that we
would like to write a 95 percent confidence interval for µ with a margin of error equal to E = 2 days.
From past studies, remaining rat lifetimes have been approximated by a normal distribution with
standard deviation σ = 8 days. How many rats should we use for the experiment?
Solution. With z0.05/2 = z0.025 ≈ 1.96, E = 2, and σ = 8, the desired sample size to estimate µ is
z 2
α/2 σ
2 1.96 × 8
n= = ≈ 61.46 ≈ 62.
B 2
73
4.5.3 Unknown population variance σ 2
In last section, we assume that we know the value of the population variance, but what if we do not
know (which is a more common)?
What if σ 2 is unknown: Suppose that X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) distri-
bution. Result 1 says the sample mean X has the following sampling distribution:
σ2
X ∼ N µ, .
n
If we standardize X, we obtain
X −µ
Z= √ ∼ N (0, 1).
σ/ n
Replacing the population standard deviation σ with the sample standard deviation S, we get a new
sampling distribution:
X −µ
t= √ ∼ t(n − 1),
S/ n
a t distribution with degrees of freedom ν = n − 1.
0.4
N(0,1)
0.3
t(2)
t(10)
f(y)
0.2
0.1
0.0
−4 −2 0 2 4
Figure 4.5.4: Probability density functions of N (0, 1), t(2), and t(10).
74
The t distribution has the following characteristics:
• It is continuous and symmetric about 0 (just like the standard normal distribution).
• As ν → ∞, t(ν) → N (0, 1); thus, when ν becomes larger, the t(ν) and the N (0, 1) distributions
look more alike.
• When compared to the standard normal distribution, the t distribution, in general, is less
peaked and has more probability (area) in the tails.
• The t pdf formula is complicated and is unnecessary for our purposes. R will compute t
probabilities and quantiles from the t distribution.
We continue to assume that X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) population distribu-
tion.
• Our goal is the same; namely, to write a 100(1−α) percent confidence interval for the population
mean µ.
• However, we will no longer make the (rather unrealistic) assumption that population variance
σ 2 is known.
Recall: If you look back in the notes at the “known σ 2 case,” you will see that to derive a 100(1 − α)
percent confidence interval for µ, we started with the following distributional result:
X −µ
Z= √ ∼ N (0, 1).
σ/ n
σ
X ± zα/2 √ .
n
The obvious problem is that, because σ 2 is now unknown, we can not calculate the interval. Not to
worry; we just need a different starting point. Recall that
X −µ
t= √ ∼ t(n − 1),
S/ n
where S is the sample standard deviation (a point estimator for the population standard deviation).
This result is all we need; in fact, it is straightforward to reproduce the “known σ 2 ” derivation and
tailor it to this (now more realistic) case. A 100(1 − α) percent confidence interval for µ is given
by
S
X ± tn−1,α/2 √ .
n
75
The symbol tn−1,α/2 denotes the upper α/2 quantile from a t distribution with ν = n − 1 degrees
of freedom.
“We are 100(1 − α) percent confident that the population mean µ is in this interval.”
Example 4.5.5. Acute exposure to cadmium produces respiratory distress and kidney and liver
damage (and possibly death). For this reason, the level of airborne cadmium dust and cadmium
oxide fume in the air, denoted by X (measured in milligrams of cadmium per m3 of air), is closely
monitored. A random sample of n = 35 measurements from a large factory are given below:
Based on past experience, engineers assume a normal population distribution (for the population of
all cadmium measurements).
Question. Find a 99 percent confidence interval for µ, the mean level of airborne cadmium.
Solution. Input this dataset into Calculator List L1 . Then press stat, go to tests, and choose
TInterval. Input, choose data, List is L1 , Freq: 1, C-level: 0.99 (since the question is asking for
a 99% confidence interval). Press Calculate. The results are
Interpretation: We are 99 percent confident that the population mean level of airborne cadmium
µ is between 0.044 and 0.054 mg/m3 .
76
4.6 Confidence interval for a population proportion p
We now switch gears and focus on a new parameter: the population proportion p. This parameter
emerges when the characteristic we measure on each individual is binary (i.e., only 2 outcomes
possible). Here are some examples:
To start our discussion, we need to recall the Bernoulli trial assumptions for each individual in the
sample:
3. the probability of “success,” denoted by p, 0 < p < 1, is the same for every individual.
Recall: If the individual success/failure statuses in the sample adhere to the Bernoulli trial assump-
tions, then
Y = the number of successes out of n sampled individuals
follows a binomial distribution, that is, Y ∼ B(n, p). The statistical problem at hand is to use the
information in Y to estimate p.
pb =
the sample proportion. This statistic is simply the proportion of “successes” in the sample (out
of n individuals).
77
Fairly simple arguments can be used to show the following results:
E(b
p) =
se(b
p) =
The first result says that the sample proportion pb is an unbiased estimator of the population
proportion p. The second (standard error) result quantifies the precision of pb as an estimator of p.
Sample Distribution: Knowing the sampling distribution of pb is critical if we are going to formalize
statistical inference procedures for p. In this situation, we appeal to an approximate result (conferred
by the CLT) which says that
p(1 − p)
pb ∼ AN p, ,
n
when the sample size n is large.
• This interval should be used only when the sample size n is “large.” A common rule of thumb
(to use this interval formula) is to require
nb
p ≥ 5
n(1 − pb) ≥ 5.
Under these conditions, the CLT should adequately approximate the true sampling distribution
of pb, thereby making the confidence interval formula above approximately valid.
“We are 100(1 − α) percent confident that the population proportion p is in this
interval.”
• The value zα/2 is the upper α/2 quantile from the N (0, 1) distribution.
78
Example 4.6.1. One source of water pollution is gasoline leakage from underground storage tanks.
In Pennsylvania, a random sample of n = 74 gasoline stations is selected and the tanks are inspected;
10 stations are found to have at least one leaking tank. Calculate a 95 percent confidence interval
for p, the population proportion of gasoline stations with at least one leaking tank.
Question: Suppose that we would like to write a 100(1 − α) percent confidence interval for p, a
population proportion. We know that
r
pb(1 − pb)
pb ± zα/2
n
is an approximate 100(1 − α) percent confidence interval for p. What sample size n should we
use?
Sample size determination: To determine the necessary sample size, we first need to specify two
pieces of information:
• the confidence level 100(1 − α)
A small problem arises. Note that B depends on pb. Unfortunately, pb can only be calculated once we
know the sample size n. We overcome this problem by replacing pb with p0 , an a priori guess at its
value. The last expression becomes
r
p0 (1 − p0 )
E = zα/2 .
n
This is the desired sample size n to find a 100(1−α) percent confidence interval for p with a prescribed
margin of error (roughly) equal to E. I say “roughly,” because there may be additional uncertainty
arising from our use of p0 (our best guess).
79
Conservative approach: If there is no sensible guess for p available, use p0 = 0.5. In this situation,
the resulting value for n will be as large as possible. Put another way, using p0 = 0.5 gives the most
conservative solution (i.e., the largest sample size, n). This is true because
z 2
α/2
n = n(p0 ) = p0 (1 − p0 ),
E
n=
Example 4.6.2. You have been asked to estimate the proportion of raw material (in a certain
manufacturing process) that is being “scrapped;” e.g., the material is so defective that it can not
be reworked. If this proportion is larger than 10 percent, this will be deemed (by management) to
be an unacceptable continued operating cost and a substantial process overhaul will be performed.
Past experience suggests that the scrap rate is about 5 percent, but recent information suggests that
this rate may be increasing.
Question. You would like to write a 95 percent confidence interval for p, the population proportion
of raw material that is to be scrapped, with a margin of error equal to E = 0.02. How many pieces
of material should you ask to be sampled?
Solution. For 95 percent confidence, we need z0.05/2 = z0.025 ≈ 1.96. In providing an initial guess,
we have options; we could use
As we can see, the “guessed” value of p0 has a substantial impact on the final sample size calculation.
80
4.7 Confidence interval for a population variance σ 2
MOTIVATION : In many situations, one is concerned not with the mean of an underlying (continu-
ous) population distribution, but with the variance σ 2 instead. If σ 2 is excessively large, this could
point to a potential problem with a manufacturing process, for example, where there is too much
variation in the measurements produced. In a laboratory setting, chemical engineers might wish
to estimate the variance σ 2 attached to a measurement system (e.g., scale, caliper, etc.). In field
trials, agronomists are often interested in comparing the variability levels for different cultivars or
genetically-altered varieties. In clinical trials, physicians are often concerned if there are substantial
differences in the variation levels of patient responses at different clinic sites.
Suppose that X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) distribution. The quantity
(n − 1)S 2
Q= ∼ χ2 (n − 1),
σ2
• The χ2 pdf formula is unnecessary for our purposes. R will compute χ2 probabilities and
quantiles from the χ2 distribution.
0.15
Chi−square; df = 5
Chi−square; df = 10
0.10
Chi−square; df = 20
f(q)
0.05
0.00
0 10 20 30 40
81
Goal: Suppose that X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) distribution. We would
like to write a 100(1 − α) percent confidence interval for σ 2 .
Notation: Let χ2n−1,α/2 denote the upper α/2 quantile and let χ2n−1,1−α/2 denote the lower α/2
quantile of the χ2 (n − 1) distribution; i.e., χ2n−1,α/2 and χ2n−1,1−α/2 satisfy
respectively. Note that, unlike the N (0, 1) and t distributions, the χ2 distribution is not symmetric.
Therefore, different notation is needed to identify the quantiles of χ2 distributions (this is nothing to
get worried about). Use the chi-square table to find these values. (Unfortunately, TI-84
cannot help you in this case).
Example 4.7.1. Using the chi-square table, find the following values: χ2n−1,α/2 , and χ2n−1,1−α/2 , for
all combination of n = 10, 20, 30 and α = 0.1, 0.05, 0.01.
is a 100(1 − α) percent confidence interval for the population variance σ 2 . We interpret the
interval in the same way.
“We are 100(1 − α) percent confident that the population variance σ 2 is in this interval.”
82
Chi-Square Distribution Table
0 χ2
df χ2.995 χ2.990 χ2.975 χ2.950 χ2.900 χ2.100 χ2.050 χ2.025 χ2.010 χ2.005
1 0.000 0.000 0.001 0.004 0.016 2.706 3.841 5.024 6.635 7.879
2 0.010 0.020 0.051 0.103 0.211 4.605 5.991 7.378 9.210 10.597
3 0.072 0.115 0.216 0.352 0.584 6.251 7.815 9.348 11.345 12.838
4 0.207 0.297 0.484 0.711 1.064 7.779 9.488 11.143 13.277 14.860
5 0.412 0.554 0.831 1.145 1.610 9.236 11.070 12.833 15.086 16.750
6 0.676 0.872 1.237 1.635 2.204 10.645 12.592 14.449 16.812 18.548
7 0.989 1.239 1.690 2.167 2.833 12.017 14.067 16.013 18.475 20.278
8 1.344 1.646 2.180 2.733 3.490 13.362 15.507 17.535 20.090 21.955
9 1.735 2.088 2.700 3.325 4.168 14.684 16.919 19.023 21.666 23.589
10 2.156 2.558 3.247 3.940 4.865 15.987 18.307 20.483 23.209 25.188
11 2.603 3.053 3.816 4.575 5.578 17.275 19.675 21.920 24.725 26.757
12 3.074 3.571 4.404 5.226 6.304 18.549 21.026 23.337 26.217 28.300
13 3.565 4.107 5.009 5.892 7.042 19.812 22.362 24.736 27.688 29.819
14 4.075 4.660 5.629 6.571 7.790 21.064 23.685 26.119 29.141 31.319
15 4.601 5.229 6.262 7.261 8.547 22.307 24.996 27.488 30.578 32.801
16 5.142 5.812 6.908 7.962 9.312 23.542 26.296 28.845 32.000 34.267
17 5.697 6.408 7.564 8.672 10.085 24.769 27.587 30.191 33.409 35.718
18 6.265 7.015 8.231 9.390 10.865 25.989 28.869 31.526 34.805 37.156
19 6.844 7.633 8.907 10.117 11.651 27.204 30.144 32.852 36.191 38.582
20 7.434 8.260 9.591 10.851 12.443 28.412 31.410 34.170 37.566 39.997
21 8.034 8.897 10.283 11.591 13.240 29.615 32.671 35.479 38.932 41.401
22 8.643 9.542 10.982 12.338 14.041 30.813 33.924 36.781 40.289 42.796
23 9.260 10.196 11.689 13.091 14.848 32.007 35.172 38.076 41.638 44.181
24 9.886 10.856 12.401 13.848 15.659 33.196 36.415 39.364 42.980 45.559
25 10.520 11.524 13.120 14.611 16.473 34.382 37.652 40.646 44.314 46.928
26 11.160 12.198 13.844 15.379 17.292 35.563 38.885 41.923 45.642 48.290
27 11.808 12.879 14.573 16.151 18.114 36.741 40.113 43.195 46.963 49.645
28 12.461 13.565 15.308 16.928 18.939 37.916 41.337 44.461 48.278 50.993
29 13.121 14.256 16.047 17.708 19.768 39.087 42.557 45.722 49.588 52.336
30 13.787 14.953 16.791 18.493 20.599 40.256 43.773 46.979 50.892 53.672
40 20.707 22.164 24.433 26.509 29.051 51.805 55.758 59.342 63.691 66.766
50 27.991 29.707 32.357 34.764 37.689 63.167 67.505 71.420 76.154 79.490
60 35.534 37.485 40.482 43.188 46.459 74.397 79.082 83.298 88.379 91.952
70 43.275 45.442 48.758 51.739 55.329 85.527 90.531 95.023 100.425 104.215
80 51.172 53.540 57.153 60.391 64.278 96.578 101.879 106.629 112.329 116.321
90 59.196 61.754 65.647 69.126 73.291 107.565 113.145 118.136 124.116 128.299
100 67.328 70.065 74.222 77.929 82.358 118.498 124.342 129.561 135.807 140.169
83
Important: A 100(1 − α) percent confidence interval for the population standard deviation σ
arises from simply taking the square root of the endpoints of the σ 2 interval. That is,
s s !
(n − 1)S 2 (n − 1)S 2
,
χ2n−1,α/2 χ2n−1,1−α/2
is a 100(1 − α) percent confidence interval for the population standard deviation σ. In practice, this
interval may be preferred over the σ 2 interval, because standard deviation is a measure of variability
in terms of the original units (e.g., dollars, inches, days, etc.). The variance is measured in squared
units (e.g., dollars2 , in2 , days2 , etc.).
Major warning: Unlike the z and t confidence intervals for a population mean µ, the χ2 interval for
σ 2 (and for σ) is not robust to departures from normality. If the underlying population distribution
is non-normal (non-Guassian), then the confidence interval formulas for σ 2 and σ are not to be used.
Therefore, it is very important to “check” the normality assumption with these interval procedures
(e.g., use a qq-plot which will be introduced later).
Example 4.7.2. Indoor swimming pools are noted for their poor acoustical properties. Suppose
your goal is to design a pool in such a way that
• the population mean time that it takes for a low-frequency sound to die out is µ = 1.3 seconds
• the population standard deviation for the distribution of die-out times is σ = 0.6 seconds.
Computer simulations of a preliminary design are conducted to see whether these standards are being
met; here are data from n = 20 independently-run simulations. The data are obtained on the time
(in seconds) it takes for the low-frequency sound to die out.
1.34 2.56 1.28 2.25 1.84 2.35 0.77 1.84 1.80 2.44
0.86 1.29 0.12 1.87 0.71 2.08 0.71 0.30 0.54 1.48
Question. Find a 95 percent confidence interval for the population standard deviation of times
σ. What does this interval suggest about whether the preliminary design conforms to specifications
(with respect to variability)?
84
4.8 Statistical Hypotheses
In the previous sections we have discussed how one could use data, observed from an experiment, to
estimate unknown parameters. Often, we will also wish to use this data to make a decision about a
statement about the parameters; e.g.,
• The mean number of car accidents that occur in Atlanta during rush hour on a given day is
greater than 3.
• The mean number of car accidents that occur in Atlanta during rush hour on a given day is
less than 5.
• The proportion of people who voted Democrat in the last election is greater than the proportion
of people who voted Republican.
In all of these situations, we are forming a hypothesis about the structure of the underlying popula-
tion(s) and we will develop data driven techniques that will allow us to decide whether or not our
hypothesis is reasonable.
Example 4.8.1. Suppose that we are interested in the burning rate of solid propellant used to power
aircrew escape systems; burning rate is a random variable that can be described by a probability
distribution. Suppose that we are interested in the mean burning rate (a parameter of this distribu-
tion). Specifically, we are interested in whether or not the mean burning rate is 50cm/s. This can
be formally expressed as
H0 : µ = 50cm/s
Ha : µ 6= 50cm/s.
The statement H0 : µ = 50cm/s is referred to as the null hypothesis, and the statement
Ha : µ = 50cm/s is called the alternative hypothesis. The alternative hypothesis, as expressed
above, is referred to as a two-sided alternative hypothesis since it specifies values of µ both greater and
less than 50cm/s. In some situations we may wish to formulate a one-sided alternative hypothesis;
e.g.,
H0 : µ = 50cm/s
Ha : µ < 50cm/s.
or
H0 : µ = 50cm/s
Ha : µ > 50cm/s.
85
In general, we will collect data and summarize it in an effort to assess the validity of either H0 or
Ha . For example, suppose that a sample of n specimens are tested an that the mean burning rate
x is observed. The sample mean is an estimate of the population mean µ. A value of x that is
“reasonable” with respect to H0 (Ha ) tends to suggest that the null(alternative) hypothesis might be
favorable. So what is reasonable? Consider defining a region such that if x falls in this region we
will decide to reject H0 in favor of Ha , we refer to this region as the critical region, and subsequently
refer to the boundaries of this region as the critical values. For example, we might decide to reject
H0 : µ = 50cm/s if x < 48 or x > 52, consequently the values of x that are less than 48 and greater
than 52 are the critical region and the critical values are 48 and 52.
This decision process can naturally lead one to make either one of the two following types of mistakes:
• Failing to reject the null hypothesis, H0 , when the alternative hypothesis, H1, is in fact true
(Type II error).
Since the decision making process is based on random variables, probabilities can be associated
with the Type I and II errors. Typically the probability of making a type I error is denoted by α,
i.e,
α = P (type I error) = P (Reject H0 when H0 is true) :
The type I error is often referred to as the significance level or size of the test. Likewise, we can
quantify the probability of making a type II error as
Example 4.8.2. The FDA’s policy is that the pharmaceutical company must provide substantial
evidence that a new drug is safe prior to receiving FDA approval, so that the FDA can confidently
certify the safety of the drug to potential customers. The null hypothesis is typically make by the
status quo. For any new drug, FDA always assume it is unsafe. For testing the new drug, the null
hypothesis is: “the new drug is unsafe”; and the alternative hypothesis is “the new drug is safe.”
(a) Given the choice of null and alternative hypothesis, describe type I and II errors in terms of this
application.
(b) If the FDA wants to very confident that the drug is safe before premitting to be marketed, is it
more important that α or 1 − β be small? Explain.
86
A few important points with regard to type I and II errors:
• Type I and II errors are related. A decrease in the probability of a type I error always results
in an increase in the probability of a type II error, and vice versa, provided the sample size n
does not change.
• When the null hypothesis is false, β decreases as the true value of the parameter approaches the
value hypothesized in the null hypothesis. The value of β increases as the difference between
the true parameter and the hypothesized value increases.
Typically, when designing a hypothesis test one controls (or specifies) the type I error probability.
It is common to specify values of 0.10, 0.05, and 0.01 for α, but these values may not be appropriate
in all situations.
There are typically three ways to test hypotheses: the confidence interval approach, the critical
value approach, and the P-value approach.
X ∼ N (µ, σ 2 /n).
To test hypotheses on the mean µ, we use this point estimator and its sample distribution.
In this course, the null hypothesis is always
H0 : µ = µ 0
Ha : µ 6= µ0
Ha : µ > µ0
Ha : µ < µ0
87
Firstly, we introduce the confidence interval approach:
Why the first two approaches are equivalent? We only show it for the two-sided alternative, i.e.,
Ha : µ 6= µ0 .
88
Finally, we have the P-value approach:
Recall that Φ(·) is the cumulative distribution function of the standard normal distribution;
i.e., Φ(z) = P (Z ≤ z).
Why the three approaches are equivalent? We only show the P-value approach and the critical value
approach are the same for the two-sided alternative, i.e., Ha : µ 6= µ0 :
With the same spirit, we can see that these three approaches are essentially the same.
TI- 84 for P-value: stat→tests→ “1: Z-test”. Input has two options: Data or Stats. Like
in confidence intervals, correctly input the rest information, and correctly choose the type of your
alternative hypothesis, press Calculate. The output p is the P-value. Then compare it with α.
Example 4.9.1. Civil engineers have found that the ability to see and read a sign at night depends
in part on its “surround luminance;” i.e., the light intensity near the sign. It is believed that the
mean surround luminance is 10 candela per m2 in a large metropolitan area. The data below are
n = 30 measurements of the random variable X, the surround luminance (in candela per m2 ). The
30 measurements constitute a random sample from all the signs in the large metropolitan area in
89
question:
10.9 1.7 9.5 2.9 9.1 3.2 9.1 7.4 13.3 13.1
6.6 13.7 1.5 6.3 7.4 9.9 13.6 17.3 3.6 4.9
13.1 7.8 10.3 10.3 9.6 5.7 2.6 15.1 2.9 16.2
Based on past experience, the engineers assume a normal population distribution (for the population
of all signs) with known population variance σ 2 = 20. From this data what conclusions should we
draw about the hypothesized mean surround luminance, at the α = 0.05 significance level.
Solution. In order to solve this problem one can complete the following 6-step procedure:
• If choosing the confidence interval approach, for the question, we should use the (two-
sided) 100 × (1 − α)% = 95% confidence interval, which is
[6.9344, 10.19].
x − µ0 8.62 − 10
z0 = √ = √ ≈ −1.69.
σ/ n 4.47/ 20
Since α = 0.05, the Ha : µ 6= µ0 , the rejection region is z0 < −zα2 = −1.96 or z0 > zα/2 =
1.96. We can see that our computed z0 is not in the rejection region.
• If choosing P-value approach: TI-84 shows that P-value is p ≈ 0.091 which is not smaller
than α = 0.05.
5. Compare: since
6. Conclusion: At the 0.05 significance level, the data do not provide sufficient evidence to con-
clude that the mean surround luminance for the specified large metropolitan area differs from
10 candela per m2 .
90
4.10 Testing hypotheses on the mean µ with unknown variance σ 2
In this case, the population variance σ 2 is not given (which is more common in real applications),
like in Section 4.5.3. Suppose X1 , . . . , Xn is a random sample of size n from a population. The
population is normally distributed, (if it is not, the conditions of the central limit theorem apply,
and n is sufficiently large.). Then, we have
X −µ
√ ∼ t(n − 1).
S/ n
Using this fact, we are now able to test the null hypothesis
H0 : µ = µ 0
Ha : µ 6= µ0
Ha : µ > µ0
Ha : µ < µ0
91
2. Find the corresponding rejection region:
The value of tn−1,a for a = α or α/2 can be found in the following T-table in the same spirit of
founding χ2n−1,a via the “Chi-squre Distribution Table”
Recall that Φ(·) is the cumulative distribution function of the standard normal distribution;
i.e., Φ(z) = P (Z ≤ z).
Using similar argument in previous section, we can see that these three approaches are essentially
the same.
TI- 84 for P-value: stat→tests→ “2: T-test”. Input has two options: Data or Stats. Like
in confidence intervals, correctly input the rest information, and correctly choose the type of your
alternative hypothesis, press Calculate. The output p is the P-value. Then compare it with α.
92
t Table upper-tail probability: t Table upper-tail probability:
df .25 .10 .05 .025 .01 .005 df .25 .10 .05 .025 .01 .005
1 1.0000 3.0777 6.314 12.706 31.821 63.657 51 0.6793 1.2984 1.6753 2.0076 2.4017 2.6757
2 0.8165 1.8856 2.9200 4.3027 6.9646 9.925 52 0.6792 1.2980 1.6747 2.0066 2.4002 2.6737
3 0.7649 1.6377 2.3534 3.1824 4.5407 5.8409 53 0.6791 1.2977 1.6741 2.0057 2.3988 2.6718
4 0.7407 1.5332 2.1318 2.7764 3.7469 4.6041 54 0.6791 1.2974 1.6736 2.0049 2.3974 2.6700
5 0.7267 1.4759 2.0150 2.5706 3.3649 4.0321 55 0.6790 1.2971 1.6730 2.0040 2.3961 2.6682
6 0.7176 1.4398 1.9432 2.4469 3.1427 3.7074 56 0.6789 1.2969 1.6725 2.0032 2.3948 2.6665
7 0.7111 1.4149 1.8946 2.3646 2.9980 3.4995 57 0.6788 1.2966 1.6720 2.0025 2.3936 2.6649
8 0.7064 1.3968 1.8595 2.3060 2.8965 3.3554 58 0.6787 1.2963 1.6716 2.0017 2.3924 2.6633
9 0.7027 1.3830 1.8331 2.2622 2.8214 3.2498 59 0.6787 1.2961 1.6711 2.0010 2.3912 2.6618
10 0.6998 1.3722 1.8125 2.2281 2.7638 3.1693 60 0.6786 1.2958 1.6706 2.0003 2.3901 2.6603
11 0.6974 1.3634 1.7959 2.2010 2.7181 3.1058 61 0.6785 1.2956 1.6702 1.9996 2.3890 2.6589
12 0.6955 1.3562 1.7823 2.1788 2.6810 3.0545 62 0.6785 1.2954 1.6698 1.9990 2.3880 2.6575
13 0.6938 1.3502 1.7709 2.1604 2.6503 3.0123 63 0.6784 1.2951 1.6694 1.9983 2.3870 2.6561
14 0.6924 1.3450 1.7613 2.1448 2.6245 2.9768 64 0.6783 1.2949 1.6690 1.9977 2.3860 2.6549
15 0.6912 1.3406 1.7531 2.1314 2.6025 2.9467 65 0.6783 1.2947 1.6686 1.9971 2.3851 2.6536
16 0.6901 1.3368 1.7459 2.1199 2.5835 2.9208 66 0.6782 1.2945 1.6683 1.9966 2.3842 2.6524
17 0.6892 1.3334 1.7396 2.1098 2.5669 2.8982 67 0.6782 1.2943 1.6679 1.9960 2.3833 2.6512
18 0.6884 1.3304 1.7341 2.1009 2.5524 2.8784 68 0.6781 1.2941 1.6676 1.9955 2.3824 2.6501
19 0.6876 1.3277 1.7291 2.0930 2.5395 2.8609 69 0.6781 1.2939 1.6672 1.9949 2.3816 2.6490
20 0.6870 1.3253 1.7247 2.0860 2.5280 2.8453 70 0.6780 1.2938 1.6669 1.9944 2.3808 2.6479
21 0.6864 1.3232 1.7207 2.0796 2.5176 2.8314 71 0.6780 1.2936 1.6666 1.9939 2.3800 2.6469
22 0.6858 1.3212 1.7171 2.0739 2.5083 2.8188 72 0.6779 1.2934 1.6663 1.9935 2.3793 2.6459
23 0.6853 1.3195 1.7139 2.0687 2.4999 2.8073 73 0.6779 1.2933 1.6660 1.9930 2.3785 2.6449
24 0.6848 1.3178 1.7109 2.0639 2.4922 2.7969 74 0.6778 1.2931 1.6657 1.9925 2.3778 2.6439
25 0.6844 1.3163 1.7081 2.0595 2.4851 2.7874 75 0.6778 1.2929 1.6654 1.9921 2.3771 2.6430
26 0.6840 1.3150 1.7056 2.0555 2.4786 2.7787 76 0.6777 1.2928 1.6652 1.9917 2.3764 2.6421
27 0.6837 1.3137 1.7033 2.0518 2.4727 2.7707 77 0.6777 1.2926 1.6649 1.9913 2.3758 2.6412
28 0.6834 1.3125 1.7011 2.0484 2.4671 2.7633 78 0.6776 1.2925 1.6646 1.9908 2.3751 2.6403
29 0.6830 1.3114 1.6991 2.0452 2.4620 2.7564 79 0.6776 1.2924 1.6644 1.9905 2.3745 2.6395
30 0.6828 1.3104 1.6973 2.0423 2.4573 2.7500 80 0.6776 1.2922 1.6641 1.9901 2.3739 2.6387
31 0.6825 1.3095 1.6955 2.0395 2.4528 2.7440 81 0.6775 1.2921 1.6639 1.9897 2.3733 2.6379
32 0.6822 1.3086 1.6939 2.0369 2.4487 2.7385 82 0.6775 1.2920 1.6636 1.9893 2.3727 2.6371
33 0.6820 1.3077 1.6924 2.0345 2.4448 2.7333 83 0.6775 1.2918 1.6634 1.9890 2.3721 2.6364
34 0.6818 1.3070 1.6909 2.0322 2.4411 2.7284 84 0.6774 1.2917 1.6632 1.9886 2.3716 2.6356
35 0.6816 1.3062 1.6896 2.0301 2.4377 2.7238 85 0.6774 1.2916 1.6630 1.9883 2.3710 2.6349
36 0.6814 1.3055 1.6883 2.0281 2.4345 2.7195 86 0.6774 1.2915 1.6628 1.9879 2.3705 2.6342
37 0.6812 1.3049 1.6871 2.0262 2.4314 2.7154 87 0.6773 1.2914 1.6626 1.9876 2.3700 2.6335
38 0.6810 1.3042 1.6860 2.0244 2.4286 2.7116 88 0.6773 1.2912 1.6624 1.9873 2.3695 2.6329
39 0.6808 1.3036 1.6849 2.0227 2.4258 2.7079 89 0.6773 1.2911 1.6622 1.9870 2.3690 2.6322
40 0.6807 1.3031 1.6839 2.0211 2.4233 2.7045 90 0.6772 1.2910 1.6620 1.9867 2.3685 2.6316
41 0.6805 1.3025 1.6829 2.0195 2.4208 2.7012 91 0.6772 1.2909 1.6618 1.9864 2.3680 2.6309
42 0.6804 1.3020 1.6820 2.0181 2.4185 2.6981 92 0.6772 1.2908 1.6616 1.9861 2.3676 2.6303
43 0.6802 1.3016 1.6811 2.0167 2.4163 2.6951 93 0.6771 1.2907 1.6614 1.9858 2.3671 2.6297
44 0.6801 1.3011 1.6802 2.0154 2.4141 2.6923 94 0.6771 1.2906 1.6612 1.9855 2.3667 2.6291
45 0.6800 1.3006 1.6794 2.0141 2.4121 2.6896 95 0.6771 1.2905 1.6611 1.9853 2.3662 2.6286
46 0.6799 1.3002 1.6787 2.0129 2.4102 2.6870 96 0.6771 1.2904 1.6609 1.9850 2.3658 2.6280
47 0.6797 1.2998 1.6779 2.0117 2.4083 2.6846 97 0.6770 1.2903 1.6607 1.9847 2.3654 2.6275
48 0.6796 1.2994 1.6772 2.0106 2.4066 2.6822 98 0.6770 1.2902 1.6606 1.9845 2.3650 2.6269
49 0.6795 1.2991 1.6766 2.0096 2.4049 2.6800 99 0.6770 1.2902 1.6604 1.9842 2.3646 2.6264
50 0.6794 1.2987 1.6759 2.0086 2.4033 2.6778 100 0.6770 1.2901 1.6602 1.9840 2.3642 2.6259
110 0.6767 1.2893 1.6588 1.9818 2.3607 2.6213
120 0.6765 1.2886 1.6577 1.9799 2.3578 2.6174
∞ 0.6745 1.2816 1.6449 1.9600 2.3264 2.5758
93
Example 4.10.1. Civil engineers have found that the ability to see and read a sign at night depends
in part on its “surround luminance;” i.e., the light intensity near the sign. It is believed that the
mean surround luminance is 10 candela per m2 in a large metropolitan area. The data below are
n = 30 measurements of the random variable X, the surround luminance (in candela per m2 ). The
30 measurements constitute a random sample from all the signs in the large metropolitan area in
question:
10.9 1.7 9.5 2.9 9.1 3.2 9.1 7.4 13.3 13.1
6.6 13.7 1.5 6.3 7.4 9.9 13.6 17.3 3.6 4.9
13.1 7.8 10.3 10.3 9.6 5.7 2.6 15.1 2.9 16.2
Based on past experience, the engineers assume a normal population distribution (for the population
of all signs). From this data what conclusions should we draw about the hypothesized mean surround
luminance, at the α = 0.05 significance level.
94
4.11 Testing hypotheses on the population proportion p
To test the null hypothesis
H0 : p = p0
Ha : p 6= p0
Ha : p > p0
Ha : p < p0
To estimate the population proportion p, we have our point estimator; i.e., sample proportion p̂.
When np̂ ≥ 5 and n(1 − p̂) ≥ 5, the CLT says we have
p(1 − p)
p̂ ∼ AN p,
n
To control the type I error, recalling that the type I error is under the assumption H0 is in fact true.
So when H0 is true, we have p = p0 . Thus,
p0 (1 − p0 )
p̂ ∼ AN p0 ,
n
Then
p̂ − p0
z0 = p ∼ AN (0, 1).
p0 (1 − p0 )/n
Based on this, we can design the testing procedures as the following.
Firstly, we introduce the critical value approach:
95
(less than, greater than, or differs from) .
Recall that Φ(·) is the cumulative distribution function of the standard normal distribution;
i.e., Φ(z) = P (Z ≤ z).
TI- 84 for P-value: stat→tests→ “5: 1-PropZTest”. Like in confidence intervals, correctly input
the rest information, and correctly choose the type of your alternative hypothesis, press Calculate.
The output p is the P-value. Then compare it with α.
Example 4.11.1. Suppose that 500 parts are tested in manufacturing and 10 are rejected. Using
both the approaches (critical value approach and P-value approach) to test the hypothesis H0 : p =
0.03 against Ha : p < 0.03 at significance level α = 0.05.
96
4.12 Testing hypotheses on the population variance σ 2
In Section 4.7, we have introduced that
Suppose that X1 , X2 , ..., Xn is a random sample from a N (µ, σ 2 ) distribution (normality is important
in this case). The quantity
(n − 1)S 2
Q= ∼ χ2 (n − 1),
σ2
a χ2 distribution with ν = n − 1 degrees of freedom.
Using this fact, we are now able to test the null hypothesis
H0 : σ = σ0 ( or H0 : σ 2 = σ02 )
Ha : σ 6= σ0 ( or Ha : σ 2 6= σ02 )
Ha : σ > σ0 ( or Ha : σ 2 > σ02 )
Ha : σ < σ0 ( or Ha : σ 2 < σ02 )
Since one can easily transform testing hypotheses on the population standard deviation σ to testing
on the population variance σ 2 . Thus, we only need know how to test on σ 2 . We have the confidence
interval approach and the critical value approach. Unfortunately, TI-84 cannot help us on this case.
Thus the P-value approach is not required.
(n−1)s2
Ha : σ 2 > σ02 χ2n−1,α
> σ02 : σ02 is less than the lower confidence bound
(n−1)s2
Ha : σ 2 < σ02 χ2n−1,1−α
< σ02 : σ02 is larger than the upper confidence bound
97
Secondly, we introduce the critical value approach:
The value of χ2n−1,a can be found via the “Chi-square Distribution Table”
Example 4.12.1. An automated filling machine is used to fill bottles with liquid detergent. A
random sample of 20 bottles results in a sample variance of fill volume of s2 = 0.0153 (fluid ounces)2 .
If the variance of fill volume exceeds 0.01 (fluid ounces)2 , an unacceptable proportion of bottles will
be underfilled or overfilled. Is there evidence in the sample data to suggest that the manufacturer
has a problem with underfilled or overfilled bottles? Use α = 0.05, and assume that fill volume has
a normal distribution (use both the confidence interval and critical value approaches).
98
5 Two-sample Statistical Inference
5.1 For the difference of two population means µ1 − µ2 : Independent samples
REMARK : In practice, it is very common to compare the same characteristic (mean, proportion,
variance) from two different distributions. For example, we may wish to compare
• the mean starting salaries of male and female engineers (compare µ1 and µ2 )
• the proportion of scrap produced from two manufacturing processes (compare p1 and p2 )
• the variance of sound levels from two indoor swimming pool designs (compare σ12 and σ22 ).
Our previous work is applicable only for a single distribution (i.e., a single mean µ, a single proportion
p, and a single variance σ 2 ). We therefore need to extend these procedures to handle two distributions.
We start with comparing two means.
GOAL: Construct statistical inference, including a 100(1 − α) percent confidence interval and hy-
pothesis test at significance level α for the difference of population means µ1 − µ2 .
99
5.1.1 Known variance case: both σ12 and σ22 are known
Goal: We want to write a confidence interval for µ1 −µ2 , but how this interval is constructed depends
on the values of σ12 and σ22 . In particular, we consider three cases:
• We do not know the values of σ12 or σ22 . However, we know σ12 = σ22 ; that is, the two population
variances are equal.
• We do not know the values of σ12 or σ22 , also σ12 6= σ22 ; that is, the two population variances are
not equal.
We first consider the equal variance case. Addressing this case requires us to start with the
following (sampling) distribution result:
(X 1· − X 2· ) − (µ1 − µ2 )
Z= q 2 ∼
σ1 σ22
n1 + n2
• The sampling distribution Z ∼ N (0, 1) should suggest to you that confidence interval quantiles
will come from the standard normal distribution; note that this distribution depends on the
sample sizes from both samples.
• In particular, because Z ∼ N (0, 1), we can find the value zα/2 that satisfies
• Substituting Z into the last expression and performing algebraic manipulations, we obtain
100
“We are 100(1 − α) percent confident that the population mean difference µ1 − µ2 is
in this interval.”
H0 : µ1 − µ2 = ∆0
H0 : µ1 − µ2 6= ∆0
H0 : µ1 − µ2 > ∆0
H0 : µ1 − µ2 < ∆0
101
Example 5.1.1. In the vicinity of a nuclear power plant, environmental engineers from the EPA
would like to determine if there is a difference between the mean weight in fish (of the same species)
from two locations. Independent samples are taken from each location and the following weights (in
ounces) are observed:
Location 1: 21.9 18.5 12.3 16.7 21.0 15.1 18.2 23.0 36.8 26.6
Location 2: 22.0 20.6 15.4 17.9 24.4 15.6 11.4 17.5
(a) Construct a 90 percent confidence interval for the mean difference µ1 − µ2 . Here, µ1 (µ2 ) denotes
the population mean weight of all fish at location 1 (2).
(b) Test whether the two distribution have the same mean at α = 0.05.
102
5.1.2 Unknown variance case, but we know they are equal; i.e., σ12 = σ22
We now consider the equal variance (but unknown) case. Addressing this case requires us to start
with the following (sampling) distribution result:
(X 1· − X 2· ) − (µ1 − µ2 )
T = r ∼ t(n1 + n2 − 2),
2 1 1
Sp n1 + n2
where
(n1 − 1)S12 + (n2 − 1)S22
Sp2 = .
n1 + n2 − 2
Some comments are in order:
• The statistic Sp2 is called the pooled sample variance estimator of the common population
variance, say, σ 2 . Algebraically, it is simply a weighted average of the two sample variances S12
and S22 (where the weights are functions of the sample sizes n1 and n2 ).
• The sampling distribution T ∼ t(n1 + n2 − 2) should suggest to you that confidence interval
quantiles will come from this t distribution; note that this distribution depends on the sample
sizes from both samples.
• In particular, because T ∼ t(n1 + n2 − 2), we can find the value tn1 +n2 −2,α/2 that satisfies
• Substituting T into the last expression and performing algebraic manipulations, we obtain
s
1 1
(X 1· − X 2· ) ± tn1 +n2 −2,α/2 Sp2 + .
n1 n2
103
“We are 100(1 − α) percent confident that the population mean difference µ1 − µ2 is
in this interval.”
H0 : µ1 − µ2 = ∆0
104
40
30
Weight (in ounces)
20
10
0
Location 1 Location 2
Example 5.1.2. In the vicinity of a nuclear power plant, environmental engineers from the EPA
would like to determine if there is a difference between the mean weight in fish (of the same species)
from two locations. Independent samples are taken from each location and the following weights (in
ounces) are observed:
Location 1: 21.9 18.5 12.3 16.7 21.0 15.1 18.2 23.0 36.8 26.6
Location 2: 22.0 20.6 15.4 17.9 24.4 15.6 11.4 17.5
(a) Construct a 90 percent confidence interval for the mean difference µ1 − µ2 . Here, µ1 (µ2 ) denotes
the population mean weight of all fish at location 1 (2).
(b) Test whether the two distribution have the same mean at α = 0.05.
105
5.1.3 Unknown and unequal variance case: σ12 6= σ22
REMARK : When σ12 6= σ22 , the problem of constructing a 100(1 − α) percent confidence interval for
µ1 − µ2 becomes more difficult theoretically. However, we can still write an approximate confidence
interval.
An approximate 100(1 − α) percent confidence interval µ1 − µ2 is given by
s
S12 S22
(X 1· − X 2· ) ± tν,α/2 + ,
n1 n2
• No one in their right mind would calculate this interval “by hand” (particularly nasty is the
formula for ν). TI-84: Stat: Tests: 2-SampTInt (Pooled: No) will produce the interval on
request.
“We are 100(1 − α) percent confident that the population mean difference µ1 − µ2 is
in this interval.”
106
• Testing hypothesis: Following the six steps:
H0 : µ1 − µ2 = ∆0
Remark: In the last two subsections, we have presented two confidence intervals for µ1 − µ2 . One
assumes σ12 = σ22 (equal variance assumption) and one that assumes σ12 6= σ22 (unequal variance
assumption). If you are unsure about which interval to use, go with the unequal variance
interval. The penalty for using it when σ12 = σ22 is much smaller than the penalty for using the
equal variance interval when σ12 6= σ22 .
107
8
6
Weight (in 100s lb)
4
2
0
Plant 1 Plant 2
Figure 5.1.2: Boxplots of discarded white paper amounts (in 100s lb) in Example 5.1.3.
Example 5.1.3. You are part of a recycling project that is examining how much paper is being
discarded (not recycled) by employees at two large plants. These data are obtained on the amount
of white paper thrown out per year by employees (data are in hundreds of pounds). Samples of
employees at each plant were randomly selected.
Plant 1: 3.01 2.58 3.04 1.75 2.87 2.57 2.51 2.93 2.85 3.09
1.43 3.36 3.18 2.74 2.25 1.95 3.68 2.29 1.86 2.63
2.83 2.04 2.23 1.92 3.02
Plant 2: 3.79 2.08 3.66 1.53 4.07 4.31 2.62 4.52 3.80 5.30
3.41 0.82 3.03 1.95 6.45 1.86 1.87 3.78 2.74 3.81
Question. Are there differences in the mean amounts of white paper discarded by employees at the
two plants? (use α = 0.05).
108
5.2 For the difference of two population proportions p1 −p2 : Independent samples
We also can extend our confidence interval procedure for a single population proportion p to two
populations. Define
• satisfied customers before and after a product design change (e.g., Facebook, etc.)
Point estimators: We assume that there are two independent random samples of individuals (one
sample from each population to be compared). Define
The point estimators for p1 and p2 are the sample proportions, defined by
Y1
pb1 =
n1
Y2
pb2 = .
n2
Goal: We would like to write a 100(1 − α) percent confidence interval and conduct hypothesis test
for p1 − p2 , the difference of two population proportions.
To accomplish this goal, we need the following distributional result. When the sample sizes n1
and n2 are large,
(b
p1 − pb2 ) − (p1 − p2 )
Z= q ∼ AN (0, 1).
p1 (1−p1 ) p2 (1−p2 )
n1 + n2
109
– the two samples to be independent
– the sample sizes n1 and n2 to be “large;” common rules of thumb are to require
ni pbi ≥ 5
ni (1 − pbi ) ≥ 5,
for each sample i = 1, 2. Under these conditions, the CLT should adequately approximate
the true sampling distribution of Z, thereby making the confidence interval formula above
approximately valid.
– Note again the form of the interval:
Ha : p1 6= p2
Ha : p1 > p2
Ha : p1 < p2
To do this, we need find a testing statistics. Remember that, while we are designing a test, we always
want to bound the Type I Error, which is defined as “reject H0 when H0 in fact is true.” So, when
H0 is in fact true, we can view the two independent samples are from the same population; i.e.,
p1 = p2 = p. Then we have Z (define above) can be written as
(b
p1 − pb2 ) − (p1 − p2 ) (b
p1 − pb2 )
Z0 = q =r ∼ AN (0, 1).
p1 (1−p1 )
+ p2 (1−p2)
n1 n2 p(1 − p) n11 + 1
n2
Y1 + Y2
pb = .
n1 + n2
110
Thus, we have
(b
p1 − pb2 )
Z0 = r ∼ AN (0, 1).
pb(1 − pb) n11 + 1
n2
Example 5.2.1. A programmable lighting control system is being designed. The purpose of the
system is to reduce electricity consumption costs in buildings. The system eventually will entail the
use of a large number of transceivers (a device comprised of both a transmitter and a receiver). Two
types of transceivers are being considered. In life testing, 200 transceivers (randomly selected) were
tested for each type.
111
5.3 For the ratio of two population variances σ22 /σ12 : Independent samples
You will recall that when we wrote a confidence interval for µ1 − µ2 , the difference of the population
means (with independent samples), we proposed two different intervals:
We now propose a confidence interval procedure that can be used to determine which assumption is
more appropriate. This confidence interval is used to compare the population variances in two
independent samples.
Suppose that we have two independent samples:
Goal: Construct a 100(1 − α) percent confidence interval and conduct hypothesis test for the ratio
of population variances σ22 /σ12 .
S12 /σ12
R= ∼ F (n1 − 1, n2 − 1),
S22 /σ22
• indexed by two degree of freedom parameters ν1 and ν2 ; these are usually integers and are
often related to sample sizes
112
0.8
0.6
F; df = 5,5
F; df = 5,10
F; df = 10,10
f(q)
0.4
0.2
0.0
0 2 4 6 8
Notation: Let Fn1 −1,n2 −1,α/2 and Fn1 −1,n2 −1,1−α/2 denote the upper and lower quantiles, respec-
tively, of the F (n1 − 1, n2 − 1) distribution; i.e., these values satisfy
Because
S12 /σ12
R= ∼ F (n1 − 1, n2 − 1),
S22 /σ22
we can write
1 − α = P Fn1 −1,n2 −1,1−α/2 ≤ R ≤ Fn1 −1,n2 −1,α/2
S12 /σ12
= P Fn1 −1,n2 −1,1−α/2 ≤ 2 2 ≤ Fn1 −1,n2 −1,α/2
S2 /σ2
2
σ22 S22
S2
= P × Fn1 −1,n2 −1,1−α/2 ≤ 2 ≤ 2 × Fn1 −1,n2 −1,α/2 .
S12 σ1 S1
113
This shows that
S22 S22
× Fn −1,n −1,1−α/2 , × Fn1 −1,n2 −1,α/2
S12 1 2
S12
is a 100(1 − α) percent confidence interval for the ratio of the population variances σ22 /σ12 . Also,
we can obtained 100(1 − α) percent confidence interval for σ12 /σ22 as
S12 S2
1 1
2 × , 12 ×
S2 Fn1 −1,n2 −1,α/2 S2 Fn1 −1,n2 −1,1−α/2
1
Fν2 ,ν1 ,a = ,
Fν1 ,ν2 ,1−a
S12 S12
1
× , × Fn2 −1,n1 −1,α/2
S22 Fn1 −1,n2 −1,α/2 S22
We interpret the interval in the same way.
“We are 100(1 − α) percent confident that the ratio σ12 /σ22 is in this interval.”
114
746
appA.indd 746
a = 0.25
f0.25, n , n
1 2
2 2.57 3.00 3.15 3.23 3.28 3.31 3.34 3.35 3.37 3.38 3.39 3.41 3.43 3.43 3.44 3.45 3.46 3.47 3.48
3 2.02 2.28 2.36 2.39 2.41 2.42 2.43 2.44 2.44 2.44 2.45 2.46 2.46 2.46 2.47 2.47 2.47 2.47 2.47
4 1.81 2.00 2.05 2.06 2.07 2.08 2.08 2.08 2.08 2.08 2.08 2.08 2.08 2.08 2.08 2.08 2.08 2.08 2.08
5 1.69 1.85 1.88 1.89 1.89 1.89 1.89 1.89 1.89 1.89 1.89 1.89 1.88 1.88 1.88 1.88 1.87 1.87 1.87
6 1.62 1.76 1.78 1.79 1.79 1.78 1.78 1.78 1.77 1.77 1.77 1.76 1.76 1.75 1.75 1.75 1.74 1.74 1.74
7 1.57 1.70 1.72 1.72 1.71 1.71 1.70 1.70 1.70 1.69 1.68 1.68 1.67 1.67 1.66 1.66 1.65 1.65 1.65
115
8 1.54 1.66 1.67 1.66 1.66 1.65 1.64 1.64 1.63 1.63 1.62 1.62 1.61 1.60 1.60 1.59 1.59 1.58 1.58
9 1.51 1.62 1.63 1.63 1.62 1.61 1.60 1.60 1.59 1.59 1.58 1.57 1.56 1.56 1.55 1.54 1.54 1.53 1.53
10 1.49 1.60 1.60 1.59 1.59 1.58 1.57 1.56 1.56 1.55 1.54 1.53 1.52 1.52 1.51 1.51 1.50 1.49 1.48
11 1.47 1.58 1.58 1.57 1.56 1.55 1.54 1.53 1.53 1.52 1.51 1.50 1.49 1.49 1.48 1.47 1.47 1.46 1.45
12 1.46 1.56 1.56 1.55 1.54 1.53 1.52 1.51 1.51 1.50 1.49 1.48 1.47 1.46 1.45 1.45 1.44 1.43 1.42
13 1.45 1.55 1.55 1.53 1.52 1.51 1.50 1.49 1.49 1.48 1.47 1.46 1.45 1.44 1.43 1.42 1.42 1.41 1.40
14 1.44 1.53 1.53 1.52 1.51 1.50 1.49 1.48 1.47 1.46 1.45 1.44 1.43 1.42 1.41 1.41 1.40 1.39 1.38
15 1.43 1.52 1.52 1.51 1.49 1.48 1.47 1.46 1.46 1.45 1.44 1.43 1.41 1.41 1.40 1.39 1.38 1.37 1.36
16 1.42 1.51 1.51 1.50 1.48 1.47 1.46 1.45 1.44 1.44 1.43 1.41 1.40 1.39 1.38 1.37 1.36 1.35 1.34
17 1.42 1.51 1.50 1.49 1.47 1.46 1.45 1.44 1.43 1.43 1.41 1.40 1.39 1.38 1.37 1.36 1.35 1.34 1.33
18 1.41 1.50 1.49 1.48 1.46 1.45 1.44 1.43 1.42 1.42 1.40 1.39 1.38 1.37 1.36 1.35 1.34 1.33 1.32
19 1.41 1.49 1.49 1.47 1.46 1.44 1.43 1.42 1.41 1.41 1.40 1.38 1.37 1.36 1.35 1.34 1.33 1.32 1.30
20 1.40 1.49 1.48 1.47 1.45 1.44 1.43 1.42 1.41 1.40 1.39 1.37 1.36 1.35 1.34 1.33 1.32 1.31 1.29
21 1.40 1.48 1.48 1.46 1.44 1.43 1.42 1.41 1.40 1.39 1.38 1.37 1.35 1.34 1.33 1.32 1.31 1.30 1.28
22 1.40 1.48 1.47 1.45 1.44 1.42 1.41 1.40 1.39 1.39 1.37 1.36 1.34 1.33 1.32 1.31 1.30 1.29 1.28
23 1.39 1.47 1.47 1.45 1.43 1.42 1.41 1.40 1.39 1.38 1.37 1.35 1.34 1.33 1.32 1.31 1.30 1.28 1.27
24 1.39 1.47 1.46 1.44 1.43 1.41 1.40 1.39 1.38 1.38 1.36 1.35 1.33 1.32 1.31 1.30 1.29 1.28 1.26
25 1.39 1.47 1.46 1.44 1.42 1.41 1.40 1.39 1.38 1.37 1.36 1.34 1.33 1.32 1.31 1.29 1.28 1.27 1.25
9/24/2013 6:28:24 PM
appA.indd 747
a = 0.10
f0.10, n , n
1 2
116
4 4.54 4.32 4.19 4.11 4.05 4.01 3.98 3.95 3.94 3.92 3.90 3.87 3.84 3.83 3.82 3.80 3.79 3.78 3.76
5 4.06 3.78 3.62 3.52 3.45 3.40 3.37 3.34 3.32 3.30 3.27 3.24 3.21 3.19 3.17 3.16 3.14 3.12 3.10
6 3.78 3.46 3.29 3.18 3.11 3.05 3.01 2.98 2.96 2.94 2.90 2.87 2.84 2.82 2.80 2.78 2.76 2.74 2.72
7 3.59 3.26 3.07 2.96 2.88 2.83 2.78 2.75 2.72 2.70 2.67 2.63 2.59 2.58 2.56 2.54 2.51 2.49 2.47
8 3.46 3.11 2.92 2.81 2.73 2.67 2.62 2.59 2.56 2.54 2.50 2.46 2.42 2.40 2.38 2.36 2.34 2.32 2.29
9 3.36 3.01 2.81 2.69 2.61 2.55 2.51 2.47 2.44 2.42 2.38 2.34 2.30 2.28 2.25 2.23 2.21 2.18 2.16
10 3.29 2.92 2.73 2.61 2.52 2.46 2.41 2.38 2.35 2.32 2.28 2.24 2.20 2.18 2.16 2.13 2.11 2.08 2.06
11 3.23 2.86 2.66 2.54 2.45 2.39 2.34 2.30 2.27 2.25 2.21 2.17 2.12 2.10 2.08 2.05 2.03 2.00 1.97
12 3.18 2.81 2.61 2.48 2.39 2.33 2.28 2.24 2.21 2.19 2.15 2.10 2.06 2.04 2.01 1.99 1.96 1.93 1.90
13 3.14 2.76 2.56 2.43 2.35 2.28 2.23 2.20 2.16 2.14 2.10 2.05 2.01 1.98 1.96 1.93 1.90 1.88 1.85
14 3.10 2.73 2.52 2.39 2.31 2.24 2.19 2.15 2.12 2.10 2.05 2.01 1.96 1.94 1.91 1.89 1.86 1.83 1.80
15 3.07 2.70 2.49 2.36 2.27 2.21 2.16 2.12 2.09 2.06 2.02 1.97 1.92 1.90 1.87 1.85 1.82 1.79 1.76
16 3.05 2.67 2.46 2.33 2.24 2.18 2.13 2.09 2.06 2.03 1.99 1.94 1.89 1.87 1.84 1.81 1.78 1.75 1.72
17 3.03 2.64 2.44 2.31 2.22 2.15 2.10 2.06 2.03 2.00 1.96 1.91 1.86 1.84 1.81 1.78 1.75 1.72 1.69
18 3.01 2.62 2.42 2.29 2.20 2.13 2.08 2.04 2.00 1.98 1.93 1.89 1.84 1.81 1.78 1.75 1.72 1.69 1.66
19 2.99 2.61 2.40 2.27 2.18 2.11 2.06 2.02 1.98 1.96 1.91 1.86 1.81 1.79 1.76 1.73 1.70 1.67 1.63
20 2.97 2.59 2.38 2.25 2.16 2.09 2.04 2.00 1.96 1.94 1.89 1.84 1.79 1.77 1.74 1.71 1.68 1.64 1.61
21 2.96 2.57 2.36 2.23 2.14 2.08 2.02 1.98 1.95 1.92 1.87 1.83 1.78 1.75 1.72 1.69 1.66 1.62 1.59
22 2.95 2.56 2.35 2.22 2.13 2.06 2.01 1.97 1.93 1.90 1.86 1.81 1.76 1.73 1.70 1.67 1.64 1.60 1.57
23 2.94 2.55 2.34 2.21 2.11 2.05 1.99 1.95 1.92 1.89 1.84 1.80 1.74 1.72 1.69 1.66 1.62 1.59 1.55
24 2.93 2.54 2.33 2.19 2.10 2.04 1.98 1.94 1.91 1.88 1.83 1.78 1.73 1.70 1.67 1.64 1.61 1.57 1.53
25 2.92 2.53 2.32 2.18 2.09 2.02 1.97 1.93 1.89 1.87 1.82 1.77 1.72 1.69 1.66 1.63 1.59 1.56 1.52
60 2.79 2.39 2.18 2.04 1.95 1.87 1.82 1.77 1.74 1.71 1.66 1.60 1.54 1.51 1.48 1.44 1.40 1.35 1.29
120 2.75 2.35 2.13 1.99 1.90 1.82 1.77 1.72 1.68 1.65 1.60 1.55 1.48 1.45 1.41 1.37 1.32 1.26 1.19
∞ 2.71 2.30 2.08 1.94 1.85 1.77 1.72 1.67 1.63 1.60 1.55 1.49 1.42 1.38 1.34 1.30 1.24 1.17 1.00
747
9/24/2013 6:28:26 PM
748
appA.indd 748
a = 0.05
f0.05, n , n
1 2
2 18.51 19.00 19.16 19.25 19.30 19.33 19.35 19.37 19.38 19.40 19.41 19.43 19.45 19.45 19.46 19.47 19.48 19.49 19.50
3 10.13 9.55 9.28 9.12 9.01 8.94 8.89 8.85 8.81 8.79 8.74 8.70 8.66 8.64 8.62 8.59 8.57 8.55 8.53
4 7.71 6.94 6.59 6.39 6.26 6.16 6.09 6.04 6.00 5.96 5.91 5.86 5.80 5.77 5.75 5.72 5.69 5.66 5.63
5 6.61 5.79 5.41 5.19 5.05 4.95 4.88 4.82 4.77 4.74 4.68 4.62 4.56 4.53 4.50 4.46 4.43 4.40 4.36
6 5.99 5.14 4.76 4.53 4.39 4.28 4.21 4.15 4.10 4.06 4.00 3.94 3.87 3.84 3.81 3.77 3.74 3.70 3.67
7 5.59 4.74 4.35 4.12 3.97 3.87 3.79 3.73 3.68 3.64 3.57 3.51 3.44 3.41 3.38 3.34 3.30 3.27 3.23
117
8 5.32 4.46 4.07 3.84 3.69 3.58 3.50 3.44 3.39 3.35 3.28 3.22 3.15 3.12 3.08 3.04 3.01 2.97 2.93
9 5.12 4.26 3.86 3.63 3.48 3.37 3.29 3.23 3.18 3.14 3.07 3.01 2.94 2.90 2.86 2.83 2.79 2.75 2.71
10 4.96 4.10 3.71 3.48 3.33 3.22 3.14 3.07 3.02 2.98 2.91 2.85 2.77 2.74 2.70 2.66 2.62 2.58 2.54
11 4.84 3.98 3.59 3.36 3.20 3.09 3.01 2.95 2.90 2.85 2.79 2.72 2.65 2.61 2.57 2.53 2.49 2.45 2.40
12 4.75 3.89 3.49 3.26 3.11 3.00 2.91 2.85 2.80 2.75 2.69 2.62 2.54 2.51 2.47 2.43 2.38 2.34 2.30
13 4.67 3.81 3.41 3.18 3.03 2.92 2.83 2.77 2.71 2.67 2.60 2.53 2.46 2.42 2.38 2.34 2.30 2.25 2.21
14 4.60 3.74 3.34 3.11 2.96 2.85 2.76 2.70 2.65 2.60 2.53 2.46 2.39 2.35 2.31 2.27 2.22 2.18 2.13
15 4.54 3.68 3.29 3.06 2.90 2.79 2.71 2.64 2.59 2.54 2.48 2.40 2.33 2.29 2.25 2.20 2.16 2.11 2.07
16 4.49 3.63 3.24 3.01 2.85 2.74 2.66 2.59 2.54 2.49 2.42 2.35 2.28 2.24 2.19 2.15 2.11 2.06 2.01
17 4.45 3.59 3.20 2.96 2.81 2.70 2.61 2.55 2.49 2.45 2.38 2.31 2.23 2.19 2.15 2.10 2.06 2.01 1.96
18 4.41 3.55 3.16 2.93 2.77 2.66 2.58 2.51 2.46 2.41 2.34 2.27 2.19 2.15 2.11 2.06 2.02 1.97 1.92
19 4.38 3.52 3.13 2.90 2.74 2.63 2.54 2.48 2.42 2.38 2.31 2.23 2.16 2.11 2.07 2.03 1.98 1.93 1.88
20 4.35 3.49 3.10 2.87 2.71 2.60 2.51 2.45 2.39 2.35 2.28 2.20 2.12 2.08 2.04 1.99 1.95 1.90 1.84
21 4.32 3.47 3.07 2.84 2.68 2.57 2.49 2.42 2.37 2.32 2.25 2.18 2.10 2.05 2.01 1.96 1.92 1.87 1.81
22 4.30 3.44 3.05 2.82 2.66 2.55 2.46 2.40 2.34 2.30 2.23 2.15 2.07 2.03 1.98 1.94 1.89 1.84 1.78
23 4.28 3.42 3.03 2.80 2.64 2.53 2.44 2.37 2.32 2.27 2.20 2.13 2.05 2.01 1.96 1.91 1.86 1.81 1.76
24 4.26 3.40 3.01 2.78 2.62 2.51 2.42 2.36 2.30 2.25 2.18 2.11 2.03 1.98 1.94 1.89 1.84 1.79 1.73
25 4.24 3.39 2.99 2.76 2.60 2.49 2.40 2.34 2.28 2.24 2.16 2.09 2.01 1.96 1.92 1.87 1.82 1.77 1.71
9/24/2013 6:28:29 PM
appA.indd 749
a = 0.025
f0.025, n , n
1 2
118
4 12.22 10.65 9.98 9.60 9.36 9.20 9.07 8.98 8.90 8.84 8.75 8.66 8.56 8.51 8.46 8.41 8.36 8.31 8.26
5 10.01 8.43 7.76 7.39 7.15 6.98 6.85 6.76 6.68 6.62 6.52 6.43 6.33 6.28 6.23 6.18 6.12 6.07 6.02
6 8.81 7.26 6.60 6.23 5.99 5.82 5.70 5.60 5.52 5.46 5.37 5.27 5.17 5.12 5.07 5.01 4.96 4.90 4.85
7 8.07 6.54 5.89 5.52 5.29 5.12 4.99 4.90 4.82 4.76 4.67 4.57 4.47 4.42 4.36 4.31 4.25 4.20 4.14
8 7.57 6.06 5.42 5.05 4.82 4.65 4.53 4.43 4.36 4.30 4.20 4.10 4.00 3.95 3.89 3.84 3.78 3.73 3.67
9 7.21 5.71 5.08 4.72 4.48 4.32 4.20 4.10 4.03 3.96 3.87 3.77 3.67 3.61 3.56 3.51 3.45 3.39 3.33
10 6.94 5.46 4.83 4.47 4.24 4.07 3.95 3.85 3.78 3.72 3.62 3.52 3.42 3.37 3.31 3.26 3.20 3.14 3.08
11 6.72 5.26 4.63 4.28 4.04 3.88 3.76 3.66 3.59 3.53 3.43 3.33 3.23 3.17 3.12 3.06 3.00 2.94 2.88
12 6.55 5.10 4.47 4.12 3.89 3.73 3.61 3.51 3.44 3.37 3.28 3.18 3.07 3.02 2.96 2.91 2.85 2.79 2.72
13 6.41 4.97 4.35 4.00 3.77 3.60 3.48 3.39 3.31 3.25 3.15 3.05 2.95 2.89 2.84 2.78 2.72 2.66 2.60
14 6.30 4.86 4.24 3.89 3.66 3.50 3.38 3.29 3.21 3.15 3.05 2.95 2.84 2.79 2.73 2.67 2.61 2.55 2.49
15 6.20 4.77 4.15 3.80 3.58 3.41 3.29 3.20 3.12 3.06 2.96 2.86 2.76 2.70 2.64 2.59 2.52 2.46 2.40
16 6.12 4.69 4.08 3.73 3.50 3.34 3.22 3.12 3.05 2.99 2.89 2.79 2.68 2.63 2.57 2.51 2.45 2.38 2.32
17 6.04 4.62 4.01 3.66 3.44 3.28 3.16 3.06 2.98 2.92 2.82 2.72 2.62 2.56 2.50 2.44 2.38 2.32 2.25
18 5.98 4.56 3.95 3.61 3.38 3.22 3.10 3.01 2.93 2.87 2.77 2.67 2.56 2.50 2.44 2.38 2.32 2.26 2.19
19 5.92 4.51 3.90 3.56 3.33 3.17 3.05 2.96 2.88 2.82 2.72 2.62 2.51 2.45 2.39 2.33 2.27 2.20 2.13
20 5.87 4.46 3.86 3.51 3.29 3.13 3.01 2.91 2.84 2.77 2.68 2.57 2.46 2.41 2.35 2.29 2.22 2.16 2.09
21 5.83 4.42 3.82 3.48 3.25 3.09 2.97 2.87 2.80 2.73 2.64 2.53 2.42 2.37 2.31 2.25 2.18 2.11 2.04
22 5.79 4.38 3.78 3.44 3.22 3.05 2.93 2.84 2.76 2.70 2.60 2.50 2.39 2.33 2.27 2.21 2.14 2.08 2.00
23 5.75 4.35 3.75 3.41 3.18 3.02 2.90 2.81 2.73 2.67 2.57 2.47 2.36 2.30 2.24 2.18 2.11 2.04 1.97
24 5.72 4.32 3.72 3.38 3.15 2.99 2.87 2.78 2.70 2.64 2.54 2.44 2.33 2.27 2.21 2.15 2.08 2.01 1.94
25 5.69 4.29 3.69 3.35 3.13 2.97 2.85 2.75 2.68 2.61 2.51 2.41 2.30 2.24 2.18 2.12 2.05 1.98 1.91
60 5.29 3.93 3.34 3.01 2.79 2.63 2.51 2.41 2.33 2.27 2.17 2.06 1.94 1.88 1.82 1.74 1.67 1.58 1.48
120 5.15 3.80 3.23 2.89 2.67 2.52 2.39 2.30 2.22 2.16 2.05 1.94 1.82 1.76 1.69 1.61 1.53 1.43 1.31
∞ 5.02 3.69 3.12 2.79 2.57 2.41 2.29 2.19 2.11 2.05 1.94 1.83 1.71 1.64 1.57 1.48 1.39 1.27 1.00
749
9/24/2013 6:28:31 PM
750
appA.indd 750
a = 0.01
f0.01, n , n
1 2
2 98.50 99.00 99.17 99.25 99.30 99.33 99.36 99.37 99.39 99.40 99.42 99.43 99.45 99.46 99.47 99.47 99.48 99.49 99.50
3 34.12 30.82 29.46 28.71 28.24 27.91 27.67 27.49 27.35 27.23 27.05 26.87 26.69 26.00 26.50 26.41 26.32 26.22 26.13
4 21.20 18.00 16.69 15.98 15.52 15.21 14.98 14.80 14.66 14.55 14.37 14.20 14.02 13.93 13.84 13.75 13.65 13.56 13.46
5 16.26 13.27 12.06 11.39 10.97 10.67 10.46 10.29 10.16 10.05 9.89 9.72 9.55 9.47 9.38 9.29 9.20 9.11 9.02
6 13.75 10.92 9.78 9.15 8.75 8.47 8.26 8.10 7.98 7.87 7.72 7.56 7.40 7.31 7.23 7.14 7.06 6.97 6.88
7 12.25 9.55 8.45 7.85 7.46 7.19 6.99 6.84 6.72 6.62 6.47 6.31 6.16 6.07 5.99 5.91 5.82 5.74 5.65
119
8 11.26 8.65 7.59 7.01 6.63 6.37 6.18 6.03 5.91 5.81 5.67 5.52 5.36 5.28 5.20 5.12 5.03 4.95 4.46
9 10.56 8.02 6.99 6.42 6.06 5.80 5.61 5.47 5.35 5.26 5.11 4.96 4.81 4.73 4.65 4.57 4.48 4.40 4.31
10 10.04 7.56 6.55 5.99 5.64 5.39 5.20 5.06 4.94 4.85 4.71 4.56 4.41 4.33 4.25 4.17 4.08 4.00 3.91
11 9.65 7.21 6.22 5.67 5.32 5.07 4.89 4.74 4.63 4.54 4.40 4.25 4.10 4.02 3.94 3.86 3.78 3.69 3.60
12 9.33 6.93 5.95 5.41 5.06 4.82 4.64 4.50 4.39 4.30 4.16 4.01 3.86 3.78 3.70 3.62 3.54 3.45 3.36
13 9.07 6.70 5.74 5.21 4.86 4.62 4.44 4.30 4.19 4.10 3.96 3.82 3.66 3.59 3.51 3.43 3.34 3.25 3.17
14 8.86 6.51 5.56 5.04 4.69 4.46 4.28 4.14 4.03 3.94 3.80 3.66 3.51 3.43 3.35 3.27 3.18 3.09 3.00
15 8.68 6.36 5.42 4.89 4.36 4.32 4.14 4.00 3.89 3.80 3.67 3.52 3.37 3.29 3.21 3.13 3.05 2.96 2.87
16 8.53 6.23 5.29 4.77 4.44 4.20 4.03 3.89 3.78 3.69 3.55 3.41 3.26 3.18 3.10 3.02 2.93 2.84 2.75
17 8.40 6.11 5.18 4.67 4.34 4.10 3.93 3.79 3.68 3.59 3.46 3.31 3.16 3.08 3.00 2.92 2.83 2.75 2.65
18 8.29 6.01 5.09 4.58 4.25 4.01 3.84 3.71 3.60 3.51 3.37 3.23 3.08 3.00 2.92 2.84 2.75 2.66 2.57
19 8.18 5.93 5.01 4.50 4.17 3.94 3.77 3.63 3.52 3.43 3.30 3.15 3.00 2.92 2.84 2.76 2.67 2.58 2.59
20 8.10 5.85 4.94 4.43 4.10 3.87 3.70 3.56 3.46 3.37 3.23 3.09 2.94 2.86 2.78 2.69 2.61 2.52 2.42
21 8.02 5.78 4.87 4.37 4.04 3.81 3.64 3.51 3.40 3.31 3.17 3.03 2.88 2.80 2.72 2.64 2.55 2.46 2.36
22 7.95 5.72 4.82 4.31 3.99 3.76 3.59 3.45 3.35 3.26 3.12 2.98 2.83 2.75 2.67 2.58 2.50 2.40 2.31
23 7.88 5.66 4.76 4.26 3.94 3.71 3.54 3.41 3.30 3.21 3.07 2.93 2.78 2.70 2.62 2.54 2.45 2.35 2.26
24 7.82 5.61 4.72 4.22 3.90 3.67 3.50 3.36 3.26 3.17 3.03 2.89 2.74 2.66 2.58 2.49 2.40 2.31 2.21
25 7.77 5.57 4.68 4.18 3.85 3.63 3.46 3.32 3.22 3.13 2.99 2.85 2.70 2.62 2.54 2.45 2.36 2.27 2.17
9/24/2013 6:28:33 PM
Hypothesis testing: Now we are interested in testing
H0 : σ12 = σ22
Ha :σ12 6= σ22
Ha :σ12 > σ22
Ha :σ12 < σ22
Thus
Some statisticians recommend to use this “equal/unequal variance test” before deciding which con-
fidence interval or testing procedure to use for µ1 − µ2 . Some statisticians do not.
Major warning: Like the χ2 interval for single population variance σ 2 , the two-sample F in-
120
terval for the ratio of two variances is not robust to departures from normality. If the underlying
population distributions are non-normal (non-Guassian), then this interval should not be used. One
easy solution, to check whether the sample is from a normal distribution or not, is the so-called
“normal qq plot.” It plots sample quantiles against theoretical normal quantiles.
• Otherwise, non-normal.
You do not need know details about how to construct it. In R program (a famous statistical software),
use commend qqnorm to draw the plot. such as
● ●
5
● ●
● ●
●
6
●
● ●
4
●●
●●●
●●
●● ●
●●●
●●● ●
●●●●
●●
Sample Quantiles
Sample Quantiles
4
●
●●●●●●
●●●
●● ●
3
●●
●●
●●
●
●●
●●
●
●
●●
●●
●●
● ●
●●
●●
●●
●●
●
●
2
●
●
●●●●● ●●
●●● ●●
2
●●●●● ●●●
●●
●●
● ●
●●● ●●●
●●
● ●●●
● ●●●
●●
●
0
●●●
●
●● ●
●●●
1
● ●
● ●
●●
●
●●
●●
●
●●
●
●
●●
●
●●
●●
●●
●●●
●●
●●●
●
●●●●●
−2
● ●●●●●●●●●●●
●
●●●●
● ● ● ● ●● ● ●●
0
−2 −1 0 1 2 −2 −1 0 1 2
121
Example 5.3.1. We consider again the recycling project in Example 5.1.3 that examined the amount
of white paper discarded per employee at two large plants. The data (presented in Example 4.13)
were obtained on the amount of white paper thrown out per year by employees (data are in hundreds
of pounds). Samples of employees at each plant (n1 = 25 and n2 = 20) were randomly selected.
Plant 1: 3.01 2.58 3.04 1.75 2.87 2.57 2.51 2.93 2.85 3.09
1.43 3.36 3.18 2.74 2.25 1.95 3.68 2.29 1.86 2.63
2.83 2.04 2.23 1.92 3.02
Plant 2: 3.79 2.08 3.66 1.53 4.07 4.31 2.62 4.52 3.80 5.30
3.41 0.82 3.03 1.95 6.45 1.86 1.87 3.78 2.74 3.81
The boxplots in Figure 5.1.2 did suggest that the population variances may be different, and the
following are the normal qq plots.
3.5
6
5
3.0
Sample Quantiles
Sample Quantiles
4
2.5
3
2.0
2
1.5
−2 −1 0 1 2 −2 −1 0 1 2
Figure 5.3.5: Normal quantile-quantile (qq) plots for employee recycle data for two plants.
Find a 95 percent confidence interval for σ22 /σ12 , the ratio of the population variances. Here, σ12 (σ22 )
denotes the population variance of the amount of white paper by employees at Plant 1 (Plant 2).
122
5.4 For the difference of two population means µ1 − µ2 : Dependent samples
(Matched-pairs)
Example 5.4.1. Creatine is an organic acid that helps to supply energy to cells in the body, pri-
marily muscle. Because of this, it is commonly used by those who are weight training to gain muscle
mass. Does it really work? Suppose that we are designing an experiment involving USC male
undergraduates who exercise/lift weights regularly.
Design 1 (Independent samples): Recruit 30 students who are representative of the popula-
tion of USC male undergraduates who exercise/lift weights. For a single weight training session, we
will
• assign 15 students an innocuous substance that looks like creatine (but has no positive/negative
effect on performance).
or s
S12 S22
(X 1· − X 2· ) ± tν,α/2 + ,
n1 n2
123
Design 2 (Matched Pairs): Recruit 15 students who are representative of the population of
USC male undergraduates who exercise/lift weights.
• Each student will be assigned first to take either creatine or the control substance.
• After a period of recovery (e.g., 1 week), we will then have each student take the other “treat-
ment” (creatine/control) and record his value of Y again (but now on the other treatment).
• In other words, for each individual student, we will measure Y under both conditions.
Note: In Design 2, because MBPW measurements are taken on the same student, the difference
between the measurement (creatine/control) should be less variable than the difference between a
creatine measurement on one student and a control measurement on a different student.
• In other words, the student-to-student variation inherent in the latter difference is not present
in the difference between MBPW measurements taken on the same individual student.
Table 5.1: Creatine example. Sources of variation in the two independent sample and matched pairs
designs.
Matched pairs: In general, by obtaining a pair of measurements on a single individual (e.g., stu-
dent, raw material, machine, etc.), where one of measurement corresponds to “Treatment 1” and the
other measurement corresponds to “Treatment 2,” you eliminate variation among the individuals.
This allows you to compare the two experimental conditions (e.g., creatine/control, biodegradabil-
ity treatments, operators, etc.) under more homogeneous conditions where only variation within
individuals is present (that is, the variation arising from the difference in the two experimental con-
ditions).
Advantage: When you remove extra variability, this enables you to do a better job at comparing
the two experimental conditions (treatments). By “better job,” I mean, you can more precisely
estimate the difference between the treatments (excess variability that naturally arises among indi-
viduals is not getting in the way). This gives you a better chance of identifying a difference between
the treatments if one really exists.
Note: In matched pairs experiments, it is important to randomize the order in which treat-
ments are assigned. This may eliminate “common patterns” that may be seen when always following,
say, Treatment 1 with Treatment 2. In practice, the experimenter could flip a fair coin to determine
which treatment is applied first.
124
Table 5.2: Creatine data. Maximum bench press weight (in lbs) for creatine and control treatments
with 15 students. Note: These are not real data.
Implementation: (Basically, you take the difference, then perform a T-test on the differences)
Data from matched pairs experiments are analyzed by examining the difference in responses of the
two treatments. Specifically, compute
Dj = Y1j − Y2j ,
for each individual j = 1, 2, ..., n. After doing this, we have essentially created a “one sample
problem,” where our data are:
D1 , D2 , ..., Dn ,
the so-called data differences. The one sample 100(1 − α) percent confidence interval
SD
D ± tn−1,α/2 √ ,
n
where D and SD are the sample mean and sample standard deviation of the differences, respectively,
is an interval estimate for
“We are 100(1 − α) percent confident that the mean difference µD is in this interval.”
125
Testing hypothesis: (Basically, you need run a T-test on the differences) Following the six steps
(now we have µD = µ1 − µ2 ), test
H0 : µD = ∆0
D−∆
t0 = √0
SD / n
Now let us finish the example of Creatine. Find a 95% confidence interval for µD and test whether
there is a difference using α = 0.05.
126
5.5 One-way analysis of variance
So far in this chapter, we have discussed confidence intervals for a single population mean µ and for
the difference of two population means µ1 − µ2 . When there are two means, we have recently seen
that the design of the experiment/study completely determines how the data are to be analyzed.
• When the two samples are independent, this is called a (two) independent-sample design.
• When the two samples are obtained on the same individuals (so that the samples are depen-
dent), this is called a matched pairs design.
More generally, the purpose of an experiment is to investigate differences between or among two
or more treatments. In a statistical framework, we do this by comparing the population means of
the responses to each treatment.
• In order to detect treatment mean differences, we must try to control the effects of error so
that any variation we observe can be attributed to the effects of the treatments rather than to
differences among the individuals.
Blocking: Designs involving meaningful grouping of individuals, that is, blocking, can help reduce
the effects of experimental error by identifying systematic components of variation among individuals.
• The matched pairs design for comparing two treatments is an example of such a design. In this
situation, individuals themselves are treated as “blocks.”
The analysis of data from experiments involving blocking will not be covered in this course (see, e.g.,
STAT 506, STAT 525, and STAT 706). We focus herein on a simpler setting, that is, a one-way
classification model. This is an extension of the two independent-sample design to more than two
treatments.
• We obtain a random sample of individuals and randomly assign them to treatments. Samples
corresponding to the treatment groups are independent (i.e., the individuals in each treatment
sample are unrelated).
• We do not attempt to group individuals according to some other factor (e.g., location, gender,
weight, variety, etc.). This would be an example of blocking.
127
Main point: In a one-way classification design, the only way in which individuals are “clas-
sified” is by the treatment group assignment. Hence, such an arrangement is called a one-way
classification. When individuals are thought to be “basically alike” (other than the possible effect
of treatment), experimental error consists only of the variation among the individuals themselves,
that is, there are no other systematic sources of variation.
Example 5.5.1. Four types of mortars: (1) ordinary cement mortar (OCM), polymer impregnated
mortar (PIM), resin mortar (RM), and (4) polymer cement mortar (PCM), were subjected to a
compression test to measure strength (MPa). Here are the strength measurements taken on different
mortar specimens (36 in all).
40
20
0
Figure 5.5.6: Boxplots of strength measurements (MPa) for four mortar types.
128
In this example,
• “Treatment” = mortar type (OCM, PIM, RM, and PCM). There are t = 4 treatment groups.
“Are the treatment (mortar type) population means equal? Or, are the treatment pop-
ulation means different?”
This question can be answered by performing a hypothesis test, that is, by testing
H0 : µ1 = µ2 = µ3 = µ4
versus
Ha : the population means µi are not all equal.
Goal: We now develop a statistical procedure that allows us to test this type of hypothesis in a
one-way classification model.
H0 : µ1 = µ2 = · · · = µt
versus
Ha : the population means µi are not all equal.
• The null hypothesis H0 says that there is “no treatment difference,” that is, all treatment
population means are the same.
• The alternative hypothesis Ha says that a difference among the t population means exists
somewhere (but does not specify how the means are different).
• The goal is to decide which hypothesis is more supported by the observed data.
129
To do this, I need introduce some notation:
Notation: Let t denote the number of treatments to be compared. Define
• Define
ni
1 X
X i· = Yij
ni
j=1
i n
1 X
Si2 = (Yij − X i· )2
ni − 1
j=1
ni
t X
1 X
X ·· = Yij .
N
i=1 j=1
The statistics X i· and Si2 are simply the sample mean and sample variance, respectively, of the
ith sample. The statistic X ·· is the sample mean of all the data (across all t treatment groups).
The procedure we develop is formulated by deriving two estimators for σ 2 . These two estimators
are formed by (1) looking at the variance of the observations within samples, and (2) looking at the
variance of the sample means across the t samples.
130
where
SSres
M Sres = .
N −t
Important: M Sres is an unbiased estimator of σ 2 regardless of whether or not H0 is true. We
call M Sres the residual mean squares.
σ2
X i· ∼ N µi , .
n
NOTE : If all the treatment population means are equal, that is,
H0 : µ1 = µ2 = · · · = µt = µ, say,
is true, then
σ2
X i· ∼ N µ, .
n
If H0 is true, then the t sample means X 1· , X 2· , ..., X t· are a random sample of size t from a
normal distribution with mean µ and variance σ 2 /n. The sample variance of this “random
sample” is given by
t
1 X
(X i· − X ·· )2
t−1
i=1
Therefore,
t
1 X
M Strt = n(X i· − X ·· )2 ,
t−1
|i=1 {z }
SStrt
is an unbiased estimator of σ 2 ; i.e., E(M Strt ) = σ 2 , when H0 is true. We call SStrt the
treatment sums of squares and M Strt the treatment mean squares. M Strt is our
second point estimator for σ 2 . Recall that M Strt is an unbiased estimator of σ 2 only when
H0 : µ1 = µ2 = · · · = µt is true (this is important!). If we have different sample sizes,
we simply adjust M Strt to
t
1 X
M Strt = ni (X i· − X ·· )2 .
t−1
|i=1 {z }
SStrt
131
Motivation:
• When H0 is true (i.e., the treatment means are the same), then
E(M Strt ) = σ 2
E(M Sres ) = σ 2 .
M Strt
F = ≈ 1.
M Sres
• When H0 is not true (i.e., the treatment means are different), then
M Strt
F = > 1.
M Sres
M Strt
F = ∼ F (t − 1, N − t).
M Sres
DECISION : We “reject H0 ” and conclude the treatment population means are different if the F
statistic is far out in the right tail of the F (t − 1, N − t) distribution. Why? Because a large value
of F is not consistent with H0 being true! Large values of F (far out in the right tail) are more
consistent with Ha . Thus, to test
H0 : µ1 = µ2 = µ3 = µ4
versus
Ha : the population means µi are not all equal,
132
0.6
0.4
PDF
0.2
0.0
0 5 10 15
Figure 5.5.7: The F (3, 32) probability density function. This is the distribution of F in Example
4.17 if H0 is true. An “×” at F = 16.848 has been added.
Example 5.5.2. (continued Example 5.5.1.) Morta Data: We now use TI-84 to do this test for
the strength/mortar type data in Example 5.5.1 at α = 0.05.
Input data in to list L1, L2, L3, and L4. Then go to Stat, to Tests, select ANOVA
‘‘ANOVA(L1, L2, L3, L4)" The output are then:
One-way ANOVA
F=16.84834325 (This the F value used for test)
p=9.5764486E-7 (P-value for the test)
Factor
df=3 (t-1)
SS=1520.87591 (SStrt)
MS=506.958637 (MStrt)
Error
df=32 (N-t)
SS=962.864785 (SSres)
MS=30.0895245 (MSres)
Sxp=5.48539192 (estimate of the common standard deviation sigma)
133
Conclusion: P-value is significantly smaller than α = 0.05 (i.e.; F = 16.848 is not an observation
we would expect from the F (3, 32) distribution (the distribution of F when H0 is true); see Figure
5.5.7). Therefore, we reject H0 and conclude the population mean strengths for the four mortar
types are different. In other words, at α = 0.05 (note that, not only for α = 0.05, but also for
α = 0.01, 0.005, 0.0005, the evidence from the data provides sufficient evidence to reject H0 .
As we have just seen (from the recent R analysis), it is common to display one-way classification
results in an ANOVA table. The form of the ANOVA table for the one-way classification is given
below:
Source df SS MS F
SStrt M Strt
Treatments t−1 SStrt M Strt = t−1 F = M Sres
SSres
Residuals N −t SSres M Sres = N −t
Total N −1 SStotal
• SStotal measures how observations vary about the overall mean, without regard to treatments;
that is, it measures the total variation in all the data. SStotal can be partitioned into two
components:
– SStrt measures how much of the total variation is due to the treatments
– SSres measures what is “left over,” which we attribute to inherent variation among the
individuals.
• Mean squares (MS) are formed by dividing sums of squares by the corresponding degrees of
freedom.
134
6 Linear regression
6.1 Introduction
IMPORTANCE : A problem that arises in engineering, economics, medicine, and other areas is that
of investigating the relationship between two (or more) variables. In such settings, the goal is to
model a continuous random variable Y as a function of one or more independent variables, say,
x1 , x2 , ..., xk . Mathematically, we can express this model as
Y = g(x1 , x2 , ..., xk ) + ,
Y = β0 + β1 x1 + β2 x2 + · · · + βk xk + ,
| {z }
g(x1 ,x2 ,...,xk )
that is, g is a linear function of β0 , β1 , ..., βk . We call this a linear regression model.
• The response variable Y is assumed to be random (but we do get to observe its value).
Y = β0 + β1 x +
| {z }
g(x)
Y = β0 + β1 x + β2 x2 +
| {z }
g(x)
Y = β0 + β1 x1 + β2 x2 + β3 x1 x2 +.
| {z }
g(x1 ,x2 )
Main point: The term “linear” does not refer to the shape of the regression function g. It refers to
how the regression parameters β0 , β1 , ..., βk enter the g function.
135
6.2 Simple linear regression
Terminology: A simple linear regression model includes only one independent variable x and
is of the form
Y = g(x) +
= β0 + β1 x + .
E(Y ) = E(β0 + β1 x + )
= β0 + β1 x + E()
= β0 + β1 x.
Example 6.2.1. As part of a waste removal project, a new compression machine for processing
sewage sludge is being studied. In particular, engineers are interested in the following variables:
Obs x Y Obs x Y
1 125.3 77.9 11 159.5 79.9
2 98.2 76.8 12 145.8 79.0
3 201.4 81.5 13 75.1 76.7
4 147.3 79.8 14 151.4 78.2
5 145.9 78.2 15 144.2 79.5
6 124.7 78.3 16 125.0 78.1
7 112.2 77.5 17 198.8 81.5
8 120.2 77.0 18 132.5 77.0
9 161.2 80.1 19 159.6 79.0
10 178.9 80.2 20 110.7 78.6
Table 6.3: Sewage data. Moisture (Y , measured as a percentage) and machine filtration rate (x,
measured in kg-DS/m/hr). There are n = 20 observations.
136
81
80
Moisture (Percentage)
79
78
77
Figure 6.2.1 displays the data in a scatterplot. This is the most common graphical display for
bivariate data like those seen above. From the plot, we see that
• the variables Y and x are positively related, that is, an increase in x tends to be associated
with an increase in Y .
• the variables Y and x are linearly related, although there is a large amount of variation that
is unexplained.
137
6.2.1 Least squares estimation
Terminology: When we say, “fit a regression model,” we mean that we would like to estimate
the regression parameters in the model with the observed data. Suppose that we collect (xi , Yi ),
i = 1, 2, ..., n, and postulate the simple linear regression model
Yi = β 0 + β 1 x i + i ,
for each i = 1, 2, ..., n. Our first goal is to estimate β0 and β1 . Formal assumptions for the error
terms i will be given later.
Least Squares: A widely-accepted method of estimating the model parameters β0 and β1 is least
squares. The method of least squares says to choose the values of β0 and β1 that minimize
n
X
Q(β0 , β1 ) = [Yi − (β0 + β1 xi )]2 .
i=1
Denote the least squares estimators by βb0 and βb1 , respectively, that is, the values of β0 and β1
that minimize Q(β0 , β1 ). A two-variable minimization argument can be used to find closed-form
expressions for βb0 and βb1 . Taking partial derivatives of Q(β0 , β1 ), we obtain
n
∂Q(β0 , β1 ) X set
= −2 (Yi − β0 − β1 xi ) = 0
∂β0
i=1
n
∂Q(β0 , β1 ) X set
= −2 (Yi − β0 − β1 xi )xi = 0.
∂β1
i=1
βb0 = Y − βb1 x
Pn
(x − x)(Yi − Y ) SSxy
βb1 = Pn i
i=1
2
= .
(x
i=1 i − x) SSxx
In real life, it is rarely necessary to calculate βb0 and βb1 by hand, TI-84 and R automate the entire
model fitting process and subsequent analysis.
Example 6.2.2. (continued Example 6.2.1). We now use R to calculate the equation of the least
squares regression line for the sewage sludge data in Example 6.1. Here is the output:
138
81
80
Moisture (Percentage)
79
78
77
From the output, we see the least squares estimates (to 3 dp) for the sewage data are
βb0 = 72.959
βb1 = 0.041.
Therefore, the equation of the least squares line that relates moisture percentage Y to the filtration
rate x is
Yb = 72.959 + 0.041x,
The textbook authors call the least squares line the prediction equation. This is because we can
predict the value of Y (moisture) for any value of x (filtration rate). For example, when the filtration
rate is x = 150 kg-DS/m/hr, we would predict the moisture percentage to be
139
6.2.2 Model assumptions and properties of least squares estimators
We wish to investigate the properties of βb0 and βb1 as estimators of the true regression parameters
β0 and β1 in the simple linear regression model
Yi = β0 + β1 xi + i ,
for i = 1, 2, ..., n. To do this, we need assumptions on the error terms i . Specifically, we will assume
throughout that
Fact 1. The least squares estimators βb0 and βb1 are unbiased estimators of β0 and β1 , respectively,
that is,
E(βb0 ) = β0
E(βb1 ) = β1 .
Fact 2. The least squares estimators βb0 and βb1 have the following sampling distributions:
where
1 x2 1
c00 = + and c11 = .
n SSxx SSxx
Knowing these sampling distributions is critical if we want to write confidence intervals and perform
hypothesis tests for β0 and β1 .
Yi = β 0 + β 1 x i + i ,
where i ∼ N (0, σ 2 ), we now turn our attention to estimating σ 2 , the error variance.
140
Terminology: In the simple linear regression model, define the ith fitted value by
where βb0 and βb1 are the least squares estimators. Each observation has its own fitted value. Geo-
metrically, an observation’s fitted value is the (perpendicular) projection of its Y value, upward or
downward, onto the least squares line.
ei = Yi − Ybi .
Each observation has its own residual. Geometrically, an observation’s residual is the vertical distance
(i.e., length) between its Y value and its fitted value.
• If an observation’s Y value is above the least squares regression line, its residual is positive.
• If an observation’s Y value is below the least squares regression line, its residual is negative.
• In the simple linear regression model (provided that the model includes an intercept term β0 ),
we have the following algebraic result:
n
X n
X
ei = (Yi − Ybi ) = 0,
i=1 i=1
that is, the sum of the residuals (from a least squares fit) is equal to zero.
SEWAGE DATA: In Table 6.2, I have used R to calculate the fitted values and residuals for
each of the n = 20 observations in the sewage sludge data set.
Table 6.4: Sewage data. Fitted values and residuals from the least squares fit.
141
Fact 3. In the simple linear regression model,
SSres
M Sres =
n−2
Example 6.2.3. SEWAGE DATA: For the sewage data in Example 6.2.1, we use R to calculate
M Sres :
M Sres ≈ 0.443.
> summary(fit)
lm(formula = moisture ~ filtration.rate)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 72.958547 0.697528 104.596 < 2e-16 ***
filtration.rate 0.041034 0.004837 8.484 1.05e-07 ***
142
6.2.4 Inference for β0 and β1
Yi = β 0 + β 1 x i + i ,
the regression parameters β0 and β1 are unknown. It is therefore of interest to construct confidence
intervals and perform hypothesis tests for these parameters.
• In practice, inference for the slope parameter β1 is of primary interest because of its connection
to the independent variable x in the model.
• Inference for β0 is less meaningful, unless one is explicitly interested in the mean of Y when
x = 0. We will not pursue this.
Confidence interval for β1 : Under our model assumptions, the following sampling distribution
arises:
βb1 − β1
t= p ∼ t(n − 2).
M Sres /SSxx
This result can be used to derive a 100(1 − α) percent confidence interval for β1 , which is given
by
p
βb1 ± tn−2,α/2 M Sres /SSxx .
• The value tn−2,α/2 is the upper α/2 quantile from the t(n − 2) distribution.
{z error} .
point estimate ± quantile × standard
√
| {z } | {z } |
βb1 tn−2,α/2 M Sres /SSxx
“We are 100(1 − α) percent confident that the population regression slope β1 is in
this interval.”
– If β1 = 0 is in the confidence interval, this suggests that Y and x are not linearly related.
– If β1 = 0 is not in the confidence interval, this suggests that Y and x are linearly related.
• The 100(1 − α) percent lower and upper confidence bounds are, respectively,
p p
βb1 − tn−2,α M Sres /SSxx and βb1 + tn−2,α M Sres /SSxx .
143
Hypothesis test for β1 : If our interest was to test
H0 : β1 = 0
Ha : β1 6= 0
Ha : β1 > 0
Ha : β1 < 0
βb1
t0 = p .
M Sres /SSxx
P-value approach
Alternative hypothesis Rejection Criterion
Ha : β1 6= 0 P-value < α
Ha : β1 > 0 P-value < α
Ha : β1 < 0 P-value < α
144
Example 6.2.4. (continued Example 6.2.1). We now use R to test
H0 : β1 = 0 versus Ha : β1 6= 0,
> summary(fit)
lm(formula = moisture ~ filtration.rate)
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 72.958547 0.697528 104.596 < 2e-16 ***
filtration.rate 0.041034 0.004837 8.484 1.05e-07 ***
Analysis: We have t = 8.484 > tn−2,α/2 = t18,0.025 = 2.1009 and P-value = 0.000000105. In
other words, there is strong evidence to reject H0 . Also a 95 percent confidence interval for β1 is
calculated as follows:
We are 95 percent confident that population regression slope β1 is between 0.0309 and 0.0511. Note
that this interval does not include “0.”
Yi = β0 + β1 xi + i ,
where i ∼ N (0, σ 2 ). We are often interested in using the fitted model to learn about the response
variable Y at a certain setting for the independent variable x = x0 , say. For example, in our sewage
sludge example, we might be interested in the moisture percentage Y when the filtration rate is
x = 150 kg-DS/m/hr. Two potential goals arise:
Y (x0 ) ∼ N (β0 + β1 x0 , σ 2 ).
Y (x0 ) ∼ N (β0 + β1 x0 , σ 2 ).
In the first problem, we are interested in estimating the mean of the response variable Y at a
certain value of x. In the second problem, we are interested in predicting the value of a new
145
random variable Y at a certain value of x. Conceptually, the second problem is far more difficult
than the first.
Goals: We would like to create 100(1 − α) percent intervals for the mean E(Y |x0 ) and for the new
value Y ∗ (x0 ). The former is called a confidence interval (since it is for a mean response) and the
latter is called a prediction interval (since it is for a new random variable).
Point estimator/Predictor: To construct either interval, we start with the same quantity:
where βb0 and βb1 are the least squares estimates from the fit of the model.
• In the confidence interval for E(Y |x0 ), we call Yb (x0 ) a point estimator.
The primary difference in the intervals arises in assessing the variability of Yb (x0 ).
Confidence interval: A 100(1 − α) percent confidence interval for the mean E(Y |x0 ) is given
by s
1 (x0 − x)2
Y (x0 ) ± tn−2,α/2 M Sres
b + .
n SSxx
Prediction interval: A 100(1 − α) percent prediction interval for the new response Y ∗ (x0 ) is
given by s
1 (x0 − x)2
Y (x0 ) ± tn−2,α/2 M Sres 1 + +
b .
n SSxx
• Comparison: The two intervals are identical except for the extra “1” in the standard error
part of the prediction interval. This extra “1” arises from the additional uncertainty associated
with predicting a new response from the N (β0 + β1 x0 , σ 2 ) distribution. Therefore, a 100(1 − α)
percent prediction interval for Y ∗ (x0 ) will be wider than the corresponding 100(1 − α) percent
confidence interval for E(Y |x0 ).
• Interval length: The length of both intervals clearly depends on the value of x0 . In fact,
the standard error of Yb (x0 ) will be smallest when x0 = x and will get larger the farther x0 is
from x in either direction. This implies that the precision with which we estimate E(Y |x0 ) or
predict Y ∗ (x0 ) decreases the farther we get away from x. This makes intuitive sense, namely,
we would expect to have the most “confidence” in our fitted model near the “center” of the
observed data.
It is sometimes desired to estimate E(Y |x0 ) or predict Y ∗ (x0 ) based on the fit of the model for
values of x0 outside the range of x values used in the experiment/study. This is called extrapolation
and can be very dangerous. In order for our inferences to be valid, we must believe that the straight
line relationship holds for x values outside the range where we have observed data. In some situations,
146
this may be reasonable. In others, we may have no theoretical basis for making such a claim without
data to support it.
Example 6.2.5. (Continued Example 6.2.1). In our sewage sludge example, suppose that we are
interested in estimating E(Y |x0 ) and predicting a new Y ∗ (x0 ) when the filtration rate is x0 = 150
kg-DS/m/hr.
• E(Y |x0 ) denotes the mean moisture percentage for compressed pellets when the machine fil-
tration rate is x0 = 150 kg-DS/m/hr. In other words, if we were to repeat the experiment
over and over again, each time using a filtration rate of x0 = 150 kg-DS/m/hr, then E(Y |x0 )
denotes the mean value of Y (moisture percentage) that would be observed.
• Y ∗ (x0 ) denotes a possible value of Y for a single run of the machine when the filtration rate is
set at x0 = 150 kg-DS/m/hr.
> predict(fit,data.frame(filtration.rate=150),level=0.95,interval="confidence")
fit lwr upr
79.11361 78.78765 79.43958
> predict(fit,data.frame(filtration.rate=150),level=0.95,interval="prediction")
fit lwr upr
79.11361 77.6783 80.54893
• A 95 percent confidence interval for E(Y |x0 = 150) is (78.79, 79.44). When the filtration
rate is x0 = 150 kg-DS/m/hr, we are 95 percent confident that the mean moisture percentage
is between 78.79 and 79.44 percent.
• A 95 percent prediction interval for Y ∗ (x0 = 150) is (77.68, 80.55). When the filtration rate
is x0 = 150 kg-DS/m/hr, we are 95 percent confident that the moisture percentage for a single
run of the experiment will be between 77.68 and 80.55 percent.
147
81
95% confidence
95% prediction
80
Moisture (Percentage)
79
78
77
Figure 6.2.3: Scatterplot of pellet moisture Y as a function of machine filtration rate x, including
the least squares regression line and ninety-five percent confidence/prediction bands.
148
6.3 Multiple linear regression
6.3.1 Introduction
Yi = β0 + β1 xi + i ,
for i = 1, 2, ..., n, where i ∼ N (0, σ 2 ). We now extend this basic model to include multiple inde-
pendent variables x1 , x2 , ..., xk . This is much more realistic because, in practice, often Y depends on
many different factors (i.e., not just one). Specifically, we consider models of the form
• There are now p = k + 1 regression parameters β0 , β1 , ..., βk . These are unknown and are to
be estimated with the observed data.
Individual Y x1 x2 ··· xk
1 Y1 x11 x12 ··· x1k
2 Y2 x21 x22 ··· x2k
.. .. .. .. .. ..
. . . . . .
n Yn xn1 xn2 ··· xnk
• Each of the n individuals contributes a response Y and a value of each of the independent
variables x1 , x2 , ..., xk .
• We also assume that the independent variables x1 , x2 , ..., xk are fixed and measured without
error.
we again use the method of least squares. Simple computing formulae for the least squares esti-
mators are no longer available (as they were in simple linear regression). This is hardly a big deal
because we will use computing to automate all analyses. For instructional purposes, it is advanta-
geous to express multiple linear regression models in terms of matrices and vectors. This streamlines
notation and makes the presentation easier.
149
6.3.2 Least square estimator
The notion of least squares is the same as it was in the simple linear regression model. To fit a
multiple linear regression model, we want to find the values of β0 , β1 , ..., βk that minimize
n
X
Q(β0 , β1 , ..., βk ) = [Yi − (β0 + β1 xi1 + β2 xi2 + · · · + βk xik )]2 .
i=1
∂Q(β0 , β1 , ..., βk )
= 0, for j = 0, . . . , k.
∂βj
And the R package can help you find the solution. See the following example.
Example 6.3.1. The taste of matured cheese is related to the concentration of several chemicals
in the final product. In a study from the LaTrobe Valley of Victoria, Australia, samples of cheddar
cheese were analyzed for their chemical composition and were subjected to taste tests. For each
specimen, the taste Y was obtained by combining the scores from several tasters. Data were collected
on the following variables:
Y = taste score (TASTE)
x1 = concentration of acetic acid (ACETIC)
x2 = concentration of hydrogen sulfide (H2S)
x3 = concentration of lactic acid (LACTIC).
Variables ACETIC and H2S were both measured on the log scale. The variable LACTIC has not
been transformed. Table 6.5 contains concentrations of the various chemicals in n = 30 specimens
of cheddar cheese and the observed taste score.
Specimen TASTE ACETIC H2S LACTIC Specimen TASTE ACETIC H2S LACTIC
1 12.3 4.543 3.135 0.86 16 40.9 6.365 9.588 1.74
2 20.9 5.159 5.043 1.53 17 15.9 4.787 3.912 1.16
3 39.0 5.366 5.438 1.57 18 6.4 5.412 4.700 1.49
4 47.9 5.759 7.496 1.81 19 18.0 5.247 6.174 1.63
5 5.6 4.663 3.807 0.99 20 38.9 5.438 9.064 1.99
6 25.9 5.697 7.601 1.09 21 14.0 4.564 4.949 1.15
7 37.3 5.892 8.726 1.29 22 15.2 5.298 5.220 1.33
8 21.9 6.078 7.966 1.78 23 32.0 5.455 9.242 1.44
9 18.1 4.898 3.850 1.29 24 56.7 5.855 10.20 2.01
10 21.0 5.242 4.174 1.58 25 16.8 5.366 3.664 1.31
11 34.9 5.740 6.142 1.68 26 11.6 6.043 3.219 1.46
12 57.2 6.446 7.908 1.90 27 26.5 6.458 6.962 1.72
13 0.7 4.477 2.996 1.06 28 0.7 5.328 3.912 1.25
14 25.9 5.236 4.942 1.30 29 13.4 5.802 6.685 1.08
15 54.9 6.151 6.752 1.52 30 5.5 6.176 4.787 1.25
Table 6.5: Cheese data. ACETIC, H2S, and LACTIC are independent variables. The response variable
is TASTE.
150
MODEL: Researchers postulate that each of the three chemical composition variables x1 , x2 , and
x3 is important in describing the taste and consider the multiple linear regression model
for i = 1, 2, ..., 30. We now use R to fit this model using the method of least squares:
Call:
lm(formula = taste ~ acetic + h2s + lactic)
Residuals:
Min 1Q Median 3Q Max
-17.390 -6.612 -1.009 4.908 25.449
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -28.8768 19.7354 -1.463 0.15540
acetic 0.3277 4.4598 0.073 0.94198
h2s 3.9118 1.2484 3.133 0.00425 **
lactic 19.6705 8.6291 2.280 0.03108 *
---
Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
151
6.3.3 Estimating the error variance
where i ∼ N (0, σ 2 ),
SSres
M Sres =
n−p
is an unbiased estimator of σ 2 , that is,
E(M Sres ) = σ 2 .
The quantity s
p SSres
b = M Sres =
σ
n−p
estimates σ and is called the residual standard error.
Example 6.3.2. For the cheese data in Example 6.3.1, we use R to calculate M Sres :
152
Table 6.6: Analysis of variance table for linear regression.
Source df SS MS F
SSreg M Sreg
Regression k SSreg M Sreg = k F = M Sres
SSres
Residual n−p SSres M Sres = n−p
Total n−1 SStotal
• SStotal is the total sum of squares. SStotal is the numerator of the sample variance of
Y1 , Y2 , ..., Yn . It measures the total variation in the response data.
• SSreg is the regression sum of squares. SSreg measures the variation in the response data
explained by the linear regression model.
• SSres is the residual sum of squares. SSres measures the variation in the response data not
explained by the linear regression model.
ANOVA TABLE : We can combine all of this information to produce an analysis of variance
(ANOVA) table. Such tables are standard in regression analysis.
• The degrees of freedom (df) add down.
– SStotal can be viewed as a statistic that has “lost” a degree of freedom for having to
estimate the overall mean of Y with the sample mean Y . Recall that n − 1 is our divisor
in the sample variance S 2 .
– There are k degrees of freedom associated with SSreg because there are k independent
variables.
– The degrees of freedom for SSres can be thought of as the divisor needed to create an
unbiased estimator of σ 2 . Recall that
SSres SSres
M Sres = =
n−p n−k−1
is an unbiased estimator of σ 2
• The sum of squares (SS) also add down. This follows from the algebraic identity noted
earlier.
153
• Mean squares (MS) are the sums of squares divided by their degrees of freedom.
• The F statistic is formed by taking the ratio of M Sreg and M Sres . More on this in a moment.
the proportion of the total variation in the data explained by the linear regression model is
SSreg SSres
R2 = =1− ..
SStotal SStotal
0 ≤ R2 ≤ 1.
The larger the R2 , the better the regression model explains the variability in the data.
IMPORTANT : It is critical to understand what R2 does and does not measure. Its value is
computed under the assumption that the multiple linear regression model is correct and assesses
how much of the variation in the data may be attributed to that relationship rather than to inherent
variation.
• If R2 is small, it may be that there is a lot of random inherent variation in the data, so that,
although the multiple linear regression model is reasonable, it can explain only so much of the
observed overall variation.
• Alternatively, R2 may be close to 1; e.g., in a simple linear regression model fit, but this may
not be the best model. In fact, R2 could be very “high,” but ultimately not relevant because
it assumes the simple linear regression model is correct. In reality, a better model may exist
(e.g., a quadratic model, etc.).
• One draw back of R2 is that, it can never decrease when a new x is added. Thus, it can be
difficult to judge whether the increase is telling us anything useful about the new x. Instead,
many regression users prefer to use an adjusted R2 statistic:
2 SSres /(n − p)
Radj =1− .
SStotal /(n − 1)
154
F statistics: The F statistic in the ANOVA table is used to test
H0 : β1 = β2 = · · · = βk = 0
versus
Ha : at least one of the βj is nonzero.
In other words, F tests whether or not at least one of the independent variables x1 , x2 , ..., xk is
important in describing the response Y . If H0 is rejected, we do not know which one or how many
of the βj ’s are nonzero; only that at least one is.
M Sreg
F = ∼ F (k, n − p).
M Sres
Therefore, we can gauge the evidence against H0 by comparing F to this distribution. Values of F
far out in the (right) upper tail are evidence against H0 . R automatically produces the value of F
and produces the corresponding p-value. Recall that small p-values are evidence against H0 (the
smaller the p-value, the more evidence).
Example 6.3.3. Example 6.3.1 (continued). For the cheese data, we fit the multiple linear regression
model
Yi = β0 + β1 xi1 + β2 xi2 + β3 xi3 + i ,
for i = 1, 2, ..., 30. The ANOVA table, obtained using R, is shown below.
155
The F statistic is used to test
H0 : β1 = β2 = β3 = 0
versus
Ha : at least one of the βj is nonzero.
Based on the F statistic (F = 16.22), and the corresponding probability value (p-value < 0.0001), we
have strong evidence to reject H0 . Interpretation: We conclude that at least one of the independent
variables (ACETIC, H2S, LACTIC) is important in describing taste.
The coefficient of determination R2 is 0.6518; i.e., about 65.2 percent of the variability in the
taste data is explained by the linear regression model that includes ACETIC, H2S, and LACTIC. The
remaining 34.8 percent of the variability in the taste data is explained by other sources.
for i = 1, 2, ..., n, where i ∼ N (0, σ 2 ). Confidence intervals and hypothesis tests for βj can help us
assess the importance of using the independent variable xj in a model with the other independent
variables. That is, inference regarding βj is always conditional on the other variables being included
in the model.
A 100(1 − α) percent confidence interval for βj , for j = 0, 1, 2, ..., k,
H0 : βj = 0
versus
Ha : βj 6= 0,
156
Hypothesis test for β1 : If our interest was to test, where j = 0, 1, . . . , k
H0 : βj = 0
Ha : βj 6= 0
Ha : βj > 0
Ha : βj < 0
where β10 is a fixed value (often, β10 = 0), we would focus our attention on
βbj
t0 = .
se(
b βbj )
P-value approach
Alternative hypothesis Rejection Criterion
Ha : βj 6= 0 P-value < α
Ha : βj > 0 P-value < α
Ha : βj < 0 P-value < α
157
Example 6.3.4. For the cheese data, Example 6.3.1 continued,
Call:
lm(formula = taste ~ acetic + h2s + lactic)
Residuals:
Min 1Q Median 3Q Max
-17.390 -6.612 -1.009 4.908 25.449
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -28.8768 19.7354 -1.463 0.15540
acetic 0.3277 4.4598 0.073 0.94198
h2s 3.9118 1.2484 3.133 0.00425 **
lactic 19.6705 8.6291 2.280 0.03108 *
---
Signif. codes: 0 ?***? 0.001 ?**? 0.01 ?*? 0.05 ?.? 0.1 ? ? 1
OUTPUT : The Estimate output gives the values of the least squares estimates:
se(
b βb0 ) = 19.735
b βb0 ) = 4.460
se(
se(
b βb0 ) = 1.248
se(
b βb0 ) = 8.629
To test
H0 : βj = 0 versus H0 : βj 6= 0,
for j = 0, 1, 2, 3, via the confidence interval approach, we have 95% confidence intervals for the
158
regression parameters β0 , β1 , β2 , and β3 , respectively, are, (since tn−p,α/2 = t26,.025 = 2.056),
The conclusions reached from interpreting these intervals are the same as those reached using the
hypothesis test p-values. Note that the β2 and β3 intervals do not include zero. Those for β0 and β1
do.
The t value output gives the t statistics
t0 = −1.463
t0 = 0.074
t0 = 3.133
t0 = 2.279
These tests can also be done use the P-values in Pr(>|t|) output. At the α = 0.05 level,
159
6.3.6 Confidence and prediction intervals for a given x = x0
GOALS : We would like to create 100(1 − α) percent intervals for the mean E(Y |x0 ) and for the new
value Y ∗ (x0 ). As in the simple linear regression case, the former is called a confidence interval
(since it is for a mean response) and the latter is called a prediction interval (since it is for a new
random variable).
CHEESE DATA: Suppose that we are interested estimating E(Y |x0 ) and predicting a new Y ∗ (x0 )
when ACETIC = 5.5, H2S = 6.0, and LACTIC = 1.4, so that x0 = (5.5, 6.0, 1.4). We use R to compute
the following:
> predict(fit,data.frame(acetic=5.5,h2s=6.0,lactic=1.4),level=0.95,interval="confidence")
fit lwr upr
23.93552 20.04506 27.82597
> predict(fit,data.frame(acetic=5.5,h2s=6.0,lactic=1.4),level=0.95,interval="prediction")
fit lwr upr
23.93552 2.751379 45.11966
• A 95 percent confidence interval for E(Y |x0 ) is (20.05, 27.83). When ACETIC = 5.5, H2S =
6.0, and LACTIC = 1.4, we are 95 percent confident that the mean taste rating is between 20.05
and 27.83.
• A 95 percent prediction interval for Y ∗ (x0 ), when x = x0 , is (2.75, 45.12). When ACETIC =
5.5, H2S = 6.0, and LACTIC = 1.4, we are 95 percent confident that the taste rating for a new
cheese specimen will be between 2.75 and 45.12.
160
6.4 Model diagnostics (residual analysis)
IMPORTANCE : We now discuss certain diagnostic techniques for linear regression. The term “di-
agnostics” refers to the process of “checking the model assumptions.” This is an important exercise
because if the model assumptions are violated, then our analysis (and all subsequent interpretations)
could be compromised.
MODEL ASSUMPTIONS : We first recall the model assumptions on the error terms in the linear
regression model
Yi = β0 + β1 xi1 + β2 xi2 + · · · + βk xik + i ,
RESIDUALS : In checking our model assumptions, we first have to deal with the obvious problem;
namely, the error terms i in the model are never observed. However, from the fit of the model, we
can calculate the residuals
ei = Yi − Ybi ,
We can think of the residuals e1 , e2 , ..., en as “proxies” for the error terms 1 , 2 , ..., n , and, therefore,
we can use the residuals to check our model assumptions instead.
161
20
10
Sample Quantiles
0
−10
−2 −1 0 1 2
Theoretical Quantiles
Figure 6.4.4: Cheese data. Normal qq-plot of the least squares residuals.
QQ PLOT FOR NORMALITY : To check the normality assumption (for the errors) in linear
regression, it is common to display the qq-plot of the residuals.
• Recall that if the plotted points follow a straight line (approximately), this supports the nor-
mality assumption.
• Substantial deviation from linearity is not consistent with the normality assumption.
• The plot in Figure 6.4.4 supports the normality assumption for the errors in the multiple linear
regression model for the cheese data.
162
RESIDUAL PLOT : By the phrase “residual plot,” I mean the plot of the residuals (on the
vertical axis) versus the predicted values (on the horizontal axis). This plot is simply the scatterplot
of the residuals and the predicted values.
• Advanced linear model arguments show that if the model does a good job at describing the
data, then the residuals and fitted values are independent.
• This means that a plot of the residuals versus the fitted values should reveal no noticeable
patterns; that is, the plot should appear to be random in nature (e.g., “a random scatter of
points”).
• On the other hand, if there are definite (non-random) patterns in the residual plot, this sug-
gests that the model is inadequate in some way or it could point to a violation in the model
assumptions.
• The plot in Figure 6.4.5 does not suggest any obvious model inadequacies! It looks completely
random in appearance.
20
10
Residuals
0
−10
0 10 20 30 40 50
Fitted values
Figure 6.4.5: Cheese data. Residual plot for the multiple linear regression model fit. A horizontal
line at zero has been added.
COMMON VIOLATIONS : Although there are many ways to violate the statistical assumptions
associated with linear regression, the most common violations are
163
• misspecifying the true regression function
2
10
Peak Demand (kWh)
Residuals
0
5
−2
−4
0
Figure 6.4.6: Electricity data. Left: Scatterplot of peak demand (Y , measured in kWh) versus
monthly usage (x, measured in kWh) with least squares simple linear regression line superimposed.
Right: Residual plot for the simple linear regression model fit.
Example 6.3. An electric company is interested in modeling peak hour electricity demand (Y )
as a function of total monthly energy usage (x). This is important for planning purposes because
the generating system must be large enough to meet the maximum demand imposed by customers.
Data for n = 53 residential customers for a given month are shown in Figure 6.4.6.
Problem: There is a clear problem with non-constant variance here. Note how the residual plot
“fans out” like the bell of a trumpet. This violation may have been missed by looking at the
scatterplot alone, but the residual plot highlights it.
Remedy: A common course of action to handle non-constant variance is to apply a transformation
√
to the response variable Y . Common transformations are logarithmic (ln Y ), square-root ( Y ), and
inverse (1/Y ).
ELECTRICITY DATA: A square root transformation is commonly applied to address non-
constant variance. Consider the simple linear regression model
Wi = β0 + β1 xi + i ,
√
for i = 1, 2, ..., 53, where Wi = Yi . It is straightforward to fit this transformed model in R as before.
We simply regress W on x (instead of regressing Y on x).
164
4.0
3.5
0.5
Peak Demand (kWh): Square root scale
3.0
0.0
2.5
Residuals
2.0
−0.5
1.5
−1.0
1.0
0.5
500 1000 1500 2000 2500 3000 3500 1.0 1.5 2.0 2.5 3.0 3.5 4.0
√
Figure 6.4.7: Electricity data. Left: Scatterplot of the square root of peak demand ( Y ) versus
monthly usage (x, measured in kWh) with the least squares simple linear regression line superim-
posed. Right: Residual plot for the simple linear regression model fit with transformed response.
ANALYSIS : Figure 6.4.7 above shows the scatterplot (left) and the residual plot (right) from
fitting the transformed model. The “fanning out” shape that we saw previously (in the untransformed
model) is now largely absent. The fitted transformed model is
W
c = 0.580831 + 0.000953x,
Further analyses can be carried out with the transformed model; e.g., testing whether peak demand
(on the square root scale) is linearly related to monthly usage, etc.
165
0.2
2.0
0.0
1.5
DC Output
Residuals
−0.2
1.0
−0.4
0.5
−0.6
4 6 8 10 1.0 1.5 2.0 2.5
Figure 6.4.8: Windmill data. Left: Scatterplot of DC output Y versus wind velocity (x, measured
in mph) with least squares simple linear regression line superimposed. Right: Residual plot for the
simple linear regression model fit.
Example 6.4. A research engineer is investigating the use of a windmill to generate electricity.
He has collected data on the direct current (DC) output Y from his windmill and the corresponding
wind velocity (x, measured in mph). Data for n = 25 observation pairs are shown in Figure 6.4.8.
Problem: There is a clear quadratic relationship between DC output and wind velocity, so a simple
linear regression model fit (as shown above) is inappropriate. The residual plot shows a pronounced
quadratic pattern; this pattern is not accounted for in fitting a straight line model.
Remedy: Fit a multiple linear regression model with two independent variables: wind velocity x
and its square x2 , that is, consider the quadratic regression model
Yi = β0 + β1 xi + β2 x2i + i ,
for i = 1, 2, ..., 25. It is straightforward to fit a quadratic model in R. We simply regress Y on x and
x2 .
166
0.2
2.0
0.1
1.5
0.0
DC Output
Residuals
1.0
−0.1
0.5
−0.2
4 6 8 10 0.5 1.0 1.5 2.0
Figure 6.4.9: Windmill data. Scatterplot of DC output Y versus wind velocity (x, measured in mph)
with least squares quadratic regression curve superimposed. Right: Residual plot for the quadratic
regression model fit.
Note that the residual plot from the quadratic model fit, shown above, now looks quite good. The
quadratic trend has disappeared (because the model now incorporates it).
167
Example 6.5. The data in Figure 6.4.10 (left) are temperature readings (in deg C) on land-air
average temperature anomalies, collected once per year from 1900-1997. To emphasize that the data
are collected over time, I have used straight lines to connect the observations; this is called a time
series plot.
• Unfortunately, it is all too common that people fit linear regression models to time series data
and then blindly use them for prediction purposes.
• It takes neither a meteorologist nor an engineering degree to know that temperature obser-
vations collected over time are probably correlated. Not surprisingly, residuals from a simple
linear regression display clear correlation over time.
• Regression techniques (as we have learned in this chapter) are generally not appropriate when
analyzing time series data for this reason. More advanced modeling techniques are needed.
0.4
2
Global temperature deviations (since 1900)
0.2
1
Residuals
0.0
0
−0.2
−1
−0.4
−2
Year Year
Figure 6.4.10: Global temperature data. Left: Time series plot of the temperature Y measured one
time per year. The independent variable x is year, measured as 1900, 1901, ..., 1997. A simple linear
regression model fit has been superimposed. Right: Residual plot from the simple linear regression
model fit.
168