18.650 - Fundamentals of Statistics
18.650 - Fundamentals of Statistics
1/32
Linear model
And 1
>
IE(Y |X = x) = µ(x) = x ,
1
Throughout we drop the boldface notation for vectors 2/32
Components of a linear model
3/32
Kyphosis
4/32
Kyphosis
5/32
Generalization
1. Random component:
Y |X = x ⇠ some distribution
2. Regression function:
>
µ(x) = x
6/32
Predator/Prey
Consider the following model for the number of preys Y that a
predator (Hawk) catches per day a predator given a number X of
preys (mice) in its hunting territory.
Random component: Y > 0 and the variance of capture rate is
known to be approximately equal to its expectation so we propose
the following model:
Y |X = x ⇠
8/32
Example 2: Prey Capture Rate
1 1
g(µ(x)) = = = 0+ 1 .
µ(x) x
9/32
Exponential Family
hX
k i
f✓ (y) = exp ⌘i (✓)Ti (y) B(✓) h(y)
i=1
10/32
Normal distribution example
I Consider Y ⇠ N (µ, 2 ), ✓ = (µ, 2 ). The density is
⇣µ 1 µ 2 ⌘ 1
2
f✓ (y) = exp 2
y 2
y 2
p ,
2 2 2⇡
which forms a two-parameter exponential family with
µ 1 2
⌘1 = 2
, ⌘ 2 = 2
, T 1 (y) = y, T 2 (y) = y ,
2
µ 2 p
B(✓) = 2
+ log( 2⇡), h(y) = 1.
2
I When 2 is known, it becomes a one-parameter exponential
family on IR:
y2
µ µ 2 e 2 2
⌘= 2
, T (y) = y, B(✓) = 2
, h(y) = p .
2 2⇡
11/32
Examples of discrete distributions
I Bernoulli(p): p (1y
p) 1 y
, y 2 {0, 1}
y
I Poisson( ): e , y = 0, 1, . . . .
y!
12/32
Examples of Continuous distributions
The following distributions form continuous exponential families
of distributions with pdf:
1 y
I Gamma(a, b): y a 1
e b;
(a)ba
I above: a: shape parameter, b: scale parameter
I reparametrize: µ = ab: mean parameter
✓ ◆a
1 a a 1
ay
y e µ.
(a) µ
↵
I Inverse Gamma(↵, ): y ↵ 1
e /y
.
(↵)
s
2 2 (y µ)2
I Inverse Gaussian(µ, 2 ): e 2µ2 y .
2⇡y 3
14/32
Normal distribution example
I Therefore ✓ = µ, = 2, b(✓) = ✓2
2 , and
1 y 2
c(y, ) = ( + log(2⇡ )).
2
15/32
Other distributions
16/32
Likelihood
@2` @` 2
IE( 2 ) + IE( ) = 0.
@✓ @✓
17/32
Expected value
Note that
Y✓ b(✓)
`(✓) = + c(Y ; ),
Therefore
@`
=
@✓
It yields
@` IE(Y ) 0
b (✓)
0 = IE( ) = ,
@✓
which leads to
IE(Y ) =
18/32
Variance
On the other hand we have we have
@2` @` 2
+ ( ) =
@✓2 @✓
and from the previous result,
Y 0
b (✓) Y IE(Y )
=
which leads to
var(Y ) =
19/32
Example: Poisson distribution
So
✓ 00
µ=e , b(✓) = b (✓) =
20/32
Link function
21/32
Examples of link functions
22/32
Examples of link functions for Bernoulli response
5
2
I in blue:
1
1 g1 (x) = f1 (x) =
x
log (logit link)
0
1 x
-1
I in red:
1 1
g2 (x) = f2 (x) = (x)
-2
(probit link)
-3
-4
-5
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
23/32
Examples of link functions for Bernoulli response
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
-5 -4 -3 -2 -1 0 1 2 3 4 5
ex
I in blue: f1 (x) =
1 + ex
I in red: f2 (x) = (x) (Gaussian CDF)
24/32
Canonical Link
g(µ) = ✓
25/32
Example: the Bernoulli distribution
I Hence we solve
0 exp(✓)
b (✓) = =µ , ✓=
1 + exp(✓)
I The canonical link for the Bernoulli distribution is the
26/32
Other examples
b(✓) g(µ)
Normal 2
✓ /2 µ
Poisson exp(✓) log µ
✓ µ
Bernoulli log(1 + e ) log 1 µ
1
Gamma log( ✓) µ
27/32
Model and notation
yi ✓ i b(✓i ) o
f✓i (yi ) = exp + c(yi , ) .
28/32
Back to
where h is defined as
0 1 1 0 1
h = (b ) g = (g b ) .
29/32
Log-likelihood
up to a constant term.
I Note that when we use the canonical link function, we obtain
the simpler expression
X Yi X > >
b(Xi )
i
`n (Y, X, ) =
i
30/32
Strict concavity
31/32
Concluding remarks
32/32