SLChapter 7
SLChapter 7
Models
Chapter 7
Generalized Linear Models
EEE 485/585 Statistical Learning and Data Analytics The classification
problem
Simple logistic
regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
Cem Tekin
Bilkent University
Response Y is categorical.
Y 2 {influenza, common cold, Covid-19} (3 classes)
Predictors: X1 , . . . , Xp
Can we use linear regression?
The classification
problem
Simple logistic
regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.2
Generalized Linear
The classification problem Models
Response Y is categorical.
Y 2 {influenza, common cold, Covid-19} (3 classes)
Predictors: X1 , . . . , Xp
Can we use linear regression?
The classification
problem
Map classes to real values? Simple logistic
regression
8
<1 if influenza
> Example 1
Multiple logistic
Y = 2 if common cold regression
>
:
3 if Covid-19 Example 2
Multiple logistic
regression - extensions
Use least squares, ridge or lasso Generalized linear
models
Poisson regression
7.2
Generalized Linear
The classification problem Models
Response Y is categorical.
Y 2 {influenza, common cold, Covid-19} (3 classes)
Predictors: X1 , . . . , Xp
Can we use linear regression?
The classification
problem
Map classes to real values? Simple logistic
regression
8
<1 if influenza
> Example 1
Multiple logistic
Y = 2 if common cold regression
>
:
3 if Covid-19 Example 2
Multiple logistic
regression - extensions
Use least squares, ridge or lasso Generalized linear
models
8
<1 if common cold
>
Y = 2 if Covid-19
>
:
3 if influenza
7.2
Generalized Linear
Problem with linear regression Models
Give some examples of classes which have a natural order Simple logistic
regression
Example 1
movie ratings
1,2191
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.3
Generalized Linear
Should we use linear regression for classification Models
Simple logistic
and regression
( Example 1
1 if influenza Example 2
Multiple logistic
regression - extensions
will produce the same result
IF 1
Generalized linear
Oi
5
models
Poisson regression
e 0+ 1X
⇡(X ) = (logistic function)
1 + e 0+ 1X The classification
problem
limX ! 1 ⇡(X )? limX !+1 ⇡(X )? Simple logistic
regression
⇡(X ) models Pr(Y = 1|X ) Example 1
Multiple logistic
L
regression
II Fat fist
0 o
Ri
Example 2
Multiple logistic
regression - extensions
Generalized linear
not
gig
models
Poisson regression
or 7.5
Generalized Linear
Simple logistic regression Models
Odds:
P.MS ⇡(X )
=e 0+ 1X
1 ⇡(X )
PrCUoHI
Log-odds (logit): The classification
problem
✓ ◆ Simple logistic
⇡(X ) regression
logit(⇡(X )) = log = 0+ 1X
1 ⇡(X ) Example 1
Multiple logistic
regression
logit is linear in X ! Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.6
Generalized Linear
MLE for simple logistic regression Models
The classification
Y Y problem
L( 0, 1) = p(D| 1, 0) = ⇡(xi ) (1 ⇡(xi 0 )) Simple logistic
regression
i:yi =1 i 0 :yi 0 =0
Example 1
j
Log-likelihood: Multiple logistic
regression
Example 2
Multiple logistic
n
X regression - extensions
0 + 1 xi
l( 0, 1) = log p(D| 1, 0) = [yi ( 0 + 1 xi ) log(1 + e )] Generalized linear
models
i=1
Poisson regression
MLE:
7.7
Generalized Linear
MLE for simple logistic regression Models
Simple logistic
regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.8
Generalized Linear
Simple logistic regression - Example Models
thresher
|||||||||||||||
|| ||||||| ||||||||||||||||||||||||| | |||| | | | | | | | | | | || | ||||| || ||| ||||||||||| ||||| |||||||||||||||||| |||||||| ||||||||||| |||||||||||||||||||||||||||| ||||||||||||||||||||||||| | |||| | | | | | |
Probability of Default
Simple logistic
regression
0.8
7 I
Example 1
0.5
0.6
Multiple logistic
regression
7 O
0.4
Example 2
pros eye
Multiple logistic
0.2
regression - extensions
Generalized linear
I
0.0
Poisson regression
Balance
ˆ MLE + ˆ MLE x
1 e 0
Orange: data points. Blue: ⇡(x) = ˆ MLE + ˆ MLE x
(logistic function)
1+e 0 1
Figure 4.2 from “An introduction to statistical learning" by James et al. 7.9
Generalized Linear
Simple logistic regression - Example Models
ˆMLE = 10.65
0
ˆMLE = 0.0055
1
Log-odds:
✓ ◆
Pr(default = 1|balance = X ) The classification
log = 10.65 + 0.0055X problem
Pr(default = 0|balance = X ) Simple logistic
regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.10
Generalized Linear
Simple logistic regression - Example Models
ˆMLE = 10.65
0
ˆMLE = 0.0055
1
Log-odds:
✓ ◆
Pr(default = 1|balance = X ) The classification
log = 10.65 + 0.0055X problem
Pr(default = 0|balance = X ) Simple logistic
regression
Multiple logistic
1 What is the probability that an individual with a balance of regression
Multiple logistic
2 What is the probability that an individual with a balance of regression - extensions
Poisson regression
7.10
Generalized Linear
Simple logistic regression - Example Models
ˆMLE = 10.65
0
ˆMLE = 0.0055
1
Log-odds:
✓ ◆
Pr(default = 1|balance = X ) The classification
log = 10.65 + 0.0055X problem
Pr(default = 0|balance = X ) Simple logistic
regression
Multiple logistic
1 What is the probability that an individual with a balance of regression
Multiple logistic
2 What is the probability that an individual with a balance of regression - extensions
Poisson regression
e 10.65+0.0055⇥1000
1 ⇡(1000) = 1+e 10.65+0.0055⇥1000
⇡ 0.0058 = 0.58%
e 10.65+0.0055⇥2000
2 ⇡(2000) = 1+e 10.65+0.0055⇥2000
⇡ 0.586 = 58.6%
7.10
Generalized Linear
Simple logistic regression - Example Models
ŷ =
( i
0 if ⇡(x) < 0.5
n too
o N96
1 if ⇡(x) 0.5 The classification
problem s P 4
96
g 0 for all data points aceracy
Risk averse: Simple logistic
regression
( Example 1
0 if ⇡(x) < 0.1
ŷ = Multiple logistic
regression
Conform matrix
1 if ⇡(x) 0.1
Example 2
predictions Twtreenegative
4 Multiple logistic
regression - extensions Fp false positive
carefullytuned
IIE
Generalized linear
models
Poisson regression
TP true positive
FPOFAY
pro Far S Faizfalsenegativerole
7.11
Generalized Linear
Simple logistic regression - Example Models
y
0 Simple logistic
regression
ˆMLE = 0.4049
1 Example 1
Example 2
e 3.5041
2 ⇡(no student) = 1+e 3.5041
⇡ 0.03 Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.12
Generalized Linear
Multiple logistic regression Models
Response: Y 2 {0, 1}
Predictors: X = [1, X1 , X2 , . . . , Xp ]T
T
Parameters: =[ 0, 1, . . . , p]
Logistic function:
The classification
problem
T
e 0 + 1 X1 +...+ p Xp e X Simple logistic
⇡(X ; ) = = regression
1 + e 0 + 1 X1 +...+ p Xp T
1+e X Example 1
Multiple logistic
⇡(X ; ) models Pr(Y = 1|X1 , X2 , . . . , Xp ; ) regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.13
Generalized Linear
Multiple logistic regression Models
Odds:
⇡(X ; ) T
X
=e
1 ⇡(X ; )
Multiple logistic
regression
logit is linear in X ! Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.14
Generalized Linear
MLE for multiple logistic regression
itt.gg
FYFEI
Models
Simple logistic
i=1
Multiple logistic
let we be or initid at no
winterguytay
regression
expansion
MLE: flints Example 2
f
ar
I
Poisson regression
7.15
eotirae
of
flail f lust
Generalized Linear
Newton-Raphson method for multiple logistic regression
2 3 2 3 2 3
Models
we w
1 x11 . . . x1p ⇡(x1 ; old ) y1
61 x21 . . . x2p 7 6⇡(x2 ; old )7 6 y2 7
6 7 6 7 6 7
cent
anti un f un f
X = 6. .. .. .. 7 ⇡ = 6 .. 7 y = 6 .. 7
4 .. . . . 5 4 . 5 4.5 Neetaupdate
1 xn1 ... xnp old yn
⇡(xn ; )
W=
2
⇡(x1 ; old
)(1
0
⇡(x1 ; old ))
⇡(x2 ; old )(1
0
old
... 0
0
3 The classification
problem Wnw anti a sty
6 ⇡(x2 ; )) . . . 7 Simple logistic
6
4 .. .. ..
.
.. 7 regression
5 Example 1 Newton update with legningrah
. . .
old old
Example 2
for f RDR
Multiple logistic
regression - extensions Newton update
Generalized linear
models
en get
offing't
cent
Poisson regression
Intl
intern
of
Heroics
II t of second
derivatives
order parts
f f
7.16
e
Hi Dif dfw
Iffy
Generalized Linear
Newton-Raphson method for multiple logistic regression Models
2 3 2 3 2 3
1 x11 . . . x1p ⇡(x1 ; old ) y1
61 x21 . . . x2p 7 6⇡(x2 ; old )7 6 y2 7
6 7 6 7 6 7
X = 6. .. .. .. 7 ⇡ = 6 .. 7 y = 6 .. 7
4 .. . . . 5 4 . 5 4.5
1 xn1 ... xnp old yn
iii i
⇡(xn ; )
W=
2 old
pied 3 The classification converges in a few iterations
⇡(x1 ; old ))
corpse cost it large
⇡(x1 ; )(1 0 ... 0 problem
iteration
6 0 ⇡(x2 ; old )(1 ⇡(x2 ; old
)) . . . 0 7 Simple logistic per
lexpecilywhen plarge
6 .. .. .. .. 7 regression
4 . . . .
5 Example 1
old old
0 0 . . . ⇡(xn ; )(1 ⇡(xn ; )) Multiple logistic
regression
Example 2
Newton
new
= old
+ (XT WX) 1
XT (y ⇡) Multiple logistic
regression - extensions
T 1 T
= (X WX) X W z Generalized linear
| {z } models
weighted least squares converged to a loadmin
Poisson regression
Offenko
old 1
where z = X +W (y ⇡) is the adjusted response.
cheaper periteration
convergence usually takes a larger of
number iterations
7.16 I I f l
Generalized Linear
Newton-Raphson method for multiple logistic regression Models
new
= (XT WX) 1
XT Wz is the solution to
new
arg min(z X )T W(z X )
Simple logistic
Update until convergence regression
withorigindleostsquo Example 1
Multiple logistic
regression
Corpor
ly Xfl
Example 2
Generalized linear
models
Poisson regression
7.17
Generalized Linear
Multiple logistic regression example Models
Simple logistic
MLE estimate:
man
regression
than.gs9E
i
Example 1
Generalized linear
students on my likely
6
models
Poisson regression
Anything wrong?
On average students are more likely to default than
non-students
in
Given a certain balance and income students are less
likely to default
7.18
Generalized Linear
Multiple logistic regression example Models
2500
0.8
1500
The classification
0.4
problem
1000
Simple logistic
regression
0.2
500
Example 1
0.0
Multiple logistic
0
regression
500 1000 1500 2000 No Yes
Example 2
Credit Card Balance Student Status
Multiple logistic
regression - extensions
Figure 4.3 from “An introduction to statistical learning" by James et al. 7.19
Generalized Linear
Boxplot example for a Gaussian distribution Models
The classification
problem
Simple logistic
regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.20
Generalized Linear
l1 regularized logistic regression
IIE
Models
Optimization problem:
0 1
BX p C
MLE ˆ MLE
B n T T
xi
X C
ˆ
( 0 , ) = arg max B
B [yi xi log(1 + e )] | j |C
C The classification
0, @ i=1 j=1 A problem
| {z } Simple logistic
regression
log-likelihood
Example 1
Multiple logistic
What happens as ! 0?, ! 1? regression
I
Example 2
a Multiple logistic
regression - extensions
Poisson regression
7.21
Generalized Linear
l2 regularized logistic regression Models
Optimization problem:
0 1
BX p C
MLE ˆ MLE
B n T T
xi
X C
2C
ˆ
( 0 , ) = arg max B
B [yi xi log(1 + e )] j C The classification
0, @ i=1 j=1 A problem
| {z } Simple logistic
regression
log-likelihood
Example 1
Multiple logistic
What happens as ! 0?, ! 1? regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.22
Generalized Linear
Logistic regression for K > 2 classses Models
Response: Y 2 C, C = {1, 2, . . . , K }.
Predictors: X = [1, X1 , X2 , . . . , Xp ]T O
T
Parameters: j =[ j0 , j1 , . . . , jp ] , j = 1, . . . , K
Example 1
1 or 12 1kt
regression
terrifies I Example 2
T T Multiple logistic
t
regression - extensions
11 1k j
Poisson regression
lossifier j j or
i
label I label o trainstgH
7.23
Alternative
J
lil
Generalized Linear
Logistic regression for K > 2 classses Models
Éf
Pr(Y = 1|X = x) regression
log
T
log = 1x Example 1
Pr(Y = K |X = x) Multiple logistic
Pr(Y = 1|X = x) T
regression
2x
by ME
log = Example 2
Pr(Y = K |X = x)
..
.
Multiple logistic
regression - extensions
Generalized linear
Q Ifj a learned
models
k
Eyes are rise c
Pr(Y = 1|X = x) T Poisson regression
log = K 1x
Pr(Y = K |X = x)
T T T
Parameter vector ✓ = { 1, 2,..., K 1}
7.23
like D
Logistic regression for K > 2 classses
Generalized Linear
Models
I
Above equations can be solved for each
⇡k (x; ✓) = Pr(Y = k |X = x)
exp( Tj x)
⇡j (x; ✓) = PK 1 T
, j = 1, . . . , K 1
1 + l=1 exp( l x)
The classification
problem
1 Simple logistic
regression
⇡K (x; ✓) = PK 1 T
1 + l=1 exp( l x)
Example 1
Multiple logistic
regression
Generalized linear
models
Poisson regression
7.24
Generalized Linear
How to solve logistic regression for K > 2 classses Models
The classification
problem
Simple logistic
regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.25
Generalized Linear
Generalized linear models (GLM) Models
Multiple logistic
T
logit(E[Y |x1 , . . . , xp ]) = x regression - extensions
Generalized linear
models
Poisson regression
7.26
Generalized Linear
Exponential family distributions Models
✓ ◆
y✓ b(✓)
p(y |✓, ) = exp + c(y , )
a( )
Multiple logistic
Bernoulli and Binomial distributions are in the exponential regression - extensions
7.27
Generalized Linear
Link function Models
T
Let ⌘ = x be the linear predictor
Let µ = E[Y |x] be the mean response
Let g be a function such that ⌘ = g(µ). g is called a link
function.
The classification
What is the link function in linear regression? problem
Simple logistic
What is the link function in logistic regression? regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.28
Generalized Linear
Linear, logistic and Poisson regression Models
Simple logistic
method! regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.29
Generalized Linear
Poisson regression Models
Response Y ⇠ Poisson(µ):
µy
Pr(Y = y ) = e µ
, y = 0, 1, 2, . . .
y!
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.30
Generalized Linear
Poisson regression Models
Response Y ⇠ Poisson(µ):
µy
Pr(Y = y ) = e µ
, y = 0, 1, 2, . . .
y!
Model: Example 1
ELY p
Multiple logistic
Y1 , . . . , Yn are independent regression
Example 2
T
log(E[Y |x]) = x Multiple logistic
To
T regression - extensions
x
E[Y |x] = e Generalized linear
models
Poisson regression
7.30
ett
Generalized Linear
MLE for Poisson regression
ELYIIB
Models
countvotes
Data: {(x i , yi )}ni=1 1t
PY4.11 141
2 3 2 3 2 T 3
1 x11 . . . x1p y1 ex 1
61 x21 . . . x2p 7 6 y2 7 6 T 7
6 7 6 7 6e x 2 7
X = 6. .. .. 6
.. 7 y = 6 .. 7 µ̂ = 6 .. 7
4 .. . . . 5 4.5 4 .
7
5 The classification
1 xn1 . . . xnp yn T
ex n problem
Log-likelihood:
Simple logistic
regression
Example 1
LCI ply Yi Ki In Il
n
X Multiple logistic
regression
Left Isplyilei.tl
l( ) = (yi x Ti exp(xi T
) log(yi !))
og
Example 2
i=1 Multiple logistic
regression - extensions
MLE:
É
Generalized linear
models
i E
Poisson regression
@l( )/@ =0
n
X
) (yi exp(xi T ))x i = 0
i=1
BILE Is predict j
ELYD ERILEE
y The classification
problem
or Simple logistic
regression
Ince
Example 1
Example 2
YE20,42
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.32
Generalized Linear
Applications of Possion regression Models
Predict:
Number of flight ticket purchases
Number of people that will attend to a concert
Number of phone calls to a call centre
The classification
problem
Simple logistic
regression
Example 1
Multiple logistic
regression
Example 2
Multiple logistic
regression - extensions
Generalized linear
models
Poisson regression
7.32
Generalized Linear
Predicting number of cavalrymen deaths from horse-kicks
S men value
Models
Pay
eÉI
Twenty years, ten army corps = 200 observations
Total number of fatalities = 122
The classification
= Mean deaths per corps-year = 122/200 = 0.61
PC07
GI
problem
PHI et
éI
Multiple logistic
(y ) deaths actual num. P(Y = y ) expected num. regression
egg
regression - extensions
1 65 0.334 66.28
eggs
Generalized linear
models
2 22 0.101 20.21 Poisson regression
3 3 0.20 4.11
4
5
1
0
0.03
0.00 I 0.62
0.07 lawof small nubs
Ben plzPoisonCIT
7.33
l