Professional Documents
Culture Documents
Lecture 4
Lecture 4
! Feature ! !
Extraction Classifier
What
•Types is machine
of Problems: learning?
•
1. Can you learn
Supervised: Classesany
areone-to-one mapping
known beforehand, given
and data enough
samples of
training
each class data?
are available
2. Unsupervised: Classes (and/or number of classes) are not known
• Supervised vs. Unsupervised
beforehand, and must be inferred from data
• Deterministic vs. Stochastic
Generative vs. Discriminative
•6.345 Automatic Speech Recognition Pattern Classification 2
Topics
• Bayes Theorem
Weight 2
Input 2
- Computes f(x) =
Activation Functions
Weight 1
- Does not saturate
Input 1 - Very computation
- Converges much
sigmoid/tanh in p
- Actually more bio
than sigmoid
Input Output
layer layer
Adding a hidden layer: universal approximator
(Backpropaga6on, 1986)
W1 W2 W3 W4
− x −1
y = (1 + e )
dy − x −2 −x
= −(1 + e ) (−e ) !
dx
Enables probabilis6c interpreta6on !
( x− µ )2
1 − 2
Ac6va6on func6ons
y=∑ i wi xi = W X !
1i For one sample: +
y = ∑1w∑= − y ) !
n n 2
i xi = n W nX2 !
E (t T
E =i 2 ∑ n (t − y ) !
2 n (training) n wi
∂E1 1 n ∂ynn 2dEn n
E ∂E= ∑ = 1(t∑−∂yy n)dE! n = − ∑ nxi (t n n
− ny )
n
xi
∂w2i =n 2 ∑n ∂wn dyn = − ∑n xi (t − y ) !
n
n ∂w i
i
i n n !
z == bb ++ ∑
i n
∂Exxw i wi ! !
Δwi = −ε∑i i =i ∑ ε xi (t − y ) !
z n n n
i∂w n
i
1
z =yy =b= + ∑−−xzzi w
1 !i !
!
11++ ee
Training with gradient descent
• Learning rate
∂E ∂y
Δw
z∂w
∂w = =
ib==+−ε∑
= 1
∑ ∑ ∑ x i∂ww
∂w
= n∑
!
dEε xn
n =∂w
(t= −− ∑
∑y∑
n−i n xnn (t n − y n )
= − )
x !
ix (t
i (t − −
y y) )
dy+ ∑ xi winn !n
n n
∂w
i
∂wiii 2 i nnn ∂w22 i n dy
i z =n b dy n i
! !
!
Learning
z = b + ∑ ∂E
∂E
x i i with n
∂Ew !
∑
i
a nlogis6c neuron
Δw Δw == −1
− ε ε ! y==∑ = = ∑ ε ε n (t n − y n ) ! !
ε xi i(t! (t −−y y) !)
n n n n
y =ii = −ε− z∂w
Δw i i 1 x x
i
1 +1e ∂w ∂wii i 1nn n+ e − z
y∂z= !
zz = == bbb1=+++xe∑ ∑ −
!
z xx xi
i
w
w w i ∂z!
i i i! ! = x !
∂w∂zi ∂wi
ii i
= 1xi !
ii y
yy∂z∂w= 11 ! ∂z
== 1=+we −−!−zzz !!
i
= wi !
∂z
∂xi = wi ! 1
1 +
+ e e
i ∂xi
∂z
∂z
∂z
∂x
f(.)
i = x ! dy
dy == xxii i!! = y(1 − y) !
∂wdyii= y(1 − y) !dz
∂w
∂w
dz i= y(1 − y) ! ∂y ∂z dy wi
∂z
∂zdz
∂z
∂y = w ∂z
===ww∂z !
ii ! dy ∂w dy = = xi y(1 − y) !
x
∂x ∂y ! = ∂w
ix y(1 − i y) !
dz i
∂x
∂x
∂w ii =
∂w
i
dz = xi y(1 − y) !n
i
∂w ii ∂w i dz ∂E ∂y ∂E
dy
dy
∂E
dy∂E =
i
= y(1
y(1 −
−
i
∂y
∂y y)
y) n !∂w
n
!
∂E ∂E
= ∑ n ∂wni ∂y
n
= − ∑ x n n
i y (1 − y n
)(t n
− y n
) !
dz
dz
∂w
dz
∂wii
= =∑
= ∑
y(1 −
∂w
y)
!
!
∂y
i
n= =
− ∑− ∑ x x
y nn n
i y
(1 − (1y−n n nn
y
)(t −
)(t nn
y −
) ! y n
) !
nn ∂w i i∂y
n i
∂y ∂z
∂y = ∂z dyback!propagation:!dy n n
! 2 j ∂y j
∂E ∂E dy j ∂E ∂E
∂E ==−(t j − y j ) =! y j (1 −∂E
back!propagation:! y j ) dy!j ∂E ∂E yj
∂y∂z
Error
! i =
deriva6ve −(t
dz ∂y
w.r.t − total
y ) !
input received= by neuron J: = y j (1 − y j )
∂y !
∂y
j j j j j ∂z j dz j ∂y j j
∂y j Layer J
∂E
∂E =1 dz i dy ∂E
j j ∂E ∂E ∂E
E ∂E == ∑∑dy(tj j ∂E − =y j=y) j∑ 2
! −w
(1 ∂Eyij j ) !∂E ! j ∂E
dz ∂E j
∂y∂zi j 2=dz j j dy
j ∂y ∂z
j = y (1
jj ∂y − ∂z=
y ∑
∂y
) j
dy ! ∂z
= ∑ wij
∂z
! z j
∂z j dzdz j ∂y j ∂y j
i j jj
i j i j j j wij
∂E
! ∂E ∂Ea weight connec6ng ∂E Layer I
∑ ∑
Error deriva6ve jw.r.t neurons i & j:
= = −(t − =
y ) ! w! ij !
∂y∂E∂E∂y ∂z dz ∂z
∂Ej ∂E ∂E ∂z ∂E y
==ij ∑ ji j = = ∑
j j i
dy ∂E
w j ∂z ! ∂E ∂E i
i
∂y dy ∂z y j
! ij =
∂z
j
= yi !
! ∂wiji∂E ∂w j dy∂z ∂zj ∂w
i
ij j ∂E
i j j j ij ∂w
j
∂E ij ∂z j ∂z j
!∂E ∂z= ∂E =∂E y j (1 − y j ) !
! deriva6ve
∂z j dz ∂y ! ∂y j
Error
∂E = j
∂z j =j y
w.r.t
∂E
hidden outputs:
∂E !
∂wjij =∂w
y z j ij ∂z y ! i
∂z y z y !
dz j ∂E= yi j ! ∂E
j
∂E i j j j i
! ! ∂w ij = ∑ ∂wij ∂z j = ∑ ∂z
! jwij !
∂yi j dyi ∂z j ∂z j
y !j z j yi !
j
!y z y !
! j j
∂z ∂E
i
∂E ∂E
Algorithm
• Ini6alize the weights
• For each weight, calculate the error deriva6ve with respect to that
weight
• Calculate the errors deriva6ves with respect to the hidden layer outputs,
do the same for the layers below
Issues: op6miza6on
• How olen to update the weights?
• Online (each training sample)
• Batch: Once for the full training set
• Mini-batch: once for a small number of training samples
• How much to update?
• fixed or variable learning rate
• adap6ve learning rate, globally, or for each neuron
separately
• Adam, SGD, RMSProp, SWA, AdaTune, …
Issues: generaliza6on
• Training data contains noise
• unreliable target values
• sampling error: accidental regulari6es, try to sample the space
as much as possible
• The model can learn the signal, but also the par6cular noise
• Prevent overfiong: Weight-decay (sparse weights), Weight-sharing,
early stopping, model averaging, Dropout, Pre-training
Output
Input
21 frequency bands
Fully connected layer
input output
231 6me x frequency 40 phonemes
Wx
40 x 231
Convolu6onal neural network (CNN)
Mul6plica6ve
Input feature
filter
Frequency
e
m
Ti
Frequency
Frequency
e
m
Ti
e
Tim
ConvNet is a series of
convolu6on and ac6va6on
+ + +
f(.) f(.) f(.)
e
m
Ti
what is 1x1 Convolu6on layer?
1x1xN
Frequency
Frequency
M such
filter m e
e Ti
m
Ti M
N
Pooling
Reducing the size of the feature space
sub-frequency
m e
Frequency
- ti
b
N su
m e
Ti
N
Downsample
Frequency
Time
Max pooling
MAX POOLING
3 2 1 0 3 4
1 2 3 4
y
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 5 - 73 April 17, 2018
Fei-Fei Li & Justin Johnson & Serena Yeung
Beyond CNNs: the need for mul6 scale speech
processing networks
Phonemes
Pitch Words
Recurrent neural networks (RNN)
RNN
RNNhidden
hiddenand
handoutput
t = (Whxlayer
output xt +activations:
layer W hh ht 1 + bh )
activations:
ytht=t =(W
h= (W
(W hhx
yhhx txt+
xt+b+yW
)Whhhhhht t 11++bbhh))
yty=
t = (W
Google(W htht++LVCSR
yhyh
Speech bby y) )with LSTM RNNs 5/37
Training an RNN
• Backpropagate through layers and 6me
input output
(ht-1 , xt) ht
(ht-1 , xt) ht
(ht-1 , xt)
Gradient can
flow through
The need for bi-direc6onal recurrence
Input waveform
(1 ms)
• Speech synthesis
• Audio compression
Topics
• Bayes Theorem