Deep Learning For NLP and Speech Recogni
Deep Learning For NLP and Speech Recogni
Deep Learning
for NLP
and Speech
Recognition
Deep Learning for NLP and Speech Recognition
Uday Kamath • John Liu • James Whitaker
123
Uday Kamath John Liu
Digital Reasoning Systems Inc. Intelluron Corporation
McLean Nashville
VA, USA TN, USA
James Whitaker
Digital Reasoning Systems Inc.
McLean
VA, USA
This Springer imprint is published by the registered company Springer Nature Switzerland AG.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To my parents Krishna and Bharathi, my wife
Pratibha, the kids Aaroh and Brandy, my
family and friends for their support.
–Uday Kamath
The publication of this book is a perfect timing. Existing books on deep learning
either focus on theoretical aspects or are largely manuals for tools. But this book
presents an unprecedented analysis and comparison of deep learning techniques for
natural language and speech processing, closing the substantial gap between the-
ory and practice. Each chapter discusses the theory underpinning the topics, and an
exceptional collection of 13 case studies in different application areas is presented.
They include classification via distributed representation, summarization, machine
translation, sentiment analysis, transfer learning, multitask NLP, end-to-end speech,
and question answering. Each case study includes the implementation and compar-
ison of state-of-the-art techniques, and the accompanying website provides source
code and data. This is extraordinarily valuable for practitioners, who can experiment
firsthand with the methods and can deepen their understanding of the methods by
applying them to real-world scenarios.
This book offers a comprehensive coverage of deep learning, from its foundations
to advanced and recent topics, including word embedding, convolutional neural net-
works, recurrent neural networks, attention mechanisms, memory-augmented net-
works, multitask learning, domain adaptation, and reinforcement learning. The book
is a great resource for practitioners and researchers both in industry and academia,
and the discussed case studies and associated material can serve as inspiration for a
variety of projects and hands-on assignments in a classroom setting.
Natural language and speech processing applications such as virtual assistants and
smart speakers play an important and ever-growing role in our lives. At the same
time, amid an increasing number of publications, it is becoming harder to iden-
tify the most promising approaches. As the Chief Analytics Officer at Digital Rea-
soning and with a PhD in Big Data Machine Learning, Uday has access to both
the practical and research aspects of this rapidly growing field. Having authored
vii
viii Foreword
Mastering Java Machine Learning, he is uniquely suited to break down both practi-
cal and cutting-edge approaches. This book combines both theoretical and practical
aspects of machine learning in a rare blend. It consists of an introduction that makes
it accessible to people starting in the field, an overview of state-of-the-art methods
that should be interesting even to people working in research, and a selection of
hands-on examples that ground the material in real-world applications and demon-
strate its usefulness to industry practitioners.
A few years ago, I picked up a few text-books to study topics related to arti-
ficial intelligence—such as natural language processing and computer vision. My
memory of reading these text-books largely consisted of staring helplessly out of
the window. Whenever I attempted to implement the described concepts and math,
I wouldn’t know where to start. This is fairly common in books written for aca-
demic purposes; they mockingly leave the actual implementation “as an exercise to
the reader.” There are a few exceptional books that try to bridge this gap, written
by people who know the importance of going beyond the math all the way to a
working system. This book is one of those exceptions—with it’s discussions, case
studies, code snippets, and comprehensive references, it delightfully bridges the gap
between learning and doing.
I especially like the use of Python and open-source tools out there. It’s an opin-
ionated take on implementing machine learning systems—one might ask the fol-
lowing question: “Why not X,” where X could be Java, C++, or Matlab? However,
I find solace in the fact that it’s the most popular opinion, which gives the read-
ers an immense support structure as they implement their own ideas. In the mod-
ern Internet-connected world, joining a popular ecosystem is equivalent to having
thousands of humans connecting together to help each other—from Stack Overflow
posts solving an error message to GitHub repositories implementing high-quality
systems. To give you perspective, I’ve seen the other side, supporting a niche com-
munity of enthusiasts in machine learning using the programming language Lua for
several years. It was a daily struggle to do new things—even basic things such as
making a bar chart—precisely because our community of people was a few orders
of magnitude smaller than Python’s.
Overall, I hope the reader enjoys a modern, practical take on deep learning sys-
tems, leveraging open-source machine learning systems heavily, and being taught
a lot of “tricks of the trade” by the incredibly talented authors, one of whom I’ve
known for years and have seen build robust speech recognition systems.
With the widespread adoption of deep learning, natural language processing (NLP),
and speech applications in various domains such as finance, healthcare, and gov-
ernment and across our daily lives, there is a growing need for one comprehensive
resource that maps deep learning techniques to NLP and speech and provides in-
sights into using the tools and libraries for real-world applications. Many books
focus on deep learning theory or deep learning for NLP-specific tasks, while oth-
ers are cookbooks for tools and libraries. But, the constant flux of new algorithms,
tools, frameworks, and libraries in a rapidly evolving landscape means that there are
few available texts that contain explanations of the recent deep learning methods
and state-of-the-art approaches applicable to NLP and speech, as well as real-world
case studies with code to provide hands-on experience. As an example, you would
find it difficult to find a single source that explains the impact of neural attention
techniques applied to a real-world NLP task such as machine translation across a
range of approaches, from the basic to the state-of-the-art. Likewise, it would be
difficult to find a source that includes accompanying code based on well-known li-
braries with comparisons and analysis across these techniques.
This book provides the following all in one place:
• A comprehensive resource that builds up from elementary deep learning, text,
and speech principles to advanced state-of-the-art neural architectures
• A ready reference for deep learning techniques applicable to common NLP and
speech recognition applications
ix
x Preface
• Practical aspects of using these techniques with tips and tricks essential for real-
world applications
This book is intended to introduce the foundations of deep learning, natural lan-
guage processing, and speech, with an emphasis on application and practical expe-
rience. It is aimed at NLP practitioners, graduate students in Engineering and Com-
puter Science, advanced undergraduates, and anyone with the appropriate mathe-
matical background who is interested in an in-depth introduction to the recent deep
learning approaches in NLP and speech. Mathematically, we expect that the reader
is familiar with multivariate calculus, probability, linear algebra, and Python pro-
gramming.
Python is becoming the lingua franca of data scientists and researchers for per-
forming experiments in deep learning. There are many libraries with Python-enabled
bindings for deep learning, NLP, and speech that have sprung up in the last few
years. Therefore, we use both the Python language and its accompanying libraries
for all case studies in this book. As it is unfeasible to fully cover every topic in a
single book, we present what we believe are the key concepts with regard to NLP
and speech that will translate into application. In particular, we focus on the inter-
section of those areas, wherein we can leverage different frameworks and libraries
to explore modern research and related applications.
The book is organized into three parts, aligning to different groups of readers and
their expertise. The three parts are:
• Machine Learning, NLP, and Speech Introduction. The first part has three
chapters that introduce readers to the fields of NLP, speech recognition, deep
learning, and machine learning with basic hands-on case studies using Python-
based tools and libraries.
Preface xi
• Deep Learning Basics. The five chapters in the second part introduce deep learn-
ing and various topics that are crucial for speech and text processing, including
word embeddings, convolutional neural networks, recurrent neural networks, and
speech recognition basics.
• Advanced Deep Learning Techniques for Text and Speech. The third part
has five chapters that discuss the latest research in the areas of deep learning that
intersect with NLP and speech. Topics including attention mechanisms, memory-
augmented networks, transfer learning, multitask learning, domain adaptation,
reinforcement learning, and end-to-end deep learning for speech recognition are
covered using case studies.
Next, we summarize the topics covered in each chapter.
• In the Introduction, we introduce the readers to the fields of deep learning, NLP,
and speech with a brief history. We present the different areas of machine learn-
ing and detail different resources ranging from books to datasets to aid readers in
their practical journey.
• In the Text and Speech Basics chapter, we introduce the fundamentals of com-
putational linguistics and NLP to the reader, including lexical, syntactic, seman-
tic, and discourse representations. We introduce language modeling and discuss
applications such as text classification, clustering, machine translation, question
answering, automatic summarization, and automated speech recognition, con-
cluding with a case study on text clustering and topic modeling.
• The Basics of Deep Learning chapter builds upon the machine learning founda-
tion by introducing deep learning. The chapter begins with a fundamental anal-
ysis of the components of deep learning in the multilayer perceptron (MLP),
followed by variations on the basic MLP architecture and techniques for training
deep neural networks. As the chapter progresses, it introduces various architec-
tures for both supervised and unsupervised learning, such as multiclass MLPs,
autoencoders, and generative adversarial networks (GANs). Finally, the mate-
rial is combined into the case study, analyzing both supervised and unsupervised
neural network architectures on a spoken digit dataset.
• The Convolutional Neural Networks chapter walks through the basics of con-
volutional neural networks and their applications to NLP. The main strand of
discourse in the chapter introduces the topic by starting from fundamental math-
ematical operations that form the building blocks, explores the architecture in
increasing detail, and ultimately lays bare the exact mapping of convolutional
neural networks to text data in its various forms. Several topics such as clas-
sic frameworks from the past, their modern adaptations, applications to different
NLP tasks, and some fast algorithms are also discussed in the chapter. The chap-
ter ends with a detailed case study using sentiment classification that explores
most of the algorithms mentioned in the chapter with practical insights.
tion mechanisms discussed in this chapter. The second case study explores and
analyzes different memory networks on the question-answering NLP task.
• The Transfer Learning: Domain Adaptation chapter probes into the area of
transfer learning where the models are subjected to constraints such as having
fewer data to train on, or situations when data on which to predict is different
from data it has trained on. Techniques for domain adaptation, few-shot learning,
one-shot learning, and zero-shot learning are covered in this chapter. A detailed
case study is presented using Amazon product reviews across different domains
where many of the techniques discussed are applied.
• In the Deep Reinforcement for Text and Speech chapter, we review the funda-
mentals of reinforcement learning and discuss their adaptation to deep sequence-
to-sequence models, including deep policy gradient, deep Q-learning, double
DQN, and DAAC algorithms. We investigate deep reinforcement learning ap-
proaches to NLP tasks including information extraction, text summarization, ma-
chine translation, and automatic speech recognition. We conclude with a case
study on the application of deep policy gradient and deep Q-learning algorithms
to text summarization.
Acknowledgments
The construction of this book would not have been possible without the tremen-
dous efforts of many people. Firstly, we want to thank Springer, especially our ed-
itor Paul Drougas, for working very closely with us and seeing this to fruition. We
want to thank Digital Reasoning for giving us the opportunity to work on many
real-world NLP and speech problems that have had a significant impact on our
work here. We would specifically like to thank Maciek Makowski and Gabrielle Liu
for reviewing and editing the content of this book, as well as those that have pro-
xiv Preface
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.1 Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.2 Unsupervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1.3 Semi-Supervised Learning and Active Learning . . . . . . . . . . . 7
1.1.4 Transfer Learning and Multitask Learning . . . . . . . . . . . . . . . 7
1.1.5 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.1 Deep Learning: A Brief History . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.2 Natural Language Processing: A Brief History . . . . . . . . . . . . 11
1.2.3 Automatic Speech Recognition: A Brief History . . . . . . . . . . 15
1.3 Tools, Libraries, Datasets, and Resources for the Practitioners . . . . . 18
1.3.1 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.3.2 Natural Language Processing . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.3.3 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.3.4 Books . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.3.5 Online Courses and Resources . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.3.6 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.4 Case Studies and Implementation Details . . . . . . . . . . . . . . . . . . . . . . . 25
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
xv
xvi Contents
Part III Advanced Deep Learning Techniques for Text and Speech
Democratization of AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
NLP Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
Speech Trends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
Closing Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
Notation
Calculus
≈ Approximately equal to
|A| L1 norm of matrix A
A L2 norm of matrix A
da
db Derivative of a with respect to b
∂a
∂b Partial derivative of a with respect to b
∇xY Gradient of Y with respect to x
∇XY Matrix of derivatives of Y with respect to X
Datasets
D Dataset, a set of examples and corresponding targets, {(x1 , y1 ),
(x2 , y2 ), . . . , (xn , yn )}
X Space of all possible inputs
Y Space of all possible outputs
yi Target label for example i
ŷi Predicted label for example i
L Log-likelihood loss
Ω Learned parameters
Functions
f :A→B A function f that maps a value in the set A to set B
f (x; θ ) A function of x parameterized by θ . This is frequently reduced to
f (x) for notational clarity.
log x Natural log of x
σ (a) Logistic sigmoid, 1+exp 1
−a
a = b A function that yields a 1 if the condition contained is true, otherwise
it yields 0
arg minx f (x) Set of arguments that minimize f (x), arg minx f (x) = {x | f (x) =
minx f (x )}
arg maxx f (x) Set of arguments that maximize f (x), arg maxx f (x) = {x | f (x) =
maxx f (x )}
xxvii
xxviii Notation
Linear Algebra
a
⎡ ⎤ Scalar value (integer or real)
a1
⎢ .. ⎥
⎣.⎦ Vector containing elements a1 to an
an
⎡ ⎤
a1,1 · · · a1,n
⎢ .. . . .. ⎥
⎣ . . . ⎦ A matrix with m rows and n columns
am,1 · · · am,n
Ai, j Value of matrix A at row i and column j
a Vector (dimensions implied by context)
A Matrix (dimensions implied by context)
A Transpose of matrix A
A−1 Inverse of matrix A
I Identity matrix (dimensionality implied by context)
A·B Dot product of matrices A and B
A×B Cross product of matrices A and B
A◦B Element-wise (Hadamard) product
A⊗B Kronecker product of matrices A and B
a; b Concatenation of vectors a and b
Probability
E Expected value
P(A) Probability of event A
P(A|B) Probability of event A given event B
X ∼ N (μ , σ 2 ) Random variable X sampled from a Gaussian (Normal) distribution
with μ mean and σ 2 variance
Sets
A A set
R Set of real numbers
C Set of complex numbers
0/ Empty set
{a, b} Set containing the elements a and b
{1, 2, . . . n} Set containing all integers from 1 to n
{a1 , a2 , . . . an } Set containing n elements
a∈A Value a is a member of the set A
[a, b] Set of real values from a to b, including a and b
[a, b) Set of real values from a to b, including a but excluding b
a1:m Set of elements {a1 , a2 , . . . , am } (used for notational convenience)
Most of the chapters, unless and otherwise specified, assume the notations given
above.
Part I
Machine Learning, NLP, and Speech
Introduction
Chapter 1
Introduction
In recent years, advances in machine learning have led to significant and widespread
improvements in how we interact with our world. One of the most portentous of
these advances is the field of deep learning. Based on artificial neural networks that
resemble those in the human brain, deep learning is a set of methods that permits
computers to learn from data without human supervision and intervention. Further-
more, these methods can adapt to changing environments and provide continuous
improvement to learned abilities. Today, deep learning is prevalent in our every-
day life in the form of Google’s search, Apple’s Siri, and Amazon’s and Netflix’s
recommendation engines to name but a few examples. When we interact with our
email systems, online chatbots, and voice or image recognition systems deployed at
businesses ranging from healthcare to financial services, we see robust applications
of deep learning in action.
Human communication is at the core of developments in many of these areas, and
the complexities of language make computational approaches increasingly difficult.
With the advent of deep learning, however, the burden shifts from producing rule-
based approaches to learning directly from the data. These deep learning techniques
open new fronts in our ability to model human communication and interaction and
improve human–computer interaction.
Deep learning saw explosive growth, attention, and availability of tools following
its success in computer vision in the early 2010s. Natural language processing soon
experienced many of these same benefits from computer vision. Speech recognition,
traditionally a field dominated by feature engineering and model tuning techniques,
incorporated deep learning into its feature extraction methods resulting in strong
gains in quality. Figure 1.1 shows the popularity of these fields in recent years.
The age of big data is another contributing factor to the performance gains with
deep learning. Unlike many traditional learning algorithms, deep learning models
continue to improve with the amount of data provided, as illustrated in Fig. 1.2.
Perhaps one of the largest contributors to the success of deep learning is the active
community that has developed around it. The overlap and collaboration between
academic institutions and industry in the open source has led to a virtual cornucopia
Fig. 1.1: Google trends for deep learning, natural language processing, and speech
recognition in the last decade
of tools and libraries for deep learning. This overlap and influence of the academic
world and the consumer marketplace has also led to a shift in the popularity of
programming languages, as illustrated in Fig. 1.3, specifically towards Python.
Python has become the go-to language for many analytics applications, due to
its simplicity, cleanliness of syntax, multiple data science libraries, and extensibility
(specifically with C++). This simplicity and extensibility have led to most top deep
learning frameworks to be built on Python or adopt Python interfaces that wrap
high-performance C++ and GPU-optimized extensions.
This book seeks to provide the reader an in-depth overview of deep learning tech-
niques in the fields of text and speech processing. Our hope is for the reader to walk
away with a thorough understanding of natural language processing and leading-
edge deep learning techniques that will provide a basis for all text and speech pro-
cessing advancements in the future. Since “practice makes for a wonderful compan-
ion,” each chapter in this book is accompanied with a case study that walks through
a practical application of the methods introduced in the chapter.
1.1 Machine Learning 5
Fig. 1.3: Google trends for programming languages such as Java, Python, and R
which are used in data science and deep learning in the last decade
Supervised learning relies on learning from a dataset with labels for each of the
examples. For example, if we are trying to learn movie sentiment, the dataset may
be a set of movie reviews and the labels are the 0–5 star rating.
There are two types of supervised learning: classification and regression
(Fig. 1.5).
Classification maps an input into a fixed set of categories, for example, classify-
ing an image as either a cat or dog.
Regression problems, on the other hand, map an input to a real number value.
An example of this is trying to predicting the cost of your utility bill or the stock
market price.
6 1 Introduction
Fig. 1.4: The area of deep learning covers multiple areas of machine learning, while
machine learning is a subset of the broader AI category
Fig. 1.5: Supervised learning uses a labeled dataset to predict an output. In a clas-
sification problem, (a) the output will be labeled as a category (e.g., positive or
negative), while in a regression problem, (b) the output will be a value
Unsupervised learning determines categories from data where there are no labels
present. These tasks can take the form of clustering, grouping similar items together,
or similarity, defining how closely a pair of items is related. For example, imagine
we wanted to recommend a movie based on a person’s viewing habits. We could
cluster users based on what they have watched and enjoyed, and evaluate whose
viewing habits most match the person to whom we are recommending the movie.
1.2 History 7
In many situations when it is not possible to label or annotate the entire dataset
due to either cost or lack of expertise or other constraints, learning jointly from
the labeled and unlabeled data is called semi-supervised learning. Instead of expert
labeling of data, if the machine provides insight into which data should be labeled,
the process is called active learning.
The basic idea behind “transfer learning” is to help the model adapt to situations
it has not previously encountered. This form of learning relies on tuning a general
model to a new domain. Learning from many tasks to jointly improve the perfor-
mance across all the tasks is called multitask learning. These techniques are becom-
ing the focus in both deep learning and NLP/speech.
1.2 History
You don’t know where you’re going until you know where you’ve been.—James Baldwin
There has been much research in both the academic and industrial fields that has led
to the current state of deep learning and its recent popularity. The goal of this section
is to give a brief timeline of research that has influenced deep learning, although we
might not have captured all the details (Fig. 1.6). Schmidhuber [Sch15] has compre-
hensively captured the entire history of neural networks and various research that
led to today’s deep learning. In the early 1940s, S. McCulloch and W. Pitts modeled
how the brain works using a simple electrical circuit called the threshold logic unit
that could simulate intelligent behavior [MP88]. They modeled the first neuron with
inputs and outputs that generated 0 when the “weighted sum” was below a threshold
and 1 otherwise. The weights were not learned but adjusted. They coined the term
connectionism to describe their model. Donald Hebb in his book “The Organization
of Behaviour (1949)” took the idea further by proposing how neural pathways can
have multiple neurons firing and strengthening over time with usage, thus laying the
foundation of complex processing [Heb49].
According to many, Alan Turing in his seminal paper “Computing Machinery
and Intelligence” laid the foundations of artificial intelligence with several criteria
to validate the “intelligence” of machines known as the “Turing test” [Tur95]. In
1959, the discovery of simple cells and complex cells that constitute the primary
visual cortex by Nobel Laureates Hubel and Wiesel had a wide-ranging influence in
many fields including the design of neural networks. Frank Rosenblatt extended the
McCulloch–Pitts neuron using the term Mark I Perceptron which took inputs, gener-
ated outputs, and had linear thresholding logic [Ros58]. The weights in the percep-
tron were “learned” by successively passing the inputs and reducing the difference
between the generated output and the desired output. Bernard Widrow and Marcian
Hoff took the idea of perceptrons further to develop Multiple ADAptive LINear El-
ements (MADALINE) which were used to eliminate noise in phone lines [WH60].
Marvin Minsky and Seymour Papert published the book Perceptrons which
showed the limitations of perceptrons in learning the simple exclusive-or function
(XOR) [MP69]. Because of a large number of iterations required to generate the
output and the limitations imposed by compute time they conclusively proved that
multilayer networks could not use perceptrons. Years of funding dried because of
this and effectively limited research in the neural networks, appropriately called the
“The First AI Winter.”
In 1986, David Rumelhart, Geoff Hinton, and Ronald Williams published the
seminal work “Learning representations by back-propagating errors” which showed
how a multi-layered neural network could not only be trained effectively using a
relatively simple procedure but how “hidden” layers can be used to overcome the
weakness of perceptrons in learning complex patterns [RHW88]. Though there was
much research in the past in the form of various theses and research, the works of
Linnainmaa, S., P. Werbos, Fukushima, David Parker, Yann Le Cun, and Rumelhart
et al. have considerably broadened the popularity of neural networks [Lin70, Wer74,
Fuk79, Par85, LeC85].
1.2 History 9
LeCun et al. with their research and implementation led to the first widespread
application of neural networks to the recognition of handwritten digits used by the
U.S. Postal Service [LeC+89]. This work was an important milestone in deep learn-
ing history, as it showed how convolution operations and weight sharing could be
effective for learning features in modern convolutional neural networks (CNNs).
George Cybenko showed how the feed-forward networks with finite neurons, a sin-
gle hidden layer, and non-linear sigmoid activation function could approximate most
complex functions with mild assumptions [Cyb89]. Cybenko’s research along with
Kurt Hornik’s work led to the further rise of neural networks and their application as
“universal approximator functions” [Hor91]. The seminal work of Yann Le Cun et
al. resulted in widespread practical applications of CNNs such as the reading bank
checks [LB94, LBB97].
Dimensionality reduction and learning using unsupervised techniques were
demonstrated in Kohen’s work titled “Self-Organized Formation of Topologi-
cally Correct Feature Maps” [Koh82]. John Hopfield with his Hopfield Networks
created one of the first recurrent neural networks (RNNs) that served as a content-
addressable memory system [Hop82]. Ackley et al. in their research showed how
Boltzmann machines modeled as neural networks could capture probability dis-
tributions using the concepts of particle energy and thermodynamic temperature
applied to the networks [AHS88]. Hinton and Zemel in their work presented var-
ious topics of unsupervised techniques to approximate probability distributions
using neural networks [HZ94]. Redford Neal’s work on the “belief net,” similar
to Boltzmann machines, showed how it could be used to perform unsupervised
learning using much faster algorithms [Nea95].
10 1 Introduction
collected a large number of images and showed the usefulness of data in important
tasks such as object recognition, classification, and clustering [Den+09].
At the same time, following Moore’s law, computers were getting faster, and
graphic processor units (GPUs) overcame many of the previous limitations of CPUs.
Mohamed et al. showed a huge improvement in the performance of a complex task
such as speech recognition using deep learning techniques and achieved huge speed
increases on large datasets with GPUs [Moh+11]. Using the previous networks such
as CNNs and combining them with a ReLU activation, regularization techniques,
such as dropout, and the speed of the GPU, Krizhevsky et al. attained the smallest
error rates on the ImageNet classification task [KSH12]. Winning the ILSVRC-2012
competition by a huge difference between the CNN-based deep learning error rate
of 15.3% and the second best at 26.2% put the attention of both academics and in-
dustry onto deep learning. Goodfellow et al. proposed a generative network using
adversarial methods that addressed many issues of learning in an unsupervised man-
ner and is considered a path-breaking research with wide applications [Goo+14].
Many companies such as Google, Facebook, and Microsoft started replacing their
traditional algorithms with deep learning using GPU-based architectures for speed.
Facebook’s DeepFace uses deep networks with more than 120 million parameters
and achieves the accuracy of 97.35% on a Labeled Faces in the Wild (LFW) dataset,
approaching human-level accuracy by improving the previous results by an unprece-
dented 27% [Tai+14]. Google Brain, a collaboration between Andrew Ng and Jeff
Dean, resulted in large-scale deep unsupervised learning from YouTube videos for
tasks such as object identification using 16, 000 CPU cores and close to one bil-
lion weights! DeepMind’s AlphGo’s beat Lee Sedol of Korea, an internationally
top-ranked Go player, highlighting an important milestone in overall AI and deep
learning.
We will give a brief overview of important events that impacted the field up until
2000 (Fig. 1.7). For a very comprehensive summary, we refer the reader to a well-
documented outline in Karen Jones’s survey [Jon94]. Since neural architectures and
deep learning have, in general, had much impact in this area and are the focus of the
book, we will cover these topics in more detail.
Though there were traces of interesting experiments in the 1940s, the IBM-
Georgetown experiment of 1954 that showcased machine translation of around
60 sentences from Russian to English can be considered an important mile-
stone [HDG55]. Though constrained with computing resources in the form of
software and hardware, some of the challenges of syntactic, semantic, and linguistic
variety were discovered, and an attempt was made to address them. Similar to
how AI was experiencing the golden age, many developments took place between
1954–1966, such as the establishment of conferences including the Dartmouth
Conference in 1956, the Washington International Conference in 1958, and the
Teddington International Conference on Machine Translation of Languages and
Applied Language Analysis in 1961. In the Dartmouth Conference of 1956, John
McCarthy coined the term “artificial intelligence.” In 1957, Noam Chomsky pub-
lished his book Syntactic Structures, which highlighted the importance of sentence
syntax in language understanding [Cho57]. The invention of the phrase-structure
grammar also played an important role in that era. Most notably, the attempts at the
Turing test by software such as LISP by John McCarthy in 1958 and ELIZA (the
first chatbot) had a great influence not only in NLP but in the entire field of AI.
In 1964, the United States National Research Council (NRC) set up a group
known as the Automatic Language Processing Advisory Committee (ALPAC) to
1.2 History 13
evaluate the progress of NLP research. The ALPAC report of 1966 highlighted the
difficulties surrounding machine translation from the process itself to the cost of im-
plementation and was influential in reducing funding, nearly putting a halt to NLP
research [PC66]. This phase of the 1960s–1970s was a period in the study of world
knowledge that emphasized semantics over syntactical structures. Grammars such as
case grammar, which explored relations between nouns and verbs, played an inter-
esting role in this era. Augmented transition networks was another search algorithm
approach for solving problems like the best syntax for a phrase. Schank’s concep-
tual dependency, which expressed language in terms of semantic primitives without
syntactical processing, was also a significant development [ST69]. SHRDLU was a
simple system that could understand basic questions and answer in natural language
using syntax, semantics, and reasoning. LUNAR by Woods et al. was the first of
its kind: a question-answering system that combined natural language understand-
ing with a logic-based system. Semantic networks, which capture knowledge as a
graph, became an increasingly common theme highlighted in the work of Silvio
Ceccato, Margaret Masterman, Quillian, Bobrow and Collins, and Findler, to name
a few [Cec61, Mas61, Qui63, BC75, Fin79]. In the early 1980s, the grammatico-
logical phase began where the linguists developed different grammar structures and
started associating meaning in phrases concerning users’ intention. Many tools and
software such as Alvey natural language tools, SYSTRAN, METEO, etc. became
popular for parsing, translation, and information retrieval [Bri+87, HS92].
The 1990s were an era of statistical language processing where many new ideas
of gathering data such as using the corpus for linguistic processing or understanding
the words based on its occurrence and co-occurrence using probabilistic-based ap-
proaches were used in most NLP-based systems [MMS99]. A large amount of data
available through the World Wide Web across different languages created a high de-
mand for research in areas such as information retrieval, machine translation, sum-
marization, topic modeling, and classification [Man99]. An increase in memory and
processing speed in computers made it possible for many real-world applications
to start using text and speech processing systems. Linguistic resources, including
annotated collections such as the Penn Treebank, British National Corpus, Prague
Dependency Treebank, and WordNet, were beneficial for academic research and
commercial applications [Mar+94, HKKS99, Mil95]. Classical approaches such as
n-grams and a bag-of-words representation with machine learning algorithms such
as multinomial logistic regression, support vector machines, Bayesian networks, or
expectation–maximization were common supervised and unsupervised techniques
for many NLP tasks [Bro+92, MMS99]. Baker et al. introduced the FrameNet
project which looked at “frames” to capture semantics such as entities and rela-
tionships and this led to semantic role labeling, which is an active research topic
today [BFL98].
In the early 2000s, the Conference on Natural Language Learning (CoNLL)
shared-tasks resulted in much interesting NLP research in areas such as chunk-
ing, named entity recognition, and dependency parsing to name a few [TKSB00,
TKSDM03a, BM06]. Lafferty et al. proposed conditional random fields (CRF),
14 1 Introduction
hierarchical structure to the sentences and are inspired by linguistic approach, be-
came another important neural architecture in the neural-based NLP world [LSM13].
Sutskever et al. proposed sequence-to-sequence learning as a general neural
framework composed of an encoder neural network processing inputs as a sequence
and a decoder neural network predicting the outputs based on the input sequence
states and the current output states [SVL14]. This framework has found a wide
range of applications such as constituency parsing, named entity recognition, ma-
chine translation, question-answering, and summarization. Google started replacing
its monolithic phrase-based machine translation models with sequence-to-sequence
neural MT models [Wu+16]. Character-based rather than word-based representa-
tions overcome many issues such as out of vocabulary and have been part of re-
search in deep learning based systems for various NLP tasks [Lam+16, PSG16]. The
attention mechanism by Bahdanau et al. is another innovation that has been widely
popular in different neural architectures for NLP and speech [BCB14b]. Memory
augmented networks with various variants such as memory networks, neural Tur-
ing machines, end-to-end memory networks, dynamic memory networks, differ-
entiable neural computers, and recurrent entity networks have become very pop-
ular in the last few years for complex natural language understanding and language
modeling tasks [WCB14, Suk+15, GWD14, Gra+16, Kum+16, Gre+15, Hen+16].
Adversarial learning and using adversarial examples have recently become com-
mon for distribution understanding, testing the robustness of models, and transfer
learning [JL17, Gan+16]. Reinforcement learning is another emerging field in deep
learning and has applications in NLP, specifically in the areas where there is a tem-
poral dependencies and non-differentiable optimization zones where gradient-based
methods fail. Modeling dialog systems, machine translation, text summarization,
and visual storytelling among others have seen the benefits of reinforcement tech-
niques [Liu+18, Ran+15, Wan+18, PXS17].
In the 1960s, small vocabulary and acoustic phonetic-based tasks became prime
research areas, leading to many techniques centered around dynamic programming
and frequency analysis. IBM’s Shoebox was able to recognize not only digits but
also words such as “sum” and “total” and use these in the arithmetic computa-
tions to give results. The researchers in University College in England could analyze
phonemes for recognition of vowels and consonants [JR05a].
In the 1970s, research moved towards medium-sized vocabulary tasks and con-
tinuous speech. The dominant techniques were various types of pattern recognition
and clustering algorithms. Dynamic time warping (DTW) was introduced to han-
dle time variability, aligning an input sequence of features to an output sequence
of classes. “Harpy,” a speech recognizer from Carnegie Mellon University, was ca-
pable of recognizing speech with a vocabulary of 1011 words. One of the main
achievements of this work was the introduction of the graph search to “decode” lex-
ical representations of words with a set of rules and a finite state network [LR90].
The methods that would optimize this capability, however, were not introduced un-
til the 1990s. A recognition system called Tangora [JBM75] was created by IBM to
provide a “voice-activated typewriter.” This effort introduced a focus on large vo-
cabularies and the sequence of words for grammars, which led to the introduction
of language models for speech. During this era, AT&T also played a significant role
in ASR, focusing heavily on speaker-independent systems. Their work, therefore,
focused more heavily on what is called the acoustic model, dealing with the anal-
ysis of the speech patterns across speakers. By the mid-late 1970s, hidden Markov
models were used to model spectral variations for discrete speech.
In the 1980s, the fundamental approach of ASR shifted to a statistical foundation,
specifically HMM methods for modeling transitions between states. By the mid-
1980s, HMMs had become the dominant technique for ASR (and remains one of
the most prominent today). This shift to HMMs allowed many other advancements
such as speech decoding frameworks with FSTs. The 1980s saw the introduction
of neural networks for speech recognition. Their ability to approximate any func-
tion made them an exciting candidate for predicting the state transitions while still
relying on the HMM to handle the temporal nature of continuous speech. Various
toolkits were created during this period to support ASR, such as Sphinx [Lee88] and
DECIPHER [Mur+89] from SRI.
In the 1990s, many advancements in machine learning were incorporated into
ASR, which led to improved accuracy. Many of these were software which became
available commercially, such as Dragon which had a dictionary of 80, 000 words
and the ability to train the software to the user’s voice. Many toolkits were created
to support ASR in the late 1980s and 1990s such as HTK [GW01] from Cambridge,
a hidden Markov model toolkit.
The time delay neural network (TDNN) [Wai+90] was one of the earliest appli-
cations of deep learning to speech recognition. It utilized stacked 2D convolutional
layers to perform phone classification. The benefits of this approach were that it was
shift-invariant (not requiring a segmentation); however, the width of the network
limits the context window. The TDNN approach was comparable to early HMM-
1.2 History 17
based approaches; however, it did not integrate with HMMs and was difficult to use
in large vocabulary settings [YD14].
1 http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC93S1.
18 1 Introduction
As with the techniques in NLP, deep learning frameworks have also seen tremendous
progress in recent years. Numerous frameworks exist, each with their specializa-
tion. The most popular deep learning frameworks are: TensorFlow, PyTorch, Keras,
MXNet, CNTK, Chainer, Caffe2, PaddlePaddle, and Matlab.2 The main component
of a modern deep learning framework is efficiency in linear algebra capabilities,
as this applies to deep learning, with support CPU and GPU computation (dedi-
cate hardware like TPUs [Jou16] are becoming increasingly popular as well). All
relevant Python frameworks encompass both CPU and GPU support. The imple-
mentation differences tend to focus on the trade-offs between the intended end user
(researcher vs. engineer vs. data scientist).
We look at the top deep learning frameworks from the ones mentioned previously
and give a brief description of them. In order to do this, we compare the Google
Trends for each of the frameworks and focus on the top 3.3 As shown in Fig. 1.9, the
top framework (worldwide) is Keras, followed by TensorFlow, and then PyTorch.
Additional information about the frameworks used in the case studies will be given
throughout the book.
2 Theano exists as another popular framework; however, major development has discontinued given
the popularity of more recent frameworks. It is therefore not included in this book.
3 Although this is a single, statistically insignificant, data point, the Google Trends mechanism
is useful and roughly correlates with other evaluations such as number of contributors, GitHub
popularity, number of articles written, and books written for the various frameworks.
1.3 Tools, Libraries, Datasets, and Resources for the Practitioners 19
The following are popular open-source frameworks for building neural networks:
• TensorFlow: TensorFlow is a computational library based on data flow graphs.
These graphs have nodes representing mathematical operations and edges rep-
resenting tensors that flow between them. Written in Python, TensorFlow was
developed by the Google Brain team.
The following resources are some of the more popular open-source toolkits for nat-
ural language processing:
• Stanford CoreNLP: A Java-based toolkit of linguistic analysis tools for process-
ing natural language text. CoreNLP was developed by Stanford University.
The following resources are some of the more popular open-source toolkits for
speech recognition.4
1.3.3.1 Frameworks
• SoX: An audio manipulation toolkit and library. It implements many file formats
and is useful for playing, converting, and manipulating audio files [NB18].
• LibROSA: A Python package for audio analysis, commonly used for feature
extraction and digital signals processing (DSP) [McF+15].
1.3.4 Books
The fields of NLP and machine learning are very broad and cannot all be con-
tained in a single resource. Here we recognize various books that provide deeper
explanations, for supplementary information. The Elements of Statistical Learn-
ing by Hastie et al. gives a good base to machine learning and statistical tech-
niques [HTF01]. Learning From Data by Abu-Mostafa et al. also provides insights
into theories in machine learning in a more simplified and understandable manner
[AMMIL12]. Deep Learning (Adaptive Computation and Machine Learning series)
[GBC16] presents an advanced theory-focused book. This is widely considered the
foundational book for deep learning, moving from the fundamentals for deep learn-
ing (Linear Algebra, Probability Theory, and Numerical computation) to an explo-
ration of many architecture implementations and approaches. Foundations of Sta-
tistical Natural Language Processing [MS99] is a comprehensive resource on sta-
tistical models for natural language processing, providing an in-depth mathematical
foundations for implementing NLP tools. Speech and Language Processing [Jur00]
provides an introduction to NLP and speech, with both breadth and depth in many
areas of statistical NLP. More recent editions explore neural network applications.
Neural Network Methods in Natural Language Processing (Synthesis Lectures
on Human Language Technologies)[Gol17] presents neural network applications
on language data, moving from the introduction of machine learning and neural net-
works to specialized neural architectures for NLP applications. Automatic Speech
Recognition: A Deep Learning Approach by Yu et al. gives a thorough introduction
to ASR and deep learning techniques [YD15].
Below we list some of the online courses where the experts in the field teach topics
related to deep learning, NLP, and speech and are extremely beneficial.
• Natural Language Processing with Deep Learning
http://web.stanford.edu/class/cs224n/
• Deep Learning for Natural Language Processing
http://www.cs.ox.ac.uk/teaching/courses/2016-2017/dl
• Neural Networks for NLP
http://phontron.com/class/nn4nlp2017/schedule.html
• Deep Learning Specialization
https://www.deeplearning.ai/deep-learning-specialization/
• Deep Learning Summer School
https://vectorinstitute.ai/2018/11/07/vector-institute-deep-learning-and-
reinforcement-learning-2018-summer-school/
• Convolutional Neural Networks for Visual Recognition
http://cs231n.stanford.edu/
22 1 Introduction
1.3.6 Datasets
Any end-to-end deep learning application is reliant on data. While most organiza-
tions rely on data collection as a fundamental part of their strategy, there are many
publicly available datasets for researchers, hobbyists, and practitioners.5
Kaggle6 is one of the most popular sources for datasets and competitions for
machine learning and data science. With thousands of datasets and competitions
available, an active community of over one million registered users, and funded
competitions through its platform, Kaggle serves as a strong starting point for not
only datasets but also techniques for many tasks.
The Linguistic Data Consortium7 combines and sells datasets from a variety of
universities, corporations, and research labs. It primarily focuses on linguistic data
and language resources, with close to 1000 datasets.
Stanford NLP group also releases a number of datasets for NLP, specifically for
the training of their CoreNLP models.8
• Text Similarity:
Dataset Description
SentEval [CK18] Evaluation library for sentence embeddings, comparing
their effectiveness on 17 downstream tasks
Quora Question Collection of 400,000 question pairs of potentially dupli-
Pairs [ZCZ] cate questions from the Quora website with the goal of
identifying duplicates
5 Additional text tasks and datasets are captured with associated papers at https://nlpprogress.com/.
6 https://www.kaggle.com.
7 https://www.ldc.upenn.edu/.
8 https://nlp.stanford.edu/data/.
1.3 Tools, Libraries, Datasets, and Resources for the Practitioners 23
• Dependency Parsing:
Dataset Description
Penn Treebank A corpus of part-of-speech annotated American English,
[MMS93] consisting of 4.5 million words
• Entity Extraction:
Dataset Description
CoNLL 2003 Newswire text tagged with various types of entities, such
[TKSDM03b] as location, organization, person, etc.
WNUT2017 A dataset composed of annotated tweets, YouTube com-
[Der+17] ments, and other internet sources, tagged with different
entities
OntoNotes A multilingual corpus with many labels, such as part-of-
[Hov+06b] speech, parse trees, and entities (two million tokens in
version 5)
• Relation Extraction:
Dataset Description
NYT Corpus New York Times corpus with relational labels for dis-
[RYM10] tantly related entities
SemEval-2010 A semantic relationship dataset, containing types of rela-
(Task 8) tionships between, such as entity-origin and cause–effect
[Hen+09]
• Machine Translation:
Dataset Description
Tatoeba A collection of multilingual sentence pairs from the
Tatoeba website https://tatoeba.org
WMT 2014 An English–French (and additional English–German)
[Sch18] dataset consisting of sentences taken from many sources,
such as common crawl, UN corpus, and new commentary
Multi30k Crowdsourced descriptions of images in multiple lan-
[Ell+16] guages
• Text Summarization:
Dataset Description
CNN/Daily Mail Approximately 300,000 news stories from CNN and
[Nal+16] Daily Mail websites with corresponding summary bullet
points
Cornell Over 1.3 million news articles and summaries from 38
Newsroom major publications between 1998 and 2017
[GNA18]
Google Dataset A sentence-compression task with 200,000 examples that
[FA13] focuses on deleting words to produce a compressed struc-
ture of the original, longer sentence
DUC A smaller sentence summarization task with newswire
and document data, containing 500 documents
Webis-TLDR-17 A dataset containing Reddit posts with the “too long;
Corpus [Sye+18] didn’t read (TL;DR)” summaries for over three million
posts
• Question Answer:
Dataset Description
bAbI [Wes+15] A collection of tasks and associated datasets to evaluate
NLP models, specifically geared towards Q&A
NewsQA A set of 100,000 challenging question–answer pairs for
[Tri+16] news articles from CNN
SearchQA A general question-answering set for search QA
[Dun+17]
1.4 Case Studies and Implementation Details 25
• Speech Recognition:
Dataset Description
AN4 [Ace90] A small, alphanumeric dataset containing randomly gen-
erated words, numbers, and letters
WSJ [PB92] A general purpose, large vocabulary, speech recognition
dataset, consisting of 400 h of spoken audio and tran-
scripts
LibriSpeech 1000 h Dataset containing read speech from audiobooks
[Pan+15] from the LibriVox audiobook project
Switchboard Conversational dataset, consisting of more than 240 h of
[GHM92] audio. The combined test with CallHome English set is
referred to as Hub5’00
TED-LIUM 452 h of TED talks with corresponding transcripts and
[RDE12] alignment information. The most recent version (3)
[Her+18] has twice as much data as the previous versions
CHiME [Vin+16] Is a speech challenge that has consisted of various tasks
and datasets throughout the years. Some of the tasks in-
clude speech separation, recognition, speech processing
in noisy environments, and multi-channel recognition
TIMIT [Gar+93] An ASR dataset for phonetic studies with 630 speakers
reading phonetically rich sentences
In this book, the goal is not only to inform, but also to enable the reader to practice
what is learned. In each of the following chapters a case study explores chapter
concepts in detail and provides a chance for hands-on practice. The case studies
and supplementary code are written in Python, utilizing a various deep learning
frameworks. In most cases, deep learning depends on high-performance C++ or
CUDA libraries to perform computation. In our experience, the installation process
can become an extremely tedious process, especially when getting started with deep
learning. We attempt to limit this difficulty by providing docker [Mer14] images and
a GitHub repository for each case study. Docker is a simple yet powerful tool that
provides a virtual machine-like environment for high-level (Python) and low-level
(C++ and CUDA) libraries to operate in isolation from the operating system.
Directions for accessing and running the code are provided in the repository.
(https://github.com/SpringerNLP).
26 1 Introduction
• Chapter 3: A text clustering, topic modeling, and text classification based on the
Reuters-21,578 dataset is used to show some fundamental approaches to NLP.
• Chapter 5: Embedding methods are introduced with a focus on text based rep-
resentations on the Open American National Corpus.
• Chapter 9: This chapter has two distinct case studies. In the first, neural machine
translation from Chap. 7 is extended, exploring various attention-based architec-
tures. In the second, memory augmented networks are compared for question-
answering tasks based on the bAbI dataset.
• Chapter 11: Transfer learning and specifically domain adaptation using various
techniques on the Amazon Review dataset is performed.
• Chapter 12: Continuing the ASR case study from Chap. 8, end-to-end tech-
niques are applied to speech recognition leveraging CTC and attention on the
Common Voice dataset.
• Chapter 13: Two popular reinforcement learning algorithms are applied to the
task of abstractive text summarization using the Cornell Newsroom dataset.
References 27
References
ICASSP ’97, Munich, Germany, April 21–24, 1997. 1997, pp. 151–
154.
[Lee88] Kai-Fu Lee. “On large-vocabulary speaker-independent continu-
ous speech recognition”. In: Speech communication 7.4 (1988), pp.
375–379.
[Lin92] Long-Ji Lin. “Reinforcement Learning for Robots Using Neu-
ral Networks”. UMI Order No. GAX93-22750. PhD thesis. Pitts-
burgh, PA, USA, 1992.
[Lin70] S. Linnainmaa. “The representation of the cumulative rounding er-
ror of an algorithm as a Taylor expansion of the local rounding
errors”. MA thesis. Univ. Helsinki, 1970.
[Liu+18] Bing Liu et al. “Dialogue Learning with Human Teaching and
Feedback in End-to-End Trainable Task-Oriented Dialogue Sys-
tems”. In: Proceedings of the 2018 Conference of the North Ameri-
can Chapter of the Association for Computational Linguistics: Hu-
man Language Technologies, Volume 1 (Long Papers). Association
for Computational Linguistics, 2018, pp. 2060–2069.
[LR90] Bruce Lowerre and Raj Reddy. “The HARPY speech understand-
ing system”. In: Readings in speech recognition. Elsevier, 1990,
pp. 576–586.
[LSM13] Minh-Thang Luong, Richard Socher, and Christopher D Manning.
“Better Word Representations with Recursive Neural Networks for
Morphology”. In: CoNLL-2013 (2013), p. 104.
[MIG02] C. Macleod, N. Ide, and R. Grishman. “The American National
Corpus: Standardized Resources for American English”. In: Pro-
ceedings of 2nd Language Resources and Evaluation Conference
(LREC). 2002, pp. 831–836.
[Man99] Inderjeet Mani. Advances in Automatic Text Summarization. Ed. by
Mark T. Maybury. MIT Press, 1999.
[MMS99] Christopher D Manning, Christopher D Manning, and Hinrich
Schütze. Foundations of statistical natural language processing.
MIT press, 1999.
[MS99] Christopher D. Manning and Hinrich Schütze. Foundations of Sta-
tistical Natural Language Processing. MIT Press, 1999.
[Mar+94] Mitchell Marcus et al. “The Penn Treebank: Annotating Predicate
Argument Structure”. In: Proceedings of the Workshop on Human
Language Technology. Association for Computational Linguistics,
1994, pp. 114–119.
[MMS93] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice San-
torini. “Building a large annotated corpus of English: The Penn
Treebank”. In: Computational linguistics 19.2 (1993), pp. 313–
330.
[Mas61] Margaret Masterman. “Semantic message detection for machine
translation using an interlingua”. In: Proceedings of the Interna-
34 1 Introduction
[NH10] Vinod Nair and Geoffrey E. Hinton. “Rectified Linear Units Im-
prove Restricted Boltzmann Machines”. In: Proceedings of the
27th International Conference on International Conference on Ma-
chine Learning. ICML’10. Omnipress, 2010, pp. 807–814.
[Nal+16] Ramesh Nallapati et al. “Abstractive text summarization using
sequence-to-sequence RNNs and beyond”. In: arXiv preprint
arXiv:1602.06023 (2016).
[Nea95] Radford M Neal. “Bayesian learning for neural networks”. PhD
thesis. University of Toronto, 1995.
[NB18] Lance Norskog and Chris Bagwell. “Sox-Sound eXchange”. In:
(2018).
[Pan+15] Vassil Panayotov et al. “LibriSpeech: an ASR corpus based on pub-
lic domain audio books”. In: Acoustics, Speech and Signal Pro-
cessing (ICASSP), 2015 IEEE International Conference on. 2015,
pp. 5206–5210.
[PLV02] Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. “Thumbs
Up?: Sentiment Classification Using Machine Learning Tech-
niques”. In: Proceedings of the ACL-02 Conference on Empirical
Methods in Natural Language Processing - Volume 10. Association
for Computational Linguistics, 2002, pp. 79–86.
[Pap+02] Kishore Papineni et al. “BLEU: A Method for Automatic Evalua-
tion of Machine Translation”. In: Proceedings of the 40th Annual
Meeting on Association for Computational Linguistics. Associa-
tion for Computational Linguistics, 2002, pp. 311–318.
[Par85] D. B. Parker. Learning-Logic. Tech. rep. TR-47. Center for Comp.
Research in Economics and Management Sci., MIT, 1985.
[PB92] Douglas B Paul and Janet M Baker. “The design for the Wall Street
Journal-based CSR corpus”. In: Proceedings of the workshop on
Speech and Natural Language. 1992, pp. 357–362.
[PXS17] Romain Paulus, Caiming Xiong, and Richard Socher. “A Deep
Reinforced Model for Abstractive Summarization”. In: CoRR
abs/1705.04304 (2017).
[PC66] John R. Pierce and John B. Carroll. Language and Machines: Com-
puters in Translation and Linguistics. Washington, DC, USA: Na-
tional Academy of Sciences/National Research Council, 1966.
[PSG16] Barbara Plank, Anders Søgaard, and Yoav Goldberg. “Multilin-
gual Part-of-Speech Tagging with Bidirectional Long Short-Term
Memory Models and Auxiliary Loss”. In: Proceedings of the 54th
Annual Meeting of the Association for Computational Linguistics
(Volume 2: Short Papers). Association for Computational Linguis-
tics, 2016, pp. 412–418.
[Pom89] Dean A. Pomerleau. “Advances in Neural Information Processing
Systems 1”. In: Morgan Kaufmann Publishers Inc., 1989. Chap.
ALVINN: An Autonomous Land Vehicle in a Neural Network, pp.
305–313.
36 1 Introduction
[SVL14] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. “Sequence to Se-
quence Learning with Neural Networks”. In: Proceedings of the
27th International Conference on Neural Information Processing
Systems - Volume 2. MIT Press, 2014, pp. 3104–3112.
[Sye+18] Shahbaz Syed et al. Dataset for generating TL;DR. Feb. 2018.
[Tai+14] Yaniv Taigman et al. “DeepFace: Closing the Gap to Human-Level
Performance in Face Verification”. In: CVPR. IEEE Computer So-
ciety, 2014, pp. 1701–1708.
[Tes95] Gerald Tesauro. “Temporal Difference Learning and TD-
Gammon”. In: Commun. ACM 38.3 (Mar. 1995), pp. 58–68.
[Thr94] Sebastian Thrun. “Learning to Play the Game of Chess”. In: Ad-
vances in Neural Information Processing Systems 7, [NIPS Con-
ference, Denver, Colorado, USA, 1994]. 1994, pp. 1069–1076.
[TKSB00] Erik F. Tjong Kim Sang and Sabine Buchholz. “Introduction to
the CoNLL-2000 Shared Task: Chunking”. In: Proceedings of the
2Nd Workshop on Learning Language in Logic and the 4th Con-
ference on Computational Natural Language Learning - Volume 7.
ConLL’00. Association for Computational Linguistics, 2000, pp.
127–132.
[TKSDM03a] Erik F. Tjong Kim Sang and Fien De Meulder. “Introduction to
the CoNLL-2003 Shared Task: Language-independent Named En-
tity Recognition”. In: Proceedings of the Seventh Conference on
Natural Language Learning at HLT-NAACL 2003 - Volume 4. As-
sociation for Computational Linguistics, 2003, pp. 142–147.
[TKSDM03b] Erik F Tjong Kim Sang and Fien De Meulder. “Introduction to
the CoNLL-2003 shared task: Language-independent named entity
recognition”. In: Proceedings of the seventh conference on Natural
language learning at HLT-NAACL 2003-Volume 4. Association for
Computational Linguistics. 2003, pp. 142–147.
[TG01] Edmondo Trentin and Marco Gori. “A survey of hybrid AN-
N/HMM models for automatic speech recognition”. In: Neurocom-
puting 37.1–4 (2001), pp. 91–126.
[Tri+16] Adam Trischler et al. “NewsQA: A machine comprehension
dataset”. In: arXiv preprint arXiv:1611.09830 (2016).
[Tur95] A. M. Turing. “Computers &Amp; Thought”. In: MIT Press, 1995.
Chap. Computing Machinery and Intelligence, pp. 11–35.
[Vin+16] Emmanuel Vincent et al. “The 4th CHiME speech separation and
recognition challenge”. In: (2016).
[Wai+90] Alexander Waibel et al. “Phoneme recognition using time-delay
neural networks”. In: Readings in speech recognition. Elsevier,
1990, pp. 393–404.
[Wan+18] Xin Wang et al. “No Metrics Are Perfect: Adversarial Reward
Learning for Visual Storytelling”. In: Proceedings of the 56th An-
nual Meeting of the Association for Computational Linguistics
38 1 Introduction
2.1 Introduction
The goal of this chapter is to review basic concepts in machine learning that are
applicable or relate to deep learning. As it is not possible to cover every aspect of
machine learning in this chapter, we refer readers who wish to get a more in-depth
overview to textbooks, such as Learning from Data [AMMIL12] and Elements of
Statistical Learning Theory [HTF09].
We begin by giving the basic learning framework for supervised machine learn-
ing and the general learning process. We then discuss some core concepts of ma-
chine learning theory, such as VC analysis and bias–variance trade-off, and how
they relate to overfitting. We guide the reader through various model evaluation,
performance, and validation metrics. We discuss some basic linear classifiers start-
ing with discriminative ones, such as linear regression, perceptrons, and logistic re-
gression. We then give the general principle of non-linear transformations and high-
light support vector machines and other non-linear classifiers. In the treatment of
these topics, we introduce core concepts, such as regularization, gradient descent,
and more, and discuss their impact on effective training in machine learning. Gener-
ative classifiers, such as naive Bayes and linear discriminant analysis, are introduced
next. We then demonstrate how basic non-linearity can be achieved through linear
algorithms via transformations. We highlight common feature transformations, such
as feature selection and dimensionality reduction techniques. Finally, we introduce
the reader to the world of sequence modeling through Markov chains. We pro-
vide necessary details in two very effective methods of sequence modeling: hidden
Markov models and conditional random fields.
We conclude the chapter with a detailed case study of supervised machine learn-
ing using a real-world problem and dataset to carry out a systematic, evidence-based
machine learning process that allows putting into practice the concepts related in this
chapter.
The population of all possible data for a particular learning problem (e.g., discrim-
inating apples from oranges) is represented by an arbitrary set X. Samples can be
drawn independently from the population X with a probability distribution P(X),
which is unknown. They can be represented formally as:
X = x1 , x2 , . . . , xn (2.1)
Note that X ⊆ X. An individual data point in the set X drawn from the input
space X, also referred to as an instance or an example, is normally represented in
vector form as xi of d dimensions. The elements of a vector xi are also referred to as
features or attributes. For example, apples and oranges can be defined in terms of
{shape, size, color}, i.e., using d = 3 features/attributes. The features can be cate-
gorical or nominal, such as color = {red, green, orange, yellow}. Alternatively, they
can be ordinal. In the latter case, the features can be discrete (taking on a finite num-
ber of values) or continuous; e.g., each feature i ∈ [d] can be a scalar in R, yielding
a feature space of Rd :
Beside the probability distribution P(X), another unknown entity is the target func-
tion or the ideal function that maps the input space X to the output space Y. This
function is formally represented as [f : X → Y]. The objective in machine learning
is to find an approximation function close to the unknown target function f .
The output space Y represents all possible values that the target function f can
map inputs to. Generally, when the values are categorical, finding an approximation
to f is known as the classification problem, and when the values are continuous,
the problem is known as regression. When the output can only take two values, the
problem is known as binary classification (e.g., apples and oranges). In regression,
yi ∈ R.
Sometimes it is advantageous to think not in terms of an exact mapping or a
deterministic output for an instance (x, y), but instead in terms of a target joint prob-
ability distribution P(x, y) = P(x)P(y|x). The latter better accommodates real-world
data that contain noise or stochasticity, and we shall have more to say on this later.
The entire learning process can now be defined in terms of finding the approximate
function h(x) from a large hypothesis space H, such that h(x) can effectively fit the
given data points Dlabeled in such a way that h(x) ≈ f (x). The measure of success is
normally quantified by error (alternatively referred to as empirical risk, loss, or cost
function), which measures the discrepancy between h(x) and the unknown target
function f (x). Specifically,
42 2 Basics of Machine Learning
1 N
Elabeled (h) = ∑ h(xn ) = yn
N n=1
(2.8)
1 N
Elabeled (h) = ∑ (h(x) − yn )2
N n=1
(2.9)
The prediction or out-of-sample error can be computed using the expected value
on the unseen datum (x, y):
error EVal , which serves as a proxy to the out-of-sample error. Finally, the test
set, DTest , is used to estimate the unbiased error of the model trained with
the best parameters over DVal and with learned parameters over DTrain . The
unbiased error gives a good estimate of the error on unseen future data.
Fig. 2.2: The labeled dataset DLabeled is split into the training DTrain , validation
DVal , and testing DTest datasets
In this section we will review basic theory associated with machine learning to ad-
dress core issues in any learning scenario.
The process of fitting a hypothesis function or a model to the labeled dataset can lead
to a core problem in machine learning that is known as overfitting. The issue here is
that we have used all the labeled data points to reduce the error, but this leads to poor
generalization. Let us consider a simple, one-dimensional dataset generated using
sin(x) as the target function with Gaussian noise added. We can illustrate the issue
of overfitting using a hypothesis set of polynomial functions. The different degrees
can be treated as parameters via which one can obtain different hypothesis functions
with which to fit the labeled data. Figure 2.3 shows how by changing the degree
parameter we can reduce the training error significantly by effectively increasing
the complexity of the model. Figure 2.3 shows how the choice of the hypothesis
function can result in underfitting or overfitting. The hypothesis function that is
44 2 Basics of Machine Learning
a polynomial of degree 1 poorly approximates the target function due to its lack of
complexity. In contrast, the hypothesis function that is so complex (degree 15 also
in Fig. 2.3) has even modeled the noise in the training data, resulting in overfitting.
Finding the right hypothesis that matches the given resources (training data) in such
a way that there is a balance between the approximation and generalization trade-off
is the holy grail of machine learning.
y
x x x
Fig. 2.3: Illustration of underfitting and overfitting in fitting a target function with
Gaussian noise with different-degree polynomials
Thus, there are two distinct errors to consider: (a) the in-sample training error,
given by Etrain (h), that measures the approximation aspect of the trade-off, and (b)
the out-of sample error, given by Eout (h), that has to be estimated and measures
the generalization aspect of the trade-off.
The probably approximately correct (PAC) learnability theorem provides the fol-
lowing theoretical bound between the two errors in terms of the probability of the
model being approximately correct:
−ε 2 N )
P[|Etrain (h) − Eout (h)| > ε ] ≤ 4mH (2N)e( 8 (2.11)
The equation bounds the probability that the absolute difference between the
two errors Etrain (h) and Eout (h) is smaller than ε by something known as the growth
function mH and the number of training data samples N. It has been shown that even
with an infinite hypothesis space of the learning algorithm (such as a perceptron, as
we will discuss later), the growth function is finite. In particular, the growth func-
tion has a tight upper bound measured in terms of the Vapnik–Chervonenkis (VC)
dimension dVC , which is the largest N that can be shattered; i.e., mH (N) = 2N . This
makes mH (N) polynomial in the number of data points [Vap95]. Thus,
dVC log(N)
Eout (h) ≤ Etrain (h) + O (2.14)
N
The VC dimension, given by dVC , correlates with the model complexity, and the
above equation can be further rewritten as:
Figure 2.4 captures the relationship between Eout (h) and Etrain (h) in the above
equations. When the model complexity is below an optimum threshold dVC ∗ , both
the training error and the out-of-sample error are decreasing. Choosing any model
to represent the data below this optimal threshold will lead to underfitting. When the
model complexity is above the threshold, the training error Etrain (h) still decreases,
but the out-of-sample error Eout (h) increases, and choosing any model with that
complexity will lead to overfitting.
Fig. 2.4: Model complexity and learning curves. (a) Relationship between training
error, out-of-sample error, and model complexity. (b) Learning curves for relation-
ship between training error, out-of-sample error, and number of data points
46 2 Basics of Machine Learning
Bias: The term ( f (x) − h̄(x))2 corresponds to the systematic error caused by
not having a good or sufficiently complex hypothesis to approximate the target
function f (x).
Noise: The term (y − f (x))2 corresponds to the inherent noise present in the data.
In general, a simple model suffers from a large bias, whereas a complex model
suffers from a large variance. To illustrate the bias–variance trade-off, we use again
a one-dimensional sin(x) as a target function with added Gaussian noise to generate
data points. We fit polynomial regression with various degrees, as shown in Fig. 2.5.
Fig. 2.5: Bias, variance, and noise errors for two hypotheses, i.e., polynomials with
degree 1 and degree 12
2.4 Machine Learning Theory 47
The bias–variance trade-off is clearly evident in Fig. 2.5 and Table 2.1, which lists
the variance, bias, and noise in each case. A simple degree 1 model has higher
bias error contributing towards underfitting while a complex degree 12 shows huge
variance error contributing towards overfitting.
Table 2.1: Bias, variance, and noise errors for polynomials of degree 1 and 12
We will consider the simple case of binary classification. In the classification do-
main, the simplest visualization of the success of a model is normally described us-
ing the confusion matrix, shown in Fig. 2.6. Accuracy and classification error are
informative measures of success when the data is balanced in terms of the classes;
that is, the classes have similar sizes. When the data is imbalanced, i.e., one class is
represented in larger proportion over the other class in the dataset, these measures
become biased towards the majority class and give a wrong estimate of success.
In such cases, base measures, such as true positive rate (TPR), false positive rate
(FPR), true negative rate (TNR), and false negative rate (FNR), become useful. For
instance, metrics such as F1 score and Matthews correlation coefficient (MCC) com-
bine the base measures to give an overall measure of success. Definitions are listed
below.
48 2 Basics of Machine Learning
6. Accuracy
TN +TP
Accuracy = (2.22)
(T P + FN + FP + T N)
7. F1 score
Precision × Recall
F1 = 2 (2.23)
(Precision + Recall)
8. Matthews correlation coefficient (MCC)
T P × T N − FP × FN
MCC = 2 (2.24)
(T P + FP) × (T P + FN) × (T N + FP) × (T N + FN)
Many classification models provide not only a prediction of the class, but also
a confidence value between 0 and 1 for each data point. The confidence thresh-
old can control the performance of the classifier in terms of TPR and FPR. The
curve that plots TPR and FPR for a classifier at various thresholds is known as the
receiver-operating characteristic (ROC) curve. Similarly, precision and recall can
be plotted at different thresholds, giving the precision-recall curve (PRC). The ar-
eas under each curve are then respectively known as auROC and auPRC and are
popular metrics of performance. In particular, auPRC is generally considered to be
an informative metric in the presence of imbalanced classes.
In the regression domain, where the predicted output is a real number that is com-
pared with the actual value (another real number), many variants of squared errors
are employed as evaluation metrics. We list a few below.
4. Relative squared error (RSE) is used when two errors are measured in differ-
ent units:
∑n (yi − ŷi )2
RSE = i=1 (2.28)
∑ni=1 (ȳi − yi )2
5. Coefficient of determination (R2 ) summarizes the explanatory power of the
regression model and is given in terms of squared errors:
n
SSEresidual = ∑ (yi − ŷi )2 (2.29)
i=1
n
SSEtotal = ∑ (yi − ȳi )2 (2.30)
i=1
SSEresidual
R2 = 1 − (2.31)
SSEtotal
Validation techniques are meant to answer the question of how to select a model(s)
with the right hyperparameter values. When there are many hypotheses in the hy-
pothesis set, then each unique hypothesis is trained on the training set Dtrain and then
evaluated on the validation set Dval ; the model(s) with the best performance metrics
is then chosen. Logically, the model h− (superscript denotes the model trained on
less data) is trained on a training dataset which has fewer points M as compared to
the entire set, as some data, K in all, are in the validation set. The performance on
the validation set is then given as:
1 K
EVal (h− ) = ∑ e(h− (xn ), yn )
K n=1
(2.32)
− − 1
Eout (h ) ≤ EVal (h ) + O √ (2.33)
K
This equation shows that the larger the number of validation data points
K, the better the estimation of the out-of-sample error is. However, from the
learning curves, we now know that the more the training data, the smaller
the training error. Thus, by removing K points from the budget of training,
we have theoretically increased the chances of having a larger training error.
2.4 Machine Learning Theory 51
− − 1
Eout (h) ≤ Eout (h ) ≤ EVal (h ) + O √ (2.34)
K
An important point to note is that when the validation set has been used for
model performance evaluation, the estimates of out-of-sample errors derived
from the validation errors are optimistic; that is, the validation set is now a
biased set, because we use it indirectly to learn the hyperparameters of the
models. The validation process is a simple method that can be used for model
selection and is independent of the model or learning algorithm.
The only drawback with the validation process is the need to have a large num-
ber of labeled data points for creating the training set and the validation set. It is
normally difficult to collect a large labeled set due to the cost or difficulty in ob-
taining the labels. In such cases, instead of physically separating the training set
and validation set, a technique known as k-fold cross-validation is used. The k-fold
cross-validation algorithm is shown in Fig. 2.8. First, the data is randomly divided
into k sets. Then, in each of the resulting k experiments, k − 1 data folds are used
k for a fold. Finally,
for training and onefold is used for validation to measure the EVal
the average of the k validation errors is used as a single estimate of the validation
error EVal .
52 2 Basics of Machine Learning
Algorithm 1: FindBestParameters
Data: DTrain [k], DVal [k]
Result: bestParameter, lowestError
begin
splitDataFolds: Create folds of labeled dataset D,k
bestParameter ← λ0
for m ∈ 1..M do
λ ← λm
for i ∈ 1..k do
trainModel(h(λ ), DTrain [i]) EVAL [i] ← testModel(h(λ ), DVal [i])
EVAL ← 1k ∑Kk=1 EVAL [k]
if current value is the best seen then
lowestError ← EVAL
bestParameter ← λm
Once the hypothesis or the model with the best parameters is selected and trained
with the entire Dtrain ∪ Dval set, the test set Dtest is then used to estimate the test
error. Recall that the PAC equation is a function of the hypotheses set size M and
dataset size N. Therefore, when only one hypothesis is considered, M = 1, in the
presence of a sufficiently large test set, the PAC equation confirms that the test error
is a good approximation of the out-of-sample error. Since the test set has not been
used for either model training or model hyperparameter selection, the error remains
an unbiased estimate in contrast to the training or validation error.
If a comparison needs to be made between two or more classifiers on one or more
datasets, to obtain statistical estimations of differences of metrics, various statistical
hypothesis testing techniques can be employed. One has to be aware of the assump-
tions and constraints of each technique before selecting one [JS11, Dem06, Die98].
Let us describe a few statistical hypothesis testing techniques.
Resampled t-test is a parametric test (in that it assumes a distribution) that is
used to compare two classifiers on a metric such as accuracy or error via n different
random resampling of training and test subsets from a single dataset. If p̂i is the
difference between the performance of the two classifiers and is assumed to have a
Gaussian distribution, and p̄ is the average performance difference, the t-statistic is
given by:
√
p̄ n
t= (2.35)
∑ni=1 ( p̂i − p̄)2
(n−1)
Table 2.2: Basic counts of errors and correct for two classifiers on a test set
The Wilcoxon signed-ranks test is a non-parametric test that is popular when two
classifiers have to be compared on multiple datasets. The test ranks the differences
between metrics of the two classifiers under comparison on the i-th dataset of N
datasets as di = pi1 − pi2 , ignoring the signs. The statistic is given by
T = min(R+ , R− ) (2.37)
where
1
R+ = ∑ rank(di ) + 2 ∑ rank(di )
di >0 di =0
and
1
R− = ∑ rank(di ) + 2 ∑ rank(di )
di <0 di =0
(N − 1)χF2
F= (2.38)
K(K − 1) − χF2
where
k
12N K(K + 1)2
χF2 =
K(K + 1) ∑ R2j −
4
j=1
Though not true in every scenario, there are many practical tips for machine
learning practitioners. We list a few here.
2.5 Linear Algorithms 55
Let us now consider similar issues in the context of regression. In this chapter, we
will limit our discussion to linear regression. We will give basic introduction and
equations around each algorithm followed by discussion points capturing advan-
tages and limitations.
Linear regression is one of the simplest linear models that has been analyzed theo-
retically and applied practically to many domains [KK62]. The dataset assumes the
56 2 Basics of Machine Learning
labels to be numeric value or a real number; for instance, one could be interested in
predicting the house price in a location where historical data of previous house sales
with important features such as structure, rooms, neighborhood, and other features
have been collected. Due to the popularity of linear regression, let us go over some
important elements, such as concepts, facets of optimization, and others.
The hypothesis h is a linear combination of input x and a weight parame-
ters w (that we intend to learn through training). In a d-dimensional input (x =
[x1 , x2 . . . xd ]), we introduce another dimension called the bias term, x0 , with value
1. Thus the input can be seen as x ∈ {1} × Rd , and the weights to be learned are
w ∈ Rd+1 .
d
h(x) = ∑ wi xi (2.39)
i=0
1 d
Etrain (h(x, w)) = ∑ (wT xn − yn )2
N i=0
(2.40)
Since the data x is given, we will write the equation in terms of weights w:
1
Etrain (w) = (Xw − y)2 , (2.41)
N
where (Xw − y)2 is the Euclidean norm of a vector.
So, we can write
1 T T
Etrain (w) = (w X Xw − 2wT XT y + yT y) (2.42)
N
We need to minimize Etrain . This is an optimization problem, as we need to find
the weights wopt that minimize the training error. That is, we need to find:
We can assume that the loss function Etrain (w) is differentiable. So, to obtain a
solution, we take the gradient of the loss function with respect to w, and set it to the
zero vector 0:
2 T
∇Etrain (w) = (X Xw − XT y) = 0 (2.44)
N
2.5 Linear Algorithms 57
XT Xw = XT y (2.45)
We also assume that XT X is invertible, and so we obtain
• Linear regression has an efficient training algorithm, with time complexity poly-
nomial in the size of the training data.
• Linear regression assumes X† is invertible. Even if this is not the case, the
pseudo-inverse can be employed, though doing so does not guarantee a unique
optimal solution. We note that there are techniques that can compute the pseudo-
inverse without inverting the matrix.
• The performance of linear regression is affected if there is correlation among the
features in the training set.
58 2 Basics of Machine Learning
2.5.2 Perceptron
Perceptrons are models based on the linear regression hypothesis composed with the
sign function that provides a classification output instead of regression, as shown
below, and illustrated in Fig. 2.9.
d
h(x) = sign ∑ wi xi (2.47)
i=0
• Perceptrons need not find the best hyperplane (maximum separation between
points) separating the two classes and suffer from noise in the datasets.
• Outliers impact the algorithm’s ability to find the best hyperplane and so address-
ing them is important.
2.5 Linear Algorithms 59
Algorithm 4: Perceptron
Data: Training Dataset (x1 , y1 ), (x2 , y2 ), ..(xn , yn ) such that xi ∈ Rd and yi ∈ (+1, −1),
MaxIterations= T
Result: Weight vector w ∈ Rd+1
begin
create a vector x from inputs and prefix the input vector with the bias term x0 = 1
create a vector y from labels
initialize weight vector w0 to be 0
bestWeight ← w0
initialize loss(w) to be 1
for t ∈ 0..T − 1 do
for i ∈ 0..N − 1 do
if sign(xi yi ) = yi then
update the weight vector w(t+1) = w(t) + xi yi
currentLoss(w) ← currentLoss(w(t+1) )
if currentLoss(w) < loss(w) then
loss(w) ← currentLoss(w)
bestWeight = w(t+1)
return bestWeight
Algorithm 5: Perceptron
Data: Test Data x such that x ∈ Rd and weight vector
Result: Prediction ŷ ∈ (+1, −1)
begin
create a vector x from inputs and adding a bias for each vector x0 = 1
ŷ = sign(xT w)
return ŷ
2.5.3 Regularization
In the augmented loss function above, the scalar parameter λ is known as the
regularization constant, and Ω(w) is known as the regularization function.
One of the popular regularization functions is the L2 norm [HK00], also known
as weight decay or ridge regularization. It can be substituted in Eq. 2.49 as shown
below:
Due to the absolute function, the above equation does not have a closed-form so-
lution and is generally represented as a constrained optimization problem as below:
exp(wT x)
h(x) = (2.56)
1 + exp(wT x)
For a binary classification, where y ∈ {−1, +1}, the hypothesis can be seen as
a likelihood of predicting y = +1, i.e., P(y = +1|x). Thus, the equation can be
rewritten as a log-odds ratio, and weights are learned to maximize the conditional
likelihood given the inputs:
P(y = +1|x)
= wT x (2.57)
P(y = −1|x)
The log-likelihood of the hypothesis can be written as:
n
log h(x) = ∑ log P(yi |xi ) (2.58)
i=0
n
log h(xi ) i f yi = +1
log L(h(x)) = ∑ (2.59)
i=0 (1 − log h(xi )) i f yi = −1
n
log L(h(x)) = ∑ (yi log h(xi ) + (1 − yi )(1 − log h(xi ))) (2.60)
i=0
be treated as our new error function Etrain , but it cannot be solved in closed form.
Instead of an analytical solution, an iterative algorithm known as gradient descent
can be employed. Gradient descent is a general optimization algorithm that is used
widely in machine learning, including deep learning. Let us discuss it at some length
below.
Let us recall that the goal is to find weights w that minimize Etrain , and that at the
minimum, the gradient of Etrain is 0. In gradient descent, the negative of the gradient
is followed in an iterative process until the gradient is zero. The gradient is a vector
containing partial derivatives over each of the dimension [Bry61], as shown below:
∂ Etrain ∂ Etrain ∂ Etrain
g = ∇Etrain (w) = , ... (2.61)
∂ w0 ∂ w1 ∂ wn
The normalized gradient ĝ can be written as:
∇Etrain (w)
ĝ = (2.62)
∇Etrain (w)
A small step size η is made in the direction of −ĝ, and the weights are updated
accordingly, leading to an optimal point. Selecting a small step size is important,
otherwise the algorithm oscillates and does not reach the optimum point. The algo-
rithm can be summarized as:
The weights w can be initialized to the 0 vector or set to random values (each
obtained from a normal distribution with 0 mean and small variance) or preset
values. Another important decision in gradient descent is the termination cri-
terion. The algorithm can be made to terminate when the number of iterations
2.5 Linear Algorithms 63
reaches a specific value or when the value of the gradient reaches a predefined
threshold, close to zero.
One of the disadvantages of gradient descent is the use of the entire training dataset
when computing the gradient. This has an implication on the memory and com-
putation speed, which increase as the number and dimension of training examples
increase. Stochastic gradient descent is a version of gradient descent that, instead
of utilizing the entire training dataset, picks a data point uniformly at random from
the training dataset (hence the name stochastic). It has been shown that with a large
number of iterations and a small step size, stochastic gradient descent generally
reaches the same optimum as the batch gradient descent algorithm [BB08].
Figure 2.10 illustrates the iterative changes in the training error for (batch) gradi-
ent descent and stochastic gradient descent for a one-dimensional linear regression
problem.
It can be shown that in logistic regression the gradient is:
1 n yi xi
∇Etrain (w) = − ∑
N i=0 (1 + expyi wT xi )
(2.63)
Fig. 2.10: One-dimensional regression with gradient descent and stochastic gradient
descent
All algorithms we have seen so far have been discriminative in their approach; that
is, they make no assumption about the underlying distribution of the data and focus
instead on the end goal of prediction accuracy. Another popular approach in machine
learning is the generative approach, which assumes an underlying distribution with
which the data is generated and tries to find parameters of this distribution in its
training.
The generative approach, though an indirect mechanism for achieving the predic-
tion accuracy, has been very successful in real-world applications. Many machine
learning algorithms, both supervised and unsupervised, naive Bayes, linear discrim-
inant analysis, expectation maximization, and Bayes networks among others, are
based on the generative approach and have a probabilistic foundation in the Bayes
theorem.
Formally, given a hypothesis h and a training dataset Dtrain , the Bayes theorem
helps in defining the probability of choosing the hypothesis given the data; that is,
it helps define P(h|Dtrain ) given the prior probability of the hypothesis P(h), the
likelihood of the data given the hypothesis P(Dtrain |h), and the probability of data
over all hypotheses P(Dtrain ) = h P(Dtrain |h) as:
2.5 Linear Algorithms 65
P(Dtrain |h)P(h)
P(h|Dtrain ) = (2.64)
P(Dtrain )
If there are multiple hypotheses, the question of which one is the most probable
one given the training data can be answered by the maximum a posteriori hypoth-
esis as:
P(Dtrain |h)P(h)
hMAP = arg max (2.66)
h∈H P(Dtrain )
Since P(Dtrain ) is independent of h, we have:
If we further assume that all the hypotheses are equally likely (i.e., P(h1 ) ≈
P(h2 ) ≈ P(hm ) for m hypotheses), the equation can be reduced to:
As stated in the assumptions, if the training examples are independent and identi-
cally distributed (i.i.d), P(Dtrain |h) can be written in terms of the training examples
as:
N N
P(Dtrain |h) = ∏ P(xi , yi |h) = ∏ P(yi |xi ; h)P(xi ) (2.69)
i=1 i=1
1 N countLabel(y)
P(Y = y) = ∑
N i=0
yi = y =
N
(2.72)
Linear discriminant analysis (LDA) is another generative model, where the assump-
tion of a Gaussian distribution for P(X|Y ) is made along with equal priors for binary
classes, i.e., P(Y = 1) = P(Y = 0) = 1/2. Formally, μ ∈ Rd is the multivariate mean,
and Σ is the covariance matrix. Then, we have:
1 −1 −1
P(X = x|Y = y)P(Y = y) = exp (x − μ ) |Σ | (x − μ )
T
(2π )d/2 |Σ |1/2 2
(2.74)
The training of LDA, similar to naive Bayes, involves estimating the parameters,
(μ and Σ here) from the training data.
1. It is always a good idea to scale the input real-valued features to the range
[0, 1] for gradient descent algorithms.
2. Binary or categorical features which are represented as one-hot vectors
can be used without any transformations. In one-hot vector represen-
tation, each categorical attribute is converted into k boolean valued at-
tributes such that only one of those k attributes has a value of one and rest
zero for a given instance.
3. Grid search over a range of values spanning multiple orders of magni-
tude should be used to determine the learning rate and the regularization
parameter.
2.6 Non-linear Algorithms 67
The algorithms we have seen so far, given by sign(wT x), are linear in the weights w,
as the inputs x are a given or constant for the training algorithm. A simple extension
is to use a non-linear transform φ (x) applied to all the features, which transforms
the points into a new space, say Z, where a linear model given by sign(wT φ (x))
can then be learned. When a prediction is needed on a new unseen data x, first the
data is transformed into the Z space using the transformation φ (x), and then linear
algorithm weights are applied in the Z space to make the prediction.
As an example, a simple non-linear, two-dimensional training dataset can
be transformed into a three-dimensional Z space, where the dimensions are
z = x1 , x2 , x12 + x22 . The Z space is linearly separable, as shown in Fig. 2.11.
space space
1.0
1.4
1.2
0.5
X1ˆ2 + X2ˆ3
1.0
0.8
X2
0.0 0.6
0.4
0.2
–0.5
0.0
1.0
–1.0 0.5
–1.0 0.0
–0.5 X2
–1.0 –0.5 0.0 0.5 1.0 0.0 –0.5
X1 X1 0.5
1.0 –1.0
1.0
1.4
0.5 1.2
X1ˆ2 + X2ˆ2
1.0
0.8
X2
0.0 0.6
0.4
0.2
–0.5
0.0
1.0
–1.0 0.5
–1.0 0.0
–0.5 X2
–1.0 –0.5 0.0 0.5 1.0 –0.5
0.0
X1 X1 0.5 –1.0
1.0
Support vector machine (SVM) is one of the most popular non-linear machine learn-
ing algorithms that can separate both linear and non-linear data using built-in trans-
formations known as kernels [Vap95]. SVMs not only separate the data but also find
the hyperplane that separates the data in the most optimal way through a principle
known as maximum margin separation, as shown in Fig. 2.12. The data points that
separate the hyperplane and lie on the margin are known as support vectors.
Fig. 2.12: Illustration of SVM finding the maximum margin separation between
labeled data
d
1
∑ |x − x |q
q
dist(x, x ) = (2.78)
d=1
Neural networks are another extension of perceptrons that are used to create non-
linear boundaries. We discuss them in detail in Chap. 4. Decision trees and many of
its extensions, such as the gradient boosted algorithm and random forest among
others, are based on the principle of finding simpler decision boundaries for the
features and combining them in hierarchical trees, as shown in Fig. 2.13.
In this section we will review some of the common techniques used for feature
transformation, selection, and reduction.
70 2 Basics of Machine Learning
In many algorithms, it is beneficial to have all the features in the same range, for
example, in [0, 1], for the algorithm to not be biased or run effectively across all
features. Some commonly used transformations are as follows.
Each feature can be transformed by subtracting the mean from its value, f f eature,i =
fi − f¯f eature :
1 N
f¯f eature = ∑ fi (2.79)
N i=0
Each feature can be transformed to be in the range of [0, 1]. For a feature f f eature ,
let f f eature Max correspond to the maximum value in the dataset, and f f eature Min
correspond to the minimum value. Then the transformation for an instance i is:
( fi − f f eature Max)
fi = (2.80)
( f f eature Max − f f eature Min)
2.7.1.3 Standardization
In this transformation, the features change to have zero mean and unit variance. The
empirical variance of a feature v f eature is calculated on a dataset by:
1 N
v f eature = ∑ ( fi − f¯f eature )2
N i=0
(2.81)
( fi − f¯f eature )
The transformation for each feature is fi = √ .
v
2.7.1.4 Discretization
Fig. 2.14: PCA process of finding reduced dimensions from original features
WCov(X)WT = I (2.84)
Covariance matrices are positive semi-definite, symmetrical, have orthogonal
eigenvectors, and real-valued eigenvalues. Matrix A can be factorized as UAUT = Λ ,
72 2 Basics of Machine Learning
where U has the eigenvectors of A in its columns, and Λ = diag(λi ), where λi are
the eigenvalues of A.
Thus the solution for WCov(X)WT is a function of the eigenvectors U and eigen-
values Λ of the covariance matrix, i.e., Cov(X). The algorithm is shown below.
Algorithm 9: PCA
Data: Dataset X = [x1 , x2 ..xN ] ∈ Rd , Components = m
Result: Transformed Data Y ∈ Rm
begin
X ← [x1 − μ , x2 − μ ..xN − μ ]
St ← N1 XXT
XT X = Vλ VT
1
U ← XVλ − 2
Um ← [u1 , u2 ..um ]
Y ← UTm X
return Y
In many sequence data analysis problems, such as language modeling, time series
analysis, and signal processing, modeling the underlying process as a Markov pro-
cess has been very successful. Many traditional NLP tasks, such a parts-of-speech
tagging, extracting information, and phrase chunking, have been modeled very suc-
cessfully using hidden Markov models (HMM), a special type of Markov pro-
cesses. In next few sections, we will discuss some important theory, properties, and
algorithms associated with Markov chains [KS+60].
Markov chains are the basic building blocks for modeling many sequential pro-
cesses. Consider a finite set of states modeled as S = {s1 , s2 , . . . , sn }, and let a vari-
able q represent the transition at any time t as qt . An illustration is provided in
Fig. 2.15.
The Markov property states that at any time t, the probability of it being in a state
si depends only on the previous k states rather than all the states from time 1 to t − 1.
This can be expressed as:
The simplest Markov chain depends only on the most recent state (k = 1) and is
represented as:
At times, the states of the Markov chain are hidden and not observed, but they
produce effects which are observable. Such Markov chains are represented using
hidden Markov models (HMM), as shown in Fig. 2.16, where new observed states
represented by set V having fixed m elements, such as V = (v1 , v2 , . . . vm ), are added
to the previous Markov chain [Rab89].
The concepts required in HMM are:
• Finite hidden states S = (s1 , s2 , ..sn ) and finite observable states V = (v1 , v2 , . . . vm ).
• For a fixed state sequence transition of length T , given by Q = q1 , q2 , . . . qT , the
observations are given by O = o1 , o2 , . . . oT .
• The parameters of HMM are λ = (A, b, π ), where
– The vector b represents the probability of observing a state vk given the hidden
state si and is independent of time, given by:
– The vector π represents the initial probability of the states and is given by:
n
πi = P(q1 = si ) s.t. ∑ πi = 1 (2.91)
1
1. Likelihood
Given an HMM (λ ) and a sequence of observations O, what is the likeli-
hood of the HMM generating the sequence; i.e., what is P(O|λ )? A dynamic
programming-based technique known as the forward algorithm, which stores
intermediate values of states and its probabilities to finally build up the proba-
bility of the whole sequence in an efficient manner, is generally employed.
2. Decoding
Given an HMM (λ ) and a sequence of observations O, what is the most
likely hidden state sequence S that generated the observations? A dynamic
programming-based technique known as the Viterbi algorithm, similar to the
forward algorithm with minor changes, is used to answer this question.
3. Learning: Supervised and Unsupervised
Given a sequence of observations O and the state sequences S, what are the
2.8 Sequence Data and Modeling 75
parameters of the HMM that could generate it? This is a supervised learning
problem and can be easily obtained from the training examples by computing
different probabilities using the Count() function for the likelihood estimates.
The individual cells Ai, j of the transition probability matrix A can be estimated
by counting the number of times the state s j is followed by the state si as:
Count(s j , si )
Ai, j = P(s j |si ) = (2.94)
Count(si )
The elements of the array b(k) can be estimated by counting the number of
times the observed state vk happens along with the hidden state s j and is given
by:
Count(vk , s j )
b j (k) = P(vk |s j ) = (2.95)
Count(s j )
And the initial probabilities computed as:
Count(q1 = si )
πi = P(q1 = si ) = (2.96)
Count(q1 )
If only the sequence of observations O is provided and we need to learn the model
that maximizes the probability of the sequence, the unsupervised learning problem
is solved using a variation of the expectation maximization (EM) algorithm known
as the Baum–Welch algorithm.
Analogous to the relation between naive Bayes and logistic regression, conditional
random field (CRF) has a similar relationship to HMM in the sequence modeling
world. HMMs have shortcomings in effectively modeling the dependency between
the inputs or the observed states and even the overlapping relationship between
them. A linear chain CRF can be considered to be an undirected graph model equiv-
alent of a linear HMM, as shown in Fig. 2.17 [LMP01]. CRF is used mostly in su-
pervised learning problems, though there are extensions that address unsupervised
learning, whereas HMMs can be easily used for unsupervised learning.
To illustrate CRFs, let us take as input a simple sentence Obama gave a speech
at the Google campus in Mountain View. Each input word will be assigned a tag
of either Person, Organization, Location, or Other, as illustrated in Fig. 2.18. The
association of these tags to words is known as the named entity recognition problem
in text processing.
76 2 Basics of Machine Learning
Feature functions are the basic units of CRF that capture the relationship between
a pair of two consecutive outputs yi−1 , yi with the entire input sequence x1 , x2 , . . . xn
as a real-valued output given by f j (yi−1 , yi , x1:n , i). A simple binary feature function
can be written for our example as:
1 i f yi = Location , yi−1 = Location and xi = View
f j (yi−1 , yi , x, i) =
0 otherwise
(2.97)
The entire labeled sequence of length n can be modeled as log-linear in terms of the
feature functions f j (yi−1 , yi , x1:n , i) and their weights λ j , similar to logistic regres-
sion, as given by:
1 n
P(y|x, λ ) = exp ∑ ∑ f j (yi−1 , yi , x, i) (2.98)
Z(x, λ ) i=0 j
where Z(x, λ ) is known as the normalization constant or the partition function and
is given by:
n
Z(x, λ ) = ∑ exp ∑ ∑ f j (yi−1 , yi , x, i) (2.99)
y∈Y i=0 j
2.8 Sequence Data and Modeling 77
Fig. 2.18: An information extraction example given with words as the inputs and the
named entity tags as the outputs of the CRF
m n
1
L(λ , D) = − ∑ log exp ∑ ∑ f j (yi−1 , yi , x, i) (2.101)
k=1 Z(x, λ ) i=0 j
The optimal parameter λopt can be estimated using the equation below, where C
acts as a prior or regularization constant.
1
λopt = arg min L(λ , D) +C |λ |2 (2.102)
λ 2
The above equation is convex, and solving for the optimum will guarantee ob-
taining the global optimum. If we rewrite the feature functions for simplicity as
below and differentiate the above equation w.r.t λ j :
∂ L(λ , D) −1 m m
∂λj
= ∑ Fj (yk , xk )
m k=1
+ ∑ EP(y|xk ,λ ) [Fj (yk , xk )] (2.104)
k=1
observed mean f eature value expected f eature value given the Model
It can be seen that this equation is not in a closed form, hence impossible to solve
analytically. Various iterative algorithms such as L-BFGS or even gradient descent
(as discussed above) are generally employed to obtain the solution.
78 2 Basics of Machine Learning
We now take the reader through a real-world application of the concepts introduced
in the chapter through a case study. The case study also equips the reader with
necessary practical hands-on tools, libraries, methods, code, and analysis that will
be useful for standard machine learning, as well as deep learning.
We use the Higgs Boson challenge which was hosted by Kaggle. The challenge
data is now available on ATLAS Higgs Challenge 2014. The case study is to classify
the events into signals and background (any other event other than the signal). This
is a binary classification problem. Most Kaggle challenges or hackathons provide
training data with labels. The models that get submitted are then evaluated on the
blind test data on a well-known metrics. Instead of the entire dataset, we have used
a sample dataset which has training data size of 10,000 and a separate testing data
of size 5000 with labels on which models will be evaluated. We will also assume the
best model is selected based on the classification accuracy achieved on the test data,
with metrics of accuracy, as the data is well balanced between the two classes.
The goal of the case study is to use various techniques and methods illustrated
in the chapter and compare the performance on the unseen test set. Various Python
libraries such as Numpy, Scipy, Pandas, and scikit-learn, which are used extensively
in machine learning, are introduced in the case study.
First, we need to describe the main open source tools and libraries we will use for
our case study.
We use basic EDA to understand the characteristics of the data through univariate
statistics, correlation analysis, and visualization.
2.9 Case Study 79
1. Exploring the number of training and testing data in terms of features and num-
ber of classes per set.
2. Exploring the data types for each feature to determine whether it is categorical,
continuous, ordinal, etc. and transforming them if needed based on the domain.
3. Finding if the features have missing or unknown values and transforming them
as needed.
4. Understanding the distribution of each feature using scatter plots, histogram
plots, box plots, etc., to see basic statistics of range, variance, and distribution
for the features. This is illustrated in Fig. 2.19.
5. Understanding similarity and differences between each of these statistics and
plots for the training and testing data features.
6. Calculating pairwise correlation between the features and correlation between
features and the labels on training set. Plotting these and visualizing them (as in
Fig. 2.19) gives a great aid to subject matter experts and data scientists.
In this section we will go over some standard machine learning techniques per-
formed on the data for learning effective models.
Understanding the impact of feature transformation and selection is one of the pre-
liminaries for training a machine learning model. As discussed earlier in the chapter
there are various dimensionality reduction techniques such as PCA, SVD, and oth-
ers, various selection techniques, such as mutual information, chi-square, and others,
and each of them have parameters that need to be tuned. These will impact the model
and training algorithms, as well. The feature selection and dimensionality reduction
techniques should be treated as hyperparameters that the model selection process
will optimize. For the sake of brevity, in this section we will only analyze two dif-
ferent feature selection algorithms. We will show two different feature selection and
analyze them in this section.
80 2 Basics of Machine Learning
Fig. 2.19: Exploratory data analysis plots. (a) Histogram plot for each feature and
label on training data. (b) Pearson correlation across features and labels on training
data
2.9 Case Study 81
We first perform PCA with two components and plot these components with
labels to see if the newly reduced training dataset with two components shows im-
proved separation. We then increase the dimensions and plot the cumulative ex-
plained variances by adding the variance captured by each transformed feature. Fig-
ure 2.20 shows the two plots and reveals that the PCA transformation and reduction
may not be useful for this dataset; the transformed features need as many dimensions
as the original features to capture the variances.
1.0
1.0
4
0.8
0.8
0.6
0.6
0
0.4
0.4
–2
0.2
0.2
–4
0.0
–2 –1 0 1 2 3 0 5 10 15 20 25
PCA component 1 number of components
(a) (b)
Fig. 2.20: PCA analysis. (a) PCA of two transformed components as scatter plot on
training data. (b) Cumulative explained variance for first 25 dimensions
We also perform chi-square analysis on the data by first scaling the features to the
range of [0, 1], as chi-square needs all features to be in positive range. The plots of
scores and feature names are shown in Fig. 2.21. Only 16 features have scores above
a threshold 0.1, and that may be a good subset to choose if reduction is pursued.
Fig. 2.21: Chi-square scores on the features plotted in the descending order
82 2 Basics of Machine Learning
We choose accuracy as the metric for performing search of hyperparameters for the
algorithms and for comparing algorithms, because that is the metric that the test
data prediction will be evaluated on. We will use cross-validation as our validation
technique in the hyperparameter search. We will use five linear and non-linear al-
gorithms for training: (a) perceptron, (b) logistic regression, (c) linear discriminant
analysis, (d) naive Bayes, and (e) support vector machines (with RBF kernel).
The code below highlights the hyperparameter search for SVM.
1 from s k l e a r n . svm i m p o r t SVC
2 i m p o r t numpy
3 # gamma p a r a m e t e r i n SVM
4 gammas = numpy . a r r a y ( [ 1 , 0 . 1 , 0 . 0 1 , 0 . 0 0 1 ] )
5 # C p a r a m e t e r f o r SVM
6 c v a l u e s = numpy . a r r a y ( [ 1 0 0 , 1 , 0 . 1 , 0 . 0 1 ] )
7 # g r i d s e a r c h f o r gamma and C
8 s v m p a r a m g r i d = { ’gamma ’ : gammas , ’C ’ : c v a l u e s }
9 # svm w i t h r b f k e r n e l
10 svm = SVC( k e r n e l = ’ r b f ’ )
11 scoring = ’ accuracy ’
12 # grid search
13 g r i d = GridSearchCV ( e s t i m a t o r =svm , p a r a m g r i d = s v m p a r a m g r i d ,
scoring=scoring )
The hyperparameters found and the validation results are given in Table 2.3. It is
interesting to observe that the simplest linear perceptron has the least score and as
the complexity of the model is increased to completely non-linear RBF kernel SVM,
the performance improves.
0.8
Classification accuracy
0.6
0.4
0.2
0.0
10 15 20
Reduced number features
Fig. 2.22: Validation accuracy of logistic regression with different feature selection
techniques
To see the impact of training examples and the variance across different validation
runs, learning curves are plotted. They provide useful comparison of the training
and validation metrics as a function of the training set size. We plot learning curves
for the tuned logistic regression and SVM, as they were high-scoring algorithms
(as shown in Fig. 2.23a and b). It can be observed that the SVM validation score
increases monotonically with the size of the training set, demonstrating that more
examples do improve the performance. It can also be observed that SVM has low
variance across runs compared to logistic regression, which indicates robustness of
the SVM classifier.
Finally, we train the best models (with the best parameters) on the entire training
data and run them on the test data for estimating the out-of-sample error (Table 2.4).
84 2 Basics of Machine Learning
Fig. 2.23: (a) Learning curves for tuned logistic regression. (b) Learning curves for
tuned SVM
References 85
Some other interesting problems readers and practitioners can attempt on their own
include:
References
3.1 Introduction
This chapter introduces the major topics in text and speech analytics and machine
learning approaches. Neural network approaches are deferred to later chapters.
We start with an overview of natural language and computational linguistics.
Representations of text that will form the basis of advanced analysis are introduced,
and the core components in computational linguistics are discussed. Readers are
guided through the broad range of applications that leverage these concepts. We
investigate the topics of text classification and text clustering, and move onto appli-
cations in machine translation, question answering, and automated summarization.
In the latter part of this chapter, acoustic models and audio representations are in-
troduced, including MFCCs and spectrograms.
A natural language is one that has evolved naturally through daily use by humans
over time, without formal construction. They encompass a broad set that includes
spoken and signed languages. By some estimates, there are about 7000 human lan-
guages currently in existence, with the top ten representing 46% of the world’s pop-
ulation [And12] (Fig. 3.1).
statistical methods that can be applied to each component in the synchronic model
form the basis of natural language processing.
Natural language processing seeks to map language to representations that cap-
ture morphological, lexical, syntactic, semantic, or discourse characteristics that can
then be processed by machine learning methods. The choice of representation can
have significant impact on later tasks and will depend on the selected machine learn-
ing algorithm for analysis.
In the following sections of this chapter, we will dive into these representations
to better understand their role in linguistics and purpose in natural language pro-
cessing. We introduce to readers the most common text-based representations, and
leave audio representations to later sections.
All natural languages have systematic structure, even sign language. In linguistics,
morphology is the study of the internal structure of words. Literally translated from
its Greek roots, morphology means “the study of shape.” It refers to the set of rules
and conventions used to form words based on their context, such as plurality, gender,
contraction, and conjugation.
3.2 Morphological Analysis 91
3.2.1 Stemming
Often, the word ending is not as important as the root word itself. This is especially
true of verbs, where the verb root may hold significantly more meaning than the
verb tense. If this is the case, computational linguistics applies the process of word
stemming to convert words to their root form (e.g., base morpheme in meaning).
Here are some stemming examples:
works → work
worked → work
workers → work
While you and I can easily recognize that each of these is related to meaning, it
would be very difficult for a computer to do so without stemming. It is important to
note that stemming can introduce ambiguity, as evident in the third example above
where “workers” has the same stem as “works,” but both words have different mean-
ings (people versus items). On the other hand, the advantage of stemming is that it is
generally robust to spelling errors, as the correct root may still be inferred correctly.
One of the most popular stemming algorithms in NLP is the Porter stem-
mer, devised by Martin Porter in 1980. This simple and efficient method uses
a series of 5 steps to strip word suffixes and find word stems. Open-source
implementations of the Porter stemmer are widely available.
92 3 Text and Speech Basics
3.2.2 Lemmatization
Lexical analysis is the task of segmenting text into its lexical expressions. In natu-
ral language processing, this means converting text into base word representations
which can be used for further processing. In the next few subsections, we provide an
overview of word-level, sentence-level, and document-level representations. As the
reader will see, these representations are inherently sparse, in that few elements are
non-zero. We leave dense representations and word embeddings to a later chapter.
Words are the elementary symbols of natural language. They are not the most
elementary, as words can consist of one or more morphemes. In natural language
processing, often the first task is to segment text into separate words. Note that we
say “often” and not “always.” As we will see later, sentence segmentation as a first
step may provide some benefits, especially in the presence of “noisy” or ill-formed
speech.
3.3.1 Tokens
The computational task of segmenting text into relevant words or units of meaning
is called tokenization. Tokens may be words, numbers, or punctuation marks. In
simplest form, tokenization can be achieved by splitting text using whitespace:
The rain in Spain falls mainly on the plain.
|The|, |rain|, |in|, |Spain|, |falls|, |mainly|, |on|, |the|, |plain|, |.|
3.3 Lexical Representations 93
Tokenization serves also to segment sentences by delineating the end of one sen-
tence and beginning of another. Punctuation plays an important role in this task, but
unfortunately punctuation is often ambiguous. Punctuation like apostrophes, hy-
phens, and periods can create problems. Consider the multiple use of the period in
this sentence:
Dr. Graham poured 0.5ml into the beaker.
|Dr.|, |Graham poured 0.|, |5ml into the beaker.|
A simple punctuation-based sentence splitting algorithm would incorrectly seg-
ment this into three sentences. There are numerous methods to overcome this am-
biguity, including augmenting punctuation-based with hand engineered rules, using
regular expressions, machine learning classification, conditional random field, and
slot-filling approaches.
Tokens do not occur uniformly in English text. Instead, they follow an exponen-
tial occurrence pattern known as Zipf’s law, which states that a small subset of
tokens occur very often (e.g., the, of, as) while most occur rarely. How rarely? Of
the 884,000 tokens in Shakespeare’s complete works, 100 tokens comprise over half
of them [Gui+06].
In the written English language, common functional words like “the,” “a,” or
“is” provide little to no context, yet are often the most frequently occurring words
in text as seen in Fig. 3.3. By excluding these words in natural language processing,
performance can be significantly improved. The list of these commonly excluded
words is known as a stop word list.
3.3.3 N-Grams
While word level representations are sometimes useful, they do not capture the re-
lationship with adjacent words that can help provide grammar or context. For in-
stance, when working with individual tokens, there is no concept of word order.
94 3 Text and Speech Basics
Fig. 3.3: Zipf’s law as it applies to the text from the complete works of Shakespeare
One simply considers the existence and occurrence of tokens within a piece of text.
This is known as a bag-of-words model (see Eq. (3.1)), and is based on a Markov
assumption. A phrase containing L tokens would be predicted with probability:
L
P (w1 w2 . . . wL ) = ∏ P(wi ) (3.1)
i=1
Notice that bigrams effectively capture the local word order of two consecutive
tokens (e.g., “lion king” is not the same as “king lion”). We can extend this concept
to capture lengths of n tokens, known as n-grams:
L
P (w1 w2 . . . wL ) = ∏ P (wi |wi−1 wi−2 . . . wi−n ) (3.3)
i=n
It is important to note that for higher values of n, n-grams become extremely infre-
quent. This will adversely impact count-based natural language processing methods
as we see later.
3.3.4 Documents
When working with a corpus of documents, there are a number of document repre-
sentation methods used in computational linguistics. Some are token based, multi-
token based, or character based. Many are sparse representations, while others are
dense. We discuss two of the most common ones in this section.
3.3 Lexical Representations 95
3.3.4.2 Bag-of-Words
One common approach is to set each element of the document-term matrix equal
to the frequency of word occurrence within each document. Imagine representing
each document as a list of counts of unique words. This is known as a bag-of-words
approach [PT13]. Obviously, there is significant information loss by simply using
a document vector to represent an entire document, but this is sufficient for many
computational linguistics applications. This process of converting a set of docu-
ments into a document-term matrix where each element is equal to word occurrence
is commonly known as count vectorization.
You may remember that the most frequent words in the English vocabulary are
generally less significant than rarer words for discerning meaning. Unfortunately,
a bag-of-words model weighs words based on occurrence. In practice, a stop word
filter is used to remove the most common words prior to count vectorization. Even
in doing so however, we will still find that rare words occurring across a set of
documents are often the most meaningful.
96 3 Text and Speech Basics
3.3.4.3 TFIDF
TFIDF is a method that provides a way to give rarer words greater weight by setting
each document-term matrix element equal to the value w of multiplying the term
frequency (TF) by the inverse document frequency (IDF) of each token [Ram99]:
w = t f × id f (3.4)
N
= (1 + log (T F t )) × log (3.5)
ni
Here, we have used the definitions of term frequency as the logarithmically scaled
ratio of the count of term t occurring in document d versus the total number of terms
in document d, and inverse document frequency as the logarithmically scaled ratio
of the total number of documents vs the count of documents with term t.
Because of the tf factor, the TFIDF value for a token increases proportionally
to the number of times it appears in a document. The idf factor reduces the TFIDF
value for a token based on the frequency of occurrence across all documents. Cur-
rently, TFIDF is the most popular weighting method, with over 80% of current dig-
ital libraries using it in production (Table 3.3).
3.4.1 Part-of-Speech
There are numerous POS subclasses in English, such as singular nouns (NN),
plural nouns (NNS), proper nouns (NP), or adverbial nouns (NR). Some languages
can have over 1000 parts of speech [PDM11]. Due to the ambiguity of the English
language, many English words belong to more than one part-of-speech category
(e.g., “bank” can be a verb, noun, or interjection), and their role depends on how
they are used within a sentence. It can be difficult to identify which category the
word belongs. Part-of-speech tagging is the process of predicting the part-of-speech
category for each word in the text based on its grammatical role and context within
a sentence (Fig. 3.4) [DeR88]. POS tagging algorithms fall into two distinct groups:
rules based and statistical methods.
The Brown corpus was the first major collection of English texts used in
computational linguistics research. It was developed by Henry Kuč era and
W. Nelson Francis at Brown University in the mid-1960s and consists of
over a million words of English prose extracted from 500 randomly chosen
98 3 Text and Speech Basics
publications of 2000 or more words. Each word in the corpus has been POS-
tagged meticulously using 87 distinct POS tags. The Brown corpus is still
commonly used as a gold set to measure the performance of POS-tagging
algorithms.
The earliest part-of-speech tagging approaches were rules based and depended on
dictionaries, lexicons, or regular expressions to predict possible POS labels for each
word. Where ambiguities arose, ad-hoc rules were often incorporated to make POS
tag decisions. This made rules-based systems brittle. For instance, a rule could de-
clare that a word that follows an adverb and comes before a conjunction should
be a noun, except it should be verb if it is not a singular common noun. The best
rules-based POS tagger to date achieved only 77% accuracy on the Brown corpus
[BM04].
Since the 1980s, hidden Markov models (HMMs) introduced in the previous chap-
ter became popular as a better approach to POS tagging. HMMs are better able to
learn and capture the sequential nature of grammar than rules-based methods. To
understand this, consider the POS tagging problem, where we seek to find the most
probable tag sequence tˆn for a given sequence of n words wn :
The equation above represents an HMM model where the Markov states are the
words wn and the hidden states t n are the POS tags. The transition matrices can be
directly computed from text data. It turns out that assigning the most common tag
to each known word can achieve fairly high accuracy. To account for more ambigu-
ous word sequences, higher-order HMMs can also be used for larger sequences by
leveraging the Viterbi algorithm. These higher-order HMMs can achieve very high
accuracy, but they require significant computation load since they must explore a
larger set of paths.
Beyond HMMs, machine learning methods have gained huge popularity for POS
tagging tasks, including CRF, SVM, perceptrons, and maximum entropy classifica-
tion approaches. Most now achieve accuracy above 97%. In the subsequent chapters,
we will examine deep learning approaches that hold even greater promise to POS
tag prediction.
3.4 Syntactic Representations 99
In a natural language, grammar is the set of structural rules by which words and
phrases are composed. Every sentence in English follows a certain pattern. These
patterns are called grammars and express a relation between a (head) word and its
dependents. Most natural languages have a rich set of grammar rules, and knowl-
edge of these rules helps us disambiguate context in a sentence. Consider the fact
that without grammar, there would be practically unlimited possibilities to combine
words together.
Parsing is the natural language processing task of identifying the syntactic rela-
tionship of words within a sentence, given the grammar rules of a language [Cov01].
There are two common ways to describe sentence structure in natural language. The
first is to represent the sentence by its constituent phrases, recursively down to the
individual word level. This is known as constituent grammar parsing, which maps
a sentence to a constituent parse tree (Fig. 3.5). The other way is to link individual
one node for every word in the sentence. Notice that the links are directional be-
tween two words in a dependency parse tree, pointing from the head word to the
dependent word to convey the relationship. Constituent and dependency parse trees
can be strongly equivalent. The appeal of dependency tree is that the links closely
resemble semantic relationships.
Because a dependency tree contains one node per word, the parsing can be
achieved with computational efficiency. Given a sentence, parsing algorithms at-
tempt to find the most likely derivation from its grammatical rules. If the sentence
exhibits structural ambiguity, more than one derivation is possible. Parsers are sub-
divided into two general approaches. Top-down parsers use a recursive algorithm
with a back-tracking mechanism to descend from the root down to all words in the
sentence. Bottom-up parsers start with the words and build up the parse tree based
on a shift/reduce or other algorithm. Top-down parsers will derive trees that will
always be grammatically consistent, but may not align with all words in a sentence.
Bottom-up approaches will align all words, but may not be always make grammati-
cal sense.
Grammar, as stated above, is the set of rules that define the syntactic structure and
pattern of words in a sentence. Because these rules are generally fixed and absolute,
a context-free grammar (CFG) can be used to represent the grammatical rules of a
language [JM09]. Context-free grammars typically have a representation known as
Backus–Naur form and are able to capture both constituency and ordering of words
in a sentence.
Unfortunately, because of the inherent ambiguity of language, CFG may generate
multiple possible parse derivations for a given sentence. Probabilistic context-free
grammars (PCFG) deal with this issue by ranking possible parse derivations and
selecting the most probable, given a set of weights learned from a distribution of
text. PCFGs generally outperform CFGs, especially for languages like English.
3.4.2.2 Chunking
For some applications, a full syntactic parse with its computational expense may
not be needed. Chunking, also called shallow parsing, is a natural language pro-
cessing task which joins words into base syntactic units rather than generating a
full parse tree. These base syntactic units are often referred to as “chunks.” For ex-
ample, given a sentence, we would like to identify just the base noun-phrases (i.e.,
phrases that serve the same grammatical function as a noun and do not contain other
noun-phrases):
NP
The winter season is depressing for NP many people .
3.5 Semantic Representations 101
3.4.2.3 Treebanks
A treebank is a text corpus that has been parsed and annotated for syntactic structure.
That is, each sentence in the corpus has been parsed into its dependency parse tree.
Treebanks are typically generated iteratively using a parser algorithm and human
review [Mar+94] [Niv+16]. Often, treebanks are built on top of a corpus that has
already been annotated with part-of-speech tags. The creation of treebanks revolu-
tionized computational linguistics, as it embodied a data-driven approach to gener-
ating grammars that could be reused broadly in multiple applications and domains.
Statistical parsers trained with treebanks are able to deal much better with structural
ambiguities [Bel+17].
The Penn Treebank is the de facto standard treebank for parse analysis and
evaluation. Initially released in 1992, it consists of a collection of articles
from Dow Jones News Service written in English, of which 1 million words
are POS-tagged and 1.6 million words parsed with a tagset. An improved
version of the Penn Treebank was released in 1995.
Whereas lexical and syntactic analyses capture the form and order of language, they
do not associate meaning with words or phrases. For instance, labeling “dog” as
a noun gives us no clue what a “dog” is. Semantic representations give a sense of
meaning to words and phrases. They attach roles to word chunks such as person,
place, or amount. Semantic analysis is interested in understanding meaning primar-
ily in terms of word and sentence relationships. There are several different kinds of
semantic relations between words (see Table 3.5).
102 3 Text and Speech Basics
Homonymy and polysemy are very similar, and the key difference is that a pol-
ysemous word is one word with different meanings, while homonyms are different
words that share a shape (usually both spelling and pronunciation). For example,
most people would consider the noun tire (the wheels on your car) and the verb tire
(what happens when you exercise) to be completely different words, even though
they look and sound the same. They’re homonyms. On the other hand, most people
agree that there is only one word offense, but that it has various meanings which are
all related: the attacking team, a criminal act, a feeling of being insulted, etc.
Named entity recognition (NER) is a task in natural language processing that seeks
to identify and label words or phrases in text that refer to a person, location, or-
ganization, date, time, or quantity. It is a subtask of information extraction and is
sometimes called entity extraction. Due to the reuse of words and ambiguity of nat-
ural language, entity recognition is hard. Take, for instance, the word “Washington”
which may be a reference to a city, a state, or a president. It would be difficult to dis-
ambiguate this word without the context of real-world knowledge. Ambiguities can
exist in two ways: different entities of the same type (George Washington and Wash-
ington Carver are both persons) or entities of different types (George Washington or
Washington state) (Table 3.6).
While regular expressions can be used to some extent for name entity recog-
nition [HN14], the standard approach is to treat it as a sequence labeling task or
HMM in similar fashion to POS-tagging or chunking [AL13]. Conditional ran-
dom fields (CRFs) have shown some success in named entity recognition. How-
ever, training a CRF model typically requires a large corpus of annotated training
data [TKSDM03c]. Even with a lot of data, name entity recognition is still largely
unsolved.
to Laura) logically follows from the first (Laura is married to George Bush), even
though it may not be explicitly stated in the text. The common approach to relation
extraction is to divide it into subtasks:
1. Identify any relations between entities
2. Classify the identified relations by type
3. Derive logical/reciprocal relations.
The first subtask is typically treated as a classification problem, where a binary
decision is made as to whether a relation is present between any two entities within
the text. The second subtask is a multiclass prediction problem. Naive Bayes and
SVM models have been successfully applied to both subtasks [BB07, Hon05]. The
last subtask is a logical inference task. Relation extraction plays an important role
in question answering tasks.
104 3 Text and Speech Basics
Events are mentions within text that have a specific location and instance or interval
in time associated with them. The task of event detection is to detect the mentions
of events in text and to identify the class to which they belong. Some examples of
events are: the Superbowl, The Cherry Blossom festival, and our 25th wedding an-
niversary celebration. Both rules-based and machine learning approaches for event
detection are similar to those for relationship extraction [Rit+12, MSM11]. Such
approaches have had mixed success due to the need for external context and the
importance of temporal relations.
Semantic role labeling (SRL), also known as thematic role labeling or shallow se-
mantic parsing, is the process of assigning labels to words and phrases that indicate
their semantic role in the sentence. A semantic role is an abstract linguistic construct
that refers to the role that a subject or object takes on with respect to a verb. These
roles include: agent, experiencer, theme, patient, instrument, recipient, source, ben-
eficiary, manner, goal, or result.
Semantic role labeling can provide valuable context [GJ02], whereas syntactic
parsing can only provide grammatical structure. The most common approach to
SRL is to parse a set of target sentences to identify predicates [PWM08]. For each of
these predicates, a machine learning classifier trained on a dataset such as PropBank
or FrameNet is used to predict a semantic role label. These labels serve as highly
useful features for further tasks such as text summarization or question answering
[JN08, BFL98b].
Discourse analysis is the study of the structure, relations, and meaning in units of
text that are longer than a single sentence. More specifically, it investigates the flow
of information and meaning by a collection of sentences taken as a whole. Discourse
presumes a sender, receiver, and message. It encompasses characteristics such as the
document/dialogue structure, topics of discussion, cohesion, and coherence of the
text. Two popular tasks in discourse analysis are coreference resolution and dis-
course segmentation.
3.6.1 Cohesion
3.6.2 Coherence
Coherence refers to the existence of semantic meaning to tie phrases and sentences
together within text. It can be defined as continuity in meaning and context, and
usually requires inference and real-world knowledge. Coherence is often based on
conceptual relationships implicitly shared by both the sender and receiver that are
used to construct a mental representation of the discourse [WG05]. An example of
coherence can be seen in the following example which presumes knowledge that a
bucket holds water:
3.6.3 Anaphora/Cataphora
Anaphora refers to the relation between two words or phrases where the interpre-
tation of one, called an anaphor, is determined by the interpretation of a word that
came before, called an antecedent. Cataphora is where the interpretation of a word
106 3 Text and Speech Basics
is determined by another word that came after in the text. Both are important char-
acteristics of cohesion in discourse.
The linguistics process by which anaphors are linked with their antecedents is
known as coreference resolution. It is a well-studied problem in discourse. When
this occurs within a document, it is commonly termed local coreference. If this
occurs across documents, it is termed global coreference. Essential when disam-
biguating pronouns and connecting them with the right individual mentions within
text, coreference also plays an important role in entity resolution [Lee+13, Sin+13].
Coreference resolution can be considered a classification task, and algorithms
for resolving coreference range in accuracy from 70% for named entities to 90% for
pronouns [PP09].
How is this useful? A language model is one that tries to predict how frequent a
phrase occurs within the natural use of a language. Having a way to estimate the
relative likelihood of different phrases is useful in many natural language processing
applications, especially ones that generate text as an output. For instance, language
models can be used for spell correction by predicting a word wL given all of the
previous words before it:
P wL wL−1 wL−2 . . . w1 (3.9)
We can extend this to the general case of n-grams. We assume that the probability
of observing the ith word wi in the context history of the preceding words can be
approximated by the probability of observing it in the shortened context history of
the preceding words (nth order Markov property).
A unigram model used in information retrieval can be treated as the combination
of several one state finite automata. It splits the probabilities of different terms in a
context, e.g., from:
L
P (w1 w2 . . . wL ) = ∏ P(wi ) (3.10)
i=1
The words bigrams and trigrams denote n-gram model language models with n = 2
and n = 3, respectively. The conditional probability can be calculated from n-gram
model frequency counts:
L
P (w1 w2 . . . wL ) = ∏ P (wi |wi−1 wi−2 . . . wi−n ) (3.11)
i=n
The sparsity of n-grams can become a problem, especially if the set of documents
used to create the n-grams language model is small. In those cases, it is not uncom-
mon for certain n-grams to have zero counts in the data. The language model would
assign zero probability to these n-grams. This creates a problem when these n-grams
occur in test data. Because of the Markov assumption, the probability of a sequence
is equal to the product of the individual probabilities of the n-grams. A single zero
probability n-gram would set the probability of the sequence to be zero.
To overcome this problem, it is common to use a technique called smoothing.
The simplest smoothing algorithm initializes the count of every possible n-gram at
1. This is known as Laplace or add-one smoothing, and guarantees that there will al-
ways be a small probability that any n-gram occurs. Unfortunately, as n-gram spar-
sity grows, this approach becomes less useful as it dramatically shifts occurrence
probabilities.
If a word was never seen in the training data, then the probability of that sentence
is zero. Clearly this is undesirable, so we apply Laplacian smoothing to help deal
with that. We add 1 to every count so it’s never zero. To balance this, we add the
number of possible words to the divisor, so the division will never be greater than 1.
Laplace smoothing is a simple, inelegant approach that provides modest improve-
ments to results for like text classification. In general, we can use a pseudocount
parameter α > 0:
xi + α
ϑi = (3.12)
N + αd
108 3 Text and Speech Basics
A more effective and wisely used method is Kneser–Ney smoothing, due to its use
of absolute discounting by subtracting a fixed value from the probability’s lower
order terms to omit n-grams with lower frequencies:
max (c(wi−1 wi ) − δ , 0)
Pabs (wi |wi−1 ) = + α pabs (wi ) (3.13)
∑w c(wi−1 w)
3.7.3 Out-of-Vocabulary
Another serious problem for language models arise when the word is not in the vo-
cabulary of the model itself. Out-of-vocabulary (OOV) words create serious prob-
lems for language models. In such a scenario, the n-grams that contain an out-of-
vocabulary word are ignored. The n-gram probabilities are smoothed over all the
words in the vocabulary even if they were not observed.
To explicitly model the probability of out-of-vocabulary words, we can intro-
duce a special token (e.g., <unk>) into the vocabulary. Out-of-vocabulary words in
the corpus are effectively replaced with this special <unk> token before n-grams
counts are accumulated. With this option, it is possible to estimate the transition
probabilities of n-grams involving out-of-vocabulary words. By doing so, however,
we treat all OOV words as a single entity, ignoring the linguistic information.
Another approach is to use approximate n-gram matching. OOV n-grams are
mapped to the closest n-gram that exists in the vocabulary, where proximity is based
on some semantic measure of closeness (we will describe word embeddings in more
detail in a later chapter).
A simpler way to deal with OOV n-grams is the practice of backoff, based on the
concept of counting smaller n-grams with OOV terms. If no trigram is found, we
instead count bigrams. If no bigram found, use unigrams.
3.7.4 Perplexity
Lower measures of perplexity imply that the model predicts the test data better,
while higher perplexity values imply lower prediction quality. Note that it is impor-
tant for the test sequence to be comprised of the same n-grams as was used to train
the language model, or else the perplexity will be very high.
With this data, we can train a classification model to predict labels of un-annotated
text samples. Popular machine learning algorithms for text classification include K-
nearest neighbor, decision trees, naive Bayes, support vector machines, and logistic
regression. The general text classification pipeline can be summarized as:
110 3 Text and Speech Basics
Models of emotion have been researched for several decades. An emotional state
model is one that captures the human states of emotion. The Mehrabian and Rus-
sell model, for instance, decomposes human emotional states into three dimensions
(Table 3.8). There are other emotional state models used in sentiment analysis, in-
within a document, and the sentiment of the document is equal to the aggregated
scores of the words. This lexical approach is very fast, but suffers from the inability
to effectively model subtlety, sarcasm, or metaphor [RR15]. Negation (e.g., “not
nice” vs. “nice”) is also problematic with pure lexical approaches.
The affective norms for English words (ANEW) dataset is a lexicon created
by Bradley and Lang containing 1000 words scored for emotional ratings of
valence, dominance, and arousal (Fig. 3.8). ANEW is very useful for longer
texts and newswire documents. Another model is the SentiStrength model
for short informal text developed by Thelwall et al., which has been applied
successfully to analyze text and Twitter messages.
Objectivity detection could help identify personal bias, track hidden viewpoints, and
alleviate the “fake news” problem existing today [WR05].
One approach to objectivity detection is to use n-grams or shallow parsing and
pattern matching with a set of learned lexical–syntactic patterns. Another is to use
lexical–syntactic features in combination with conversational discourse features to
train a classifier for subjectivity.
3.8.3 Entailment
Textual entailment is the logical concept that truth in one text fragment leads to
truth in another text fragment. It is a directional relation, analogous to the “if-then”
clause. Mathematically, given text fragments X and Y, entailment is given by:
where P(Y |X) is considered the entailment confidence. Note that the relation X
entails Y does not give any certainty that Y entails X (logical fallacy).
Entailment is considered a text classification problem. It has widespread use in
many NLP applications (e.g., question answering). Initial approaches toward entail-
ment were logical-form based methods that required a many axioms, inference rules,
and a large knowledge base. These theorem-proving methods performed poorly in
comparison to other statistical NLP approaches [HMM16].
3.9 Text Clustering 113
Currently, the most popular entailment approaches are syntax based [AM10].
Parse trees are used to generate and compare similarity scores that are combined
with an SVM or LR classifier to detect entailment. Such approaches are quite ca-
pable of capturing shallow entailment but do poorly on more complex text (such as
text that switches between active and passive voice).
Recent semantic approaches have shown better ability to generalize by incorpo-
rating semantic role labeling in addition to lexical and syntactic features [Bur+07].
Even so, the gap between human level entailment and computational approaches is
still significant. Entailment remains an open research topic.
While text classification is the usual go-to approach for text analytics, we are often
presented with a large corpus of unlabeled data in which we seek to find texts that
share common language and/or meaning. This is the task of text clustering [Ber03].
The most common approach to text clustering is via the k-means algorithm
[AZ12]. Text documents are tokenized, sometimes stemmed or lemmatized, stop
words are removed, and text is vectorized using bag-of-words or TFIDF. K-means
is applied to the resulting document-term matrix for different k values.
There are two main considerations when using k-means. The first is the notion
of distance between two text fragments. For k-means, this is the Euclidean distance,
but other measures like cosine distance could theoretically be used. The second is
determining the value of k—how many different clusters of text exist within a cor-
pus. As in standard k-means, the elbow method is most widely used for determining
the value of k.
114 3 Text and Speech Basics
Often, we have a collection of documents and want to broadly know what is dis-
cussed within the collection. Topic modeling provides us the ability to organize,
understand, and summarize large collections of text. A topic model is a statistical
model used to discover abstract “topics” within in a collection of documents. It is a
form of text mining, seeking to identify recurring patterns of words in discourse.
3.9.2.1 LSA
3.9.2.2 LDA
Machine translation (abbreviated MT) refers to the process of translating text from
a source language to a different target language. Language translation is hard even
for humans to be able to fully capture meaning, tone, and style. Languages can
have significantly different morphology, syntax, or semantic structure. For instance,
it will be rare to find English words with more than 4 morphemes, but it is quite
common in Turkish or Arabic. German sentences commonly follow the subject-
verb-object syntactic structure, while Japanese mostly follows a subject-object-verb
order, and Arabic prefers a verb-subject-object order. With machine translation, we
typically focus on two measures:
• Faithfulness = preserving the meaning of text in translation
• Fluency = natural sounding text or speech to a native speaker.
Statistical machine translation adopts a probabilistic approach to map from one lan-
guage to another. Specifically, it builds two types of models by treating the problem
as one similar to a Bayesian noisy channel problem in communications:
• Language model (fluency) = P(X)
• Translation model (faithfulness) = P(Y |X).
The language model measures the probability that any sequence of words X is an ac-
tual sentence—that is, there is consistency within a language. The translation model
measures the conditional probability that a sequence of words Y in the target lan-
guage is a true translation of a sequence of words X in the source language. A statis-
tical machine translation model will find the best translation to the target language
Y by optimizing for:
Statistical models are based on the notion of word alignment, which is a mapping
of a sequence of words from the source language to those of a target language.
Because of differences between languages, this mapping will almost never be one-
to-one. Furthermore, the order of words may be quite different.
Question answering (QA) is the NLP task of answering questions in natural lan-
guage. It can leverage expert system, knowledge representation, and information
retrieval methods. Traditionally, question answering is a multi-step process where
relevant documents are retrieved, useful information is extracted from these docu-
ments, possible answers are proposed and scored against evidence, and a short text
answer in natural language is generated as a response.
Question: Who won the Tournament of Champions on Jeopardy in 2011?
Answer: IBM Watson debuted a system named DeepQA in 2011 that went on to win first
place against the legendary champions on Jeopardy.
3.11 Question Answering 117
Web-based question answering systems like Google Search are based on informa-
tion retrieval (IR) methods that leverage the web. These text-based systems seek to
answer questions by finding short texts from the internet or some other large collec-
tion of documents. Typically, they map queries into a bag-of-words and use methods
like LSA to retrieve a set of relevant documents and extract passages within them.
Depending on the question type, answer strings can be generated with a pattern-
extraction approach or n-gram tiling methods. IR-based QA systems are entirely
statistical in nature and are unable to truly capture meaning beyond distributional
similarity.
118 3 Text and Speech Basics
3.11.2 Knowledge-Based QA
DBpedia is a free semantic relation database with 4.6 million entities ex-
tracted from Wikipedia pages in multiple languages. It contains over 3 billion
relational triples expressed in the resource description framework (RDF) for-
mat. DBpedia is often considered the foundation for the semantic web, also
known as the linked open data cloud. First released in 2007, DBpedia contin-
ues to evolve through crowdsourced updates in similar fashion to Wikipedia.
1 N 1
MRR = ∑ ranki
N i=1
(3.20)
Automatic summarization is a useful NLP task that identifies the most relevant in-
formation in a document or group of documents and creates a summary of the con-
tent. It can be an extraction task that takes the most relevant phrases or sentences
in original form and uses them to generate the summary, or an abstraction task that
generates natural language summaries from the semantic content [AHG99, BN00].
Both approaches mirror how humans tend to summarize text, though the former
extracts text while the latter paraphrases text.
based on lexical measures (e.g., TFIDF). Another is to use discourse measures (e.g.,
coherence) to identify key sentences. Centroid-based methods evaluate word prob-
ability relative to the background corpus to determine importance. A creative ap-
proach called TextRank takes a graph-based approach to assign sentence scores
based on lexical similarity of words. As long as plagiarism is not a concern,
extraction-based summarization is the more popular approach.
Here, P(W ) represents the probability that a string of words is an English sentence—
that is, P(W ) is the language model. The quantity P(S|F) is known as the pronunci-
ation model and the quantity P(F|W ) is the acoustic model.
3.13.1.1 Spectrograms
3.13.1.2 MFCC
each representing a small window in time. Consider the cepstrum, which is the in-
verse fast-Fourier transform of the log of the fast-Fourier transform of an audio
signal (Fig. 3.11):
2
C = F −1 log F f (t) (3.23)
MFCCs is similar to the cepstrum and is given by taking the discrete cosine trans-
form of the log of the fast-Fourier transform of an audio signal where a triangular
filter of Mel frequency banks has been applied. MEL filters are placed linearly for
frequencies less than 1000Hz and on a log scale for frequencies above 1000Hz,
closely corresponding to the response of the human ear:
C = DCT log MEL F f (t) (3.24)
MFCCs contain both time and frequency information about the audio signal. They
are particularly useful for ASR because cepstral features are effectively orthogonal
to each other and robust to noise.
study will reinforce the fundamental principles of text analytics as well as identify
key gaps in classical NLP.
For this case study, we will use Python and the following libraries:
• Pandas (https://pandas.pydata.org/) is a popular open source implementation for
data structures and data analysis. We will use it for data exploration and some
basic processing.
• scikit-learn (http://scikit-learn.org/) is a popular open source for various ma-
chine learning algorithms and evaluations. We will use it only for sampling, cre-
ating datasets, and machine learning implementations of linear and non-linear
algorithms in our case study.
• NLTK (https://www.nltk.org/) is a suite of text and natural language processing
tools. We will use it to convert text into vectors for processing.
• Matplotlib (https://matplotlib.org/) is a popular open source library for visual-
ization. We will use it to visualize performance.
3.14.2 EDA
Our first task is to take a close look at the dataset by loading and performing ex-
ploratory data analysis. To do so, we must extracting metadata and the text body
from each document in the corpus. If we take a close look at the corpus, we find
(Figs. 3.12, 3.13 and 3.14):
1. There are 11,367 documents that have one or more topic annotations.
2. The greatest number of topics in a single document is 16.
3. There are a total of 120 distinct topic labels in the corpus.
4. There are 147 distinct place and 32 organization labels.
So far so good. But before we perform any NLP analysis, we will want to perform
some cursory text normalization:
124 3 Text and Speech Basics
300
250
200
150
100
50
0
[ec]
[opec]
[mf]
[co-coffee]
[gatt]
[worldbank]
[oecd]
[cco]
[imf, worldbank]
[adb-asia]
[ec, gatt]
[tc]
[gatt, ec]
[eib]
Fig. 3.12: Document count by organization
(uk,)
(canada,)
(japan,)
(west-germany,)
(france,)
(australia,)
(usa, japan)
(switzerland,)
(brazil,)
(netherlands,)
(japan, usa,)
(belgium,)
(china,)
0 200 400 600 800
(eam,)
(acq,)
(crude,)
(trade,)
(money-fx,)
(interest,)
(money-supply,)
(ship,)
(grain, wheat)
(sugar,)
(money-fx, interest)
(coffee,)
(gold,)
(grain, corn)
0 500 1000 1500 2000 2500 3000 3500 4000
7 from s k l e a r n . f e a t u r e e x t r a c t i o n . t e x t i m p o r t T f i d f V e c t o r i z e r
8
9 n l t k . download ( ” p u n k t ” )
10 n l t k . download ( ” s t o p w o r d s ” , ” d a t a ” )
11 n l t k . d a t a . p a t h . append ( ’ d a t a ’ )
12
13 labelBinarizer = MultiLabelBinarizer ()
14 data target = labelBinarizer . fit transform ( data set [u ’ topics ’
])
15
16 s t o p W o r d s = s t o p w o r d s . words ( ’ e n g l i s h ’ )
17 c h a r f i l t e r = r e . c o m p i l e ( ’ [ a−zA−Z]+ ’ ) ;
18
19 def SimpleTokenizer ( t e x t ) :
20 words = map ( lambda word : word . l o w e r ( ) , w o r d t o k e n i z e ( t e x t ) )
21 words = [ word f o r word i n words i f word n o t i n s t o p W o r d s ]
22 t o k e n s = ( l i s t ( map ( lambda t o k e n : P o r t e r S t e m m e r ( ) . s t e m ( t o k e n ) ,
words ) ) )
23 n t o k e n s = l i s t ( f i l t e r ( lambda t o k e n : c h a r f i l t e r . match ( t o k e n ) ,
tokens ) )
24 return ntokens
25
26 vec = T f i d f V e c t o r i z e r ( t o k e n i z e r = SimpleTokenizer ,
27 m a x f e a t u r e s =1000 ,
28 norm= ’ l 2 ’ )
29
30
31 m y t o p i c s = [ u ’ c o c o a ’ , u ’ t r a d e ’ , u ’ money−s u p p l y ’ , u ’ c o f f e e ’ , u ’ g o l d
’]
32 d a t a s e t = d a t a s e t [ d a t a s e t [ u ’ t o p i c s ’ ] . map ( s e t ( m y t o p i c s ) .
intersection )
33 . a p p l y ( lambda x : l e n ( x ) >0 ) ]
34 d o c s = l i s t ( d a t a s e t [ u ’ body ’ ] . v a l u e s )
35
36 dtm = v e c . f i t t r a n s f o r m ( d o c s )
126 3 Text and Speech Basics
We want to see if clusters exist in the documents, so let’s create some document
representations through TFIDF. This gives us a document-term matrix, but typically
the dimensions of this matrix are too large and the representations are sparse. Let’s
first apply principal component analysis (PCA) to reduce the dimensionality. The
original TFIDF vectors have dimension = 1000. Let’s take a look at the effect of
dimensionality reduction by plotting the proportion of explained variance of the
data as a function of the number of principal components (Fig. 3.15):
1 from s k l e a r n . d e c o m p o s i t i o n i m p o r t PCA
2
3 explained var = []
4 f o r components i n range ( 1 , 1 0 0 , 5 ) :
5 p c a = PCA( n c o m p o n e n t s = c o m p o n e n t s )
6 p c a . f i t ( dtm . t o a r r a y ( ) )
7 e x p l a i n e d v a r . a p p e n d ( p c a . e x p l a i n e d v a r i a n c e r a t i o . sum ( ) )
8
0.6
Proportion Explained Variance
0.5
0.4
0.3
0.2
0.1
0 20 40 60 80 100
Number of Components
The graph above shows that half of the variance can be explained by 60 components.
Let’s apply this to the dataset, and visualize the results by plotting the first two PCA
components of each document (Fig. 3.16).
1 from s k l e a r n . d e c o m p o s i t i o n i m p o r t PCA
2 import seaborn as sns
3
4 c o m p o n e n t s = 60
5
3.14 Case Study 127
6 p a l e t t e = np . a r r a y ( s n s . c o l o r p a l e t t e ( ” h l s ” , 1 2 0 ) )
7
8 p c a = PCA( n c o m p o n e n t s = c o m p o n e n t s )
9 p c a . f i t ( dtm . t o a r r a y ( ) )
10 p c a d t m = p c a . t r a n s f o r m ( dtm . t o a r r a y ( ) )
11
15 e x p l a i n e d v a r i a n c e = p c a . e x p l a i n e d v a r i a n c e r a t i o . sum ( )
16 p r i n t ( ” E x p l a i n e d v a r i a n c e o f t h e PCA s t e p : {}%” . f o r m a t (
17 i n t ( e x p l a i n e d v a r i a n c e ∗ 100) ) )
0.6
0.4
0.2
0.0
–0.2
–0.4
We know that there are 5 distinct topics (though some documents might have over-
lap), so let’s run the k-means algorithm with k = 5 to examine document grouping
(Fig. 3.17).
1 from s k l e a r n . c l u s t e r i m p o r t KMeans
2 p a l e t t e = np . a r r a y ( s n s . c o l o r p a l e t t e ( ” h l s ” , 5 ) )
3
4 model = KMeans ( n c l u s t e r s =5 , m a x i t e r = 1 0 0 )
5 c l u s t e r e d = model . f i t ( p c a d t m )
6 c e n t r o i d s = model . c l u s t e r c e n t e r s
7 y = model . p r e d i c t ( p c a d t m )
8
9 ax = p l t . s u b p l o t ( )
10 s c = ax . s c a t t e r ( p c a d t m [ : , 0 ] , p c a d t m [ : , 1 ] ,
11 c= p a l e t t e [ y . a s t y p e ( np . i n t ) ] )
How does this compare with the manually annotated labels? (Fig. 3.18)
1 p a l e t t e = np . a r r a y ( s n s . c o l o r p a l e t t e ( ” h l s ” , 5 ) )
2
128 3 Text and Speech Basics
0.6
0.4
0.2
0.0
–0.2
–0.4
–0.2 0.0 0.2 0.4 0.6
0.6
0.4
0.2
0.0
–0.2
–0.4
–0.2 0.0 0.2 0.4 0.6
3 g o l d l a b e l s = d a t a s e t [ ’ t o p i c s ’ ] . map ( s e t ( m y t o p i c s ) .
intersection )
4 . ( lambda x : x . pop ( ) ) . a p p l y ( lambda x : m y t o p i c s . i n d e x ( x ) )
5
6 ax = p l t . s u b p l o t ( )
7 s c = ax . s c a t t e r ( p c a d t m [ : , 0 ] , p c a d t m [ : , 1 ] , c= p a l e t t e [
gold labels ])
3.14 Case Study 129
In addition to the lexical clustering of documents, let’s see if we can discern any
natural topic structure within the corpus. We apply the LSA and LDA algorithms,
which will associate words to a set of topics, and topics to our set of documents.
3.14.4.1 LSA
We start with the LSA algorithm and set the number of dimensions to 60 (Fig. 3.19):
1 from s k l e a r n . d e c o m p o s i t i o n i m p o r t TruncatedSVD
2 import seaborn as sns
3
4 c o m p o n e n t s = 60
5
6 p a l e t t e = np . a r r a y ( s n s . c o l o r p a l e t t e ( ” h l s ” , 1 2 0 ) )
7
8 l s a = TruncatedSVD ( n c o m p o n e n t s = c o m p o n e n t s )
9 l s a . f i t ( dtm )
10 l s a d t m = l s a . t r a n s f o r m ( dtm )
11
1 from s k l e a r n . c l u s t e r i m p o r t KMeans
2 p a l e t t e = np . a r r a y ( s n s . c o l o r p a l e t t e ( ” h l s ” , 8 ) )
3
4 model = KMeans ( n c l u s t e r s =5 , m a x i t e r = 1 0 0 )
5 c l u s t e r e d = model . f i t ( l s a d t m )
6 c e n t r o i d s = model . c l u s t e r c e n t e r s
7 y = model . p r e d i c t ( l s a d t m )
8
9 ax = p l t . s u b p l o t ( )
10 s c = ax . s c a t t e r ( l s a d t m [ : , 0 ] , l s a d t m [ : , 1 ] , c= p a l e t t e [ y . a s t y p e (
np . i n t ) ] )
130 3 Text and Speech Basics
0.6
0.4
0.2
0.0
–0.2
0.6
0.4
0.2
0.0
–0.2
3.14.4.2 LDA
Let’s see if the LDA algorithm can do better as a Bayesian approach to document
clustering and topic modeling. We set the number of topics to the known number of
topics = 5.
1 i m p o r t numpy a s np
2 import seaborn as sns
3 from s k l e a r n . c l u s t e r i m p o r t KMeans
4 from s k l e a r n . d e c o m p o s i t i o n i m p o r t L a t e n t D i r i c h l e t A l l o c a t i o n
5
6 components = 5
7 n t o p w o r d s = 10
8
9 p a l e t t e = np . a r r a y ( s n s . c o l o r p a l e t t e ( ” h l s ” , 1 2 0 ) )
10
11 d e f p r i n t t o p w o r d s ( model , f e a t u r e n a m e s , n t o p w o r d s ) :
12 f o r t o p i c i d x , t o p i c i n e n u m e r a t e ( model . c o m p o n e n t s ) :
13 m e s s a g e = ” T o p i c #%d : ” % t o p i c i d x
14 m e s s a g e += ” ” . j o i n ( [ f e a t u r e n a m e s [ i ]
15 f o r i i n t o p i c . a r g s o r t ( ) [: − n t o p w o r d s − 1 : − 1 ] ] )
16 p r i n t ( message )
17 print ()
18
19 l d a = L a t e n t D i r i c h l e t A l l o c a t i o n ( n c o m p o n e n t s = components ,
20 m a x i t e r =5 , l e a r n i n g m e t h o d = ’ o n l i n e ’ )
21 l d a . f i t ( dtm )
22 l d a d t m = l d a . t r a n s f o r m ( dtm )
23
24 v e c f e a t u r e n a m e s = vec . g e t f e a t u r e n a m e s ( )
25 p r i n t t o p w o r d s ( lda , vec feature names , n top words )
Topic 0 said trade u.s. deleg quota brazil export year coffe market
Topic 1 gold mine ounc ton said ltd compani ore feet miner
Topic 2 fed volcker reserv treasuri bank borrow pct rate growth dlr
Topic 3 said trade u.s. export japan coffe would ec market offici
Topic 4 billion dlr mln pct januari februari rose bank fell year
The LDA results are encouraging, and we can easily discern 4 of the 5 original
topics from the list of words associated with each topic.
Now let’s see if we can build classifiers to possibly identify the topics above. We
first randomize and split our dataset into train and test sets.
1 from s k l e a r n . m o d e l s e l e c t i o n i m p o r t t r a i n t e s t s p l i t
2
132 3 Text and Speech Basics
9 X t r a i n = X t r a i n [ u ’ body ’ ]
10 X t e s t = X t e s t [ u ’ body ’ ]
We then create a pipeline that builds classifiers based on 5 models: naive Bayes,
logistic regression, SVM, K-nearest neighbor, and random forest.
7 models = [ ( ’ m u l t i n o m i a l n b ’ , MultinomialNB ( ) ) ,
8 ( ’ log reg ’ , LogisticRegression () ) ,
9 ( ’ l i n e a r s v c ’ , LinearSVC ( ) ) ,
10 ( ’ knn ’ , K N e i g h b o r s C l a s s i f i e r ( n n e i g h b o r s = 6 ) ) ,
11 ( ’ r f ’ , R a n d o m F o r e s t C l a s s i f i e r ( n e s t i m a t o r s =6) ) ]
We then train each model on the training set and evaluate on the test set. For each
model, we want to see the precision, recall, F1 score, and support (number of sam-
ples) for each topic class.
1 from s k l e a r n . p i p e l i n e i m p o r t P i p e l i n e
2 from s k l e a r n . m e t r i c s i m p o r t c l a s s i f i c a t i o n r e p o r t
3
4 f o r m name , model i n m o d e l s :
5 p i p e l i n e = P i p e l i n e ( [ ( ’ vec ’ , T f i d f V e c t o r i z e r ( t o k e n i z e r =
S i m p l e T o k e n i z e r ) ) , ( m name , model ) ] )
6 pipeline . f i t ( X train , y t r a i n )
7 test y = pipeline . predict ( X test )
8 p r i n t ( c l a s s i f i c a t i o n r e p o r t ( y t e s t , t e s t y , d i g i t s =6) )
The results seem to indicate that a linear SVM model seems to perform the best,
with random forest a close second. This is a bit misleading, since we didn’t tune
any of these models to obtain our results. Hyperparameter tuning can significantly
affect how well a classifier performs. Let’s try tuning the LinearSVC model. We
want to tune parameters by using grid search with cross-validation. Note that cross-
validation is important as we do not want to tune with our test set, which we will
use only at the end to assess performance. Note also that this can take a while!
1 from s k l e a r n . m o d e l s e l e c t i o n i m p o r t GridSearchCV
2
3 p i p e l i n e = P i p e l i n e ( [ ( ’ vec ’ , v e c t o r i z e r ) ,
4 ( ’ model ’ , model ) ] )
3.14 Case Study 133
11 g r i d s e a r c h = GridSearchCV ( p i p e l i n e , p a r a m e t e r s , v e r b o s e = 1)
12 grid search . f i t ( X train , y t r a i n )
13
As you see, the SVM model typically outperforms other machine learning algo-
rithms, and often provides state-of-the-art quality (Fig. 3.21). Unfortunately, SVM
suffers from several major drawbacks, including the inability to scale to large
datasets. As we will learn in later chapters, neural networks can bypass the limi-
tations of SVMs.
References
4.1 Introduction
One of the most talked-about concepts in machine learning both in the academic
community and in the media is the evolving field of deep learning. The idea of neural
networks, and subsequently deep learning, gathers its inspiration from the biological
representation of the human brain (or any brained creature for that matter).
The perceptron is loosely inspired by biological neurons (Fig. 4.1), connecting
multiple inputs (signals to dendrites), combining and accumulating these inputs (as
would take place in the cell body proper), and producing an output signal that re-
sembles an axon.
illustrated in Fig. 4.2. Each of these neurons learns a different function of its input,
giving the network of neurons an extremely diverse representational power.
The last 6–7 years have seen exponential growth in the popularity and applica-
tion of deep learning. Although the foundations of neural networks can be traced
back to the late 1960s [Iva68], the AlexNet architecture [KSH12c] ushered in an
explosion of interest in the deep learning when it handily won the 2012 Imagenet
image classification competition [Den+09b] with a 5-layer convolutional neural net-
work. Since then deep learning has been applied to a multitude of domains and has
achieved state-of-the-art performance in most of these areas.
The purpose of this chapter is to introduce the reader to deep learning. By the end
of this chapter, the reader should be able to understand the basics of neural networks
and how to train them. We begin this chapter with a review of the perceptron algo-
rithm that was introduced in Chap. 2, where neural networks found their origin. We
then introduce the multilayer perceptron (MLP) classifier, the most simplistic form
of feed-forward neural networks. Following this is a discussion of the essential com-
ponents of training an MLP. This section contains an introduction to both forward
and back propagation and explains the overall training process for neural networks.
We then move toward an exploration of the essential architectural components: ac-
tivation functions, error metrics, and optimization methods. After this section, we
broaden the MLP concept to the deep learning domain, where we introduce addi-
tional considerations when training deep neural networks, such as computation time
and regularization. Finally, we conclude with a practical discussion of common deep
learning framework approaches.
4.2 Perceptron Algorithm Explained 143
Fig. 4.3: The step function performs perfectly adequately for the perceptron; how-
ever, the lack of a non-zero gradient makes it useless for neural networks
0 if v < 0.5
f (v) = (4.2)
1 if v ≥ 0.5
The step function takes a real number input and yields a binary value of 0 or 1,
indicating a positive or negative classification if it exceeds the threshold of 0.5.
4.2.1 Bias
The perceptron algorithm learns a hyperplane that separates two classes. However,
at this point, the separating hyperplane cannot shift away from the origin, as shown
in Fig. 4.4a. Restricting the hyperplane in this fashion causes issues, as we can see
in Fig. 4.4b.
144 4 Basics of Deep Learning
Fig. 4.4: (a) The perceptron algorithm is able to separate the two classes with the
line passing through the origin. (b) Although the data is linearly separable, the per-
ceptron algorithm is not able to separate the data. This is due to the restriction of the
separating plane needing to pass through the origin
One solution is to ensure that our data is learnable if we normalize the method
to center around the origin as a potential solution to alleviate this issue or add a
bias term b to Eq. 4.1, allowing the classification hyperplane to move away from the
origin, as shown in Fig. 4.5.
Fig. 4.5: (a) The perceptron algorithm is able to separate the two classes after cen-
tering the data at the origin. Note the location of the origin in the figure. (b) The
bias allows the perceptron algorithm to relocate the separating plane, allowing it to
correctly classify the data points
4.2 Perceptron Algorithm Explained 145
Fig. 4.6: Perceptron classifier diagram. (a) Perceptron classifier diagram drawn
without the bias. (b) Perceptron diagram including the bias
The bias term is a learned weight that removes the restriction that the separat-
ing hyperplane must pass through the origin.
The learning process for the perceptron algorithm is to modify the weights w to
achieve 0 error on the training set. For example, suppose we need to separate sets
of points A and B. Starting with random weights w, we incrementally improve the
boundary through each iteration with the aim of achieving E(w, b) = 0. Thus, we
would minimize the error of the following function over the entire training set.
Two sets of data are linearly separable if a single decision boundary can separate
them. For example, two sets, A and B, are linearly separable if, for some decision
threshold t, every xi ∈ A satisfies the inequality ∑i wi xi ≥ t and every y j ∈ B satisfies
∑i wi yi < t. Conversely, two sets are not linearly separable if separation requires a
non-linear decision boundary.
If we apply the perceptron to a non-linearly separable dataset, like the dataset
shown in Fig. 4.7a, then we are unable to separate the data as shown in 4.7b since
we are only able to learn three parameters, w1 , w2 , and b.
Fig. 4.7: (a) Non-linearly separable dataset (generalization of the XOR function).
(b) Result of training the perceptron algorithm on the non-linearly separable dataset
in (a). The linear boundary is incapable of classifying the data correctly
Unfortunately, most data that we tend to encounter in NLP and speech is highly
non-linear. One option (as we saw in Chap. 2) is to create non-linear combinations
of the input data and use them as features in the model. Another option is to learn
non-linear functions of the raw data, which is the principal aim of neural networks.
Fig. 4.8: Illustration of the multilayer perceptron network with an input layer, one
hidden layer containing two neurons, and an output layer. The hidden layer, h, is
the result of h = g(W(1) x), where g(x) is the activation function. The output of
the network ŷ = f (W(2) h), where f (x) is the output function, such as the step or
sigmoid function
The hidden layer provides two outputs, h1 and h2 , which may be non-linear com-
binations of their input values x1 and x2 . The output layer weighs its inputs from the
hidden layer, now a potential non-linear mapping, and makes its prediction.
Training the weights of the MLP (and by extension, a neural network) relies on four
main components.
148 4 Basics of Deep Learning
We will walk through each of these components with the network shown in
Fig. 4.8.
The first step in training an MLP is to compute the output of the network for an
example from the dataset. We use the sigmoid function, represented by σ (x), as the
activation function for the MLP. It can be thought of as a smooth step function and
is illustrated in Fig. 4.14. Additionally, it is continuously differentiable, which is a
desirable property for backpropagation, as is shown momentarily. The definition of
the sigmoid function is:
1
σ (x) = . (4.7)
1 + e−x
The forward propagation step is very similar to steps 3 and 4 of the perceptron
algorithm. The goal of this process is to compute the current network output for
a particular example x, with each output connected as the input to the next layer’s
neuron(s).
For notational and computational convenience, the layer’s weights are combined
into a single weight matrix, Wl , representing the collection of weights in that layer,
where l is the layer number. The linear transform performed by the layer computa-
tion for each weight is an inner product computation between x and Wl . This type
is regularly referred to as a “fully connected,” “inner product,” or “linear” layer be-
cause a weight connects each input to each output. Computing the prediction ŷ for
an example x where h1 and h2 represent the respective layer outputs becomes:
f (v) = σ (v)
h1 = f (W1 x + b1 ) (4.8)
h2 = f (W2 h1 + b2 )
ŷ = h2 .
4.3 Multilayer Perceptron (Neural Networks) 149
Note the bias b1 is a vector because there is a bias value associated with each
neuron in the layer. There is only one neuron in the output layer, so the bias b2 is a
scalar.
By the end of the forward propagation step, we have an output prediction for our
network. Once the network is trained, a new example is evaluated through forward
propagation.
The error computation step verifies how well our network performed on the example
given. We use mean squared error (MSE) as the loss function used in this example
(treating the training as a regression problem). MSE is defined as:
1 n
E(ŷ, y) = ∑ (ŷi − yi )2 .
2n i=1
(4.9)
1
The 2 simplifies backpropagation. With a single output this quantity is reduced
to:
1
E(ŷ, y) = (ŷ − y)2 . (4.10)
2
Error functions will be explored more in Sect. 4.4.2.
This error function is commonly used for regression problems, measuring the
average of the square errors for the target. The squaring function forces the error
to be non-negative and functions as a quadratic loss with the values closer to zero,
yielding a polynomially smaller error than values further from zero.
The error computation step produces a scalar error value for the training example.
We will talk more about error functions in Sect. 4.4.2.
Fig. 4.9: Output neuron of Fig. 4.12 showing the full computation of the pre-
activation and post-activation output
150 4 Basics of Deep Learning
Figure 4.9 shows the forward propagation step and error propagation for the out-
put neuron of Fig. 4.8.
4.3.4 Backpropagation
During forward propagation, an output prediction ŷ is computed for the input x and
the network parameters θ . To improve our prediction, we can use SGD to decrease
the error of the whole network. Determining the error for each of the parameters
can be done via the chain rule of calculus. We can use the chain rule of calculus to
compute the derivatives of each layer (and operation) in the reverse order of forward
propagation as seen in Fig. 4.10.
∂E ∂ E ∂ ŷ
= · (4.11)
∂ W2 ∂ ŷ ∂ W2
The chain rule allows us to compute the gradient of the error for each of the
learnable parameters θ , allowing us to update the network using stochastic gradient
descent.
4.3 Multilayer Perceptron (Neural Networks) 151
We begin by computing the gradient on the output layer with respect to the pre-
diction.
∂E
∇ŷ E(ŷ, y) = = (ŷ − y) (4.12)
∂ ŷ
We can then compute error with respect to the layer 2 parameters.
We currently have the “post-activation” gradient, so we need to compute the pre-
activation gradient:
∂E ∂ E ∂ ŷ
∇a2 E = = ·
∂ a2 ∂ ŷ ∂ a2
(4.13)
∂E
= f (W2 h1 + b2 )
∂ ŷ
Now we can compute the error with respect to W2 and b2 .
∂E ∂ E ∂ ŷ ∂ a2
∇W2 E = = · ·
∂ W2 ∂ ŷ ∂ a2 ∂ W2
(4.14)
∂E
= h
∂ a2 1
∂E ∂ E ∂ ŷ ∂ a2
∇b2 E = = · ·
∂ b2 ∂ ŷ ∂ a2 ∂ b2
(4.15)
∂E
=
∂ a2
We can also compute the error for the input to layer 2 (the post-activation output
of layer 1).
∂E ∂ E ∂ ŷ ∂ a2
∇h1 E = = · ·
∂ h1 ∂ ŷ ∂ a2 ∂ h1
(4.16)
∂E
= W2
∂ a2
We then repeat this process to calculate the error for layer 1’s parameters W1 and
b1 , thus propagating the error backward throughout the network.
Figure 4.11 shows the backward propagation step for the output neuron of the
network shown in Fig. 4.8. We leave numerical exploration and experimentation for
our notebook exercises.
152 4 Basics of Deep Learning
The last step in the training process is the parameter update. After obtaining the
gradients with respect to all learnable parameters in the network, we can complete
a single SGD step, updating the parameters for each layer according to the learning
rate α .
θ = θ − α ∇θ E (4.17)
The simplicity of the SGD update rule presented here does come at a cost. The
value of α is particularly vital in SGD and affects the speed of convergence, the
quality of convergence, and even the ability for the network to converge at all. Too
small of a learning rate and the network converges very slowly and can potentially
get stuck in local minima near the random weight initialization. If the learning rate
is too large, the weights may grow too quickly, becoming unstable and failing to
converge at all. Furthermore, the selection of the learning rate depends on a combi-
nation of factors such as network depth and normalization method. The simplicity of
the network presented here alleviates the tedious nature of selecting a learning rate,
but for deeper networks, this process can be much more difficult. The importance
of choosing a good learning rate has led to an entire area of research around gradi-
ent descent optimization algorithms. We discuss some of these techniques more in
Sect. 4.4.3.
4.3 Multilayer Perceptron (Neural Networks) 153
for i ←
− l to 1 do
gai = ∇ai E = ghi+1 ◦ f (ai )
∇Wi E = gai hi−1
∇bi E = gai
ghi = ∇hi−1 E = Wi gai
θ = θ − α ∇θ E
1 The universal approximation theorem was initially proved for neural network architectures us-
ing the sigmoid activation function, but was subsequently shown to apply to all fully connected
networks [Cyb89b, HSW89].
154 4 Basics of Deep Learning
The topography of the parameter space becomes more varied as machine learning
problems become more complex. It is typically non-convex with many local min-
ima. A simple gradient descent approach may struggle to learn the specific function.
Instead, multiple layers of neurons are stacked consecutively and trained jointly with
backpropagation. The network of layers then learns multiple non-linear functions to
fit the training dataset. Deep learning refers to many neural network layers con-
nected in sequence.
The term “deep learning” is somewhat ambiguous. In many circles deep learning is
a re-branding term for neural networks or is used to refer to neural networks with
many consecutive (deep) layers. However, the number of layers to distinguish a deep
network from a shallow network is relative. For example, would the neural network
shown in Fig. 4.12 be considered deep or shallow?
In general, deep networks are still neural networks (trained with backpropaga-
tion, learning hierarchical abstractions of the input, optimized using gradient-based
4.4 Deep Learning 155
learning), but typically with more layers. The distinguishing characteristic of deep
learning is its application to problems previously infeasible to traditional methods
and smaller neural networks, such as the MLP shown in Fig. 4.8. Deeper networks
allow for more layers of hierarchical abstractions to be learned for the input data,
thus becoming capable of learning higher-order functions in more complex domains.
For this book however we utilize the term deep learning as described above—a neu-
ral network with more than one hidden layer.
The flexibility of neural networks is what makes them so compelling. Neural
networks are applied to many types of problems given the simplicity and effective-
ness of backpropagation and gradient-based optimization methods. In this section,
we introduce additional methods and considerations that impact the architecture de-
sign and model training for deep neural networks (DNN). In particular, we focus
on activation functions, error functions, optimization methods, and regularization
approaches.
Fig. 4.13: The step function performed perfectly adequate for the perceptron, how-
ever its derivative makes it bad for gradient descent methods
When computing the gradient of the output layer, it becomes apparent that the
step function is not exactly helpful when trying to compute a gradient. As shown
in Fig. 4.13, the derivative is 0 everywhere which means any gradient descent is
useless. Therefore we wish to use a non-linear activation function that provides a
meaningful derivative in the backpropagation process.
156 4 Basics of Deep Learning
4.4.1.1 Sigmoid
smooth, continuous derivative ideal for gradient descent methods. It is also zero-
centered, creating a simple decision boundary for binary classification tasks, and
the derivative of the sigmoid function is mathematically convenient:
4.4.1.2 Tanh
The tanh function is another common activation function. It also acts as a squashing
function, bounding its output in the range(−1, 1) as shown in Fig. 4.15.
The tanh function solves one of the issues with the sigmoid non-linearity because
it is zero-centered. However, we still have the same issue with the gradient saturation
at the extremes of the function, shown in Fig. 4.16.
4.4.1.3 ReLU
The rectified linear unit (ReLU) is a simple, fast activation function typically
found in computer vision. The function is a linear threshold, defined as:
This simple function has become popular because it has shown faster conver-
gence compared to sigmoid and tanh, possibly due to its non-saturating gradient in
the positive direction.
In addition to faster convergence, the ReLU function is much faster computation-
ally. The sigmoid and tanh functions require exponentials which take much longer
than a simple max operation.
One drawback from the simplicity of the gradient updates being 0 or 1 is that it
can lead to neurons “dying” during training. If a large gradient is backpropagated
through a neuron, the neuron’s output can become so affected that the update pre-
vents the neuron from ever updating again. Some have shown that as many as 40%
of the neurons in a network can “die” with the ReLU activation function if the learn-
ing rate is set too high.
Other activation functions have been incorporated to limit the effects of those pre-
viously described, displayed in Fig. 4.17.
• Hard tanh
The hard tanh function is computationally cheaper than the tanh. It does, how-
ever, re-introduce the disadvantage of gradient saturation at the extremes.
• Leaky ReLU
The Leaky ReLU introduces an α parameter that allows small gradients to be
backpropagated when the activation is not active, thus eliminating the “death” of
neurons during training.
x if x ≥ 0
f (x) = . (4.24)
αx if x < 0
• PRELU
The parametric rectified linear unit, similar to the Leaky ReLU, uses an α pa-
rameter to scale the slope of the negative portion of the input; however, an alpha
parameter is learned for each neuron (doubling the number of learned weights).
Note that when the value of α = 0 this is the ReLU function and when the α is
fixed, it is equivalent to the Leaky ReLU.
x if x ≥ 0
f (x) = . (4.25)
αx if x < 0
4.4 Deep Learning 159
• ELU
The ELU is a modification of the ReLU that allows the mean of activations to
push closer to 0, which therefore potentially speeds up convergence.
x if x > 0
f (x) = . (4.26)
α (ex − 1) if x ≤ 0
• Maxout
The maxout function takes a different approach to activation functions. It differs
from the element-wise application of a function to each neuron output. Instead,
it learns two weight matrices and takes the highest output for each element.
Fig. 4.17: Additional activation function. (a) Hard tanh. (b) Leaky ReLU. (c)
PReLU. (d) ELU
160 4 Basics of Deep Learning
4.4.1.5 Softmax
Fig. 4.18: The output of a neural network can be mapped to a multi-class classi-
fication task (three classes shown here). The softmax function maps the real-value
network output to a probability distribution over the number of classes, where the
number of classes equals the number of neurons in the final layer
occurs between layers. Therefore, the softmax is often treated as the last layer of a
network for multiclass classification rather than an activation function.
As the number of classes begins to grow, as is often the case in language tasks,
the computation of the softmax function can become expensive to compute. For
4.4 Deep Learning 161
example, in a language modeling task, our output layer may be trying to predict
which word will be next in the sequence. Therefore, the output of the network would
be a probability distribution over the number of terms in our vocabulary, which
could be thousands or hundreds of thousands. The Hierarchical Softmax [MB05]
approximates the softmax function by representing the function as a binary tree
with the depth yielding less probable class activations. The tree must be balanced as
the network is training, but it will have a depth of log2 (K) where K is the number
of classes, which means only log2 (K) states need to be evaluated to compute the
output probability of a class.
Another important aspect of training neural networks is the choice of error functions
often referred to as the criteria/criterion. The selection of the error function depends
on the type of problem being addressed. For a classification problem, we want to
predict a probability distribution over a set of classes. In regression problems, how-
ever, we want to predict a specific value rather than a distribution. We present the
basic, most commonly used loss functions here.
Mean squared error(MSE) computes the squared error between the classification
prediction and the target. Training with it minimizes the difference in magnitude.
One drawback to MSE is that it is susceptible to outliers since the difference is
squared.
1 n
E(ŷ, y) = ∑ (yi − ŷi )2
n i=1
(4.29)
So far, we have been using the MSE or L2 for its simplicity as the loss for a binary
classification problems, classifying it as a 0 if ŷ ≥ 0.5 or 1 if ŷ < 0.5; however, it is
typically used for regression problems and could be easily extended for the simple
problems that we have been working with.
Mean absolute error gives a measure of the absolute difference between the target
value and prediction. Using it minimizes the magnitude of the error without consid-
ering direction, making it less sensitive to outliers.
1 n
E(ŷ, y) = ∑ |yi − ŷi |
n i=1
(4.30)
162 4 Basics of Deep Learning
Negative log likelihood (NLL), is the most common loss function used for multi-
class classification problems. It is also known as the multiclass cross-entropy loss.
The softmax provides a probability distribution over the output classes. The entropy
computation is a weighted-average log probability over the possible events or clas-
sifications in a multiclass classification problem. This causes the loss to increase as
the probability distribution of the prediction diverges from the target label.
1 n
E(ŷ, y) = − ∑ (yi log(ŷi ) − (1 − yi ) log(1 − ŷi ))
n i=1
(4.31)
The hinge loss is a max-margin loss classification taken from the SVM loss. It at-
tempts to separate data points between classes by maximizing the margin between
them. Although it is not differentiable, it is convex, which makes it useful to work
with as a loss function.
n
E(ŷ, y) = ∑ max(0, 1 − yi ŷi ) (4.32)
i=1
1 n
E(ŷ, y) = ∑ DKL (yi ||ŷi )
n i=1
(4.33)
1 n 1 n
= ∑
n i=1
yi · log(yi ) − ∑ yi · log(ŷi )
n i=1
θt+1 = θt − α ∇θ E. (4.34)
4.4.3.2 Momentum
One issue that commonly arises with SGD is that there are areas of feature space that
have long shallow ravines, leading up to the minima. SGD will oscillate back and
forth across the ravine because the gradient will point down the steepest gradient
on one of the sides rather than in the direction of the minima. Thus, SGD can yield
slow convergence.
Momentum is one modification of SGD to move the objective more quickly to
the minima. The parameter update equation for momentum is
vt = γ vt−1 + η ∇θ E
(4.35)
θt+1 = θt − vt
4.4.3.3 Adagrad
gt,i = ∇θ E(θt,i )
η
θt+1,i = θt,i − ◦ gt,i (4.36)
Gt,ii + ε
4.4.3.4 RMS-Prop
RMS-prop [TH12] developed by Hinton was also introduced to solve the inadequa-
cies of adagrad. It also divides the learning rate by an average of squared gradients,
but it also decays this quantity exponentially.
where ρ = 0.9 and the learning rate η = 0.001 is suggested in the presented lecture.
4.4.3.5 ADAM
mt = β1 mt−1 + (1 − β1 )gt
vt = β2 vt−1 + (1 − β2 )gt2
mt
m̂t =
1 − β1t (4.38)
vt
v̂t =
1 − β2t
η
θt+1 = θt − √ m̂t
v̂t + ε
4.5 Model Training 165
Empirical results show that Adam works well in practice in comparison with
other gradient-based optimization techniques.
While Adam has been a popular technique, some criticisms of the original
proof have surfaced showing convergence to sub-optimal minima in some situa-
tions [BGW18, RKK18]. Each work proposes a solution to the issue, however the
subsequent methods remain less popular than the original Adam technique.
Achieving the best generalization error (best performance on the test set) is the main
goal for machine learning, which requires finding the best position on the spectrum
between overfitting and underfitting. Deep learning is more prone to overfitting.
With many free parameters, it can be relatively easy to find a path to achieve E = 0.
It has been shown that many standard deep learning architectures can be trained on
random labeling of the training data and achieve E = 0 [Zha+16].
In contrast to overfitting, for many complex functions there are diverse local
minima that may not be the optimal solution, and it is common to settle in a local
minima. Deep learning relies on finding a solution to a non-convex optimization
problem which is NP-complete for a general non-convex function [MK87]. In prac-
tice, we see that computing the global minimum for a well-regularized deep network
is mostly irrelevant because local minima are usually roughly similar and get closer
to the global minimum as the complexity of the model increases [Cho+15a]. In a
poorly regularized network, however, the local minima may yield a high loss, which
is undesirable.
The best model is one that achieves the smallest gap between its training loss
and validation loss; however, selecting the correct architecture configuration and
training technique can be taxing. Here we discuss typical training and regularization
techniques to improve model generalization.
One of the more practical ways that we can prevent a model from overfitting is “early
stopping.” Early stopping hinges on the assumption: “As validation error decreases,
test error should also decrease.” When training we compute the validation error at
distinct points (usually at the end of each epoch) and keep the model with the lowest
validation error, as shown in Fig. 4.19.
166 4 Basics of Deep Learning
The learning curve shows that the training error will continue to decrease towards
0. However, the model begins to perform worse on the validation set as it overfits
to the training data. Therefore, to maintain the generalization of the model on the
test set, the model (learned parameters of model) that performed best on our vali-
dation set would be selected. It is also important to point out here that this requires
a dataset that is split into training, validation, and testing sets with no overlap. The
test set should be kept separate from the training and validation, as, otherwise, this
compromises the integrity of the model.
The simplicity of early stopping makes it the most commonly used form of reg-
ularization in deep learning.
When training neural networks with many layers with backpropagation, the issue of
vanishing/exploding gradients arises. During backpropagation, we are multiplying
the gradient by the output of each successive layer. This means that the gradient can
Fig. 4.19: Early stopping point is shown when validation error begins to diverge
from training error
get larger and larger if ∇E > 1 or ∇E < 1 and smaller if the gradient is 1 < ∇E < 0
as it is multiplied by each successive layer. Practically, this means, in the case of
vanishing gradients, very little of the error is propagated back to the earlier layers
of the network causing learning to be very slow or nonexistent. For exploding gra-
dients, this causes the weights to eventually overflow which prevents learning. The
deeper a neural network becomes, the greater a problem this becomes.
4.5 Model Training 167
Batch gradient decent is a variant of gradient descent that evaluates the error on
the whole dataset before updating the model by accumulating the error after each
example. This alleviates some of the problems of SGD, such as the noise introduced
from each example, but the frequency of the updates can cause a higher variance
between training epochs, which can create significant differences in the models.
This approach is rarely used in practice with deep learning.
A suitable compromise between these two strategies is mini-batch gradient de-
scent. Mini-batch gradient descent splits the dataset into batches, and the model
accumulates the error over a mini-batch before making an update. This approach
provides a variety of advantages, including:
• Reduced noise in each model update due to accumulating the gradients from
multiple training examples
• Greater efficiency than SGD
• Faster training by taking advantages of matrix operations to reduce IO time
4.5.4 Regularization
forms. Some methods focus on reducing the capacity of the models by penalizing
the abnormal parameters in the objective function by adding a regularization term
where W is the weights of the network. Some approaches focus on limiting the in-
formation provided to the network (e.g., dropout) or normalizing the output of layers
(batch normalization), while others may make changes to the data directly. Here we
will explore a variety of regularization methods, and it is typically suggested to
incorporate multiple into every problem.
4.5.4.2 L1 Regularization
As training progresses many of the weights will become zero, introducing spar-
sity into the model weights. This is often used in feature selection but is not always
a desirable quality with neural networks.
4.5.4.3 Dropout
Fig. 4.20: Dropout when applied to a fully connected neural network. (a) Standard
2-layer (hidden) neural network. (b) Standard 2-layer (hidden) neural network with
dropout
170 4 Basics of Deep Learning
In all machine learning tasks, we are optimizing for a specific error metric or func-
tion. Therefore, to perform well on various tasks simultaneously, we usually train
a model for each metric and then ensemble, linearly combine, or connect them in
some other meaningful way to perform well on our overall collection of tasks. Be-
cause deep learning is achieved via gradient-based computation and descent, we
can simultaneously optimize for a variety of optimization functions. This allows
our underlying representation to learn a general representation that can accomplish
multiple tasks. Multitask learning has become a widely used approach recently. The
addition of auxiliary tasks can help improve the gradient signal to the learned pa-
rameters leading to better quality on the overall task [Rud17a].
During the process of training, there may be a lot of variation in the training exam-
ples leading to the introduction of noise in the training process. One of the ways that
we recommended in the introduction is normalizing our data before training. Nor-
malization reduces the amount that weights need to shift to accommodate a specific
example, maintaining the same distribution properties. With deep learning we have
multiple layers of computation with hidden values that are passed to subsequent lay-
ers. The output of each of these layers is likely to be a non-normalized input, and the
distribution is likely to change frequently during the training process. This process
is commonly referred to as “internal covariate shift.” Batch normalization [IS15]
aims to reduce internal covariate shift in a network by normalizing the outputs of
intermediate layers during training. This speeds the training process and allows for
higher learning rates without risking divergence.
Batch normalization achieves this by normalizing the output of the previous hid-
den layer by the batch’s (mini-batch’s) mean and variance. This normalization, how-
4.5 Model Training 171
ever, would affect the inference phase, and, thus, batch normalization captures a
moving average of the mean and variance and fixes them at inference time.
For an input mini-batch β = {x1:m }, we learn parameters γ and β via:
1 m
μβ = ∑ xi
m i=1
1 m
σβ2 = ∑ (xi − μβ )2
m i=1 (4.45)
xi − μβ
x̂i = !
σβ2 + ε
yi = γ x̂i + β .
Most learning techniques and regularization methods have some form of training
configuration parameters associated with them. Learning rate, momentum, dropout
probability, and weight decay, for example, all need to be selected for each model.
Selecting the best combination of these hyperparameters can be a challenging task.
Automatic hyperparameter selection is a much faster and robust method for op-
timizing a training configuration. Grid search, introduced in Chap. 2, is the most
172 4 Basics of Deep Learning
Regularization is the most common technique to prevent overfitting, but it can also
be accomplished with increasing the amount of data. Data is the most important
component of any machine learning model. Although it may seem obvious, this is
often one of the most overlooked components in real-world scenarios. Abstractly,
neural networks are learning from the experiences they encounter. In binary classifi-
cation, for example, the positive example-label pairs are encouraged, while negative
pairs are discouraged. Tuning the neural network’s hyperparameters is typically the
best appropriate step to improve generalization error. If a performance gap still ex-
ists between the training and generalization error, it may be necessary to increase
the amount of data (or quality in some cases).
Neural networks can be robust to some amount of noise in a dataset, and during
the training process, the effects of outliers are typically lessened. However, erro-
neous data can cause many issues. Poor model performance in real-world applica-
tions can be caused by consistently incorrect labels or insufficient data.
This typically manifests itself in one of two ways: overfitting or poor conver-
gence. In the case of overfitting, the model may learn an anomaly of the data (such
as the presence of a user name in many negative sentiment reviews).
4.5 Model Training 173
Deep learning in particular benefits more from larger datasets than other machine
learning algorithms. Much of the quality improvements achieved by deep learning
are directly attributable to the increase in the size of datasets used. Large datasets
can act as a regularization technique to prevent a model from overfitting to specific
examples.
One of the easiest ways to improve model performance is to introduce more train-
ing data. Practically, this can be expensive, but if the data can be augmented in a
meaningful way, this method can be quite useful. This technique can be particularly
beneficial to reduce over-fitting to specific anomalies in a dataset.
In the case of images, we can imagine rotating and horizontal flipping as creat-
ing a different (X, y) pair, without having to re-label any data. This, however, would
not be the case for handwritten numbers, where a horizontal flip might corrupt the
interpretation of the label (think of the 5 and the 2). When incorporating data aug-
mentation, make sure to keep the constraints of the example and target relationship
in mind.
4.5.6.2 Bagging
bustness of the network by providing training examples that focus on the areas of
uncertainty in the parameter space.
4.5.7 Discussion
Broadly speaking, there are typically four pillars in tension when configuring
and training neural networks:
• Data availability (and quality)
• Computation speed
• Memory requirements
• Quality
In practice, it is generally a good idea to establish the end goal and work backwards
to figure out the boundaries for each of the constraints.
Generally speaking, the initial stage of model selection ensures the model has the
capacity to learn reliably. This inevitably leads to overfitting on the training dataset,
at which point regularization is introduced to decrease the gap between the training
loss and validation loss. In practice, it is usually not necessary (nor feasible) to start
from scratch for each new model type or task. However, we believe introducing
complexity gradually is best with highly dynamic systems. It is common to start
with empirically verified architecture sizes and apply regularization directly from
the beginning, however it is best to remove complexity when unexpected situations
arise.
While numerous advancements made deep learning possible, one of the most sig-
nificant contributors to the recent growth in adoption is undoubtedly hardware im-
provements, particularly specialized computer architectures (GPUs). The process-
ing speeds accomplished with GPUs have been among one of the most significant
contributing factors to the popularity and practicality of deep learning. Speed ad-
vantages through matrix optimizations and the ability to batch compute make the
problems of deep learning ideal for GPU architectures. This development made it
possible to move beyond shallow architectures to the deep, complex architectures
that we see today.
Large datasets and deep learning architectures have led to significant quality im-
provements; however, the computational cost of deep learning models is typically
higher other machine learning methods, which needs to be considered in limited re-
source environments (such as mobile devices). The model requirements also impact
4.6 Unsupervised Deep Learning 175
the amount of hyperparameter optimization that can be done. It is unlikely that a full
grid search can be performed for models that take days or weeks to train.
The same reasoning applies to memory concerns, with larger models requir-
ing more space. Although, many quantization techniques are being introduced to
shrink model sizes, such as quantizing parameters or using hashing parameter val-
ues [Jou+16b].
Energy-based models (EBMs) gain their inspiration from physics. The free energy
in a system can be correlated with the probability of an observation. High energies
are associated with a low probability observation and low energies are associated
with a high probability observation. Thus, in EBMs, the aim is to learn an energy
function that results in low energies for observed examples from the dataset and
higher energies for unobserved examples [LeC+06].
For an energy-based model, a probability distribution is defined through the en-
ergy function, similar to:
e−E(x)
p(x) = (4.46)
Z
176 4 Basics of Deep Learning
Z = ∑ e−E(x) (4.47)
x
e−E(x)
P(x) = ∑ P(x, h) = ∑ (4.48)
h h Z
This function yields a negative log-likelihood gradient with two parts, commonly
referred to as the positive phase and negative phase. The positive phase increases the
probability of training data.
Fig. 4.21: Illustration of an RBM. Note that this can be seen as a fully connected
layer as shown previously with only visible-hidden connections in the network. Con-
nections are only shown for one visible neuron and one hidden neuron for the sake
of clarity
where W represents the weight matrix connecting the visible units and the hidden
units, b is the bias of the hidden unit, and c is the bias of the probability for each xi .
We then get the probability from the energy function:
e−E(x,h)
p(x, h) = (4.52)
Z
Furthermore, if x, h ∈ {0, 1}, we can further reduce the equation to:
Once we have samples of the function p(x), we can run a Markov chain with
Gibbs sampling.
The effectiveness of RBMs showed that these architectures can be stacked and
trained together to create a deep belief network (DBN) [HOT06b]. Each sub-
network is trained in isolation, with a hidden layer serving as the visible layer to
the next network. The concept of this layer-by-layer training led to one of the first
effective approaches to deep learning. A deep belief network is shown in Fig. 4.22.
Fig. 4.22: Illustration of a three layer, deep belief network. Each rbm layer is trained
individually, starting with the lowest layer
4.6.4 Autoencoders
vector, learning what is most important for the reconstruction to minimize a recon-
struction error.
The autoencoder is split into two components, the encoder and the decoder. The
encoder converts the input, x into an embedding,2 z. The decoder maps the encoding,
z, back to the original input x. Thus, for a neural network encoder, Enc(x), and
decoder, Dec(z), the loss L (mean squared error) is minimized by:
z = Enc(x)
x̂ = Dec(z) (4.56)
L(x, x̂) = x − x̂ . 2
Fig. 4.23: Architecture diagram of an autoencoder with six input values and an em-
bedding of size 4
2 This output of the encoder is sometimes referred to as the code, encoding or embedding.
3 If the task has real-valued inputs between 0 and 1, then Bernoulli cross-entropy is a better choice
for the objective function.
180 4 Basics of Deep Learning
together between the encoder and decoder, with the decoder weights W∗ = WT . In
this scenario, the gradients for the weights W will be the sum of two gradients, one
from the encoder and one from the decoder.
In general, there are four types of autoencoders:
• Undercomplete autoencoders (standard)
• Sparse
• Denoising autoencoder
• Variational autoencoders (VAE)
with variants of each depending on the application.
An undercomplete autoencoder is the most common type. As shown in Fig. 4.23, the
encoder narrows the network to produce an encoding that is smaller than the input.
This operates as a learned dimensionality reduction technique. Ideally, the encoder
learns to compress the most essential information into the encoding, so the decoder
can reconstruct the input.
x = N(x)
z = Enc(x )
(4.57)
x̂ = Dec(z)
L(x, x̂ ) = x − x̂ 2 .
function can be described as L(x, x̂) = x − x̂2 + λ ∑ |zi |, where λ sets the weight
i
of the sparsity.
Fig. 4.24: The variational autoencoder learns a vector of means, μ , and a vector of
standard deviations, σ . The sampled latent vector, z, is computed by z = μ + σ ◦ ε
where ε is sampled from a normal distribution, N(0, 1)
z = N(μ , σ 2 ) (4.58)
to the reparameterized:
z = μ + σε, (4.59)
where ε is sampled from a normal distribution, N(0, 1). Now, although ε is still
stochastic, μ and σ do not depend on it for backpropagation.
Training the VAE requires optimizing two components in the loss function. The
first component is the reconstruction error that we have optimized for normal au-
toencoders and the second part is the KL-divergence. The KL-divergence loss en-
sures the learned mean and variance parameters stay close to N(0, 1).
The overall loss is defined as:
where DKL is the KL-divergence, p(z) is prior distribution, and q j (z|x) is the learned
distribution.
Sparse coding [Mai+10] aims to learn a set of basis vectors to represent the data.
These basis vectors can then be used to form linear combinations to represent the
input x. The technique of learning basis vectors to represent our data is similar to
techniques like PCA that we explored in Chap. 2. However, with sparse coding, we
instead learn an over-complete set that will allow the learning of a variety of patterns
and structures within the data.
Sparse coding itself is not a neural network algorithm, but we can add a penalty to
our network to enforce sparsity of an autoencoder that creates a sparse autoencoder.
This is merely the addition of an L1 penalty to the loss function that forces most of
weights to be 0.
At training time both true and generated examples are provided to the discrimi-
nator. The discriminator and generator are trained jointly, with the generator’s ob-
jective to increase the error of the discriminator, and the discriminator’s objective
to decrease its error. This is related to the minimax decision rule used in statistics
and decision theory in zero-sum games. This technique has been used as both a
regularization technique and a way to generated synthetic data.
For a generator G and a discriminator D the objective function is given by:
where Pr and Pg represent the real data distribution and generated data distribution,
respectively, and "x = G(z), where z is drawn from a noise distribution such as the
Gaussian distribution.
GANs tend to be used more commonly in computer vision rather than NLP. For
example, some amount of Gaussian noise can be added to an image, while still
maintaining the overall structure and meaning of the image’s content. Sentences
are typically mapped to a discrete space instead of a continuous space, as a word is
discrete (present or not), where noise cannot be readily applied without changing the
meaning. However, a form of character-level language modeling was accomplished
in [Gul+17] by using a latent vector to generate 32 one-hot character vectors through
a convolutional neural network.
The majority of the architectural and algorithmic considerations that have been dis-
cussed are already implemented in deep learning frameworks, with CPU and GPU
support. Many of the differences center on the implementation language, target
users, and abstractions. The most common implementation language is C++ with
a Python interface. The target users can vary broadly and with that variation, the
decisions on abstractions. A key abstraction is how deep networks are composed.
184 4 Basics of Deep Learning
Earlier, we briefly introduced the concept of the layer abstraction, referring to the
linear transformation operation as a “linear layer.” Conceptually, we can continue
the layer abstraction to include all portions of the neural network, representing the
MLP in Fig. 4.8 as three layers with one hidden layer as shown in Fig. 4.26.
Note that although we have represented the inputs, non-linearities, and output as
layers this is still a single hidden layer network.
This makes it easier to split a neural network into logical blocks that can be
composed together. Early deep learning frameworks took this approach for com-
posing neural networks. One could create any layer by implementing a minimal set
of functions, namely the forward propagation step and backward propagation step.
The layers are connected to form a neural network.
This abstraction is useful when constructing standard neural networks, with de-
fined behavior and has been a common approach for frameworks. It is reasonably
straightforward to reason about the interaction of the layers and make guarantees
about the computational requirements. A downside to this approach, as we will see,
the layer abstraction becomes difficult when dealing with complex network struc-
tures. For example, if we wanted recursive connections in a network, we usually
have to implement all recurrent computation in a single layer block (we will explore
this more in the Chap. 7).
4.7 Framework Considerations 185
Many frameworks have since moved beyond the layer abstraction to computational
graphs. The computational graph approach is similar in concept to abstract symbol
trees (AST) in compilers. A dependency graph of inputs and outputs can be repre-
sented with symbols in a tree. This allows a compiler to generate assembly instruc-
tions linking libraries and functions for an executable model. Data flows through the
AST based on the dependencies present in the graph.
In deep learning, a computational graph is a directed graph which defines the
order of computations. The nodes of the graph correspond to operations or vari-
ables. The inputs of a specific node into the graph are the dependencies present in
the computational graph. Subsequently, the backpropagation process can readily be
determined by following the operations in the reverse order from which they were
computed in the forward propagation step. An example of a neural network compu-
tational graph is shown in Fig. 4.27.
Fig. 4.27: Illustration of a 3-layer neural network with the corresponding backward
computation graph. Notice how certain operations can still be combined program-
matically for optimization (e.g., Addmm combines the addition and the multiplica-
tion into a single operation.) (a) Illustration of a 3-layer neural network with sigmoid
activation functions and a softmax output for 10 classes. (b) Computational graph
built from the network shown in (a)
186 4 Basics of Deep Learning
Not only is the computational graph approach convenient for complex functions, it
can be extended to allow for simpler gradient approximations in a complex neural
network. Gradient computation is central in neural networks. One of the most dif-
ficult parts of programming deep neural networks is the gradient computation for
a specific layer or operation. However, the graph-based approach to deep learning
allows for the efficient and automatic computation of gradients in the reverse mode
over the computational graph.
Computational graphs make it much easier to leverage reverse-mode automatic
differentiation methods. Automatic differentiation (AD) [GW08] is a method used
to compute the derivative of a function numerically. AD leverages the concept that,
in computers, all mathematical computation is executed as a sequence of basic math-
ematical operations (addition, subtraction, multiplication, exp, log, sin, cos, etc.).
The AD approach leverages the chain rule of differentiation to decompose a function
into the differentials for each basic operation in the function. This allows derivatives
to be applied automatically and accurately (within a small precision of the theoret-
ical derivative). This approach is typically straightforward to implement achieving
much simpler implementations for complex architectures.
The algorithm of reverse mode AD [Spe80] is the select approach to AD for
deep learning, because it differentiates a single scalar loss. The forward propagation
operation can be seen through the computational graph. This graph can be finely
decomposed to the primitive operations, and, during the backward pass, the gradient
for the output can be computed with respect to the scalar error.
Static computational graphs are graphs that have been created with a static view
of memory. The static structure allows for the optimization of the graph before it
is computed, allowing parallel computation and optimal sequencing of operations.
For example, fusing certain operations may reduce the time needed for memory
IO or efficient optimization of the computation across a collection of GPUs that
may improve the overall performance. This upfront optimization cost is beneficial
when there are resource constraints such as in embedded applications, or when the
network architecture is relatively rigid, as it repeatedly executes the same graph with
little variability in the input.
One of the disadvantages of static computational graphs is that once they are
created, they cannot be modified. Any modifications would eliminate potential ad-
vantages in the applied optimization strategy.
4.8 Case Study 187
Dynamic computational graphs take a different approach where the operations are
computed dynamically at run-time. This is useful in situations where you do not
know what the computation will be beforehand or where we would like to execute
different computations on given data points. A clear example of this is recursive
computation in recurrent neural networks that are based on time sequence inputs of
often variable-length. Dynamic computation is often desirable in NLP applications
where sentence lengths differ and similarly in ASR with the variable lengths of
audio files.
Each of these approaches has trade-offs, much like comparing dynamic typed
programming languages with statically typed languages. Two current examples of
each of these approaches are TensorFlow [Aba+15] and PyTorch [Pas+17]. Tensor-
Flow relies on static computational graphs while PyTorch utilizes dynamic compu-
tational graphs.
In this section, we will apply the concepts of this chapter to the common Free Spo-
ken Digit Dataset5 (FSDD). FSDD is a collection of 1500 recordings of spoken
digits, 0–9, from 3 speakers. We increase the number of files by performing data
augmentation. We discuss this in the next section.
The spoken words are relatively short (most less than 1.5 s). In its raw form,
audio is a single series of samples in the time domain, however it is typically more
useful to convert it to the frequency domain using an FFT. We convert each audio
file to a logMel Spectrogram.
A spectrogram shows the features in a two-dimensional representation with the
intensity of a frequency at a point in time. These representations will be discussed
more in Chap. 8. A set of logMel spectrogram samples from the FSDD dataset are
shown in Fig. 4.28.
In these sections, we will use PyTorch for our example code. We find that the code
used for PyTorch mixes effortlessly with Python, making it easier to focus on the
deep learning concepts rather than the syntax associated with other frameworks.
In addition to PyTorch, we also use librosa to perform the audio manipulation and
augmentation.
5 https://github.com/Jakobovski/free-spoken-digit-dataset.
188 4 Basics of Deep Learning
Fig. 4.28: FSDD sample, showing logMel spectrograms for spoken digits
The original FSDD dataset contains 1500 examples, with no dedicated validation or
testing set. This is a relatively small number of examples, when considering deep
learning, so we scale the dataset by using data augmentation. We focus on two types
of augmentation time stretching and pitch shifting. Time stretching either increases
or decreases the length of the file, while pitch shifting moves the frequencies higher
or lower. For time stretching we move the file 25% faster or slower, and with pitch
shifting we shift up or down one half-step. Every combination of these is applied to
each file, yielding 13, 500 examples, a 9× increase in the amount of data.
1 samples , s a m p l e r a t e = l i b r o s a . load ( f i l e p a t h )
2 for ts in [0.75 ,1 ,1.25]:
3 f o r ps i n [ −1 ,0 ,+1]:
4 samples new = l i b r o s a . e f f e c t s . t i m e s t r e t c h ( samples ,
rate=ts )
5 y new = l i b r o s a . e f f e c t s . p i t c h s h i f t ( s a m p l e s n e w ,
s a m p l e r a t e , n s t e p s =ps )
The neural networks described so far are only able to take fixed length inputs. The
temporal nature of speech makes that difficult, as some of the files are longer than
others. In order to alleviate this constraint, we choose to trip all files to a maximum
duration of 1.5 s. This allows us to work with a fixed representation for all files. This
also helps when batching, as all files in a batch should typically be the same length
for computational efficiency.
After increasing the total amount of data and limiting the length, we randomly
split into training, validation, and testing sets. 80% of the data is used for training,
10% for validation, and 10% for testing.
We use librosa to obtain the logMel Spectrogram, with 128 mel filters applied
(typically < 40 is still fine).
4.8 Case Study 189
1 m a x l e n g t h = 1 . 5 # Max l e n g t h i n s e c o n d s
2 samples , s a m p l e r a t e = l i b r o s a . load ( f i l e p a t h )
3 s h o r t s a m p l e s = l i b r o s a . u t i l . f i x l e n g t h ( samples , s a m p l e r a t e ∗
max length )
4 melSpectrum = l i b r o s a . f e a t u r e . melspectrogram ( s h o r t s a m p l e s .
a s t y p e ( np . f l o a t 1 6 ) , s r = s a m p l e r a t e , n m e l s = 1 2 8 )
5 l o g M e l S p e c t r o g r a m = l i b r o s a . p o w e r t o d b ( melSpectrum , r e f =np .
max )
In addition to saving the audio files in a raw, wav format, we also save them as
numpy arrays. Loading numpy arrays is much faster during training, especially if
we are applying any augmentation. The input data will be the scaled pixel input
from the spectrograms. The dimensionality of the input will be d × t, where d is
the number of mel features extracted and t, the number of time steps. At loading
time, we normalize the logMel spectrogram to be between 0 and 1. Converting the
data range from the power decibel range [−80, 0] to be continuous in the range [0, 1]
alleviates the need for the network to learn higher weights in the early stages of
training. This typically makes training more stable, as there is less internal covariate
shift.
In theory, scaling and normalization is not necessarily required in neural net-
works. Any normalization can be converted by changing the weights and bias asso-
ciated with the input to achieve the same outcome. However, some gradient descent
methods are very sensitive to scaling, standardizing the input data reduces the need
for the network to learn extreme values for outliers. This typically improves training
times, because it reduces the dependency on the scale of the initial weights.
The next thing that we would like to look for in our data is if there is a class or
dataset imbalance. If there is a substantial class imbalance, then we would want to
ensure that we have a representative sample across our datasets. Figure 4.29 shows
a histogram for our dataset splits. From the histograms, we can see that each class
is well represented in each of our sets, and that all classes are relatively balanced in
the number of examples per class. This usually is true for academic datasets, but is
infrequently the case in practice.
Now that we have a good representation of our data, we will show an example of
a supervised classification problem with a neural network as well as an unsupervised
learning method using an autoencoder.
1000 140
120
Number of examples
Number of examples
800
100
600 80
60
400
40
200
20
0 0
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9
Example label Example label
140
120
Number of examples
100
80
60
40
20
0
0 1 2 3 4 5 6 7 8 9
Example label
Fig. 4.29: Histograms for the FSDD training, validation, and testing sets. Each ex-
ample has a spoken label of 0–9. The distribution between the classes is roughly
consistent across the datasets
3 # PyTorch Network D e f i n i t i o n
4 c l a s s Model ( nn . Module ) :
5 def init ( self ) :
6 s u p e r ( Model , s e l f ) . init ()
7 s e l f . f c 1 = nn . L i n e a r ( 3 0 7 2 , 1 2 8 )
8 s e l f . f c 2 = nn . L i n e a r ( 1 2 8 , 1 2 8 )
9 s e l f . f c 3 = nn . L i n e a r ( 1 2 8 , 1 0 )
10
16 h = s e l f . fc2 ( h )
17 h = torch . relu (h)
18
19 h = s e l f . fc3 ( h )
20 o u t = t o r c h . l o g s o f t m a x ( h , dim = 1 )
21 r e t u r n out
In the network definition, we only need to instantiate the learned layers, and the
forward function then defines the order of computation that will be executed.
4.8 Case Study 191
Fig. 4.30: 3-layer neural network for FSDD classification. A ReLU layer is used as
the activation function after the first two hidden layers and a log-softmax transfor-
mation after the output layer
6 Note, PyTorch can still train in mini-batch mode. The view function converts the input tensor
into the dimensions [n, 1, 1, 3072], where n is the mini-batch size.
192 4 Basics of Deep Learning
4 # N e u r a l Network T r a i n i n g i n PyTorch
5 model = Model ( )
6 model . t r a i n ( )
7 i f use cuda :
8 model . c u d a ( )
9 o p t i m i z e r = o p t i m . Adam ( model . p a r a m e t e r s ( ) , l r = 0 . 0 1 )
10 n e p o c h = 40
11 f o r epoch i n range ( n epoch ) :
12 for data , t a r g e t in t r a i n l o a d e r :
13 # Get S a m p l e s
14 i f use cuda :
15 data , t a r g e t = d a t a . cuda ( ) , t a r g e t . cuda ( )
16
17 # Clear gradients
18 optimizer . zero grad ()
19
20 # Forward P r o p a g a t i o n
21 y p r e d = model ( d a t a )
22
23 # Error Computation
24 l o s s = t o r c h . c r o s s e n t r o p y ( y pred , t a r g e t )
25
26 # Backpropagation
27 l o s s . backward ( )
28
29 # P a r a m e t e r Update
30 optimizer . step ()
This code snippet is not complete because it does not incorporate validation eval-
uation during the training process. A more robust example is given in the accompa-
nying notebook. It is left to the reader to experiment with different hyper-parameter
configurations in the exercises. During the training process, we save a copy of the
model with the best validation loss. This model is used to compute the error on the
test set. The training curves and test set results are shown in Fig. 4.31.
We can additionally modify our network to include some of the regularization
techniques and activation functions that we discussed previously, such as batch nor-
malization, dropout, and ReLUs. Incorporating these features is a simple modifi-
cation of the model architecture described previously. The training graph for this
model is also given in Fig. 4.31.
1 # PyTorch Network D e f i n i t i o n
2 c l a s s Model ( nn . Module ) :
3 def init ( self ) :
4 s u p e r ( Model , s e l f ) . init ()
5 s e l f . f c 1 = nn . L i n e a r ( 3 0 7 2 , 1 2 8 )
6 s e l f . bc1 = nn . BatchNorm1d ( 1 2 8 )
7
8 s e l f . f c 2 = nn . L i n e a r ( 1 2 8 , 1 2 8 )
9 s e l f . bc2 = nn . BatchNorm1d ( 1 2 8 )
10
11 s e l f . f c 3 = nn . L i n e a r ( 1 2 8 , 1 0 )
4.8 Case Study 193
12
20 h = s e l f . fc2 ( h )
21 h = s e l f . bc2 ( h )
22 h = torch . relu (h)
23 h = F . dropout (h , p =0.2 , t r a i n i n g = s e l f . t r a i n i n g ) #
Disabled during evaluation
24
25 h = s e l f . fc3 ( h )
26 o u t = t o r c h . l o g s o f t m a x ( h , dim = 1 )
27 r e t u r n out
For the unsupervised example, we will train a simple autoencoder on the FSDD
dataset. This autoencoder learns a low-dimensional encoding of the input data that
the decoder is able to produce examples, and the architecture that we will use in this
example is shown in Fig. 4.32.
Because this is an unsupervised task, we will use the MSE error function com-
paring our input with the output of our decoder. The output of our network must be
the same size as our input, d = 3072, thus the final layer of our network must ensure
that the dimensionality matches the input.
The network architecture is a very simple definition with four linear layers
learned for each of the encoder and the decoder. The PyTorch autoencoder defi-
nition is show below.
1 i m p o r t t o r c h . nn a s nn
2 i m p o r t t o r c h . nn . f u n c t i o n a l a s F # I n p l a c e o p e r a t i o n s f o r non
−l i n e a r i t i e s
3
4 # PyTorch Network D e f i n i t i o n
5 c l a s s a u t o e n c o d e r ( nn . Module ) :
6 def init ( self ) :
7 super ( autoencoder , s e l f ) . init ()
8
2.25
11.0 Training Loss
2.00 Validation Loss
Validation Accuracy
1.75 10.5
1.50
Accuracy
10.0
Loss
1.25
1.00 9.5
0.75 9.0
0.50
8.5
0.25
0 5 10 15 20 25 30 35 40
(a)
Training/Validation Curves
0.5
Training Loss
98 Validation Loss
Validation Accuracy
0.4
96
Accuracy
0.3 94
Loss
92
0.2
90
0.1
88
0 5 10 15 20 25 30 35 40
(b)
Fig. 4.31: Learning curve for a 40 epoch run with two different architecture defini-
tions. Notice the stability of the regularized architecture in (b) compared to (a). (a)
Learning curve for a 40 epoch run of the 2-hidden layer network shown in Fig. 4.30.
On the test set, the best performing validation model achieves a loss of 2.3050 an
accuracy of 10%, statistically the same as random guessing. (b) Learning curve for
a 40 epoch run of the 2-hidden layer network shown in Fig. 4.30 with the incor-
poration of batch normalization, and dropout. On the test set, the best performing
validation model achieves a loss of 0.0825 an accuracy of 98%
18
33 r e t u r n out
4.8 Case Study 195
Fig. 4.32: Autoencoder for the FSDD dataset. Note: layer sizes define the output
size of that layer
The training algorithm is very similar to the one that was introduced for the
classification example. We use the Adam optimizer and add a weight decay term
for regularization. Additionally, as we will be using the same size input as output,
we will move the 2D to 1D transformation outside of the model. The rest of the
algorithm is the same as previously shown. The training algorithm is shown below.
1 import t o r c h . optim as optim
2 i m p o r t t o r c h . nn . f u n c t i o n a l a s F
3
4 # N e u r a l Network T r a i n i n g i n PyTorch
5 model = a u t o e n c o d e r ( )
6 o p t i m i z e r = o p t i m . Adam (
7 model . p a r a m e t e r s ( ) , l r = l e a r n i n g r a t e , w e i g h t d e c a y =1e −5)
8
14 # Forward P r o p a g a t i o n
15 o u t p u t = model ( i n p u t )
16
17 # Error Computation
18 loss = F . mse loss ( output , input )
19
20 # Clear gradients
21 optimizer . zero grad ()
22
23 # Backpropagation
24 l o s s . backward ( )
25
26 # P a r a m e t e r Update
27 optimizer . step ()
196 4 Basics of Deep Learning
Fig. 4.33: Autoencoder output after n epoch(s) on the training data. Notice how
the horizontal lines in the spectrogram are starting to form differently for separate
inputs. (a) Autoencoder reconstruction of its input after 1 epoch. (b) Autoencoder
reconstruction of its input after 100 epochs
When examining the reconstructed inputs, we notice that they appear to be less
sharp than the examples shown in Fig. 4.28. This is mainly due to the MSE loss
function. Because it is computing the squared error, it tends to pull all values toward
the mean prioritizing the average over specific areas of the input.
The RBM learns unsupervised features during the training process. Once these un-
supervised features are learned, we can create a low-dimensional, labeled dataset
using these features to be used in a supervised classifier. In our example, we train a
RBM and then use the learned features as input to a logistic regression classifier.
We can define an RBM with the following code:
1 c l a s s RBM( nn . Module ) :
2 def init ( s e l f , n v i s =3072 , n h i n =128 , k =5) :
3 s u p e r (RBM, s e l f ) . init ()
4 s e l f .W = nn . P a r a m e t e r ( t o r c h . r a n d n ( n h i n , n v i s ) ∗ 1 e −2)
5 s e l f . v b i a s = nn . P a r a m e t e r ( t o r c h . z e r o s ( n v i s ) )
6 s e l f . h b i a s = nn . P a r a m e t e r ( t o r c h . z e r o s ( n h i n ) )
7 self .k = k
8
12 def v to h ( self , v ) :
13 p h = F . s i g m o i d ( F . l i n e a r ( v , s e l f .W, s e l f . h b i a s ) )
14 sample h = s e l f . sample from p ( p h )
15 r e t u r n p h , sample h
16
4.8 Case Study 197
17 def h to v ( self , h ) :
18 p v = F . s i g m o i d ( F . l i n e a r ( h , s e l f .W. t ( ) , s e l f . v b i a s ) )
19 sample v = s e l f . sample from p ( p v )
20 r e t u r n p v , sample v
21
25 h = h1
26 for in range ( s e l f . k ) :
27 pre v , v = s e l f . h to v ( h )
28 pre h , h = s e l f . v to h ( v )
29
30 return v , v
31
We train the model with Adam. The sample code to do this is as follows:
1 rbm = RBM( n v i s =3072 , n h i n =128 , k =1)
2
3 t r a i n o p = o p t i m . Adam ( rbm . p a r a m e t e r s ( ) , 0 . 0 1 )
4 f o r epoch i n range ( epochs ) :
5 loss = []
6 for , ( data , t a r g e t ) in enumerate ( t r a i n l o a d e r ) :
7 d a t a = V a r i a b l e ( d a t a . view ( −1 , 3 0 7 2 ) )
8 sample data = data . bernoulli ()
9
10 v , v1 = rbm ( s a m p l e d a t a )
11 l o s s = rbm . f r e e e n e r g y ( v ) − rbm . f r e e e n e r g y ( v1 )
12 l o s s . append ( l o s s . d a t a [ 0 ] )
13 train op . zero grad ()
14 l o s s . backward ( )
15 train op . step ()
After training our RBM features, we can create a logistic regression classifier to
classify our examples based on the unsupervised features we have learned.
1 from s k l e a r n . l i n e a r m o d e l i m p o r t L o g i s t i c R e g r e s s i o n
2
3 clf = LogisticRegression ()
4 clf . fit ( train features , train labels )
5 predictions = clf . predict ( test features )
Fig. 4.34: Confusion matrices for a logistic regression classifier with RBM features
on the FSDD dataset. (a) Confusion matrix for FSDD. (b) Normalized confusion
matrix for FSDD
4.8.6 Results
Some other interesting problems readers and practitioners can try on their own in-
clude:
Table 4.1: End-to-end speech recognition performance on FSDD test set. High-
lighted result indicates best performance
Approach Accuracy
2-layer MLP 10.38
2-layer MLP (with regularization) 98.44
RBM + Logistic Regression 71.04
1. What is the effect of training the FSDD classifier with each of the learning rates
[0.001, 0.1, 1.0, 10]? What is the effect when switching the optimization method?
2. What is the result of a learning rate of 0.1 for the FSDD autoencoder?
3. How would the architecture change if we want to learn a set of sparse features
instead of a low-dimensional encoding of the handwritten digits?
4. What effect does batch size have on the learning process? Does it effect the
learning rate?
References 199
5. What additional data augmentations could be applied to the audio for the system
to be more robust?
6. Train a classifier with the trained autoencoder’s encoding as the features. How
does the accuracy compare to the supervised model?
7. Change the autoencoder to a variational autoencoder. Does it improve the visible
quality of the generated output? Vary the inputs to the decoder to understand the
features that have been learned.
8. Extend the RBM to create a deep belief network for classifying the FSDD
dataset.
References
5.1 Introduction
In this chapter, we introduce the notion of word embeddings that serve as core
representations of text in deep learning approaches. We start with the distributional
hypothesis and explain how it can be leveraged to form semantic representations of
words. We discuss the common distributional semantic models including word2vec
and GloVe and their variants. We address the shortcomings of embedding models
and their extension to document and concept representation. Finally, we discuss
several applications to natural language processing tasks and present a case study
focused on language modeling.
on the key property that the proximity of points in the hyperspace is a measure of
the semantic similarlity of the documents. In other words, documents with similar
vector representations imply that they are semantically similar. VSMs have found
widespread adoption in information retrieval applications, where a search query is
achieved by returning a set of nearby documents sorted by distance. We have already
seem VSMs in the form of the bag-of-words term-frequency or TFIDF example back
in Chap. 3.
VSMs can suffer from a major drawback if they are based on high-dimensional
sparse representations. Here, sparse means that a vector has many dimensions with
zero values. This is termed the curse of dimensionality. As such, these VSMs re-
quire large memory resources and are computationally expensive to implement and
use. For instance, a term-frequency based VSM would theoretically require as many
dimensions as the number of words in the dictionary of the entire corpus of docu-
ments. In practice, it is common to set an upper bound on the number of words and
hence, dimensionality of the VSM. Words that are not within the VSM are termed
out-of-vocabulary (OOV). This is a meaningful gap with most VSMs in that they
are unable to attribute semantic meaning to new words that they haven’t seen before
and are OOV.
The distributional hypothesis says that the meaning of a word is derived from
the context in which it is used, and words with similar meaning are used in
similar contexts.
5.2 Distributional Semantics 205
One of the earliest use of word representations dates back to 1986. Word vec-
tors explicitly encode linguistic regularities and patterns. Distributional semantic
models can be divided into two classes, co-occurrence based and predictive mod-
els. Co-occurrence based models must be trained over the entire corpus and cap-
ture global dependencies and context, while predictive models capture local depen-
dencies within a (small) context window. The most well-known of these models,
word2vec and GloVe, are known as word models since they model word depen-
dencies across a corpus. Both learn high-quality, dense word representations from
large amounts of unstructured text data. These word vectors are able to encode lin-
guistic regularities and semantic patterns, which lead to some interesting algebraic
properties.
5.2.2.1 Co-occurrence
The distributional hypothesis tells us that co-occurrence of words can reveal much
about their semantic proximity and meaning. Computational linguistics leverages
this fact and uses the frequency of two words occurring alongside each other within
a corpus to identify word relationships. Pointwise Mutual Information (PMI) is a
commonly used information-theoretic measure of co-occurrence between two words
w1 and w2 :
p(w1 , w2 )
PMI(w1 , w2 ) = log (5.1)
p(w1 )p(w2 )
where p(w) is the probability of the word occurring, and p(w1 , w2 ) is joint probabil-
ity of the two words co-occurring. High values of PMI indicate collocation and co-
incidence (and therefore strong association) between the words. It is common to es-
timate the single and joint probabilities based on word frequency and co-occurrence
within the corpus. PMI is a useful measure for word clustering and many other tasks.
5.2.2.2 LSA
Latent semantic analysis (LSA) is a technique that effectively leverages word co-
occurrence to identify topics within a set of documents. Specifically, LSA analyzes
word associations within a set of documents by forming a document-term matrix
(see Fig. 5.2), where each cell can be the frequency of occurrence or TFIDF of
a term within a document. As this matrix can be very large (with as many rows
as words in the vocabulary of the corpus), a dimensionality reduction technique
such as singular-value decomposition is applied to find a low-rank approximation.
This low-rank space can be used to identify key terms and cluster documents or for
information retrieval (as discussed in Chap. 3).
206 5 Distributed Representations
Recall that language models seek to learn the joint probability function of sequences
of words. As stated above, this is difficult due to the curse of dimensionality—the
sheer size of the vocabulary used in the English language implies that there could be
an impossibly huge number of sequences over which we seek to learn. A language
model estimates the conditional probability of the next word wT given all previous
words wt :
T
p (wT ) = ∏ p(wt |w1 , . . . , wt−1 ) (5.2)
t=1
5.2.3.1 Bengio
In 2003, Bengio et al. [Ben+03] presented a neural probabilistic model for learn-
ing a distributed representation of words. Instead of sparse, high-dimensional rep-
resentations, the Bengio model proposed representing words and documents in
lower-dimensional continuous vector spaces by using a multilayer neural network
5.2 Distributional Semantics 207
to predict the next word given the previous ones. This network is iteratively trained
to maximize the conditional log-likelihood J over the training corpus using back-
propagation:
T
1
J=
T ∑ log f (v(wt ), v(wt−1 ), . . . , v(wt−n+1 ); θ ) + R(θ ) (5.3)
t=1
where v(wt ) is the feature vector for word wt , f is the mapping function representing
the neural network, and R(θ ) is the regularization penalty applied to weights θ
of the network. In doing so, the model concurrently associates each word with a
distributed word feature vector as well as learning the joint probability function of
word sequences in terms of the feature vectors of the words in the sequence. For
instance, with a corpus of vocabulary size of 100,000, a one-hot encoded 100,000-
dimensional vector representation, the Bengio model can learn a much smaller 300-
dimensional continuous vector space representation (Fig. 5.4).
In 2008, Collobert and Weston [CW08] applied word vectors to several NLP tasks
and showed that word vectors could be trained in an unsupervised manner on a
corpus and used to significantly enhance NLP tasks. They used a multilayer neu-
ral network trained in an end-to-end fashion. In the process, the first layer in the
network learned distributed word representations that are shared across tasks. The
output of this word representation layer was passed to downstream architectures that
were able to output part-of-speech tags, chunks, named entities, semantic roles, and
sentence likelihood. The Collobert and Weston’s model is an example of multitask
learning enabled through the adoption of dense layer representations.
208 5 Distributed Representations
5.2.4 word2vec
Note that word2vec models are fast—they can quickly learn vector represen-
tations of much larger corpora than previous methods.
5.2.4.1 CBOW
where c is the number of context words to each side of the target word (Fig. 5.6).
For the simple CBOW model, the average vector representation from the output of
the projection layer is fed into a softmax that predicts over the entire vocabulary of
the corpus, using backpropagation to maximize the log probability objective:
exp vwt vwt+ j
p(wt |wt+ j ) = V (5.5)
∑w=1 exp vw vwt+ j
where V is the number of words in the vocabulary. Note that after training, the matrix
W are the learned word embeddings of the model.
210 5 Distributed Representations
5.2.4.2 Skip-Gram
Whereas the CBOW model is trained to predict a target word based on the nearby
context words, the skip-gram model is trained to predict the nearby context words
based on the target word (Fig. 5.7). Once again, word order is not considered. For
a context size c, the skip-gram model is trained to predict the c words around the
target word. The objective of the skip-gram model is to maximize the average log
probability:
1
T ∑ ∑ log (p (wt+ j |wt )) (5.6)
t=1 −c< j<c, j=0
where c is the size of the training context (Fig. 5.8). Higher values of c result in
more training examples and thus can lead to a higher accuracy, at the expense of the
training time. The most simple skip-gram formulation utilizes the softmax function:
exp vwt+ j vwt
p(wt+ j |wt ) = V (5.7)
∑w=1 exp (vw vwt )
where V is the number of words in the vocabulary.
It is interesting to note that shorter training contexts result in vectors that cap-
ture syntactic relationships well, while larger context windows better capture
semantic relationships. The intuition behind this is that syntactic information
is typically dependent on the immediate context and word order, whereas se-
mantic information can be non-local and require larger window sizes.
212 5 Distributed Representations
The simple versions of CBOW and skip-gram use a full softmax output layer, which
can be computationally expensive when the vocabulary is large. A more compu-
tationally efficient approximation to the full softmax is the hierarchical softmax,
which uses a binary tree representation of the output layer. Each word w can be
reached by an appropriate path from the root of the tree:
L(w)−1
p (w|wt ) = ∏ σ (b (n (w, j + 1) = ch (n (w, j)))) vn(w, j) vwt (5.8)
j=1
where
1
σ (x) = (5.9)
1 + e−x
V
∑ p (w|wt ) = 1 (5.10)
w=1
In practice, it is common to use a binary Huffman tree, which assigns short codes
to the frequent words and results in fast training as it requires only calculating over
log2 (V ) words instead of V words for the softmax.
5.2 Distributional Semantics 213
k
log σ vWO
vWI + ∑ Ewi ∼pn (w) log σ −vwi vWI (5.11)
i=1
When choosing the number of negative samples k, note that word2vec’s per-
formance will decrease as this number increases in most cases. In practice, k
in the range of 5–20 can be used.
The main difference between negative sampling and NCE is that NCE needs both
samples and the numerical probabilities of the noise distribution, while negative
sampling uses only samples.
∑ ∑ (s (wt , wc )) + ∑ (−s (wt , n)) (5.12)
t=1 c∈ct n∈Nt,c
Subsampling can considerably speed up training times as well as increase the accu-
racy of the learned vectors of rare words.
214 5 Distributed Representations
count(wi , w j )
score(wi , w j ) = log (5.16)
count(wi )count(w j )
such that words are combined and replaced by a single token whenever the score
rises above a threshold value. This equation is an approximation to the pointwise-
mutual information.
Interestingly, word2vec models and in particular the skip-gram model have
shown the vector compositionality—the ability to use simple vector additions can
often produce meaningful phrases. Adding the vectors for Philadelphia and Eagles
can yield a vector that is in closest proximity to other sports teams.
We will derive equations for forward and backward propagation for CBOW to give
the readers insight into the training mechanisms and how the weights are updated.
Let a single input word be represented as a one-hot vector given by x ∈ RV where
V is the vocabulary and many such word vectors given by {x1 , x2 , · · · , xC } of size C
form the context. Let the vectors flow into a single hidden layer h ∈ RD , where D
is the dimension of the embeddings to be learned through training, with the identify
activation function and the input values are averaged across the context words. Let
W ∈ RV ×D be the weight matrix that captures weights between input and the hidden
layer. Figure 5.9 shows the different layers and the connections as described above.
The hidden layer can be given as:
1 C
C∑
h = W xc (5.17)
c=1
C
1
We will represent the C ∑ xc as the average input vector given by x̄. Thus:
c=1
h = W x̄ (5.18)
5.2 Distributional Semantics 215
Fig. 5.9: word2vec CBOW with one-hot encoded inputs, single hidden layer, output
layer and a softmax layer
u = W h (5.19)
u = W W x̄ (5.20)
The output layer is then mapped to a softmax output layer y ∈ RV given by:
y = softmax(u) = softmax(W W x̄) (5.21)
When we are training the model with target (wt )-context words(wc,1 , wc,2 , · · · , wc,C ),
the output value should match the target in the one-hot encoded representation, i.e.
at position j∗ the output has value 1 and 0 elsewhere. The loss in terms of conditional
probability of target word given context words is given by:
L = − log P(wt |wc,1 , wc,2 , · · · , wc,C ) = − log(y j ∗) = − log(softmax(u j ∗)) (5.22)
exp(u ∗)
j
L = − log(y j ∗) = − log (5.23)
∑i exp(ui )
L = −u j ∗ + log ∑ exp(ui ) (5.24)
i
The idea of training through gradient descent as discussed in Chap. 4 is to find values
of W and W that minimize the loss function given by Eq. (5.24). The loss function
depends on W and W through the output variable u. So to find the values we differ-
entiate the loss function L with respect to both W and W . Since L = L(u(W, W ))
the two derivatives can be written as:
∂L V
∂ L ∂ uk
= ∑
∂ Wi j k=1 ∂ uk ∂ Wij
(5.25)
216 5 Distributed Representations
∂L V
∂ L ∂ uk
=∑ (5.26)
∂ Wi j k=1 ∂ uk ∂ Wi j
Let us consider Eq. (5.25) where Wij is the connection between hidden layer i and
output layer j and since the output is one-hot encoded affects only at value k = j
and will be 0 in all other places. Thus the equation reduces to:
∂L ∂L ∂uj
= (5.27)
∂ Wij ∂ u j ∂ Wij
∂L
Now ∂uj can be written as:
∂L
= −δ j j∗ + y j = e j (5.28)
∂uj
∂uj V
= ∑
∂ Wij k=1
Wki x̄k (5.29)
Thus combining:
∂L V
∂ Wij
= (− δ j j∗ + y j ) ∑ Wki x̄k (5.30)
k=1
For Eq. (5.26) after we fix the input, the output y j at node j depends on all the
connections from that input and thus
∂L V
∂L ∂ 1 D V C
=∑ ∑ ∑ Wmk ∑ Wlm xlc (5.33)
∂ Wi j k=1 ∂ uk ∂ Wi j C d=1 l=1 c=1
5.2 Distributional Semantics 217
∂L 1 V C
= ∑ ∑ (−δkk∗ + yk )W jk xic (5.34)
∂ Wi j C k=1 c=1
Thus the new values Wnew and W new using a learning rate η is given by:
∂L
Wnew = Wold − η (5.36)
∂W
and
∂L
W new = W old − η (5.37)
∂ W
As we have defined that the skip-gram model is the inverse of the CBOW, i.e., the
center word is given in the input and the context words are predicted at the output as
shown in Fig. 5.10. We will derive the skip-gram equations on the same lines using
a simple network similar to CBOW above.
Fig. 5.10: word2vec skip-gram with input word, single hidden layer, generating C
context words as output that maps to the softmax function generating the one-hot
representation for each
218 5 Distributed Representations
The input x ∈ RV goes into the hidden layer h ∈ RD through the weights W ∈
RV ×D and unit activation function. The hidden layer then generates C context word
vectors uc ∈ RD×V as the output and that can be mapped to a softmax function to
generate one-hot representation y ∈ RV that maps a word in the vocabulary for each
embedding output.
h = W x (5.38)
uc = W h (5.39)
uc = W W x c = 1, · · · ,C (5.40)
yc = softmax(uc ) = softmax(W W x) c = 1, · · · ,C (5.41)
The loss function for skip-gram can be written as:
where the word wi is the input word and wc,1 , wc,2 , · · · , wc,C are the output context
words.
C
L = − log ∏ P(wc,i |wi ) (5.43)
c=1
C C V
L= ∑ exp(uc, j∗ ) + ∑ log ∑ exp(uc, j ) (5.45)
c=1 c=1 j=1
The loss function is dependent on uc and each u dependent on (W, W ). This can be
expressed as:
L = L(u1 (W, W ) · · · uC (W, W )) (5.46)
L = L(u1,1 (W, W ) · · · uC,V (W, W )) (5.47)
The chain-rule applied to skip-gram gives:
∂L V C
∂ L ∂ uc,k
=∑∑ (5.48)
∂ Wi j k=1 c=1 ∂ uc,k ∂ Wij
∂L V C
∂ L ∂ uc,k
=∑∑ (5.49)
∂ Wi j k=1 c=1 ∂ uc,k ∂ Wi j
5.2 Distributional Semantics 219
∂L V C
∂ L ∂ uc,k C
∂ L ∂ uc, j
=∑∑ =∑ (5.50)
∂ Wi j k=1 c=1 ∂ uc,k ∂ Wi j c=1 ∂ uc, j ∂ Wij
∂L C V
= ∑ (−δ j jc ∗ + yc, j ) ∑ Wk,i xk (5.51)
∂ Wi j c=1 k=1
∂L
where ∂ uc, j = −δ j jc ∗ + yc, j = ec, j
∂L C
= (W x) ⊗ ∑ ec (5.52)
∂W c=1
∂L V C
∂L ∂ D V
=∑∑ ∑ ∑ WmkWld xl (5.53)
∂ Wi j k=1 c=1 ∂ uc,k ∂ Wi j m=1 l=1
∂L V C
= ∑ ∑ (−δkk∗c + yc,k )W jk xi (5.54)
∂ Wi j k=1 c=1
Now, representing −δkk∗c + yc,k = eck we can simplify it as
∂L C
= x ⊗ W ∑ ec (5.55)
∂W c=1
5.2.5 GloVe
The global co-occurrence based models can be the alternative to predictive, local-
context window methods like word2vec. Co-occurrence methods are usually very
high dimensional and require much storage. When dimensionality reduction meth-
ods are used like in LSA, the resulting representations typically perform poorly in
capturing semantic word regularities. Furthermore, frequent co-occurrence terms
tend to dominate. Predictive methods like word2vec are local-context based and
generally perform poorly in capturing the statistics of the corpus. In 2014, Pen-
nington et al. [PSM14] proposed a log-bilinear model that combines both global
220 5 Distributed Representations
co-occurrence and shallow window methods. They termed this the GloVe model,
which is play on the words Global and Vector. The GloVe model is trained via least
squares using the cost function:
V 2
J= ∑ f (Xi j ) ui v j − log (Xi j ) (5.56)
i=1, j=1
where V is the size of the vocabulary, Xi j is the count of times that words i and j co-
occur in the corpus (Fig. 5.11), f is a weighting function that acts to reducethe im-
pact of frequent counts, and ui and v j are word vectors. Typically, a clipped power-
law form is assumed for weighting function f :
X a
ij
Xmax if Xi j < Xmax
f (Xi j ) = (5.57)
1 otherwise
with Xmax is set at training time based on the corpus. Note that the model trains
context vectors U and word vectors V separately, and GloVe embeddings are the
given by the sum of these two vector representations U + V. Similar to word2vec,
GloVe embeddings can express semantic and syntactic relationships through vec-
tor addition and subtraction [SL14]. Furthermore, word embeddings generated by
GloVe are superior to word2vec in performance over many NLP tasks, especially in
situations where global context is important such as named entity recognition.
where
It is well known that the distributional hypothesis holds for most human lan-
guages. This implies that we can train word embedding models in many languages
[Cou+16, RVS17], and companies such as Facebook and Google have released
pre-trained word2vec and GloVe vectors for up to 157 languages [Gra+18]. These
embedding models are monolingual—they are learned on a single language. Sev-
eral languages exist with multiple written forms. For instance, Japanese possesses
three distinct writing systems (Hiragana, Katakana, Kanji). Mono-lingual embed-
ding models cannot associate the meaning of a word across different written forms.
The term word alignment is used to describe the NLP process by which words are
related together across two written forms across languages (translational relation-
ships) (Fig. 5.12) [Amm+16]. Embedding models have provided a path for deep
learning to make major breakthroughs in word alignment tasks, as we will learn in
Chap. 6 and beyond.
The Zipfian distributional nature of the English language is such that there exists
a huge number of infrequent words. Learning representations for these rare words
would require huge amounts of (possibly unavailable) data, as well as potentially
5.3 Limitations of Word Embeddings 223
5.3.2 Antonymy
where the first term are the co-occurrence pairs within a context window and the sec-
ond term represents negative sampling. Given a set of synonyms Sw and antonyms
Aw of a word w, we can modify the skip-gram model objective function to the form:
This objective can be optimized to learn embeddings that can distinguish synonyms
from antonyms. Studies have shown that embeddings learned in this manner to in-
corporate both distributional and thesauri information perform significantly better
in tasks such as question-answering.
5.3.3 Polysemy
In the English language, words can sometimes have several meanings. This is known
as polysemy. Sometimes these meanings can be very different or complete oppo-
sites of each other. Look up the meaning of the word bad and you might find up to
46 distinct meanings. As models such as word2vec or GloVe associate each word
with a single vector representation, they are unable to deal with homonyms and pol-
ysemy. Word sense disambiguation is possible but requires more complex models.
In linguistics, word sense relates to the notion that, in the English language and
many other languages, words can take on more than one meaning. Polysemy is the
concept that a word can have multiple meanings. Homonymy is a related concept
where two words are spelled the same but have different meanings. For instance,
compare the usage of the word play in the sentences below:
She enjoyed the play very much.
She likes to play cards.
She made a play for the promotion.
For NLP applications to differentiate between the meanings of a polysemous word,
it would require separate representations to be learned for the same word, each as-
sociated with a particular meaning [Nee+14]. This is not possible with word2vec or
GloVe embedding models since they learn a single embedding for a word. Embed-
ding models must be extended in order to properly handle word sense.
5.3 Limitations of Word Embeddings 225
One approach to deal with word sense disambiguation is to start by building an in-
ventory of senses for words within a corpus. Each instance of a word wi is associated
with a representation based on context words surrounding it. These representations,
termed context embeddings, are then clustered together. The centroid of each cluster
is the representation Swi for the different senses of the word:
sense(wi ) = arg min d(ci , s j ) (5.65)
j:s j ∈Swi
+ ∑
∑
log P(D = 0|vs (wt , st ), vg (c )) (5.67)
(wt ,ct )∈D− c ∈ct
5.3.3.2 Sense2vec
The sense2vec model can learn different word senses of this word by combin-
ing a single-sense embedding model with POS labels (Fig. 5.14). Given a corpus,
sense2vec will create a new corpus for each word for each sense by concatenating a
word with its POS label. The new corpus is then trained using word2vec’s CBOW
or skip-gram to create word embeddings that incorporate word sense (as it relates
to their POS usage). Sense2vec has been shown to be effective for many NLP tasks
beyond word-sense disambiguation (Fig. 5.15).
Recently, we have become aware of the potential biases that may implicitly exist
within embedding models. Learned word representations are only as good as the
data that they were trained on—that is, they will capture the semantic and syntac-
tic context inherent in the training data. For instance, recent studies have revealed
racial and gender biases within popular word embedding models such as GloVe and
word2vec trained on a broad news corpus:
v(nurse) ≈ v(doctor) − v(father) + v(mother)
v(Leroy) ≈ v(Brad) − v(happy) + v(angry)
A further limitation of word embedding models relates to the batch nature of train-
ing and the practicality of augmenting an existing model with new data or ex-
panded vocabulary. Doing so requires us to retrain an embedding model with both
the original data and new data—the entire data needs to be available and embed-
dings recomputed. An online learning approach to word embeddings would allow
them to be more practical.
Recent interest in word embedding models has led to practical adaptations that can
leverage word compositionality (subword embeddings) and address memory con-
straints (word2bits). Others have extended word2vec to learn distributed represen-
tations of sentences, documents (DM and DBOW), and concepts (RDF2Vec). Inter-
est has also given rise to Bayesian approaches that map words to latent probability
228 5 Distributed Representations
Methods such as word2vec or GloVe ignore the internal structure of words and
associate each word (or word sense) to a separate vector representation. For mor-
phologically rich languages, there may be a significant number of rare word forms
such that either a very large vocabulary must be maintained or a significant num-
ber of words are treated as out-of-vocabulary (OOV). As previously stated, out-of-
vocabulary words can significantly impact performance due to the loss of context
from rare words [Bak18]. An approach that can help deal with this limitation is
the use of subword embeddings [Boj+16], where vector representations zg are as-
sociated with character n-grams g and words wi are represented by the sum of the
n-gram vectors (Fig. 5.16).
wi = ∑ zg (5.68)
g∈Gw
For instance, the vector for the word indict consists of the sum of the vectors for the
n-grams {ind,ndi,dic,ict,indi,ndic,dict,indic,ndict,indict} when n ∈ (3, 6). Thus, the
set of n-grams is a superset of the vocabulary of the corpus (Fig. 5.17). As n-grams
are shared across words, this allows for representation of even unseen words since
an OOV word will still consist of n-grams that will have representations. Subword
embeddings can significantly boost NLP tasks such as language modeling and text
classification.
Even for a small vocabulary, word models can require a significant amount of mem-
ory and storage. Consider a 150,000-word vocabulary. A 300-dimensional continu-
ous 64-bit representation of these words can easily occupy over 360 megabytes. It is
possible to learn a compact representation by applying quantization to word vectors.
In some cases, compression ratios of 8x-16x are possible relative to full-precision
word vectors while maintaining comparable performance [Lam18]. Furthermore,
the quantization function can act as a regularizer that can improve generalization
[Lam18].
Word2Bits is an approach that adapts the word2vec CBOW method by introduc-
ing a quantization element to its loss function:
Jquantized uo , v̂c = − log σ (uo ) v̂c
(q) (q) (q) (q)
k
− ∑ log σ (−ui ) v̂c
(q) (q)
(5.69)
i=1
5.4 Beyond Word Embeddings 229
where
(q)
uo = Qbitlevel (uo ) (5.70)
∑
(q)
v̂c = Qbitlevel (vi ) (5.71)
−w+i≤i≤w+o,i=o
Here, w is the context window width, Qbitlevel is the quantization function, uo and v̂c
(q) (q)
are the target and context word vectors, respectively, and uo and v̂c are their quan-
tized equivalents. The Heaviside step function is commonly chosen as the quanti-
zation function Qbitlevel . Similar to the standard CBOW algorithm, the loss function
is optimized over the target ui and context v j over the corpus. The gradient updates
for the target word uo , negative sampling word ui , and context word vi are given by:
(q) (q) (q) (q)
∂ Jquantized uo , v̂c ∂ Jquantized uo , v̂c
uo : = (5.72)
∂ uo ∂ uo
(q)
(q) (q) (q) (q)
∂ Jquantized uo , v̂c ∂ Jquantized uo , v̂c
ui : = (5.73)
∂ ui (q)
∂ ui
230 5 Distributed Representations
(q) (q) (q) (q)
∂ Jquantized uo , v̂c ∂ Jquantized uo , v̂c
vi : = (5.74)
∂ vi (q)
∂ vi
The final vector for each word is expressed as Qbitlevel (ui + v j ) whose elements can
take on one of 2bitlevel values and requires only bitlevel bits to represent in compar-
ison with full-precision 32/64 bits. Studies have shown that quantized vectors can
perform comparably on word similarity tasks and question answering tasks even
with 16x compression.
While word embedding models capture semantic relationships between words, they
lose this ability at the sentence level. Sentence representations are usually expressed
the sum of the word vectors of the sentence. This bag-of-words approach has a
major flaw in that different sentences can have identical representations as long
as the same words are used. To incorporate word order information, people have
attempted to use bag-of-n-grams approaches that can capture short order contexts.
However, at the sentence level, they are limited by data sparsity and suffer from
poor generalization due to high dimensionality.
Le and Mikolov in 2014 [LM14] proposed an unsupervised algorithm to learn
useful representations of sentences that capture word order information. Their ap-
proach was inspired by Word2Vec for learning word vectors and is commonly
known as doc2vec. It generates fixed-length feature representations from variable-
length pieces of text, making it useful for application to sentences, paragraphs, sec-
tions, or entire documents. The key to the approach is to associate every paragraph
5.4 Beyond Word Embeddings 231
with a unique paragraph vector ui , which is averaged with the word vectors wij of
the J words in the paragraph to yield a representation of the paragraph pi :
pi = ui + ∑ wij (5.75)
j=1,J
Note that the term paragraph can refer to a sentence or document as well. This
approach is termed a distributed memory model (DM) (Fig. 5.18). The paragraph
vector ui can be thought of acting as a memory that remembers word order context.
During training, a sliding window of context words C and the paragraph vector
pi are used to predict the next word in the paragraph context. Both paragraph vec-
tors and word vectors are trained via backpropagation. While the paragraph vector
is unique to each paragraph and shared across all contexts generated from the same
paragraph, the word vectors are shared across the entire corpus. It is notable that the
DM architecture resembles the CBOW architecture of word2vec, except with the
added paragraph context vector. Le and Mikolov also presented an architecture they
called distributed bag-of-words (DBOW) which used only the paragraph context
vector to predict the words in the paragraph (Fig. 5.19). This simple model is anal-
ogous to the skip-gram version of word2vec, except the paragraph vector is used to
predict all the words paragraph instead of using the target word to predict the con-
text words. As in the skip-gram model, DBOW is very computationally and memory
efficient. Empirical results have shown that both DM and DBOW outperform bag-
of-words and bag-of-n-gram models for text representations. Furthermore, averag-
ing the DM and DBOW vector representations often yields the best performance
overall.
232 5 Distributed Representations
entities in knowledge graphs (Fig. 5.20). An RDF is a statement with three con-
stituent parts: a subject, predicate, and object. A collection of these can be used to
build a knowledge graph. RDF2Vec converts RDF graphs into a set of sequences
using graph walks/subtree graph kernels and then applies the word2vec algorithm
to map entities to latent representations. In the resulting embedding space, entities
that share a background concept are clustered close to each other, such that entities
such as “New York” are close to entities such as “city.”
5.4 Beyond Word Embeddings 233
in Fig. 5.18 embedded by the TransE method. For instance, the following transla-
tions hold:
v(teller) ≈ v(doctor)—v(Jill) + v(Jack)
v(Jill) ≈ v(Jack)—textbfv(Dover) + v(Erie)
With the ability to scale to large datasets, TransE and related methods [Bor+13] are
useful for both relation extraction and linked prediction as well as NLP tasks.
word embeddings such that lexically linked words have similar vector represen-
tations. This refinement method, commonly called retrofitting with semantic lexi-
cons, makes no assumptions on how these vector representations are learned and
is applicable across spectral and neural approaches. Given a vocabulary of words
(w1 , w2 , . . . , wn ), a set of semantic relations expressed as an undirected graph with
edges (wi , w j ), and a set of learned word vectors q̂i for each wi , the goal is to infer a
new set of word vectors qi such that they are close in distance to their counterparts in
q̂i and to adjacent vertices w j . With a Euclidean distance measure, this is equivalent
to minimizing the objective function:
n
J = ∑ αi ||qi − q̂i ||2 + ∑ βi j ||qi − q̂ j ||2 (5.78)
i=1 (i, j)∈E
Rather than assuming that embedding models map words to point vectors in la-
tent representation space, words can be mapped to continuous probability densities.
This gives rise to several interesting advantages, as they can inherently capture un-
certainty and asymmetry of semantic relationships between words.
5.4.6.1 Word2Gauss
One such approach is Word2Gauss, which maps each word w in the vocabulary
D and context word c in the dictionary C to a Gaussian distribution over a latent
embedding space. The vectors of this space are termed word types and the words
observed are word instances. Word2Gauss presents two methods for generating em-
beddings. The first way is to replace the notion of cosine distance for point vectors
in latent density space by inner product E between two Gaussian densities:
#
E(Pi , Pj ) = N(x; μi , Σi )N(x; μ j , Σ j )dx = N(0; μi − μ j , Σi + Σ j ) (5.80)
x∈Rn
where N(x; μi , Σi ) and N(x; μ j , Σ j ) are the densities of a target and context word.
This is a symmetric measure that is computationally efficient, but it cannot model
5.4 Beyond Word Embeddings 235
A recent approach that builds upon the notion of words embeddings as probability
densities takes a generative Bayesian approach. The Bayesian skip-gram model
(BSG) models each word representation in the form of a Bayesian model generated
from prior densities associated with each occurrence of a given word in the corpus.
By incorporating context, the BSG model can overcome the polysemy limitation
of Word2Gauss. In fact, it can potentially model an infinite set of continuous word
senses.
For a target word w and a set of context words c, the BSG model assumes a prior
distribution pθ (z|w) and posterior distribution qθ (z|c, w) in the form of a Gaussian
distribution:
with diagonal covariance matrices Σw and Σq and z are latent vectors drawn from
the prior (Fig. 5.22). The larger the covariance matrix Σq values, the more uncertain
the meaning of target word w in context c.
The BSG model aims to maximize the probability pθ (c|w) of words in a context
window given a target word w, which is analogous to the skip-gram model. It is
trained by taking a target word w, sampling its latent meaning z from the prior, and
drawing context words c from pθ (c|z). The goal is to maximize the log-likelihood
function:
# C
log pθ (c|w) = log ∏ pθ (c j |z)pθ (z|w)dz (5.84)
j=1
236 5 Distributed Representations
where C is the context window size and c j are the context words for target word w.
For computational simplicity, the BSG model is trained by optimizing on the lower
bound of the log-likelihood, given by:
J(θ ) = ∑ DKL qφ ||N(z; μc̃k , Σc̃k ) − DKL qφ ||N(z; μc j , Σc j )
( j,k)
− DKL qφ ||pθ (z|w) (5.85)
where the sum is over pairs of positive c j and negative c̃k context words. The re-
sult of training are embeddings associated with the prior pθ that represent a word
type and associated with the posterior qθ that encode dynamic context. In compari-
son with Word2Gauss, BSG can provide better context-sensitivity, as in the case of
polysemy.
−1 ||u − v||2
d(u, v) = cosh 1+2 (5.86)
(1 − ||u||2 ) (1 − ||v||2 )
Note that as ||x|| approaches 1, the distance to other points grows exponentially. The
notion of straight lines in Euclidean space map to geodesics in Bd (Fig. 5.24). Such a
formulation allows the modeling of trees by placing the root node at/near the origin
and leaf nodes at the boundary. During training, the model learn representations Θ :
by minimizing the loss function L(θ ) using a negative sampling approach on a set
of data D = {(u, v)}:
e−d(u,v)
L (Θ ) = ∑ log
∑v ∈N(u) e−d(u,v )
(5.88)
(u,v)∈D
238 5 Distributed Representations
Fig. 5.24: Whereas shortest paths are straight lines in Euclidean space, they are
curved lines within a hyperbolic space and are called geodesics
5.5 Applications
provide simple examples of how word embeddings can be applied to NLP, while
leaving deep learning approaches to later chapters.
5.5.1 Classification
Text classification forms the basis of many important tasks in NLP. Traditional
linear-classifier bag-of-word approaches such as Naive Bayes or logistic regression
can perform well for text classification. However, they suffer from the inability to
generalize to words and phrases unseen in the training data. Embedding models
provide the ability to overcome this shortcoming. By leveraging pretrained em-
beddings—learning word representations on a separate large corpus—we can build
classifiers that generalize across text.
The FastText model proposed by Joulin et al. [Jou+16a] is an example of a text-
classification model that leverages word embeddings (Fig. 5.25). The first phase of
FastText learns word representations on a large corpus, in effect capturing the se-
mantic relationships on a wide range of vocabulary. These embeddings are then used
during the classifier training phase where words of a document are mapped to vec-
tors using these embeddings and these vectors are subsequently averaged together
to form a latent representation of the document. These latent representations and
their labels form the training set for a softmax or hierarchical softmax classifier.
FastText, as its name implies, is computationally efficient, and reportedly able to
train on a billion words in 10 min while achieving near state-of-the-art performance
[Jou+16a].
As noted previously, language models are strongly related to the training of em-
bedding models, given that both predict a target word given a set of context
words. An n-gram language model predicts a word wt given the previous words
wt−1 , wt−2 , . . . , wt−n . Training of an n-gram language model is equivalent to maxi-
mizing the negative log-likelihood:
T
J(θ ) = ∑ log p(wt |wt−1 , wt−2 , . . . , wt−n+1 ) (5.89)
t=1
So the language model predicts a target word based on the previous n context words,
while CBOW predicts a target word based on the n context words on each side.
Embedding methods excel and language modeling tasks and have led to deep
neural network approaches leading the state-of-the-art in performance [MH09].
5.5 Applications 241
A density-based approach could count the number of entities within a small neigh-
borhood of vd and flag the document if the count fell below a threshold T .
242 5 Distributed Representations
We start off by a detailed look into the word2vec algorithm and examine a python
implementation of the skip-gram model with negative sampling. Once the concepts
underpinning word2vec are examined, we will use the Gensim package to speed
up training time and investigate the translational properties of word embeddings.
We will examine GloVe embeddings as an alternative to word2vec. Both methods,
however, are unable to handle antonymy, polysemy, and word-sense disambigua-
tion. We consider document clustering by using an embeddings approach. Lastly,
we study how an embedding method like sense2vec can better handle word sense
disambiguation.
In this case study, we will be examining the inner operations of word2vec’s skip-
gram and negative sampling approach as well as GloVe embeddings with python.
We will also leverage the popular nltk, gensim, glove, and spaCy libraries for our
analysis. NLTK is a popular open-source toolkit for natural language processing
and text analytics. The gensim library is an open-source toolkit for vector space
modeling and topic modeling implemented in Python with Cython performance ac-
celeration. The glove library is an efficient open-source implementation of GloVe in
python. SpaCy is a fast open-source NLP library written in Python and Cython for
part-of-speech tagging and named entity recognition.
For our analysis, we will leverage the Open American National Corpus, which
consists of roughly 15 million spoken and written words from a variety of sources.
Specifically, we will be using the subcorpus which consists of 4531 Slate magazine
articles from 1996 to 2000 (approximately 4.2 million words).
Let’s take a look at some basic statistics on this dataset, such as document length
and sentence length (Figs. 5.28 and 5.29). By examining word-frequency by looking
at the top 1000 terms in this corpus (Fig. 5.30), we see that the top 100 terms are
what we typically consider stop-words (Table 5.1). They are common across most
sentences and do not capture much, if any, semantic meaning. As we move further
down the list, we start to see words that play a more important role in conveying the
meaning within a sentence or document.
244 5 Distributed Representations
document length
500
400
300
200
100
0
0 1000 2000 3000 4000
sentence length
14000
12000
10000
8000
6000
4000
2000
0
0 20 40 60 80 100
Our goal is to train a set of word embeddings for the corpus above. Let’s build a
skip-gram model with negative sampling, followed by a GloVe model. Before we
train either model, we see that there are 77,440 unique words in the preprocessed
4.86 million word corpus.
5.6 Case Study 245
Word Frequency
0 the 266,007
1 of 115,973
2 – 114,156
3 to 107,951
4 a 100,993
5 and 96,375
6 in 74,561
7 that 64,448
8 is 51,590
9 it 38,175
.. .. ..
. . .
990 Eyes 500
991 Troops 499
992 Raise 499
993 Pundits 499
994 Calling 498
995 de 498
996 Sports 498
997 Strategy 497
998 Numbers 496
999 Argues 496
frequency
250000
200000
150000
100000
50000
0
0 50 100 150 200 250 300 350 400
5.6.3.1 Word2Vec
We are now ready to train the neural network of the word2vec model. Let’s define
our model parameters:
• dim = dimension of the word vectors
• win = context window size (number of tokens)
• start alpha = starting learning rate
• neg = number of samples for negative sampling
• min count = minimum mentions to be included in vocabulary
We can reduce the size of this vocabulary by filtering out rare words. If we apply a
minimum count threshold of 5 mentions in the corpus, we find that our vocabulary
size drops down to 31,599, such that 45,842 words will be considered OOV. We will
be mapping all of these words to a special out-of-vocabulary token.
1 truncated = []
2 t r u n c a t e d . a p p e n d ( VocabWord ( ’<unk > ’ ) )
3 unk hash = 0
4
5 count unk = 0
6 for token in vocab items :
7 i f token . count < min count :
8 c o u n t u n k += 1
9 t r u n c a t e d [ u n k h a s h ] . c o u n t += t o k e n . c o u n t
10 else :
11 t r u n c a t e d . append ( token )
12
13 t r u n c a t e d . s o r t ( key = lambda t o k e n : t o k e n . c o u n t , r e v e r s e = T r u e )
14
15 v o c a b h a s h = {}
16 for i , token in enumerate ( t r u n c a t e d ) :
17 v o c a b h a s h [ t o k e n . word ] = i
18
To speed up training, let’s create a negative sampling lookup table that we will use
during training.
1 power = 0 . 7 5
2 norm = sum ( [ math . pow ( t . c o u n t , power ) f o r t i n v o c a b i t e m s ] )
3
4 t a b l e s i z e = i n t ( 1 e8 )
5.6 Case Study 247
5 t a b l e = np . z e r o s ( t a b l e s i z e , d t y p e =np . i n t )
6
7 p = 0
8 i = 0
9 f o r j , unigram in enumerate ( vocab items ) :
10 p += f l o a t ( math . pow ( u n i g r a m . c o u n t , power ) ) / norm
11 w h i l e i < t a b l e s i z e and f l o a t ( i ) / t a b l e s i z e < p :
12 table [ i ] = j
13 i += 1
14
We are now ready to train the word2vec model. The approach is to train a two-
layer (syn0, syn1) neural network by iterating over the sentences in the corpus and
adjusting lawyer weights to maximize the probabilities of context words given a
target word (skip-gram) with negative sampling. After completion, the weights of
the hidden layer syn0 are the word embeddings that we seek.
5 tmp = np . z e r o s ( s h a p e = ( v o c a b s i z e , dim ) )
6 s y n 1 = np . c t y p e s l i b . a s c t y p e s ( tmp )
7 s y n 1 = np . a r r a y ( s y n 1 )
8
9 current sent = 0
10 t r u n c a t e d v o c a b u l a r y = [ x . word f o r x i n v o c a b i t e m s ]
11 corpus = df [ ’ t e x t ’ ] . t o l i s t ( )
12
40 embedding = d i c t ( z i p ( t r u n c a t e d v o c a b u l a r y , s y n 0 ) )
The semantic translation properties of these embeddings are noteworthy. Let’s ex-
amine the cosine similarity between two similar words (man, woman) and two dis-
similar words (candy, social). We would expect the similar words to exhibit higher
similarity.
• dist(man, woman) = 0.01258108
• dist(candy, social) = 0.05319491
We can visualize the word embeddings using the T-SNE algorithm to map the em-
beddings to 2D space. Note that T-SNE is a dimensionality reduction technique that
preserves notions of proximity within a vector space (points close together in 2D are
close in proximity in higher dimensions). The figure below shows the relationships
of a 300-word sample from the vocabulary (Fig. 5.31).
The python code above is useful for understanding principles, but is not the fastest
to run. The original word2vec package was written in C++ to facilitate rapid training
speed over multiple cores. The gensim package provides an API to the word2vec li-
brary, as well as several useful methods to examine vectors neighborhoods. Let’s see
how we can use gensim to train on the sample data corpus. Gensim expects us to pro-
vide a set of documents as a list of list of tokens. We will call the simple preprocess()
method of gensim to remove punctuation, special and uppercase characters. With
the wrapper API provided by the gensim package, training word2vec is as simple as
defining a model and passing the set of training documents.
5.6 Case Study 249
1 documents = [ gensim . u t i l s . s i m p l e p r e p r o c e s s ( df [ ’ t e x t ’ ] . i l o c [ i
] ) for i in range ( len ( df ) ) ]
2 model = g e n s i m . m o d e l s . Word2Vec ( docume nt s ,
3 s i z e =100 ,
4 window =10 ,
5 m i n c o u n t =2 ,
6 workers =10)
7 model . t r a i n ( documents , t o t a l e x a m p l e s = l e n ( d o c u m e n t s ) , e p o c h s
=10)
5.6.3.6 Similarity
Let’s assess the quality of the learned word embeddings by examining word neigh-
borhoods. If we look at the most similar words to “man” or “book’,’ we find highly
similar words in their neighborhoods. So far so good.
1 model . wv . m o s t s i m i l a r ( ”man” , t o p n = 5 )
2 [ ( ’ guy ’ , 0 . 6 8 8 0 4 6 3 3 6 1 7 4 0 1 1 2 ) ,
3 ( ’woman ’ , 0 . 6 3 0 1 9 3 5 3 1 5 1 3 2 1 4 1 ) ,
4 ( ’ person ’ , 0.6296881437301636) ,
5 ( ’ s o l d i e r ’ , 0.5808842182159424) ,
6 ( ’ someone ’ , 0 . 5 5 5 2 0 1 1 7 2 8 2 8 6 7 4 3 ) ]
250 5 Distributed Representations
1 model . wv . m o s t s i m i l a r ( ” book ” , t o p n = 5 )
2 [ ( ’ books ’ , 0.7232613563537598) ,
3 ( ’ novel ’ , 0.6448987126350403) ,
4 ( ’ biography ’ , 0.6039375066757202) ,
5 ( ’ memoir ’ , 0 . 6 0 1 0 3 2 1 3 7 8 7 0 7 8 8 6 ) ,
6 ( ’ c h a p t e r ’ , 0.5646576881408691) ]
Let’s look at some polysemous words. The similar words to the word “bass” reflect
the music definition of bass. That is, they only capture a single word sense (there
are no words related to the aquatic definition of bass). Similarly, words similar to
“bank” all reflect its financial word sense, but no seashores or riverbeds. This is one
of the major shortcomings of word2vec.
1 model . wv . m o s t s i m i l a r ( ” b a s s ” , t o p n = 5 )
2 [ ( ’ g u i t a r ’ , 0.6996911764144897) ,
3 ( ’ solo ’ , 0.6786242723464966) ,
4 ( ’ b l a z e r ’ , 0.6665750741958618) ,
5 ( ’ r o a r s ’ , 0.6658747792243958) ,
6 ( ’ corduroy ’ , 0.6525936126708984) ]
1 model . wv . m o s t s i m i l a r ( ” bank ” , t o p n = 5 )
2 [ ( ’ banks ’ , 0.6580432653427124) ,
3 ( ’ bankers ’ , 0.5862468481063843) ,
4 ( ’ imf ’ , 0.5782995223999023) ,
5 ( ’ r e s e r v e s ’ , 0.5546875) ,
6 ( ’ loans ’ , 0.5457302331924438) ]
We can examine the semantic translation properties in more detail with some vec-
tor algebra. If we start with the word “son” and subtract “man” and add “woman,”
we indeed find that “daughter” is the closest word to the resulting sum. Similarly,
if we invert the operation and start with the word “daughter” and subtract “woman”
and add “man,” we find that “son” is closest to the sum. Note that reciprocity is not
guaranteed with word2vec.
1 model . wv . s i m i l a r b y v e c t o r ( model . wv [ ’ s o n ’ ]− model . wv [ ’ man ’ ]
2 + model . wv [ ’woman ’ ] ,
topn =5)
3 [ ( ’ daughter ’ , 0.7489624619483948) ,
4 ( ’ s i s t e r ’ , 0.7321654558181763) ,
5 ( ’ mother ’ , 0.7243343591690063) ,
6 ( ’ b o y f r i e n d ’ , 0.7229076623916626) ,
7 ( ’ l o v e r ’ , 0.7120637893676758) ]
1 model . wv . s i m i l a r b y v e c t o r ( model . wv [ ’ d a u g h t e r ’ ]
2 −model . wv [ ’woman ’ ]
3 + model . wv [ ’ man ’ ] , t o p n = 5 )
4 [ ( ’ son ’ , 0.7144862413406372) ,
5 ( ’ daughter ’ , 0.6668421030044556) ,
6 ( ’ man ’ , 0 . 6 6 5 2 4 9 9 4 3 7 3 3 2 1 5 3 ) ,
7 ( ’ g r a n d f a t h e r ’ , 0.5896619558334351) ,
5.6 Case Study 251
8 ( ’ f a t h e r ’ , 0.585667073726654) ]
We can also see that word2vec captures geographic similarities by taking the word
“paris,”subtracting “france” and adding “russia.” There resulting sum is close to
what we expect—“moscow.”
1 model . wv . s i m i l a r b y v e c t o r ( model . wv [ ’ p a r i s ’ ]
2 −model . wv [ ’ f r a n c e ’ ]
3 + model . wv [ ’ r u s s i a ’ ] , t o p n = 5 )
4 [ ( ’ r u s s i a ’ , 0.7788714170455933) ,
5 ( ’ moscow ’ , 0 . 6 2 6 9 0 5 3 2 2 0 7 4 8 9 0 1 ) ,
6 ( ’ b r a z i l ’ , 0.6154285669326782) ,
7 ( ’ japan ’ , 0.592476487159729) ,
8 ( ’ gazeta ’ , 0.5799405574798584) ]
1 model . wv . m o s t s i m i l a r ( ” l a r g e ” , t o p n = 5 )
2 [ ( ’ small ’ , 0.726446270942688) ,
3 ( ’ enormous ’ , 0 . 5 4 3 9 9 3 4 1 3 4 4 8 3 3 3 7 ) ,
4 ( ’ huge ’ , 0 . 5 0 7 0 8 8 7 2 0 7 9 8 4 9 2 4 ) ,
5 ( ’ vast ’ , 0.5017688870429993) ,
6 ( ’ size ’ , 0.48968151211738586) ]
1 model . wv . m o s t s i m i l a r ( ” h a r d ” , t o p n = 5 )
2 [ ( ’ easy ’ , 0.6564798355102539) ,
3 ( ’ d i f f i c u l t ’ , 0.6085934638977051) ,
4 ( ’ tempting ’ , 0.5201482772827148) ,
5 ( ’ impossible ’ , 0.5099537372589111) ,
6 ( ’ e a s i e r ’ , 0.4868208169937134) ]
Whereas word2vec captures the local context of words within sentences, GloVe
embeddings can additionally account for global context across the corpus. Let’s
take a deeper dive on how to calculate GloVe embeddings. We begin by building a
vocabulary dictionary from the corpus.
1 from c o l l e c t i o n s i m p o r t C o u n t e r
2
Let’s build the word co-occurrence matrix from the corpus. Note that word occur-
rences go both ways, from the main word to context, and vice versa. For smaller
values of the context window, this matrix is expected to be sparse.
1 # B u i l d co−o c c u r r e n c e m a t r i x
2 from s c i p y i m p o r t s p a r s e
3
4 m i n c o u n t = 10
5 window size = 5
6
7 v o c a b s i z e = l e n ( vocab )
8 i d 2 w o r d = d i c t ( ( i , word ) f o r word , ( i , ) i n vocab . i t e m s ( ) )
9 occurrence = sparse . l i l m a t r i x ( ( vocab size , vocab size ) , dtype=
np . f l o a t 6 4 )
10
15 for c e n t e r i , c e n t e r i d in enumerate ( t o k e n i d s ) :
16 c o n t e x t i d s = t o k e n i d s [ max ( 0 , c e n t e r i −−−w i n d o w s i z e ) :
center i ]
17 contexts len = len ( context ids )
18
19 for l e f t i , l e f t i d in enumerate ( c o n t e x t i d s ) :
20 d i s t a n c e = c o n t e x t s l e n −−− l e f t i
21 increment = 1.0 / f l o a t ( distance )
22 o c c u r r e n c e [ c e n t e r i d , l e f t i d ] += i n c r e m e n t
23 o c c u r r e n c e [ l e f t i d , c e n t e r i d ] += i n c r e m e n t
24 i f i % 10000 == 0 :
25 p r i n t ( ” P r o c e s s i n g s e n t e n c e %d ” % i )
26
27 d e f o c c u r m a t r i x ( vocab , c o c c u r r e n c e , m i n c o u n t ) :
28 f o r i , ( row , d a t a ) i n e n u m e r a t e ( z i p ( c o c c u r r e n c e . rows ,
coccurrence . data ) ) :
29 i f m i n c o u n t i s n o t None and v o c a b [ i d 2 w o r d [ i ] ] [ 1 ] <
min count :
30 continue
31 f o r d a t a i d x , j i n e n u m e r a t e ( row ) :
32 i f m i n c o u n t i s n o t None and v o c a b [ i d 2 w o r d [ j ] ] [ 1 ] <
min count
33 : continue
34 yield i , j , data [ data idx ]
5.6 Case Study 253
We can now train the embeddings by iterating over the documents (sentences) in the
corpus.
1 from random i m p o r t s h u f f l e
2 from math i m p o r t l o g
3 import pickle
4
5 i t e r a t i o n s = 30
6 dim = 100
7 l e a r n i n g r a t e = 0.05
8 x max = 100
9 alpha = 0.75
10
11 v o c a b s i z e = l e n ( vocab )
12 W = ( np . random . r a n d ( v o c a b s i z e ∗ 2 , dim ) −−−0.5) / f l o a t ( dim + 1 )
13 b i a s e s = ( np . random . r a n d ( v o c a b s i z e ∗ 2 ) −−−0.5) / f l o a t ( dim + 1 )
14
15 g r a d i e n t s q u a r e d = np . o n e s ( ( v o c a b s i z e ∗ 2 , dim ) , d t y p e =np .
float64 )
16 g r a d i e n t s q u a r e d b i a s e s = np . o n e s ( v o c a b s i z e ∗ 2 , d t y p e =np .
float64 )
17
18 d a t a = [ (W[ i m a i n ] , W[ i c o n t e x t + v o c a b s i z e ] ,
19 biases [ i main : i main + 1] ,
20 biases [ i context + vocab size : i context +
vocab size + 1] ,
21 gradient squared [ i main ] , gradient squared [
i context + vocab size ] ,
22 gradient squared biases [ i main : i main + 1] ,
23 gradient squared biases [ i context + vocab size
24 : i context + vocab size
+ 1] ,
25 cooccurrence )
26 f o r i main , i c o n t e x t , cooccurrence in comatrix ]
27
28 for i in range ( i t e r a t i o n s ) :
29 global cost = 0
30 shuffle ( data )
31 f o r ( v main , v c o n t e x t , b main , b c o n t e x t , gradsq W main ,
gradsq W context ,
32 gradsq b main , gradsq b context , cooccurrence ) in
data :
33
34 w e i g h t = ( c o o c c u r r e n c e / x max ) ∗∗ a l p h a i f
c o o c c u r r e n c e < x max e l s e 1
35
36 c o s t i n n e r = ( v main . dot ( v c o n t e x t )
37 + b main [ 0 ] + b c o n t e x t [ 0 ]
38 −−−l o g ( c o o c c u r r e n c e ) )
39 c o s t = w e i g h t ∗ ( c o s t i n n e r ∗∗ 2 )
40 g l o b a l c o s t += 0 . 5 ∗ c o s t
41
254 5 Distributed Representations
47 v m a i n −= ( l e a r n i n g r a t e ∗ g r a d m a i n / np . s q r t (
gradsq W main ) )
48 v c o n t e x t −= ( l e a r n i n g r a t e ∗ g r a d c o n t e x t / np . s q r t (
gradsq W context ) )
49
50 b m a i n −= ( l e a r n i n g r a t e ∗ g r a d b i a s m a i n / np . s q r t (
gradsq b main ) )
51 b c o n t e x t −= ( l e a r n i n g r a t e ∗ g r a d b i a s c o n t e x t / np .
sqrt (
52 gradsq b context ) )
53
54 g r a d s q W m a i n += np . s q u a r e ( g r a d m a i n )
55 gradsq W context += np . s q u a r e ( g r a d c o n t e x t )
56 g r a d s q b m a i n += g r a d b i a s m a i n ∗∗ 2
57 gradsq b context += g r a d b i a s c o n t e x t ∗∗ 2
The learned weight matrix consists of two sets of vectors, one if the word is in the
main word position and one for the context word position. We will average them to
generate the final GloVe embeddings for each word.
3 v o c a b s i z e = i n t ( l e n (W) / 2 )
4 f o r i , row i n e n u m e r a t e (W[ : v o c a b s i z e ] ) :
5 merged = m e r g e f u n ( row , W[ i + v o c a b s i z e ] )
6 merged / = np . l i n a l g . norm ( merged )
7 W[ i , : ] = merged
8
9 r e t u r n W[ : v o c a b s i z e ]
10
11 embedding = m e r g e v e c t o r s (W)
Let’s examine the translational properties of these vectors. We define a simple func-
tion that returns the 5 most similar words to the word “man.”
Interestingly, the similarity results fall into two categories. Whereas “woman” and
“girl” have similar semantic meaning to “man,” the words “dead” and “young” do
not. But these words do co-occur often, with phrases such as “young man” or “dead
man.” GloVe embeddings can capture both contexts. We can see this when we visu-
alize the embeddings using T-SNE (Fig. 5.32).
While useful, our python implementation is too slow to run with a large corpus. The
glove library is a python package that implements the GloVe algorithm efficiently.
Let’s retrain our embeddings using the glove package.
1 from g l o v e i m p o r t Corpus , Glove
2
3 c o r p u s = Corpus ( )
4 c o r p u s . f i t ( documents , window = 5 )
5
6 g l o v e = Glove ( n o c o m p o n e n t s =100 , l e a r n i n g r a t e = 0 . 0 5 )
7 g l o v e . f i t ( c o r p u s . m a t r i x , e p o c h s =30 , n o t h r e a d s =4 , v e r b o s e = T r u e )
8 glove . a d d d i c t i o n a r y ( corpus . d i c t i o n a r y )
256 5 Distributed Representations
1 g l o v e . m o s t s i m i l a r ( ’ man ’ , number = 6 )
2 [ ( ’ woman ’ , 0 . 9 4 1 7 1 5 5 1 4 2 1 7 6 4 3 1 ) ,
3 ( ’ young ’ , 0 . 8 5 4 1 7 5 2 2 5 2 2 4 3 2 0 2 ) ,
4 ( ’ guy ’ , 0 . 8 1 3 8 9 2 0 6 3 4 1 8 8 7 8 1 ) ,
5 ( ’ person ’ , 0.8044470112897205) ,
6 ( ’ g i r l ’ , 0.793038798219135) ]
1 g l o v e . m o s t s i m i l a r ( ’ n i c e ’ , number = 6 )
2 [ ( ’ guy ’ , 0 . 7 5 8 3 1 5 0 8 0 9 8 9 9 1 9 4 ) ,
3 ( ’ very ’ , 0.7071106359169386) ,
4 ( ’ seems ’ , 0 . 7 0 4 8 2 1 1 0 9 2 7 3 7 8 0 7 ) ,
5 ( ’ t e r r i b l e ’ , 0.697033427158236) ,
6 ( ’ fun ’ , 0 . 6 8 9 8 1 1 1 3 0 3 1 9 4 3 0 8 ) ]
1 g l o v e . m o s t s i m i l a r ( ’ a p p l e ’ , number = 6 )
2 [ ( ’ i n d u s t r y ’ , 0.6965166116455955) ,
3 ( ’ employee ’ , 0 . 6 7 2 4 0 6 4 7 9 7 6 7 2 1 7 8 ) ,
4 ( ’ fbi ’ , 0.6280345651329606) ,
5 ( ’ gambling ’ , 0.6276268857034702) ,
6 ( ’ indian ’ , 0.6266591982382662) ]
Once again, the most similar words exhibit both semantic similarity and high co-
occurrence probability. Even with the additional context, GloVe embeddings still
lack the ability to handle antonyms and word sense disambiguation.
The use of word embeddings provides a useful and efficient means for document
clustering in comparison with traditional approaches such as LSA or LDA. The
simplest approach is a bag-of-words method where a document vector is created by
averaging the vectors of each of the words in the document. Let’s take our Slate
corpus and see what we can find with this approach.
We create a set of document vectors by adding the vectors of each word in the
document and dividing by the total number of words.
6 glove . a d d d i c t i o n a r y ( corpus . d i c t i o n a r y )
7 p r i n t ( ” Glove e m b e d d i n g s t r a i n e d . ” )
8
9 doc vectors = []
10 f o r doc i n d o c u m e n t s :
11 v e c = np . z e r o s ( ( dim , ) )
12 f o r t o k e n i n doc :
13 v e c += g l o v e . w o r d v e c t o r s [ g l o v e . d i c t i o n a r y [ t o k e n ] ]
14 i f l e n ( doc ) > 0 :
15 v e c = v e c / l e n ( doc )
16 d o c v e c t o r s . append ( vec )
17
18 p r i n t ( ” P r o c e s s e d documents = ” , l e n ( d o c v e c t o r s ) )
If we visualize these embeddings using T-SNE, we can see there are several pro-
nounced clusters (Fig. 5.33).
The sense2vec algorithm is an improved approach that can deal with polysemy or
antonymy through supervised disambiguation. Moreover, sense2vec is computation-
ally inexpensive and can be implemented as a preprocessing task prior to training a
word2vec or GloVe model. To see this, let’s apply the sense2vec algorithm to our
corpus by leveraging the spaCy library to generate part-of-speech labels that will
serve as our supervised disambiguation labels.
Let’s process the sentences in our corpus using the spaCy NLP annotations. We
create a separate corpus where each word is augmented by its part-of-speech label.
For instance, the word he is mapped to he PRON.
1 import spacy
2 n l p = s p a c y . l o a d ( ’ en ’ , d i s a b l e = [ ’ p a r s e r ’ , ’ n e r ’ ] )
3 corpus = df [ ’ t e x t ’ ] . t o l i s t ( )
4 p r i n t ( ” Number o f d o c s = ” , l e n ( c o r p u s ) )
5
6 docs = [ ]
7 count = 0
8 for item in corpus :
9 docs . append ( nlp ( item ) )
10 c o u n t += 1
11 i f c o u n t % 10000 == 0 :
12 p r i n t ( ” P r o c e s s e d document # ” , c o u n t )
13
14 s e n s e c o r p u s = [ [ x . t e x t +” ”+x . p o s f o r x in y ] f o r y in docs ]
With the new preprocessed corpus, we can proceed with training word2vec. We can
use this trained model to look at how words like “run” or “lie” can be disambiguated
based on their part-of-speech.
1 model . wv . m o s t s i m i l a r ( ” run NOUN ” , t o p n = 5 )
2 [ ( ’ runs NOUN ’ , 0 . 5 4 1 8 1 7 2 4 7 8 6 7 5 8 4 2 ) ,
3 ( ’ term NOUN ’ , 0 . 5 0 8 5 5 6 3 0 6 3 6 2 1 5 2 1 ) ,
4 ( ’ ropy VERB ’ , 0 . 5 0 2 7 1 1 4 1 5 2 9 0 8 3 2 5 ) ,
5 ( ’ distance NOUN ’ , 0 . 4 9 7 8 7 6 7 6 3 3 4 3 8 1 1 0 4 ) ,
6 ( ’ sosa NOUN ’ , 0 . 4 9 4 2 4 9 6 1 2 0 9 2 9 7 1 8 ) ]
References
[Boj+16] Piotr Bojanowski et al. “Enriching Word Vectors with Subword Infor-
mation”. In: CoRR abs/1607.04606 (2016).
[Bor+13] Antoine Bordes et al. “Translating Embeddings for Modeling Mul-
tirelational Data.” In: NIPS. 2013, pp. 2787–2795.
[CP18] José Camacho-Collados and Mohammad Taher Pilehvar. “From Word
to Sense Embeddings: A Survey on Vector Representations of Mean-
ing”. In: CoRR abs/1805.04032 (2018).
[Che+16] Ting Chen et al. “Entity Embedding-Based Anomaly Detection for
Heterogeneous Categorical Events.” In: IJCAI. IJCAI/AAAI Press,
2016, pp. 1396–1403.
[CW08] Ronan Collobert and Jason Weston. “A Unified Architecture for Nat-
ural Language Processing: Deep Neural Networks with Multitask
Learning”. In: Proceedings of the 25th International Conference on
Machine Learning. ACM, 2008, pp. 160–167.
[CJF16] Marta R. Costa-Jussà and José A. R. Fonollosa. “Character-based
Neural Machine Translation.” In: CoRR abs/1603.00810 (2016).
[Cou+16] Jocelyn Coulmance et al. “Trans-gram, Fast Cross-lingual Word em-
beddings”. In: CoRR abs/1601.02502 (2016).
[Dev+18] Jacob Devlin et al. “BERT: Pre-training of Deep Bidirectional Trans-
formers for Language Understanding.” In: CoRR abs/1810.04805
(2018).
[DFU11] Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. “Multiview
learning of word embeddings via cca”. In: In Proc. of NIPS. 2011.
[Dhi+18] Bhuwan Dhingra et al. “Embedding Text in Hyperbolic Spaces”.
In: Proceedings of the Twelfth Workshop on Graph-Based Methods
for Natural Language Processing (TextGraphs-12). Association for
Computational Linguistics, 2018, pp. 59–69.
[Far+14] Manaal Faruqui et al. Retrofitting Word Vectors to Semantic Lexicons.
2014.
[Gra+18] Edouard Grave et al. “Learning Word Vectors for 157 Languages”. In:
CoRR abs/1802.06893 (2018).
[Gu+16] Jiatao Gu et al. Incorporating Copying Mechanism in Sequence-to-
Sequence Learning. 2016.
[HR18] Jeremy Howard and Sebastian Ruder. “Universal Language Model
Fine-tuning for Text Classification”. In: Association for Computa-
tional Linguistics, 2018.
[Jou+16a] Armand Joulin et al. “Bag of Tricks for Efficient Text Classification”.
In: CoRR abs/1607.01759 (2016).
[Kan+17] Ramakrishnan Kannan et al. “Outlier Detection for Text Data: An
Extended Version.” In: CoRR abs/1701.01325 (2017).
[Kim+16] Yoon Kim et al. “Character-Aware Neural Language Models”. In:
AAAI. 2016.
[KB16] Anoop Kunchukuttan and Pushpak Bhattacharyya. “Learning variable
length units for SMT between related languages via Byte Pair Encod-
ing.” In: CoRR abs/1610.06510 (2016).
References 261
6.1 Introduction
In the last few years, convolutional neural networks (CNNs), along with recurrent
neural networks (RNNs), have become a basic building block in constructing com-
plex deep learning solutions for various NLP, speech, and time series tasks. LeCun
first introduced certain basic parts of the CNN frameworks as a general NN frame-
work to solve various high-dimensional data problems in computer vision, speech,
and time series [LB95]. ImageNet applied convolutions to recognize objects in im-
ages; by improving substantially on the state of the art, ImageNet revived interest in
deep learning and CNNs. Collobert et al. pioneered the application of CNNs to NLP
tasks, such as POS tagging, chunking, named entity resolution, and semantic role
labeling [CW08b]. Many changes to CNNs, from input representation, number of
layers, types of pooling, optimization techniques, and applications to various NLP
tasks have been active subjects of research in the last decade.
The initial sections of this chapter describe CNNs, starting with the basic op-
erations, and demonstrate how these networks address the reduction in parameters
while creating an inductive bias towards local patterns. Later sections derive the for-
ward and backward pass equations for the basic CNN. Applications of CNNs and
their adaptations to text inputs are introduced next. Classic CNN frameworks are
then presented, followed by modern frameworks, in order to provide readers with
examples of the diversity of ways in which CNNs are used in different domains.
Special attention is paid to popular applications of CNNs to various NLP tasks.
This chapter also describes specific algorithms that allow deep CNN frameworks
to run more efficiently on modern GPU-based hardware. To provide readers with
a practical, hands-on experience, the chapter concludes with a detailed case study
of airline tweet sentiment analysis using many of the discussed CNN frameworks
using Keras and TensorFlow for implementation. In this case study, readers are pro-
vided with a detailed exploratory data analysis, preprocessing, training, validation,
and evaluation, similar to what one can experience in a real-world project.
The next few sections introduce fundamental concepts and building blocks of CNNs.
Note that since CNNs originated in computer vision applications, many of the terms
and examples in their building blocks refer to images or two-dimensional (2d) ma-
trices. As the chapter continues, these will be mapped to one-dimensional (1d) text
input data.
Similarly in two dimensions, mostly used in computer vision with still images:
Many CNNs employ the cross-correlation operator, but the operation is called
convolution. We will use the terms synonymously, as the main idea of both is
to capture similarity in input signals. Many of the terms used in CNNs have
their roots in image processing.
In a basic NN, all the units of the input layer connect to all the units of the next layer.
This connectivity negatively impacts both computational efficiency and the ability to
266 6 Convolutional Neural Networks
capture certain local interactions. Consider an example shown in the Fig. 6.1, where
the input layer has m = 9 dimensions, and the hidden layer has n = 4 dimensions;
in a fully connected NN, as shown in Fig. 6.1a, there will be m × n = 36 connec-
tions (thus, weights) that the NN has to learn. On the other hand, if we allow only
k = 3 spatially proximal inputs to connect to a single unit of the hidden layer, as
shown in Fig. 6.1b, the number of connections reduces to n × k = 12. Another ad-
vantage of limiting the connectivity is that restricting the hidden layer connections
to spatial-proximal inputs forces the feed-forward system to learn local features
through backpropagation. We will refer to the matrix of dimension k as our filter,
or kernel. The spatial extent of the connectivity or the size of the filter (width and
height) is generally called the receptive field, due to its computer vision heritage.
In a 3d input space, the depth of the filter is always equal to the depth of the input,
but the width and height are the hyperparameters that can be obtained via search.
Note that the number, arrangement, and connections of the filters are hyperparame-
ters in a model. The depth of the CNN is the number of layers, whereas the number
of filters in a layer determines the “depth” of the subsequent layer. How the filters
get moved, that is, how many inputs get skipped before the next instance of the filter,
is known as the stride of the filter. For example, when the stride is 2, two inputs are
6.2 Basic Building Blocks of CNN 267
Fig. 6.1: Local connectivity and sparse interactions. (a) Fully connected layers. (b)
Locally connected layers
Fig. 6.2: Local connectivity and parameter sharing. (a) Locally connected layers.
(b) Locally connected layers with parameter sharing
268 6 Convolutional Neural Networks
Fig. 6.3: Spatial arrangement and relationship between hyperparameters. (a) Spatial
arrangement leading to N = 9−3+02 + 1 = 4 neurons in the next layer. (b) Spatial
9−3+2
arrangement leading to N = 2 + 1 = 5 neurons in the next layer
skipped before the next instance of the filter is connected to the subsequent layer.
The inputs can be zero-padded at the edges, so that filters can fit around the edge
units. The number of padding units is another hyperparameter for tuning. Figure 6.3
illustrates different paddings between layers. Adding paddings to the edges with
values of 0 is called zero-padding; the convolution performed with zero-padding is
called wide convolution, and one without is called narrow convolution.
The relationship between the number of inputs (input volume) W , receptive field
(filter size) F, stride size S, and the zero-padding P leads to the number of neurons
in the next layer, as provided in Eq. (6.9).
W − F + 2P
N= +1 (6.9)
S
6.2 Basic Building Blocks of CNN 269
Figure 6.4 illustrates how a 2d filter convolves with the input generating the final
output for that layer, with the convolution steps broken down. Figure 6.4 visual-
izes all of the above spatial arrangements and illustrates how changing the padding
affects the number of neurons according to the equation above.
Figure 6.5 extends the convolution process to a 3d input with three channels
(similar to RGB channels in an image) and two filters, showing how the linear con-
volution reduces the volume between layers. As shown in Fig. 6.6, these filters act
as a complex feature detection mechanism. Given a large volume of training data, a
CNN can learn filters, such as horizontal edge, vertical edge, outline, and more in
the classification process.
The number of channels in the filters should match the number of channels in
its input.
is the most popular function because of its easy computation and simple differen-
tiation. ReLU has also been shown to lead to better generalization when used in
CNNs.
The outputs of one layer can be further downsampled to capture summary statistics
of local neurons or sub-regions. This process is called pooling, subsampling or
downsampling based on the context. Usually, the outputs from the detector stage
become the inputs for the pooling layer.
Pooling has many useful effects, such as reduction of overfitting and reduction of
the number of parameters. Specific types of pooling can also result in invariance.
Invariance allows identification of a feature irrespective of its precise location and
is an essential property in classification. For example, in a face detection problem,
the presence of features that indicate an eye is not only essential but it also exists
irrespective of its location in an image.
There are different pooling methods, each with its own benefits, and the particular
choice depends on the task at hand. When the bias of the pooling method matches
the assumptions made in a particular CNN application, such as in the example of
face detection, one can expect significant improvement in the results. Some of the
more popular pooling methods are listed below.
As the name suggests, the max pooling operation chooses the maximum value of
neurons from its inputs and thus contributes to the invariance property discussed
above. This is illustrated in Fig. 6.7. Formally, for a 2d output from the detection
stage, a max pooling layer performs the transformation:
where p and q denote the coordinates of the neuron in its local neighborhood and l
represents the layer. In k-max pooling, k values are returned instead of a single value
in the max pooling operation.
In average or mean pooling, the local neighborhood neuron values are averaged to
give the output value, as illustrated in Fig. 6.7. Formally, average pooling performs
the transformation:
272 6 Convolutional Neural Networks
1
m2 ∑
hli, j = l−1
hi+p, j+q (6.11)
p,q
There are indeed many variants of pooling, such as k-max pooling, dynamic pool-
ing, dynamic k-max pooling, and others.
Fig. 6.7: Examples of pooling operations. (a) Max pooling. (b) Average pooling
In stochastic pooling, instead of picking the maximum, the picked neuron is drawn
from a multinomial distribution. Stochastic pooling works similar to dropout in ad-
dressing the issue of the overfitting [ZF13b].
In spectral pooling, the spatial input is transformed into a frequency domain through
a discrete Fourier transform (DFT) to capture important signals in the lower dimen-
sion. For example, if the input is x ∈ Rm×m and has to be reduced to the size h × w, a
DFT operation is performed on the input, so that the frequency representation main-
tains the central h × w submatrix; then, an inverse DFT is performed to transform
6.3 Forward and Backpropagation in CNN 273
back into the spatial domain [RSA15]. This transformation has the effect of dimen-
sionality reduction on the space and can be very effective in certain applications.
Now that all basic components are covered, they will be connected. This section
will also go through the step-by-step process to clearly understand the different
operations involved in the forward and the backward pass in a CNN.
In the interest of clarity, we will consider a basic CNN block consisting of one
convolutional layer, a nonlinear activation function, such as ReLU that performs a
non-linear transformation, and a pooling layer. In real-world applications, multiple
blocks like this are stacked together to form network layers. The output of these
blocks is then flattened out and connected to a fully connected output layer. The
flattening out process converts a multidimensional tensor to a mono-dimensional
vector, for example a three-dimensional (W, H, N) to a vector of dimension d =
W × H × N.
Let us start the derivation for the layer l, which is the output of convolution on
layer l − 1. Layer l has height h, width w, and channels c. Let us assume that there is
one channel, i.e. c = 1, and has iterators i, j for the input dimensions. We will con-
sider the filter k1 × k2 dimensions with iterators m, n for the convolution operation.
The weight matrix with weights Wm,n l and the bias bl transforms the previous layer
l − 1 into the layer l via the convolution operation. The convolution layer is fol-
lowed by a non-linear activation function, such as ReLU f (·). The output for layer
l is denoted by Oli, j .
Thus forward propagation for layer l can be written as:
Xi,l j = ∑ ∑ Wm,n
l l−1
Oi+m, j+n + b
l
(6.14)
m n
and
∂E
6.3.1 Gradient with Respect to Weights ∂W
We will first consider the impact of single pixel (m , n ) of the kernel, given by Wm ,n
on the error E, using the chain rule:
∂ E ∂ Xi, j
h−k1 w−k2 l
∂E
∂W l
= ∑ ∑ ∂ Xi,l j ∂ W l
(6.16)
m ,n i=0 j=0 m ,n
If we consider (δi,l j ) as the gradient error for the layer l w.r.t input, the above
equation can be rewritten as:
Writing the change in output in terms of the inputs (the previous layer) we get:
∂ Xi,l j ∂
∂W l
= ∑ ∑W O l l−1
∂ W l m n m,n i+m, j+n
+b l
(6.18)
m ,n m ,n
When we take partial derivatives with respect to Wm ,n , all values become zero
except for the components mapping to m = m and n = n .
∂ Xi,l j ∂
= W0,0 Oi+0, j+0 + · · · +Wm ,n O
l l−1 l l−1
+···+b
l
(6.19)
∂W l ∂W l i+m , j+n
m ,n m ,n
∂ Xi,l j ∂
= Wml ,n Ol−1 (6.20)
∂W l ∂W l
i+m , j+n
m ,n m ,n
∂ Xi,l j
= Ol−1 (6.21)
∂W l
i+m , j+n
m ,n
∂E
= rotate180 {δi,l j } × Ol−1
(6.23)
∂W l
m ,n
m ,n
∂E
6.3.2 Gradient with Respect to the Inputs ∂X
Next, we are interested in how a change to single input, given by Xi , j affects the
error E. Borrowing from computer vision, the input pixel Xi , j after convolution
affects a region bounded by top left (i + k1 − 1, j + k2 − 1) and bottom right (i , j ).
So, we obtain:
∂E k1 −1 k2 −1
∂E ∂ X l+1
i −m, j −n
∂ X l
= ∑ ∑ ∂ X l
(6.24)
m=0 n=0 ∂ Xi −m, j −n
l+1
i ,j i ,j
∂E k1 −1 k2 −1 ∂ X l+1
i −m, j −n
∂ X l
= ∑ ∑ δ l+1
i −m, j −n ∂ X l
(6.25)
i ,j m=0 n=0 i ,j
All partial derivatives result into zero except where m = m and n = n,
f (X l ) = f (X l ), and W l+1 l+1
= Wm,n in the relevant output regions.
i −m +m, j −n +n i ,j m ,n
∂ X l+1
i −m, j −n ∂
= ∂ X l
W l+1
f X
l
+ · · · +Wm,n
l+1 f X l
∂ X l i ,j
m ,n 0−m+m ,0−n+n i ,j
i ,j
+ · · · + bl+1 (6.28)
276 6 Convolutional Neural Networks
∂ X l+1
∂
i −m, j −n l+1
= Wm,n f Xil , j (6.29)
∂ X l ∂ X l
i ,j i ,j
∂ X l+1
i −m, j −n l+1
= Wm,n f Xil , j (6.30)
∂ X l
i ,j
∂E k1 −1 k2 −1
∂ X l
= ∑ ∑ δ l+1
i −m, j −n
l+1
Wm,n f Xil , j (6.31)
i ,j m=0 n=0
The term
k1 −1 k2 −1 l+1
∑m=0 ∑n=0 δi −m, j −nWm,n
l+1 can be seen as the flipped filter, or the filter ro-
As we saw in the pooling section, the pooling layer does not have any weights but
just reduces the size of the input based on the spatial operations performed. In the
forward pass, the pooling operation leads to conversion of a matrix or a vector to a
single scalar value.
In max pooling, the winner neuron or cell is remembered; when the backpropa-
gation needs to be done, the entire error is passed on to that winner neuron, as others
have not contributed. In average pooling, an n × n pooling block during backprop-
1
agation divides the total scalar value by n×n and distributes it equally across the
block.
Various NLP and NLU tasks in text mining use a CNN as the feature engineering
block. We will start with basic text classification to highlight some important analo-
gies between images in computer vision and mappings in text and the necessary
changes in the CNN process.
6.4 Text Inputs and CNNs 277
Let us assume all training data are in the form of sentences with labels and of
given maximum length s. The first transformation is to convert the sentences into a
vector representation. One way is to perform a lookup function for each word in the
sentence for its fixed-dimensional representation, such as Word Embeddings. Let us
assume that the lookup for word representation in a fixed vocabulary size V yields a
vector of fixed dimension and let that be d; thus, each vector can be mapped to Rd .
The rows of the matrix represent words of the sentences, and the columns can be the
fixed-length vector corresponding the representation. The sentence is a real matrix
X ∈ Rs×d .
The general hypothesis, especially in classification tasks, is that the words which
are local in the sequence, similar to n-grams, form complex higher-level features
when combined. This combination of local words in proximity is analogous to com-
puter vision, where local pixels can be combined to form features such as lines,
edges, and real objects. In computer vision with image representations, the con-
volution layer had filters smaller in size than the inputs performing convolution
operations via sliding across an image in patches. In text mining, the first layer of
convolution generally has filters of the same dimension d as the input but has vary-
ing height h, normally referred to as the filter size.
Figure 6.8 illustrates this on the sentence “The cat sat on the mat” which is to-
kenized into s = 6 words {T he, cat, sat, on,the, mat}. A lookup operation obtains a
3-dimensional word embeddings (d = 3) for each word. A single convolution filter
with height or size h = 2 starts producing the feature map. The output goes through a
non-linear activation of ReLU with 0.0 threshold that then feeds into a 1-max pool-
ing. Figure 6.8 illustrates the end state obtained by using the same shared kernel
across all the inputs, producing the single output at the end of the 1-max pooling.
Figure 6.9 gives a generalized methodology from sentences to output in a simple
CNN framework. In the real world, many representations of the input words can
exist similar to images having color channels in the computer vision field. Different
word vectors can map to the channels that may be static (i.e., pre-trained) using a
278 6 Convolutional Neural Networks
well-known corpus and do not change. They can also be dynamic, where even if
they were pre-trained, backpropagation can fine-tune it. There are various applica-
tions that use not only word embeddings for the representation but also POS tags
for the words or the position in the sequence. The application and the NLP task at
hand determine the particular representations.
Generally, outputs result in regions, and there are multiple of these varying in
size, generally from 2 to 10, i.e., sliding over 2–10 words. For each region size,
there can be multiple learned filters, given by n f . Similar to the image representation
s−h
calculations derived above, there are stride + 1 regions, where stride is the number
of words filters slide across. Thus, the output of the convolution layers are vectors
of dimensions Rs−h+1 for a stride of 1. Formally,
oi = W · X[i : i + h − 1, :] (6.33)
This equation represents how a filter matrix W of height h slides over the region
matrix given by X[i : i + h − 1, :] with unit stride. For each filter, the weights are
shared as in image-based frameworks giving the local feature extraction. The output
then flows into a non-linear activation function f (a), most commonly ReLU. The
output generates a feature map ci with bias b for each filter or region as in:
ci = f (oi ) + b (6.34)
The output vector of each of the feature maps goes through a pooling layer, as dis-
cussed before. The pooling layer provides the downsampling and invariance prop-
erty, as mentioned above. The pooling layer also helps in addressing the variable
length of words in sentences by yielding a reduced dimension ĉ for the entire vec-
tor. Since vectors represent words or sequences over time, max pooling in the text
domain is referred to as max pooling over time. How two different sizes of texts
get reduced to same-dimensional representation through max pooling over time is
shown in Fig. 6.10.
The output of the pooling layers is concatenated and passed on to the softmax
layer, which performs classification based on the number of categories or labels.
There are many hyperparameter choices when representing text and performing
CNN operations on text, such as type of word representation, choice of how many
word representations, strides, paddings, filter width, number of filters, activation
function, pooling size, pooling type, and number of CNN blocks before the final
softmax layer, to name a few.
The total number of hyperparameters for one simple block of CNN with output
for text processing can be given by:
6.4 Text Inputs and CNNs 279
Fig. 6.9: A simple text and CNN mapping. Sentence to word representations, map-
ping to the embeddings with different channels act as an input. Different filters for
each height, i.e., 2 filters each of sizes 2 and 3, shown in two shades, capture differ-
ent features that are then passed to non-linear processing to generate outputs. These
go through max pooling operation, which selects the maximum values from each
filter. The values are concatenated to form a final output vector
Now, we will provide a formal treatment from input to output for a simple CNN
in the text domain with word representations. For a variable-length sentence from
training data, with words of maximum length s having a similar lookup for word
embeddings of vector size d, we obtain an input vector in Rd . For all other sentences
which have fewer than s words, we can use padding with 0 or random values in the
inputs. As in the previous section, there can be many representations of these words,
such as static or dynamic, different embeddings for the words as word2vec, GloVe,
etc., and even different embeddings such as positional, tag-based (POS-tag), etc.,
all forming the input channels with the only constraint that they are all of the same
dimension.
The convolution filter can be seen as weights in 1d of the same length as the word
embedding but sliding over words of size h. This can create different word windows
W1:h ,W2:h ..Wn−h+1:n . These feature vectors are represented by [o1 , o2 , · · · , on−h+1 ]
in Rn−h+1 , which go through non-linear activation, such as ReLU. The output of
non-linear activation has the same dimension as the input and can be represented
as [c1 , c2 , · · · , cn−h+1 ] in Rn−h+1 . Finally, the outputs of the non-linear activation
280 6 Convolutional Neural Networks
Fig. 6.10: Two simple sentences with padding and 3-dimensional word embeddings,
going through same single filter of size or height 2, a ReLU with a threshold of 0,
and max pooling over time, result in the similar output value corresponding the
2-gram of “The cat”
function are further passed to a max pooling layer which finds a single scalar value
ĉ in R for each filter. In practice, there are multiple filters for each size h, and there
are multiple-sized filters for h varying from 2 to 10 in the general case. The output
layer connects all the max pooling layer outputs ĉ into a single vector and uses
the softmax function for classification. The considerations of padding sentences at
the beginning and the end, stride lengths, narrow or wide convolutions, and other
hyperparameters are as in the general convolution mapping process.
In many classification-related tasks, the vocabulary size grows large, and taking
into account unseen words in training even with embeddings results in suboptimal
performance. Work by Zhang et al. [ZZL15] uses character-level embeddings in-
stead of word-level embeddings in the training input to overcome such issues. The
researchers show that character embeddings result in open vocabulary, a way to
handle misspelled words, to name a few benefits. Figure 6.11 shows the designed
CNN to have many convolution blocks of 1d convolution layer, non-linear activa-
6.5 Classic CNN Architectures 281
tions, and k-max pooling, all stacked together to form a deep convolution layer. The
representation uses the well-known set of 70 alphanumeric characters with all low-
ercase letters, digits, and other characters using one-hot encoding. Combination of
characters with fixed size l = 1024 as a chunk of the input at a given time forms
the input for variable-length text. A total of 6 blocks of 2 sets of CNNs, each of
different size, the larger of 1024 dimensions and the smaller of 256 dimensions are
used. Two layers have filter height of 7, and the rest have height of 3, with 3-pooling
in the first few and strides of 1. The last 3 layers are fully connected. The final layer
is a soft-max layer for classification.
In this section, we will visit some of the standard CNN architectures. We will discuss
their structure in detail and will give some historical context for each. Though many
of them have been popular in the computer vision domain, they are still applicable
with variations in the text and speech domains, as well.
282 6 Convolutional Neural Networks
6.5.1 LeNet-5
LeCun et al. presented LeNet-5 as one of the first implementations of CNNs and
showed impressive results on the handwritten digit recognition problem [LeC+98].
Figure 6.12 shows the complete design of LeNet-5. LeCun demonstrated the concept
of decreasing the height and width via convolutions, increasing the filter/channel
size, and having fully connected layers with a cost function to propagate the errors,
which are now the backbone of all CNN frameworks. LeNet-5 used the MNIST
dataset of 60K training data for training and learning the weights. In all of our
discussions from now on, the representation of a layer will be given by nw × nh × nc ,
where nw , nh , nc are the width, height, and the number of channels/filters. Next, we
will give the details of the design regarding inputs, outputs, number of filters, and
pooling operations.
• The input layer uses only the grayscale pixel values and is 32 × 32 × 1 in size. It
is normalized to mean 0 and variance 1.
• A filter of 5 × 5 × 6 with no padding and stride s = 1 is used to create a layer with
size 28 × 28 × 6.
• This is followed by an average pooling layer with filter width f = 2 and stride
s = 2, resulting in a layer of size 14 × 14 × 6.
• Another convolution layer of 5 × 5 × 16 is applied with no padding and stride
s = 1 to create a layer of size 10 × 10 × 16.
• Another average pooling layer with filter width f = 2 and stride s = 2, resulting
in reduced height and width, yields a layer of size 5 × 5 × 16.
• This is then connected to a fully connected layer of size 120 and followed by
another fully connected layer of size 84.
• These 84 features are then fed to an output function which uses the Euclidean
radial basis function for determining which of the 10 digits are represented by
these features.
6.5.2 AlexNet
AlexNet, designed by Krizhevsky et al. [KSH12a], was the first deep learning archi-
tecture that won the ImageNet challenge in 2012 by a large margin (around 11.3%).
AlexNet was responsible in many ways for focusing attention on deep learning
research [KSH12a]. Its design is very similar to LeNet-5, but with more layers
and filters resulting in a larger network and more parameters to learn. The work
in [KSH12a] showed that with deep learning frameworks, features can be learned
instead of manually generated with deep domain understanding. The details of the
AlexNet designs are listed below (Fig. 6.13):
• Unlike LeNet-5, AlexNet used all the three channels of inputs. The size 227 ×
227 × 3 is convolved with 11 × 11 × 96 filter, stride of s = 4, and with non-
linearity performed by ReLU giving an output of 55 × 55 × 96.
• This goes through a max pooling layer with size 3×3 and stride s = 2 and reduces
the output to 27 × 27 × 96.
• This layer goes through a local response normalization (LRN) which effectively
normalizes the values across the depth of the channels and then another convo-
lution of size 5 × 5 × 256, stride s = 1, padding f = 2, with ReLU applied to get
an output of 27 × 27 × 256.
• This goes through a max pooling layer with size 3 × 3 and stride s = 2, reducing
the output to 13 × 13 × 256.
• This is followed by LRN, another convolution of size 3 × 3 × 384, stride s = 1,
padding f = 1, with ReLU applied to get an output of 13 × 13 × 384.
• This is followed by another convolution of size 3 × 3 × 384, stride s = 1, padding
f = 1, with ReLU applied to get an output of 13 × 13 × 384.
284 6 Convolutional Neural Networks
node and sent information to the other. Even with GPUs, the 90 training
epochs required 5–6 days of training on the 1.2 million training examples.
6.5.3 VGG-16
VGG-16, also known as VGGNet, by Simonyan and Zisserman, is known for its
uniform design and has been very successful in many domains [SZ14]. The unifor-
mity in having all the convolutions being 3 × 3 with stride s = 1 and max pooling
with 2× 2 along with the channels increasing from 64 to 512 in multiples of 2 makes
it very appealing and easy to set up. It has been shown that stacking convolutions of
3 × 3 with stride s = 1 in three layers is equivalent to 7 × 7 convolutions and with
a significantly reduced number of computations. VGG-16 has two fully connected
layers at the end with a softmax layer for classification. VGG-16’s only disadvan-
tage is the huge network with approximately 140 million parameters. Even with a
GPU setup, a long time would be required for training the model (Fig. 6.14).
We will discuss changes that have led to modern CNN architectures in different
domains, including in text mining.
286 6 Convolutional Neural Networks
The basic CNN mapping to sentences with a convolution filter of size k is shown
to be analogous to the ngram token detector in classic NLP settings. The idea of
a stacked or hierarchical CNN is to extend the principle by adding more layers.
In doing so, the receptive field size is increased, and larger windows of words or
contexts will be captured as features, as shown in Fig. 6.15.
If we consider (W, b) as the parameters corresponding to weights and biases,
⊕ as the concatenation operation done on sentence of length n with sequence of
the word embeddings e1:n each of d dimension, and k as the window size of the
convolution, the output is given by: c1:m
ci = f (⊕(wi:i+k−1 ) · W + b) (6.37)
c11:m1 = CONVk(W
1
1 ,b1 ) (w1:n )
c21:m2 = CONVk(W
2 1
2 ,b2 ) (c1:m1 )
(6.38)
···
p p k p−1
c1:m p
= CONV(W p ,b p ) (c1:m
p−1
)
6.6 Modern CNN Architectures 287
As the layers feed into each other, the effective window size or the receptive field
to capture the signal increases. For instance, if in sentiment classification there is a
sentence—The movie is not a very good one, a convolution filter with size 2 will not
capture the sequence “not a very good one” but a stacked CNN with the same size
2 will capture this in the higher layers.
In the stacked CNN, we assumed strides of size 1, but if we generalize the stride to
size s, we can then write the convolution operation as:
k,s
c1:m = CONV(W,b) (w1:n ) (6.39)
ci = f (⊕(w1+(i−1)s:(s+k)i ) · W + b) (6.40)
A dilated CNN can be seen as a special version of the stacked CNN. One way is
to have the stride size of each layer be k − 1 when the kernel size is k.
k,k−1
c1:m = CONV(W,b) (w1:n ) (6.41)
Fig. 6.16: A dilated CNN showing the increase in the receptive fields with dilations
changing to 1, 2, and 4
Inception networks by Szegedy et al. [Sze+17] are currently one of the best-
performing CNNs, especially in computer vision. The core of an inception network
is a repetitive inception block. An inception block uses many filters, such as 1 × 1,
3 × 3, and 5 × 5, as well as max pooling without the need to choose any one of
them. The central idea behind using 1 × 1 filters is to reduce the volume and hence
the computation before feeding it to a larger filter such as 3 × 3.
Figure 6.17 shows a sample 28×28×192 output from a previous layer convolved
with a 3 × 3 × 128 filter to give an output of 28 × 28 × 128; i.e., 128 filters at the
output result in about 174 million MACs. By having 1 × 1 × 96 filters to reduce the
volume and then convolve with 3 × 3 × 128 filters, the total computations reduce
to approximately 100 million MACs, almost a saving of 60%. Similarly, by using
1 × 1 × 16 filters preceding 5 × 5 × 32 filters, the total computations can be reduced
from approximately 120 to 12 million, a reduction by a factor of ten. The 1 × 1 filter
that reduces the volume is also called a bottleneck layer.
Figure 6.18 gives a pictorial view of a single inception block with a sample
2d input with width and height of 28 × 28 and depth of 192, producing an out-
put of width and height of 28 × 28 and depth of 256. The 1 × 1 filter plays a dual
role in reducing the volume for other filters, as well as producing the output. The
1 × 1 × 64 filter is used to produce an output of 28 × 28 × 64. The input, when
passed through the 1 × 1 × 96 filter and convolved with 3 × 3 × 128, generates a
28 × 28 × 128 output. The input, when passed through the 1 × 1 × 16 filter and
then through the 5 × 5 × 32 filter, produces a 28 × 28 × 32 output. Max pooling
with stride 1 and padding is used to produce an output of size 28 × 28 × 192. Since
the max pooling output has many channels, each channel passes through another
1 × 1 × 32 filter for reducing the volume. Thus, each filtering operation can occur
in parallel and generate an output that is concatenated together for a final size of
28 × 28 × 256.
6.6 Modern CNN Architectures 289
Fig. 6.17: Computational cost savings with having a 1 × 1 filter precede a 3 × 3 filter
In many NLP tasks, such as in sentiment analysis, features using syntactic ele-
ments and other structural information in the language have yielded improved per-
formance. Figure 6.19 shows a syntactic and semantic representation connected
290 6 Convolutional Neural Networks
Fig. 6.20: A tree-based structured CNN that can capture syntactic and semantic
dependencies
such as answer selection, paraphrase identification, and text entailment. The twin
networks, each consisting of CNN and pooling blocks, process one of the two sen-
tences and the final layer solves the sentence pair task. The Input layer has word em-
beddings from word2vec concatenated for the words in each sentence. Each block
uses wide convolutions so that all words in the sentence can provide signals as com-
pared to a narrow convolution. The tanh activation function tanh(Wxi + b) is used
as the nonlinear transformation. Next, an average pooling operation is performed on
each. Finally, the output of both average pooling layers is concatenated and passed
to a logistic regression function for binary classification.
Fig. 6.22: A basic bi-CNN by [Yin+16a] for sentence pair tasks using wide convo-
lutions, average pooling, and logistic regression for binary classification
In this section, we will discuss some of the applications of CNN in various text min-
ing tasks. Our goal is to summarize this research and provide insights into popular
CNNs and modern designs. CNNs with various modifications and even combina-
tions with other frameworks, such as LSTMs, have indeed been used in different
NLP tasks. Since a CNN by itself can capture local features and combinations of
these through further combinations, CNNs have been primarily used in text/doc-
ument classification, text categorization, and sentiment classification tasks. CNNs
lose the order of the sequences, though they have been used in sequence-based tasks,
such as POS tagging, NER, chunking, and more. To be truly effective in such set-
6.7 Applications of CNN in NLP 293
tings, they either need to be combined with other frameworks or have positional
features encoded.
Many text classification tasks which employ n-grams of words to capture local in-
teractions and features have seen lots of success using CNN frameworks in the last
few years. CNN-based frameworks can easily capture temporal and hierarchical fea-
tures in variable-length text sequences. Word or character embeddings are generally
the first layers in these frameworks. Based on the data and type, either pre-trained
or static embeddings are used to obtain a vector representation of the words in sen-
tences.
Collobert et al. [CW08c], [Col+11] uses a one-layer convolution block for mod-
eling sentences to perform many NLP tasks. Yu et al. [Yu+14] also use a one-layer
CNN to model a classifier to select question answer mappings. Kalchbrenner et al.
extend the idea to form a dynamic CNN by stacking CNN and using dynamic k-
max pooling operations over long sentences [KGB14b]. This research significantly
improved over existing approaches at the time in much short text and multiclass
classification. Kim extends the single-block CNN by adding multiple channels in
the input and multiple kernels of various lengths to give higher-order combinations
of ngrams [Kim14b]. This work also performed various impact analysis of static vs.
dynamic channels, the importance of max pooling, and more to detect which basic
blocks yielded lower error rates. Yin et al. extend Kim’s multi-channel, variable ker-
nel framework to use hierarchical CNN, obtaining further improvements [YS16].
Santos and Gatti use character-to-sentence level representation with CNN frame-
works for effective sentiment classification [SG14]. Johnson and Zhang explore the
usage of region embeddings for effective short text categorization, due to the ability
to capture contexts over a larger span where word embeddings fail [JZ15]. Wang and
others perform semantic clustering using density peaks on pre-trained word embed-
dings forming a representation they call semantic cliques, with such semantic units
used further with convolutions for short-text mining [Wan+15a].
Zhang et al. explore the use of character-level representations for a CNN instead
of word-level embeddings [ZZL15]. On a large dataset, character-level modeling of
sentences performs very well when compared to traditional sparse representations
or even deep learning frameworks, such as word embedding CNNs or RNNs. Con-
neau et al. design a very deep CNN along with modifications such as shortcuts to
learn more complex features [Con+16]. Xiao and Cho’s research further extends
the character level encoding for entire document classification task when combined
with RNNs with a lower number of parameters [XC16].
294 6 Convolutional Neural Networks
In their work, Xu et al. use a CNN for short-text clustering in a completely unsuper-
vised manner [Xu+17]. The original keyword features from the text are used to gen-
erate compact binary code with locality-preserving constraints. Deep feature repre-
sentation is obtained using word embeddings with the dynamic CNN in [KGB14b].
The outputs of the CNN are made to fit the binary codes during the training process.
The deep features thus evolved from the CNN layers are passed to normal cluster-
ing algorithms, such as k-means, to give the final clustering of the data. Experiments
show that this method does significantly better than traditional feature-based meth-
ods on various datasets.
Lau et al. jointly learn the topics and language models using CNN-based
frameworks that result in better coherent, effective, and interpretable topic mod-
els [LBC17]. Document context is captured using a CNN framework, and the
resulting document vectors are combined with topic vectors to give an effective
document-topic representation. The language model is composed of the same doc-
ument vectors from above and LSTMs.
In seminal work by Collobert et al., many NLP tasks, such as POS Tagging, Chunk-
ing, Named Entity Resolution, and Semantic Role Labeling are performed for the
first time using word embeddings and CNN blocks [Col+11]. The research shows
the strength of using CNNs in finding features in an automated way rather than hand-
crafted task-specific features used for similar tasks. Zheng et al. show that character-
based representations, CNN-based frameworks, and dynamic programming can be
very effective in performing syntactic parsing without any task-specific feature engi-
neering in languages as complex as Chinese [Zhe+15]. Santos and Zadrozny show
that using character-level embeddings jointly with word-level embeddings and deep
CNNs can further improve POS Tagging tasks [DSZ14] in English and Portuguese.
Zhu et al. propose a recursive CNN (RCNN) to capture complex structure in the de-
pendency trees. An RCNN has a basic k-ary tree as a unit that can capture the parsing
tree with the relationship between nodes and children. This structure can be applied
recursively to map the representation for the entire dependency tree. The RCNN is
shown to be very effective as a re-ranking model in dependency parsers [Zhu+15].
traction, coreference resolution, and entity linking to name a few. In the work of
Chen et al., instead of using hand-coded features, the researchers employ word-
based representations to capture lexical and sentence level features that work with
a modified CNN for superior multiple event extractions in text [Che+15]. The re-
searchers use word context, positions, and event type in their representations and
embeddings flowing into a CNN with multiple feature maps. Instead of a CNN
with max pooling, which can miss multiple events happening in a sentence, the
researchers employ dynamic multi pooling. In dynamic multi pooling, the feature
maps are split into three parts, finding a maximum for each part.
As discussed above in the section on NLP, Zheng et. al [Zhe+15] and Santos et.
al [DSZ14] employ CNNs for relation classification without any hand-coded fea-
tures. In the work of Vu et al., relation classification uses a combination of CNNs
and RNNs. In a sentence which has entities and relations between them, the re-
searchers perform a split between left and middle, and middle and right part of the
sentence, flowing into two different word embeddings and CNN layers with max
pooling. This design gives special attention to the middle part, which is an impor-
tant aspect in relation classification as compared to previous research. Bi-directional
RNNs with an additional hidden layer are introduced to capture relation arguments
from succeeding words. The combined approach shows significant improvements
over traditional feature-based and even independently used CNNs and RNNs.
Nguyen and Grisham use a CNN-based framework for relation extraction
[NG15b]. They use word embeddings and position embeddings concatenated as
the input representation of sentences with entities having relations. They employ a
CNN with multiple filter sizes and max pooling. It is interesting to see that the per-
formance of their framework is better than all handcrafted feature engineering-based
machine learning systems that use many morphological and lexical features.
In the work of Adel et al., the researchers compare many techniques from a tra-
ditional feature-based machine learning to CNN-based deep learning for relation
classification in the context of slot filling [AS17a]. Similar to the above work, they
break the sentences into three parts for capturing the contexts and use a CNN with
k-max pooling.
Hu et al. highlight how CNNs can be used to encode both semantic similarities and
contexts in translation pairs and thus yield a more effective translations [Hu+15].
Another interesting aspect of this research is its employment of a curriculum train-
ing, where the training data is categorized from easy to difficult, and uses phrase to
sentence for contexts encoding for effective translations.
Meng et al. build a CNN-based framework for guiding signals from both source
and target during machine translation [Men+15]. Using CNNs with gating provides
296 6 Convolutional Neural Networks
guidance on which parts of source text have influence on the target words. Fusing
them with entire source sentence for context yields a better joint model.
Gehring et al. use a CNN with an attention module, thus showing not only a
fast performing and parallelizable implementation, but also a more effective model
compared to LSTM-based ones. Using word and positional embeddings in the input
representation, stacking the filters on top of each other for hierarchical mappings,
gated linear units, and multi-step attention module gives the researchers a real edge
on English-French and English-German translation [Geh+17b].
6.7.6 Summarizations
Denil et. al show that by having a hierarchy of word embeddings that compose
sentence embeddings, which in turn compose document embeddings, and using dy-
namic CNNs gives useful document summarizations, as well as effective visualiza-
tions [Den+14]. The research also highlights that the composition can capture from
low-level lexical features to high-level semantic concepts in various tasks, including
summarization, classification, and visualization.
Cheng and Lapata develop a neural framework combining CNN for hierarchi-
cal document encoding and attention extractor for effective document summariza-
tions [CL16]. Mapping the representations very close to the actual data, where there
is the composition of words into sentences, sentences to paragraphs, paragraphs to
entire document using CNN, and max pooling gives the researchers a clear advan-
tage in capturing both local and global sentential information.
Dong et al. use multi-column CNNs for analyzing questions from various aspects
such as answer path, answer type, and answer contexts [Don+15b]. The embed-
ding of answer layers using entities and relations using low-dimensional embedding
space is utilized along with a scoring layer on top to rank candidate answers. The
work shows that without hand-coded or engineered features, the design provides a
very effective question answering system.
Severyn and Moschitti show that using relational information given by the
matches between the words used in the question and answers with a CNN-based
framework gives very effective of question-answering system [SM15]. If we can
map the question to finding facts from the database, Yin et al. show in their work
that a two-stage approach using a CNN-based framework can yield excellent re-
sults [Yin+16b]. The facts in the answers are mapped to subject, predicate, and
object. Then entity linking from the mention in the question to the subject employ-
6.8 Fast Algorithms for Convolutions 297
ing a character-level CNN is the first stage of the pipeline. Matching the predicate
in the fact with the question using a word-level CNN with attentive max pooling is
the second stage of the pipeline.
CNNs, in general, are more parallel in nature as compared to other deep learning
architectures. However, as training data size has increased, need for faster predic-
tions in near real-time, and GPU based hardware for parallelizing operations are be-
coming more widespread, convolution operations in CNN have gone through many
enhancements. In this section, we will discuss some fast algorithms for CNN and
give insights into how convolutions can be made faster with fewer floating point
operations [LG16].
This theorem states that convolution in the time domain (any discrete input such as
image or text) is equivalent to pointwise multiplications in the frequency domain.
We can represent this transformation as taking fast Fourier transform (FFT) of the
input and the kernel, multiplying it and taking inverse FFT.
tions in normal. The minimal filtering algorithms F(m, r) can be shown to need
only μ (F(m, r)) = m + r − 1 multiplications. Let us consider a simple example with
input [d0 , d1 , d2 , d3 ] of size 4 convolved with filter [g0 , g1 , g2 ] of size 3 to give size 2
outputs [m1 , m2 ]. In traditional convolution, we would need 6 multiplications but by
arranging the inputs as shown:
⎡ ⎤
g0
d0 d1 d2 ⎣ ⎦ m + m2 + m3
g1 = 1 (6.43)
d1 d2 d3 m2 − m3 − m4
g2
where
Thus, multiplications are reduced to only m + r − 1, i.e., 4, and savings are only
6
4 = 1.5. The additions (g0 +g21 +g2 ) and (g0 −g21 +g2 ) can be precomputed from the fil-
ters, giving additional performance benefits. These fast filtering algorithms can be
written in matrix form as:
Here are some practical tips in regards to CNNs especially for classification
tasks.
• For the classification task, it is always good to start with Yoon Kim et al.
proposed CNN with word representation [Kim14b].
• Using pre-trained embeddings with word2vec or GloVe as compared to 1-
hot vector representation as a single static channel for mapping sentences
should be done before fine-tuning or introducing multiple channels.
• Having multiple filters such as [3, 4, 5], number of feature maps ranging
from 60 to 500, and ReLU as the activation function often gives a good
performance [ZW17].
• 1-max pooling as compared to average pooling and k-max pooling gives
better results [ZW17]
• Choice of regularization technique, i.e., L1 or L2 or dropout, etc. depends
on the dataset and always good to try without the regularization and with
regularization and compare the validation metrics.
• Learning curves and variances in them across multiple cross-validation
gives an interesting idea of “robustness” of the algorithm. Flatter the curves
and smaller the variances, highly likely the validation metric estimates are
accurate.
• Understand the predictions from the model on the validation data to look
for the patterns in false positives and false negatives. Is it because of
spelling mistakes? Is it because of dependencies and orders? Is it because
of lack of training data to cover the cases?
• Character-based embeddings can be useful provided there is enough train-
ing data.
• Adding other structures such as LSTM, hierarchical, attention-based
should be done incrementally to see the impact of each combination.
300 6 Convolutional Neural Networks
First, we need to describe the main open source tools and libraries we will use for
our case study.
• Keras (www.keras.io) is a high-level deep learning API written in Python which
gives a common interface to various deep learning backends, such as Tensor-
Flow, CNTK, and Theano. The code can run seamlessly on CPUs and GPUs. All
experiments with CNN are done using Keras API.
• TensorFlow (https://www.tensorflow.org/) is a popular open source machine
learning and deep learning library. We use TensorFlow as our deep learning li-
brary but Keras API as the basic API for experimenting.
• Pandas (https://pandas.pydata.org/) is a popular open source implementation for
data structures and data analysis. We will use it for data exploration and some
basic processing.
• scikit-learn (http://scikit-learn.org/) is a popular open source for various ma-
chine learning algorithms and evaluations. We will use it only for sampling and
creating datasets for estimations in our case study.
• Matplotlib (https://matplotlib.org/) is a popular open source for visualization.
We will use it to visualize performance.
Now we are ready to focus on the four following sub-tasks.
• Exploratory data analysis
• Data preprocessing and data splits
• CNN model experiments and analysis
• Understanding and improving models
6.9 Case Study 301
The total data has 14, 640 labeled data, 15 features/attributes of which only the at-
tribute text will be used for learning. The classes are in three categories of positive,
negative, and neutral. We will take 15% of the total data for testing from the whole
dataset in a stratified way and similarly 10% from training data for validation. Nor-
mally cross-validation (CV) is used for both model selection and parameter tuning,
and we will use the validation set to reduce the time to run. We did compare CV
estimates with separate validation set and both looked comparable.
Figure 6.23 shows the class distribution. We see that there is a skew in class
distribution towards negative sentiments as compared to positive and neutral.
Class Distribution
Number of Sentences in Class
8000
6000
4000
2000
0
negative neutral positive
Class
One interesting step in EDA is to plot the word cloud for positive and negative
sentiment data from the entire dataset to understand some of the most frequent words
used and maybe correlating with that sentiment. Tokens that are over-represented
in the cloud are mostly adjectives such as “thanks,” “great,” “good,” “appreciate,”
etc. in the positive tweets while the negative sentiment word cloud in has reasons
“luggage,” “canceled flight,” “website,” etc. as shown in Fig. 6.24.
We perform some further basic data processing to remove stop words and mentions
from the text as they are basically not useful in our classification task. The listing
below shows the basic data cleanup code.
1 # remove s t o p words w i t h e x c e p t i o n s
2 def remove stopwords ( i n p u t t e x t ) :
3 s t o p w o r d s l i s t = s t o p w o r d s . words ( ’ e n g l i s h ’ )
4 # Some words which m i g h t i n d i c a t e a c e r t a i n s e n t i m e n t a r e
kept
5 w h i t e l i s t = [ ” n ’ t ” , ” n o t ” , ” no ” ]
6 words = i n p u t t e x t . s p l i t ( )
7 c l e a n w o r d s = [ word f o r word i n words i f (
302 6 Convolutional Neural Networks
12
16
Next, we will split the entire data into training and testing with 85% for training and
15% for testing. We will build various models using same training data and evaluate
with respect to same test data to get a clear comparison.
We then perform tokenization, splits of training data into training and validation and
sequence mapping with fixed size. We use the maximum text length in our corpus
to determine the sequence length and do padding.
1 # t o k e n i z a t i o n w i t h max words d e f i n e d and f i l t e r s t o remove
characters
2 t k = T o k e n i z e r ( num words =NB WORDS,
3 f i l t e r s = ’ ! ” # $%&() ∗ + , −./:; < = > ?@[ \ \ ] ˆ ‘ { | } ˜ \ t \ n ’ ,
4 l o w e r = True ,
5 s p l i t =” ” )
6 tk . f i t o n t e x t s ( X train )
6.9 Case Study 303
1 # c o n v e r t t r a i n and t e s t t o s e q u e n c e u s i n g t h e t o k e n i z e r
t r a i n e d on t h e t r a i n i n g d a t a
2 X t r a i n t o t a l = tk . texts to sequences ( X train )
3 X t e s t t o t a l = tk . texts to sequences ( X test )
4
5 # pad t h e s e q u e n c e s t o a maximum l e n g t h
6 X t r a i n s e q = p a d s e q u e n c e s ( X t r a i n t o t a l , maxlen =MAX LEN)
7 X t e s t s e q = p a d s e q u e n c e s ( X t e s t t o t a l , maxlen =MAX LEN)
8
9 # perform encoding of
10 l e = LabelEncoder ( )
11 y train le = le . fit transform ( y train )
12 y t e s t l e = le . transform ( y test )
13 y train one hot = to categorical ( y train le )
14 y test one hot = to categorical ( y test le )
Once we have preprocessed and created training, validation and test sets, we will
perform various modeling analysis on the data. We will first do some basic analysis
using simple CNN and then proceed to run various configurations and modifications
of CNN discussed in the chapter.
Next, we show a basic code of CNN which input layer with maximum length of
sentence 24, which outputs a 100-dimensional vectors which are convoluted with
64 filters each of height or size 3, equivalent to 3-gram, going through a ReLU non-
linear activation function, then a max pooling layer that gets flattened so that it is
input of a fully connected layer which outputs to soft max layer with 3 outputs for
3 classes.
1 # b a s i c CNN Model t o u n d e r s t a n d how i t works
2 def base cnn model ( ) :
3 # Embedding
4 # Layer −>C o n v o l u t i o n 1 D −>MaxPooling1D−> F l a t t e n −>
F u l l y C o n n e c t e d −> C l a s s i f i e r
5 model = S e q u e n t i a l (
6 [ Embedding ( i n p u t d i m =10000 , o u t p u t d i m =100 ,
i n p u t l e n g t h =24) ,
7 C o n v o l u t i o n 1 D ( f i l t e r s =64 , k e r n e l s i z e =3 , p a d d i n g = ’
same ’ , a c t i v a t i o n = ’ r e l u ’ ) ,
8 MaxPooling1D ( ) ,
9 Flatten () ,
10 Dense ( 1 0 0 , a c t i v a t i o n = ’ r e l u ’ ) ,
11 Dense ( 3 , a c t i v a t i o n = ’ s o f t m a x ’ ) ] )
12 model . summary ( )
13 r e t u r n model
14
304 6 Convolutional Neural Networks
15
16 # t r a i n and v a l i d a t e
17 base cnn model = base cnn model ( )
18 base history = train model (
19 base cnn model ,
20 X train seq ,
21 y train ,
22 X valid seq ,
23 y valid )
A single channel using pre-trained embeddings and Yoon Kim’s CNN model
with multiple filters is shown here
1 # c r e a t e embedding m a t r i x f o r t h e e x p e r i m e n t
2 emb matrix = c r e a t e e m b e d d i n g m a t r i x ( tk , 100 , embeddings )
3
4 # s i n g l e c h a n n e l CNN w i t h m u l t i p l e f i l t e r s
5
15 f i l t e r s i z e s = [3 , 4 , 5]
16 convs = [ ]
17 # p a r a l l e l l a y e r s f o r e a c h f i l t e r s i z e w i t h conv1d and max
pooling
18 for f i l t e r s i z e in f i l t e r s i z e s :
19 l conv = Convolution1D (
20 f i l t e r s =128 ,
21 kernel size=filter size ,
22 p a d d i n g = ’ same ’ ,
23 a c t i v a t i o n = ’ relu ’ ) ( text embedding )
24 l p o o l = MaxPooling1D ( f i l t e r s i z e ) ( l c o n v )
25 convs . append ( l p o o l )
26 # c o n c a t e n a t e o u t p u t s from a l l cnn b l o c k s
27 merge = c o n c a t e n a t e ( convs , a x i s = 1 )
28 c o n v o l = C o n v o l u t i o n 1 D ( 1 2 8 , 5 , a c t i v a t i o n = ’ r e l u ’ ) ( merge )
29 pool1 = GlobalMaxPooling1D ( ) ( convol )
30 d e n s e = Dense ( 1 2 8 , a c t i v a t i o n = ’ r e l u ’ , name= ’ Dense ’ ) ( p o o l 1 )
31 # classification layer
32 o u t = Dense ( 3 , a c t i v a t i o n = ’ s o f t m a x ’ ) ( d e n s e )
33 model = Model (
34 i n p u t s =[ t e x t s e q i n p u t ] ,
35 o u t p u t s =out ,
36 name=” KimSingleChannelCNN ” )
37 model . summary ( )
38 r e t u r n model
39
6.9 Case Study 305
40
We will list different experiments with its name and their purpose before high-
lighting the results from each.
1. Base CNN. A basic single block of CNN with convolution with filter of size 3,
max pooling and a softmax layer.
2. Base CNN + Dropout. To see the impact of dropout on the base CNN.
3. Base CNN + Regularization. To see the impact of L2 regularization on the
base CNN.
4. Multi-filters. To see the impact of adding more filters [2,3,4,5] to CNN.
5. Multi-filters + Increased Maps. To see the impact of increasing the filter maps
from 64 to 128.
6. Multi-filters + Static Pre-trained Embeddings. To see the impact of using
pre-trained word embeddings in CNN.
7. Multi-filters + Dynamic Pre-trained Embeddings. To see the impact of using
pre-trained word embeddings in CNN that are trained on the training set.
8. Yoon Kim’s Single Channel. Single channel CNN using widely known archi-
tecture [Kim14b].
9. Yoon Kim’s Multiple Channel. Multiple channel CNN using widely known
architecture [Kim14b] to see the impact on increasing the channels. Static and
dynamic embeddings are used as two different channels.
10. Kalchbrenner et al. Dynamic CNN. Kalchbrenner et al. based dynamic CNN
with K-max pooling [KGB14b].
11. Multichannel Variable MVCNN. We use two embedding layers with static
and dynamic channels [YS16].
12. Multigroup MG-CNN. We use three different channels (two inputs with em-
bedding layers using GloVe and one using fastText) with different dimensions
(100 and 300) [ZRW16].
13. Word-level Dilated CNN. Exploring the concept of dilations with the reduced
parameters and larger coverage using word-level inputs [YK15].
14. Character-level CNN. We explore the character-level embeddings instead of
the word-level embeddings [ZZL15].
15. Very Deep Character-level CNN. Impact of a very deep level CNN with mul-
tiple layers [Con+16].
16. Character-level Dilated CNN. Exploring the concept of dilations with the re-
duced parameters and larger coverage using character-level inputs [YK15].
17. C-LSTM. Exploring C-LSTM to verify how CNN can be used to capture the
local features of phrases and RNN to capture global and temporal sentence se-
mantics [Zho+15].
18. AC-BiLSTM Exploring the bi-directional LSTM with CNN [LZ16].
306 6 Convolutional Neural Networks
We use some practical deep learning aspects while training the models as high-
lighted below:
1 # u s e t h e v a l i d a t i o n l o s s t o d e t e c t t h e b e s t w e i g h t s t o be
saved
2 c h e c k p o i n t s . append ( ModelCheckpoint ( c h e c k p o i n t f i l e , monitor = ’
v a l l o s s ’ , v e r b o s e =0 , s a v e b e s t o n l y = True ,
s a v e w e i g h t s o n l y = True , mode= ’ a u t o ’ , p e r i o d = 1 ) )
3 # output to TensorBoard
4 c h e c k p o i n t s . append ( TensorBoard ( l o g d i r = ’ . / l o g s ’ , w r i t e g r a p h =
True , w r i t e i m a g e s = F a l s e ) )
5 # i f no i m p r o v e m e n t s i n 10 e p o c h s , t h e n q u i t
6 c h e c k p o i n t s . append ( E a r l y S t o p p i n g ( monitor = ’ v a l l o s s ’ , p a t i e n c e
=10) )
In the Table 6.1 we will highlight the results of running different CNN architectures
given above and the results we track with accuracy and average precision.
We will list some high level analysis and observations from Table 6.1 and our
analysis from various runs below:
• Multiple layers and multiple filters seem to improve both accuracy and
average precision by more than 2%.
• Using pre-trained embeddings which get trained in the data gives one of
the best performances and is very much in line with many research.
• Multigroup Norm constraint MG-CNN show the best results in both ac-
curacy and average precision in word-based representation. Using three
embedding channels with two different embeddings with different sizes
seems to give the edge.
• Yoon Kim’s model with two channels has second best performance and
confirms that it should be always a model to try in classification problems.
The performance of dual channel along with MG-CNN confirms that in-
creasing number of channels helps the model in general.
• Increasing the depth and complexity of CNN and hence the parameters
has not much effect on generalization, again can be accounted for small
training data size.
• Character-based representation shows relatively poor performance and that
is well in line with most research because of limited corpus and training
size.
• Introducing complexity by combining CNN and LSTM does not improve
the performance and again can be attributed to the task complexity and the
size of training data.
In this section, we will give some practical tips and tricks to help the researchers
gain insights into model behaviors and improve them further.
One way of understanding the model behavior is to look at the predictions at various
layers using some form of dimensionality reduction techniques and visualization
techniques. To explore the behavior of last but one layer before the classification
layer, we first create a clone of the model by removing the last layer and using the
test set to generate high dimensional outputs from this layer. We then use PCA to get
30 components from 128-dimensional outputs and finally project this using TSNE.
As shown in Fig. 6.25, we see a clear reason why Yoon Kim’s Single Channel
CNN performs better than the Basic CNN.
Next, we will analyze the false positives and false negatives to understand the pat-
terns and the causes, to improve the models further.
Table 6.2 highlights some of the text and probable cause. Having words such
as late flight, overhead, etc. are so overrepresented in the negatives that it
308 6 Convolutional Neural Networks
40 negative negative
neutral neutral
positive positive
40
30
20 20
10
0
–20
–10
–20 –40
–30
–60
(a) (a)
Fig. 6.25: A layer before last with 128 dimensions is used to visualize the test data
with PCA and TSNE. (a) Hidden layer from B̄asic CNN. (b) Hidden layer from
Yoon Kim’s Single Channel CNN
Table 6.3 highlights some of the text and probable causes. Having words such
as awesome, thanks, etc. are so overrepresented in the positives that it causes
even sentences which have these to be classified positives. Adding more neg-
atives with similar language and using average pooling might help. Having
sarcasm based datasets, training embeddings on them and using it as input
channel can improve the performance.
6.9 Case Study 309
Using Lime for model explanations, especially false positives and false negatives
gives insights into the reason with weights associated with the keywords as shown
in Fig. 6.26.
6 exp = e x p l a i n e r . e x p l a i n i n s t a n c e ( ’ f o r g e t r e s e r v a t i o n s t h a n k
g r e a t company i h a v e c a n c e l l e d f l i g h t e d f l i g h t once
a g a i n t h a n k you ’ ,
7 keras wrapper ,
8 n u m f e a t u r e s =10 ,
9 l a b e l s =[0 , 2]
10 )
flight Negative
i Positive
again
forget
Features
company
flighted
reservations
cancelled
great
thank
Fig. 6.26: Lime outputs weights for words for both positive and negative class
Some other interesting problems and research questions that readers and practition-
ers can further attempt are listed below:
310 6 Convolutional Neural Networks
6.10 Discussion
Geoffery Hinton in his talk “What is wrong with convolutional neural nets?” given
at MIT highlights some of the issues with CNNs, especially around max pooling.
The talk explains how max pooling can “ignore” some of the important signals be-
cause of the bias it has towards finding the “key” features. Another issue highlighted
was around how the filters can capture different features and build higher level fea-
tures but fail to capture the “relationship” between these features to some extent. An
example being, presence of features detecting eye, ears, mouth in face recognition
using CNNs with different filters and layers cannot separate out an image with pres-
ence of these but not in the right place. Capsule Networks, using capsules as a basic
building block is seen as an alternative design to overcome these issues [SFH17].
References
7.1 Introduction
In the previous chapter, CNNs provided a way for neural networks to learn a hierar-
chy of weights, resembling that of n-gram classification on the text. This approach
proved to be very effective for sentiment analysis, or more broadly text classifica-
tion. One of the disadvantages of CNNs, however, is their inability to model contex-
tual information over long sequences.1 In many situations in NLP, it is desirable to
capture long-term dependencies and maintain the contextual order between words to
resolve the overall meaning of a text. In this chapter, we introduce recurrent neural
networks (RNNs) that extend deep learning to sequences.
Sequential information and long-term dependencies in NLP traditionally relied
on HMMs to compute context information, for example, in dependency parsing.
One of the limitations of using a Markov chain for sequence focused tasks is that
the generation of each prediction is limited to a fixed number of previous states.
RNNs, however, relax this constraint, accumulating information from each time step
into a “hidden state.” This allows sequential information to be “summarized” and
predictions can be made based on the entire history of the sequence.
Another advantage of RNNs is their ability to learn representations for variable
length sequences, such as sentences, documents, and speech samples. This allows
two samples of differing lengths to be mapped into the same feature space, allowing
them to be comparable. In the context of language translation, for example, an input
sentence may have more words than its translation, requiring a variable number of
computational steps. Thus, it is highly beneficial to have knowledge of the entire
length of the sentence before predicting the translation. We will study this example
more at the end of this chapter.
In this chapter, we begin by describing the basic building blocks of RNNs and
how they retain memory. We then describe the training process for RNNs and dis-
cuss the vanishing gradient problem, regularization, and RNN variants. Next we
1 This statement is made in a basic context of CNNs and RNNs. The CNN vs. RNN superiority
debate in sequential contexts is an active area of research.
© Springer Nature Switzerland AG 2019 315
U. Kamath et al., Deep Learning for NLP and Speech Recognition,
https://doi.org/10.1007/978-3-030-14596-5 7
316 7 Recurrent Neural Networks
show how to incorporate text input in recurrent architectures, leveraging word and
character representations. We then introduce some traditional RNN architectures in
NLP, and then move towards more modern architectures. The chapter is concluded
with a case study on neural machine translation and a discussion about the future
directions of RNNs.
First we will look at the idea of recurrence conceptually. Let us define a T length
input sequence as X, where X = {x1 , x2 , . . . , xT }, such that xt ∈ RN is a vector input
at time t. We then define our memory or history up to and including time t as ht .2
Thus, we can define our output ot as:
where the function f maps memory and input to an output. The memory from the
previous time step is ht−1 , and the input is xt . For the initial case x1 , h0 is the zero
vector 0.
Abstractly, the output ot is considered to have summarized the information from
the current input xt and the previous history from ht−1 . Therefore, ot can be con-
sidered the history vector for the entire sequence up to and including time t. This
yields the equation:
Here we see where the term “recurrence” comes from: the application of the same
function for each instance, wherein the output is directly dependent on the previous
result.
More formally, we can extend this concept to neural networks by redefining the
transformation function f as follows:
2 This history vector will be called the hidden state later on for obvious reasons.
7.2 Basic Building Blocks of RNNs 317
The code snippet below illustrates a PyTorch implementation of the simple RNN
previously described. It illustrates the recurrent computation in a modern frame-
work.
1 # PyTorch RNN D e f i n i t i o n
2 i m p o r t t o r c h . nn a s nn
3 from t o r c h . a u t o g r a d i m p o r t V a r i a b l e
4 import t o r c h . optim as optim
5
6 c l a s s RNN( nn . Module ) :
7
15 s e l f . U = nn . L i n e a r ( i n p u t s i z e , s e l f . h i d d e n s i z e )
16 s e l f .W = nn . L i n e a r ( s e l f . h i d d e n s i z e , s e l f . o u t p u t s i z e )
17
24 r n n = RNN( i n p u t s i z e )
25
26 # T r a i n i n g the network
27 o p t i m i z e r = o p t i m . Adam ( r n n . p a r a m e t e r s ( ) , l r = l e a r n i n g r a t e ,
w e i g h t d e c a y =1e −5)
28
318 7 Recurrent Neural Networks
34 # Forward P r o p a g a t i o n
35 hidden = Vari abl e ( t o r c h . zeros (1 , rnn . h i d d e n s i z e ) )
36 for i in range ( input . s i z e ( ) [ 0 ] ) :
37 output = rnn ( i n p u t [ i ] , hidden )
38 hidden = output
39
40 # Error Computation
41 loss = F . n l l l o s s ( output , t a r g e t )
42
43 # Clear gradients
44 optimizer . zero grad ()
45
46 # Backpropagation
47 l o s s . backward ( )
48
49 # P a r a m e t e r Update
50 optimizer . step ()
RNNs are trained through backpropagation and gradient descent similar to feed-
forward networks we have seen previously: forward propagating an example, cal-
culating the error for a prediction, computing the gradients for each set of weights
via backpropagation, and updating the weights according to the gradient descent
optimization method.
The forward propagation equations for ht and the output prediction ŷt are:
7.3 RNNs and Properties 319
ht = tanh(Uxt + Wht−1 )
(7.4)
ŷt = softmax(Vht )
where the learnable parameters are U, W, and V.3 U incorporates the information
from xt , W incorporates the recurrent state, and V learns a transformation to the
output size and classification. A diagram of this RNN is shown in Fig. 7.2.
We compute the error using cross-entropy loss at each time step t, where yt is the
target.
The gradients are computed by evaluating every path that contributed to the pre-
diction ŷt . This process is called backpropagation through time (BPTT). This pro-
cess is illustrated in Fig. 7.3.
The parameters of our RNN are U, V, and W, so we must compute the gradient of
our loss function with respect to these matrices. Figure 7.4 shows backpropagation
through a step of the RNN.
3It is common to split the single weight matrix W of an RNN in Eq. (7.3) into two separate weight
matrices, here U and W. Doing this allows for a lower computational cost and forces separation
between the hidden state and input in the early stages of training.
320 7 Recurrent Neural Networks
Fig. 7.3: Backpropagation through time shown with respect to the error at t = 3. The
error E3 is comprised of input from each previous time step and the inputs to those
time steps. This figure excludes backpropagation with respect to E1 , E2 , and E4
The weight matrix V controls the output dimensionality of ŷ, and does not contribute
to the recurrent connection. Therefore, computing the gradient is the same as a linear
layer.
For convenience, let
qt = Vht . (7.7)
Then,
∂ Et ∂ Et ∂ ŷtk ∂ qtl
= . (7.8)
∂ Vi, j ∂ ŷ tk ∂ qtl ∂ Vi, j
∂ Et yt
=− k. (7.9)
∂ ŷtk ŷtk
If we combine (7.9) and (7.10) we obtain the sum over all values of k to produce
∂ Et
∂ qt :
l
ytl ytk
− ŷtl 1 − ŷtl + ∑ − −ŷtk ŷtl = −ytl + ytl ŷtl + ∑ ytk ŷtl (7.11a)
ŷtl k=l ŷtk k=l
Recall that all yt are one-hot vectors, meaning that all values are in the vector are
zero except for one indicating the class. Thus, the sum is 1, so
∂ Et
= ŷtl − ytl (7.12)
∂ qtl
∂ qtl ∂
= Vl,m htm (7.13a)
∂ Vi, j ∂ Vi, j
= δil δ jm htm (7.13b)
= δil ht j . (7.13c)
∂ Et
= (ŷti − yti ) ht j , (7.14)
∂ Vi, j
∂ Et
= (ŷt − yt ) ⊗ ht , (7.15)
∂V
where ⊗ is the outer product.
The parameter W appears in the argument for ht , so we will have to check the
gradient in both ht and ŷt . We must also make note that ŷt depends on W both
directly and indirectly (through ht−1 ). Let zt = Uxt + Wht−1 . Then ht = tanh(zt ).
At first it seems that by the chain rule we have:
Note that of these four terms, we have already calculated the first two, and the third
is simple:
∂ qtl ∂
= Vl,b htb (7.17a)
∂ htm ∂ htm
= Vl,b δb,m (7.17b)
= Vl,m (7.17c)
The final term, however, requires an implicit dependence of ht on Wi, j through ht−1
as well as a direct dependence. Hence, we have:
This process continues until we reach h(−1) , which was initialized to a vector of
∂ htm ∂ ht−2n
zeros (0). Notice that the last term in (7.19) collapses to ∂ ht−2n ∂ Wi, j and we can
∂ htm ∂ htn
turn the first term into ∂ htn ∂ Wi, j . Then, we arrive at the compact form:
where we sum over all values of r less than t in addition to the standard dummy
index n. More clearly, this is written as:
∂ htm t
∂ htm ∂ hrn
=∑ ,. (7.21)
∂ Wi, j r=0 ∂ hrn ∂ Wi, j
This term is responsible for the vanishing/exploding gradient problem: the gradient
exponentially shrinking to 0 (vanishing) or exponentially growing larger (explod-
∂h ∂h
ing). The multiplication of the term ∂ htrm by the term ∂ Wri,nj means that the product
n
will be smaller if both terms are less than 1 or larger if the terms are greater than 1.
We will address this problem in more detail momentarily.
Combining all of these yields:
∂ Et t
∂ htm ∂ hrn
= ŷtl − ytl Vl,m ∑ . (7.22)
∂ Wi, j r=0 ∂ hrn ∂ Wi, j
7.3 RNNs and Properties 323
Taking the gradient of U is similar to doing it for W since they both require taking
sequential derivatives of the ht vector. We have:
Note that we only need to calculate the last term now. Following the same procedure
as for W , we find that:
∂ htm t
∂ htm ∂ hrn
=∑ , (7.24)
∂ Ui, j r=0 ∂ hrn ∂ Ui, j
and thus we have:
∂ Et t
∂ htm ∂ hrn
= ŷtl − ytl Vl,m ∑ . (7.25)
∂ Ui, j r=0 ∂ hrn ∂ Ui, j
The difference between U and W appears in the actual implementation since the
∂h ∂h
values of ∂ Uri,nj and ∂ Wri,nj differ.
The error for all time steps is the summation of Et ’s according to our loss func-
tion (7.6). Therefore, we can sum the gradients for each of the weights in our net-
work (U, V, and W) and update then with the accumulated gradients.
One of the most difficult parts of training RNNs is the vanishing/exploding gradient
problem (often referred to as just the vanishing gradient problem).4 During back-
propagation, the gradients are multiplied by the weight’s contribution to the error
at each time step, shown in Eq (7.21). The impact of this multiplication at each
time step dramatically reduces or increases the gradient propagated to the previous
time step which will in turn be multiplied again. The recurrent multiplication in the
backpropagation step causes an exponential effect for any irregularity.
• If the weights are small, the gradients will shrink exponentially.
• If the weights are large, the gradients will grow exponentially.
In the case when the contribution is very small, the weight update may be a negligi-
ble change, potentially causing the network to stop training. Practically, this usually
4 The tanh activation function bounds the gradient between 0 and 1. This has the effect of shrinking
the gradient in these circumstances.
324 7 Recurrent Neural Networks
leads to underflow or overflow errors when not considered. One way to alleviate this
issue is to use the second-order derivatives to predict the presence of vanishing/ex-
ploding gradients by using Hessian-free optimization techniques. Another approach
is to initialize the weights of the network carefully. However, even with careful ini-
tialization, it can still be challenging to deal with long-range dependencies.
A common initialization for RNNs is to initialize the initial hidden state to 0 . The
performance can typically be improved by allowing this hidden state to be learned
[KB14].
Adaptive learning rate methods, such as Adam [KB14], can be useful in recur-
rent networks, as they cater to the dynamics of individual weights, which can vary
significantly in RNNs.
There are many methods used to combat the vanishing gradient problem, many
of them focusing on careful initialization or controlling the size of the gradient be-
ing propagated. The most commonly used method to combat vanishing gradients
is the addition of gates to RNNs. We will focus more on this approach in the next
section. RNN sequences can be very long. For example, if an RNN used for speech
recognition samples 20 ms windows with a stride of 10 ms will produce an output
sequence length of 999 time steps for a 10-s clip (assuming no padding). Thus, the
gradients can vanish/explode very easily [BSF94b].
Long short-term memory (LSTM) utilizes gates to control the gradient propa-
gation in the recurrent network’s memory [HS97b]. These gates (referred to as the
input, output, and forget gates) are used to guard a memory cell that is carrying the
hidden state to the next time step. The gating mechanisms are themselves neural
network layers. This allows the network to learn the conditions for when to forget,
ignore, or keep information in the memory cell. Figure 7.5 shows a diagram of an
LSTM.
7.3 RNNs and Properties 325
it = σ (Wi xt + Ui ht−1 + bi )
ft = σ (W f xt + U f ht−1 + b f )
ot = σ (Wo xt + Uo ht−1 + bo )
(7.26)
c̃t = tanh(Wc xt + Uc ht−1 )
ct = ft ◦ ct−1 + it ◦ c̃t
ht = ot ◦ tanh(ct )
The forget gate controls how much is remembered from step to step. Some recom-
mend initializing the bias of the forget gate to 1 in order for it to remember more
initially [Haf17].
The gated recurrent unit (GRU) is another popular gating structure for RNNs
[Cho+14]. The GRU combines the gates in the LSTM to create a simpler update
rule with one less learned layer, lowering the complexity and increasing efficiency.
The choice between using LSTM or GRU is largely decided empirically. Despite a
number of attempts to compare the two methods, no generalizable conclusion has
been reached [Chu+14]. The GRU uses fewer parameters, so it is usually chosen
when performance is equal between the LSTM and GRU architectures. The GRU is
shown in Fig. 7.6. The equations for the update rules are shown below:
zt = σ (Wz xt + Uz ht−1 )
rt = σ (Wr xt + Ur ht−1 )
(7.27)
h̃t = tanh(Wh xt + Uh ht−1 ◦ rt )
ht = (1 − zt ) ◦ h̃t + zt ∗ ht−1
326 7 Recurrent Neural Networks
In the GRU, the new candidate state, h̃t , is combined with the previous state,
with zt determining how much of the history is carried forward or how much the
new candidate replaces the history. Similar to setting the LSTM’s forget gate bias
for improved memory in the early stages, the GRU’s reset gate biases can be set to
−1 to achieve a similar effect [Haf17].
A simple way to limit gradient explosion is to force the gradients to a specific range.
Limiting the gradient’s range can solve a number of problems, specifically prevent-
ing overflow errors when training. It is typically good practice to track the gradient
norm to understand its characteristics, and then reduce the gradient when it exceeds
the normal operating range. This concept is commonly referred to as gradient clip-
ping.
The two most common ways to clip gradients are:
• L2 norm clipping with a threshold t.
t
∇new = ∇current ◦ (7.28)
L2 (∇)
• Fixed range
⎧
⎨ tmin if ∇ < tmin
∇new = ∇ (7.29)
⎩
tmax if ∇ > tmax
Recurrent networks, like other deep learning networks, are prone to overfitting.
Dropout, being a common regularization technique, is an intuitive choice to apply
to RNNs as well, however, the original form must be modified. If the original form
of dropout is applied at each step, then the combination of masks can cause little
signal to be passed over longer sequences. Instead, we can reuse the same mask at
each step [SSB16] to prevent loss of information between time steps.
Additional techniques such as variational dropout [GG16] and zoneout [Kru+16]
have similar aims, by dropping out input or output gates in LSTMs or GRUs.
As with the entire field of deep learning, many of the architectures and techniques
are an area of active research. In this section, we describe a few architectural variants
to illustrate the expressive power and extensions of the basic RNN concepts that
have been introduced so far.
Just as we have stacked multiple fully connected and convolutional layers, we can
also stack layers of recurrent networks [EHB96]. The hidden state in a stacked RNN
composed of l vanilla RNN layers can be defined as follows:
) *
(l) (l) (l−1)
ht = f W ht−1 ; ht (7.30)
(l−1)
where ht is the output of the previous RNN layer at time t. This is illustrated
in Fig. 7.7. Anecdotally, when convolutional layers were stacked, the network was
learning a hierarchy of spatially correlated features. Similarly, when recurrent net-
works are stacked it allows longer ranges of dependencies and more complex se-
quences to be learned [Pas+13].
328 7 Recurrent Neural Networks
Because the weights in RNNs are quadratic in size, it can also be more efficient
to have multiple smaller layers rather than larger ones. Another benefit is computa-
tional optimization for fused RNN layers [AKB16].
A common problem with stacking RNNs, however, is the vanishing gradient
problem due to the depth and number of time steps. However, RNNs have been
able to gain inspiration from other areas of deep learning, incorporating residual
connections and highway networks seen in deep convolutional networks.
In Prakash et al. [Pra+16], the authors used residual connections between layers of
the LSTM to provide a stronger gradient to lower layers for the purpose of para-
phrase generation. Residual layers, typically applied in convolutional networks, al-
low “residuals” of lower level information to pass on to later layers of the network.
This provides lower level information to higher layers and also allows a larger gra-
dient to be passed to the earlier layers, because there is a more direct connection to
the output. In Kim et al. [KEL17], the authors used residual connections to improve
word error rates on a deep speech network and concluded that the lack of accu-
mulation on the highway path, while using a projection matrix to scale the LSTM
output.
In the LSTM definition in Eq. (7.26), ht is changed to:
ht = ot · (W p · tanh(ct ) + Wh xt ) (7.31)
where W p is the projection matrix and Wh is an identity matrix that matches the
sizes of xt to ht . When the dimensions of xt and ht are the same, this equation
becomes:
ht = ot · (W p · tanh(ct ) + xt ). (7.32)
Note that the output gate is applied after the addition of the input xt .
7.4 Deep RNN Architectures 329
Recurrent highway networks (RHN) [Zil+16] offer an approach to gate the gradient
propagation between recurrent layers in multilayer RNN architectures. The authors
present an extension of the LSTM architecture that allows for gated connections
between the recurrent layers, allowing an increase in the number of layers that can
be stacked for deep RNNs.
For an RHN with L layers and output s(L) , the networks is is described as:
(l) (l) (l) (l−1) (l)
st = ht · tt + st · ct
(l) (l−1)
ht = tanh WH xt 1{l=1} + RH l st + bH l
(7.33)
(l) (l−1)
tt = σ WT xt 1{l=1} + RT l st + bT l
(l) (l−1)
ct = σ WC xt 1{l=1} + RCl st + bCl
So far we have only considered the accumulation of a memory context in the forward
direction. In many situations it is desirable to know what will be encountered in
future time steps to inform the prediction at time t. Bidirectional RNNs [SP97]
allow for both the forward context and “backward” context to be incorporated into
a prediction. This is accomplished by running two RNNs over a sequence, one in
330 7 Recurrent Neural Networks
Fig. 7.9: A diagram of a two layer highway LSTM. Not that the highway connection
uses a learned gate along the connection to the next layer
the forward direction and one in the backward direction. For an input sequence
X = {x1 , x2 , . . . , xT }, the forward context RNN receives the inputs in forward order
t = {1, 2, . . . , T }, and the backward context RNN receives the inputs in reverse order
t = {T, T − 1, . . . , 1}. These two RNNs together constitute a single bidirectional
layer. Figure 7.10 shows a diagram of a bidirectional RNN.
Fig. 7.10: Diagram of a bidirectional RNN. Here the outputs are concatenated to
form a single output vector holding the forward and backward context
The output of the two RNNs, h f and hr , is often joined to form a single out-
put vector either by summing the two vectors, concatenating, averaging, or another
method.
In NLP, there are many uses for this type of structure. For example, this has
proven very useful for the task of phoneme classification in speech recognition,
where knowledge of the future context can better inform the predictions at any for-
ward time step. Bidirectional networks typically outperform forward-only RNNs in
most tasks. Furthermore, this approach can be extended to other forms of recurrent
networks such as bidirectional LSTMs (BiLSTM). These techniques follow logi-
cally, with one LSTM network operating over the inputs in the forward direction
and another with the inputs in the reverse direction, combining the outputs (con-
catenation, addition, or another method).
One limitation of bidirectional RNNs is that full input sequence must be known
before prediction, because the reverse RNN requires xT for the first computation.
Thus bidirectional RNNs cannot be used for real-time applications. However, de-
pending on the requirements of the application, having a fixed buffer for the input
can alleviate this restriction.
7.4 Deep RNN Architectures 331
The recurrent connections restrict the amount of computation that can be paral-
lelized, because the information must be processed sequentially. Thus, the com-
putational cost of RNNs is high compared to CNNs. Two techniques introduced
to speed up computation involve eliminating some of the sequential dependencies.
These techniques allow networks to become much deeper for a lower computa-
tional cost. The first technique introduces a semi-recurrent unit (SRU) [LZA17].
The approach processes the input at each time step simultaneously and applies a
light-weight recurrent computation afterward. The SRU incorporates skip and high-
way connections to improve the feature propagation in the network. The SRU is
defined as:
x̃t = Wxt
ft = σ (W f xt + b f )
rt = σ (Wr xt br ) (7.34)
ct = ft ◦ ct−1 + (1 − ft ) ◦ x̃t
ht = rt ◦ g(ct ) + (1 − rt ) ◦ xt
where f is the forget gate, r is the reset gate, and c is the memory cell.
This approach was applied to text classification, question answering, language
modeling, machine translation, and speech recognition, achieving competitive re-
sults with a reduction in training times by up to 10× over the LSTM counterpart.
The quasi-recurrent neural network (QRNN) [Bra+16] is a different approach
with the same goal. The QRNN applies convolutional layers to parallelize the input
computation that is being supplied to the reduced recurrent component. This net-
work was applied to the task of sentiment analysis and also achieved competitive
results with a significant reduction in training and prediction time.
Recursive neural networks (RecNN) are a generalized form of recurrent neural net-
works that allow effective manipulation of graphical structures. A recursive neural
network can learn information related to labeled directed acyclic graphs, while re-
current networks only process ordered sequences [GK96]. In NLP, the main appli-
cation for recursive neural networks is dependency parsing [SMN10] and learning
morphological word vectors [LSM13b].
The data is structured as a tree with parent nodes at the top and children nodes
stemming from them. The aim is to learn the appropriate graphical structure of
the data by predicting a tree and reducing the error with respect to the target tree
structure.
332 7 Recurrent Neural Networks
where W is the weight matrix for the shared layer and U is the weight matrix for the
score computation.
The score of a tree is the sum of the scores at each node:
S= ∑ sn (7.36)
n∈nodes
The error computation for recursive neural networks uses max-margin parsing:
The quantity Δ (y, yi ) computes the loss for all incorrect decisions
Backpropagation through structure (BPTS) similar to BPTT computes the deriva-
tives at each node in the graph. The derivatives are split at each node and passed on
7.5 Extensions of Recurrent Networks 333
to the children nodes. In addition to the gradient with respect to the predicted node,
we also compute the gradient with respect to the score values as well.
LSTM and GRU units have also been applied to recursive networks to combat the
vanishing gradient problem [TSM15]. Recursive networks have been used in areas,
such as relation classification [Soc+12], sentiment analysis [Soc+13], and phrase
similarity [TSM15].
Recursive neural networks demonstrate powerful extensions to sequence-based
neural architectures. Although their use has decreased in popularity with the in-
troduction of attention-based architectures, the concepts they present for improved
computational efficiency are useful.
Recurrent neural networks can be used to accomplish many types of sequence prob-
lems. Until now, we have focused on a many-to-many example with a one-to-one
mapping from an input to an output with the same number of time steps. However,
RNNs can be used for many types of sequence oriented problems by modifying
where the error is computed. Figure 7.12 shows the types of sequence problems that
can be solved with RNNs.
Fig. 7.12: Recurrent neural networks can address a variety of sequence-based prob-
lems. (a) shows a one-to-one sequence (this would be equivalent to deep neural
networks with shared weights). (b) illustrates a one-to-many sequence task, gener-
ating a series of outputs given one input. (c) is a many-to-one task, which could
represent a text classification task, predicting a single classification after the en-
tire text has been seen. (d) shows a many-to-many sequence task with a one-to-one
alignment between the number of input and output time steps. This structure is com-
mon in language modeling. (e) shows a many-to-many without a specific alignment
between the input and output. The number of inputs and outputs steps can also be
different lengths. This technique is commonly referred to as sequence-to-sequence
and is commonly seen in neural machine translation
334 7 Recurrent Neural Networks
RNNs have tremendous flexibility and can be extended to address a wide range of
sequence tasks. The limitations of feed-forward neural networks remain: a tendency
to overfit without proper regularization, need for large datasets, and computational
requirements. Additionally, sequence models introduce other considerations, such
as vanishing gradients with longer sequence lengths and “forgetting” of earlier con-
text. These difficulties have led to various extensions, best practices, and techniques
to alleviate these issues.
7.5.1 Sequence-to-Sequence
Many NLP and speech tasks are sequence oriented. One of the most common ar-
chitectural approaches for these tasks is sequence-to-sequence, often abbreviated
seq-to-seq or seq2seq. The seq-to-seq approach resembles an autoencoder, having a
recurrent encoder and a recurrent decoder, shown in Fig. 7.13. The final hidden state
of the encoder functioning as the “encoding” passed to the decoder; however, it is
typically trained in a supervised fashion with a specific output sequence. The seq-to-
seq approach was born out of neural machine translation, having an input sentence
in one language and a corresponding output sentence in a separate language. The
aim is to summarize the input with encoder and decode to the new domain with the
decoder.
Fig. 7.13: Seq-to-seq model with an RNN-based encoder and decoder. The first
hidden state of the decoder is the last hidden state of the encoder (shown in yellow)
One difficulty of this approach is that the hidden state tends to reflect the most
recent information, losing memory of earlier content. This forces a limitation for
long sequences, where all information about that sequence must be summarized
into a single encoding.
7.5 Extensions of Recurrent Networks 335
7.5.2 Attention
Forcing a single vector to summarize all the information from the previous time
steps is a drawback. In most applications, the information from a generated sequence
from the decoder will have some correlation with the input sequence. For example,
in machine translation, the beginning of the output sentence likely depends on the
beginning of the input sentence and less so on the end of the sentence, which has
been seen more recently. In many situations, it would be helpful to have not only the
summarized knowledge, but also the ability to focus on different parts of the input
to better inform the output at a particular time step.
Attention [BCB14a] has been one of the most popular techniques to address this
issue by paying specific attention to parts of the sequence for each word in the output
sequence. Not only does this allow us to improve the quality of our predictions, it
also allows insight into the network by viewing what inputs were relied upon for the
prediction.
If si is the attention augmented hidden state at time i, it takes three inputs:
• the previous hidden state of the decoder si−1 ,
• the prediction from the previous time step yi−1 , and
• a context vector ci which weighs the appropriate hidden states for the given time
step.
si = f (si−1 , si , ci ) (7.38)
The context vector, ci , is defined as:
Tx
max − marginparsingi = ∑ αi j h j . (7.39)
j=1
exp(ei j )
αi j = Tx
(7.40)
∑k=1 exp(eik )
and
ei j = a(si−1 , h j ). (7.41)
The function a(s, h) is referred to as the alignment model. This function scores how
influential input h j should be on the output at position i.
It is fully differentiable and deterministic because it is considering all time steps
that have contributed to the output. A drawback of using all of the previous time
steps is that it requires a large amount of computation for long sequences. Other
techniques relax this dependency by being selective about the number of states that
inform the context vector. Doing this creates a non-differentiable loss, however, and
training requires Monte Carlo sampling for the estimation of the gradient during
backpropagation.
336 7 Recurrent Neural Networks
An additional benefit of attention is that it provides a score for each time step,
identifying what inputs were most useful for the prediction. This can be very useful
for interpretability when inspecting the quality of a network or gaining intuition
about what the model is learning, as shown in Fig. 7.15. Attention mechanisms are
covered in more detail in Chap. 9.
Fig. 7.14: Attention is applied to the first step of decoding for a neural machine
translation model. A similarity score is computed for the hidden state at each time
step in the encoder and the current hidden state of the decoder. These scores are
used to weigh the contribution of that time step. These weights are used to produce
the context vector that is supplied to the decoder
uij = v tanh(W1 e j + W2 di )
(7.42)
p(Ci |C1 , . . . ,Ci−1,P ) = softmax(ui )
where e j is the output of the encoder at time j ∈ {1, . . . , n}, di is the decoder output at
time step i, and Ci is the index at time i, and v, W1 , and W2 are learnable parameters.
should
<eos>
<sos>
you
see
try
to
it
.
vous
devriez 0.8
essayer
0.6
de
le
0.4
voir
. 0.2
<eos>
0.0
This model showed success finding planar convex hulls, computing Delaunay
triangulations, and producing solutions to the traveling salesman problem.
The success of attention on seq-to-seq tasks prompts the question of whether it can
be directly applied to the input, reducing or even eliminating the need for recurrent
connections in the network. Transformer networks [Vas+17b] applied this attention
directly to the input with great success, beating both recurrent and convolutional
models in machine translation. Instead of relying on RNNs to accumulate a memory
of previous states as in sequence-to-sequence models, the transformer uses “multi-
headed” attention directly on the input embeddings. This alleviates the sequential
dependencies of the network allowing much of the computation to be performed in
parallel.
Attention is applied directly to the input sequence, as well as the output sequence
as it is being predicted. The encoder and decoder portions are combined, using an-
338 7 Recurrent Neural Networks
other attention mechanism before predicting a probability distribution over the out-
put dictionary.
Multi-head attention, shown in Fig. 7.16, is defined by three input matrices: Q
the set of queries packed into a matrix, keys K, and values V.
QK
Attention(Q, K, V) = softmax √ V (7.43)
dk
where
headi (Q, K, V) = Attention QWQ
i , KW K
i , VVQ
i . (7.45)
The encoder and decoder apply multiple layers of multi-head attention with resid-
ual connections and additional fully connected layers. Because much of the compu-
tation is happening in parallel, a masking technique and offsetting are used to ensure
that the network only uses information that is available up to time t − 1 when pre-
dicting for time t. The transformer network reduces the number of steps required for
prediction that significantly improve the computation time, while achieving state-of-
the-art results on the translation task.
7.6 Applications of RNNs in NLP 339
Figure 7.17 shows the structure of a simple text classification task for an input sen-
tence. With a recurrent network we are able to sequentially encode the word em-
beddings at each time step. Once the entire sequence is encoded, we use the last
hidden state to predict the class. The network is trained using BPTT and learns to
sequentially weigh the words for the classification task.
In Lee and Dernoncourt [LD16], the authors compared CNN and RNN architec-
tures for short-text classification. The addition of sequential information via CNN
and RNN architectures significantly improved the results on dialog act characteri-
zation.
340 7 Recurrent Neural Networks
In Huang et al. [HXY15], word features and embeddings were applied to POS, NER,
and chunking tasks with a bidirectional LSTM with CRF to boost performance. In
Ma and Hovy [MH16] a bidirectional LSTM was used for end-to-end classification
for POS on WSJ. Ma and Hovy [MH16] used an end-to-end method to improve on
these results. Their method does not rely on context features that were applied in
other works, such as POS, lexicon features, and task-dependent preprocessing. In
Lample et al. [Lam+16b], a bidirectional LSTM was used in conjunction with CRF
to achieve state-of-the-art performance on NER in four languages on the CoNLL-
2003 dataset. This work also extended the base RNN-CRF architecture to stack-
LSTM (LSTM units used to mimic a stack data structure with pushing and pulling
capabilities). Character embeddings are often incorporated in addition to word em-
beddings to capture additional information about a word’s semantic structure as well
as to inform predictions on OOV words.
In Dyer et al. [Dye+15], stack-LSTMs that allow for pushing and pulling operations
were used to predict dependency parsing for variable-length text by predicting the
dependency tree transitions. Kiperwasser and Goldberg [KG16] simplified the archi-
tecture by removing the need for the stack-LSTM, relying on bidirectional LSTMs
to predict the dependency tree transitions.
In Ghosh et al. [Gho+16], the contextual LSTM (C-LSTM) was introduced for word
prediction, sentence selection, and topic prediction. The C-LSTM concatenates a
topic embedding with the word embedding at each time step in the training of the
network. This work functions similarly to language model training in that the goal
is to predict the next word; however, it is extended to include the topic context into
the prediction as well. Thus, the aim is to predict the next word and the topic of the
sentence so far.
7.6 Applications of RNNs in NLP 341
In Tan et al. [Tan+15], the authors train a question RNN and an answer RNN to yield
a respective embedding for each. The two networks are then trained simultaneously
by using a hinge loss objective to enforce a cosine similarity between the two most
probable pairs. Another approach, dynamic memory networks [XMS16], incorpo-
rates a series of components to make up a question answering system. This system
used a combination of recurrent networks and attention mechanisms to construct
input, question, and answer modules that utilize an episodic memory to condition
the predictions.
7.6.6 Multi-Modal
The effectiveness of deep learning in other applications such as images and video
has led to a variety of multi-modal applications. These applications require gener-
ating language based on an input medium. These applications include image and
video captioning, visual question answering, and visual speech recognition.
Image captioning was one of the first ways that deep convolutional networks for
images were combined with text. In Vinyals et al. [Vin+15b], the authors utilized
a pre-trained convolutional network for image classification to generate an image
embedding for the initial state of an LSTM network. The LSTM network was trained
to predict each word of the caption. The initial approach led to advancements in
RNN architectures [Wan+16a].
Video captioning showed a similar development, with [Ven+14] utilizing a pre-
trained CNN model to extract image features for each video frame to be used as
input into a recurrent network for text generation. Pan et al. [Pan+15a] extended
this method by striding over the output frames of the earlier recurrent layers to
create a “hierarchical recurrent neural encoder” to reduce the number of time steps
considered in the output layer of the stacked RNN.
In visual question answering, language generation is used to generate an answer
for a textual question related to a visual input. An end-to-end approach was shown
with the neural-image-QA network in Malinowski et al. [MRF15], where the input
image and question conditioned the LSTM network to generate a textual answer.
In the previous chapters we have briefly discussed language models. Recall that a
language model provides a way to determine the probability of a sequence of words.
For example, an n-gram language model determines the probability of a sequence of
words P(w1 , . . . , wm ) by looking at the probability of each word given its n preceding
words:
m
P(w1 , . . . , wm ) ≈ ∏ P(wi |wi−(n−1) , . . . , wi−1 ). (7.46)
i=1
342 7 Recurrent Neural Networks
Language models are particularly interesting in NLP as they can provide addi-
tional contextual information to situations where a prediction might be semantically
similar, yet syntactically different. In the case of speech recognition, two words that
sound the same such as “to” and “two” have different meanings. But the phrase “set
a timer for to minutes” doesn’t make sense, whereas “set a timer for two minutes”
does.
Language models are often used when language is being generated and for do-
main adaptation (where there may be large amounts of unlabeled text data and lim-
ited labeled data). The concept of n-gram language models can also be implemented
with RNNs, which benefit from not having to set a hard cutoff for the number of
grams considered. Additionally, similar to word vectors, these models can be trained
in an unsupervised manner over a large corpus of data.
The language model is trained to predict the next word in the sequence given
the previous context, i.e., the hidden state. This allows the language model to target
learning:
m
P(w1 , . . . , wm ) = ∏ P(wi |w1 , . . . , wi−1 ). (7.47)
i=1
Fig. 7.18: An RNN language model trained to predict the next word in the sequence
given the entire history of the sequence. Note that each time step is focused on
classification, therefore the target outputs are the size of the vocabulary, not the size
of the input word embeddings
when the output contains a large number of elements. This projection layer reduces
the size of the final linear projection, as is often the case in language modeling
[MDB17].
7.6.7.1 Perplexity
Perplexity is a measure of how well a model can represent the domain, shown by
its ability to predicting a sample. For language models, perplexity can quantify the
language model’s ability to predict the validation or test data. The language model
performs well if it produces a high probability for a sentence in the test set. Perplex-
ity is the inverse probability normalized by the number of words.
We can define the perplexity measure for a test set of sentences (s1 , . . . , sm ) with:
1
PP(s1 , . . . , sm ) = 2− M ∑i=1 log2 p(si )
m
(7.48)
where M is the vocabulary size of the test set. Because perplexity gives the inverse
probability of the dataset, a lower perplexity implies a better result.
Machine translation has been one of the largest benefactors of the success of recur-
rent neural networks. Traditional approaches were based around statistical models
that were computationally expensive and required heavy domain expertise to tune
them. Machine translation is a natural fit for RNNs because input sentences may
differ in length and order from the desired output. Early architectures for neural ma-
chine translation (NMT) relied on a recurrent encoder–decoder architecture. A very
simple illustration of this is shown in Fig. 7.19.
NMT takes an input sequence of words X = (x1 , . . . , xm ) and maps them to an
output sequence Y = (y1 , . . . , yn ). Note n is not necessarily the same as m. Using
an embedding space, input X is mapped to a vector representation that is utilized
by a recurrent network to encode the sequence. A decoder then uses the final RNN
hidden state (the encoded input) to predict the translated sequence of words, Y (some
have also shown success with subword translation [DN17]).
344 7 Recurrent Neural Networks
Fig. 7.19: Diagram of a one hidden layer encoder–decoder neural machine trans-
lation architecture. Note how the input and output sequences can have different
lengths, and are truncated when the end of sentence (<EOS>)tag is reached
Fig. 7.20: A one hidden layer encoder–decoder neural machine translation architec-
ture with the previous word of the prediction being used as the input at the following
time step. Note that the embedding matrix has entries for both languages
7.6 Applications of RNNs in NLP 345
In practice, the encoder and decoder do not need to be more than 2–4 layers deep,
and bidirectional encoders usually outperform unidirectional ones [Bri+17].
7.6.8.1 BLEU
The most common metric used to evaluate machine translation is BLEU. BLEU
(bilingual evaluation understudy) is a quality evaluation metric for machine trans-
lation, designed to align with human evaluations of natural language. It allows a
translation to be compared with a set of target translations to evaluate the quality.
The score is bound between 0 and 1 with higher values indicating better per-
formance. Often in literature, the score will be multiplied by 100 to approximate a
percentage correlation. At its core the BLEU score is a precision measurement. It
computes the precision for reference n-grams in the targets.
A perfect match would look like the following:
1 from n l t k . t r a n s l a t e . b l e u s c o r e i m p o r t s e n t e n c e b l e u
2 t a r g e t s = [ [ ’ i ’ , ’ had ’ , ’ a ’ , ’ cup ’ , ’ of ’ , ’ b l a c k ’ , ’ c o f f e e ’ , ’
at ’ , ’ the ’ , ’ cafe ’ ] ]
3 p r e d i c t i o n = [ ’ i ’ , ’ had ’ , ’ a ’ , ’ cup ’ , ’ of ’ , ’ b l a c k ’ , ’ c o f f e e ’ ,
’ at ’ , ’ the ’ , ’ cafe ’ ]
4 s c o r e = s e n t e n c e b l e u ( t a r g e t s , p r e d i c t i o n ) ∗ 100
5 print ( score )
6
7 > 100.0
Alternatively, if none of the reference words are present in the prediction, then
we get a score of 0.
1 from n l t k . t r a n s l a t e . b l e u s c o r e i m p o r t s e n t e n c e b l e u
2 t a r g e t s = [ [ ’ i ’ , ’ had ’ , ’ a ’ , ’ cup ’ , ’ of ’ , ’ b l a c k ’ , ’ c o f f e e ’ , ’
at ’ , ’ the ’ , ’ cafe ’ ] ]
3 p r e d i c t i o n = [ ’ what ’ , ’ a r e ’ , ’we ’ , ’ d o i n g ’ ]
4 s c o r e = s e n t e n c e b l e u ( t a r g e t s , p r e d i c t i o n ) ∗ 100
5 print ( score )
6
7 > 0
If we change one or two words in the transcript, then we see a drop in the score.
1 from n l t k . t r a n s l a t e . b l e u s c o r e i m p o r t s e n t e n c e b l e u
2 t a r g e t s = [ [ ’ i ’ , ’ had ’ , ’ a ’ , ’ cup ’ , ’ of ’ , ’ b l a c k ’ , ’ c o f f e e ’ , ’
at ’ , ’ the ’ , ’ cafe ’ ] ]
3 p r e d i c t i o n = [ ’ i ’ , ’ had ’ , ’ a ’ , ’ cup ’ , ’ of ’ , ’ b l a c k ’ , ’ t e a ’ , ’
at ’ , ’ the ’ , ’ cafe ’ ]
4 s c o r e = s e n t e n c e b l e u ( t a r g e t s , p r e d i c t i o n ) ∗ 100
5 print ( score )
6
7 > 65.8037
8
9 t a r g e t s = [ [ ’ i ’ , ’ had ’ , ’ a ’ , ’ cup ’ , ’ of ’ , ’ b l a c k ’ , ’ c o f f e e ’ , ’
at ’ , ’ the ’ , ’ cafe ’ ] ]
346 7 Recurrent Neural Networks
10 p r e d i c t i o n = [ ’ i ’ , ’ had ’ , ’ a ’ , ’ cup ’ , ’ of ’ , ’ b l a c k ’ , ’ t e a ’ , ’
a t ’ , ’ t h e ’ , ’ house ’ ]
11 s c o r e = s e n t e n c e b l e u ( t a r g e t s , p r e d i c t i o n ) ∗ 100
12 print ( score )
13
14 > 58.1430
If we predict the most likely word at each step, we may not yield the best sequence
probability over all. The best decision early in the process may not maximize the
overall probability of the sequence. In fact there is a decision tree of possibilities
to be decoded for the best possible outcome. Because of the tree-like structure in
language model outputs, there are a variety of methods to parse them.
Another way we can parse the output of our model is by using a random search. In
a random search, the next word in the sentence is chosen according to the probabil-
ity distribution of the next state. The random sampling technique can help achieve
diversity in the results. However, sometimes the predictions for language models
can be very confident, making the output results look similar to the greedy search
results. A common way to improve the diversity of the predictions is to use a con-
cept called temperature. Temperature is a method that exponentially transforms the
probabilities and renormalizes to redistribute the highest probabilities among the top
classes.
One method of sampling from the language model is to use “temperature sam-
pling.” This method selects an output prediction by applying a freezing function,
defined by:
1
piτ
fτ (p)i = 1 (7.49)
∑ j p jτ
7.6 Applications of RNNs in NLP 347
where τ ∈ [0, 1] is the temperature parameter that controls how “warm” the predic-
tions are. The lower the temperature, the less diverse the results.
Another desirable quality for NLP is language generation. In Sutskever et al.
[SVL14b], a deep RNN-based encoder–decoder architecture is used to generate
unique sentences. The networks encode a “source” word sequence into a fixed length
“encoding,” a vector via an RNN. The decoder uses the “encoding” as the initial hid-
den state, and produces the response.
Greedy search makes an independence assumption between each time step for the
decoding. We are relying on our RNNs to correctly inform the dependency between
each time step. We can provide a prior to our predictions to ensure that we avoid
simple errors (such as conjugation). We can do this by biasing our prediction on
a scoring mechanism that informs whether a particular sequence is more probable
than another.
When using our trained model to predict on new data, we are relying on the
model to produce the correct output given the most confident prediction. However,
in many situations it is desirable to impose a prior on the output, biasing it towards
a particular domain. For example, in speech recognition an acoustic model’s perfor-
mance can be greatly improved by incorporating a language model as the bias for
the output predictions.
The output that we are obtaining from our machine translation model, for exam-
ple, is a probability distribution over the vocabulary at each time step, thus creating
a tree of possibilities for how we could parse the output.
Often it is too computationally expensive to explore the entire tree of possibil-
ities, so the most common search method is the beam search. Beam search is a
searching algorithm that keeps a fixed maximum on the number of possible states
in memory. This provides a flexible approach to optimizing the output of a network
after it is trained, balancing speed and quality.
If we consider the output sequence of our network (y1 , . . . , ym ) where yt is a
softmax output over our vocabulary, then we can compute the probability of the
overall sequence with the product of the probabilities at each time step :
m
P (y1 , . . . , ym ) = ∏ pi (yi ) (7.50)
i=1
Let y be a sequence of words and P(y) be the probability of that sequence ac-
cording to our language model. We will use a beam search to explore multiple hy-
potheses of sequences at time t, Ht−1 , with a beam size of k.
+ ,
Ht := w11 , . . . , wt1 , . . . , wk1 , . . . , wtk
H3 := {(cup of tea), (cup of coffee)}
With beam search we keep track of our top k hypotheses, and choose the path
that maximizes P(y). We will collect the probability of each hypothesis P(ht ) in Pt .
The index order of Ht and Pt should be tied to keep them in sequence when sorting.
We begin each hypothesis with the <SOS> token and end the hypothesis once the
<EOS> token is reached. The hypothesis with the highest score is the one that is
selected.
Ht + = ŷ
Pt + = P(ŷ)
Here, we apply the concepts of recurrent neural networks for neural machine transla-
tion. Specifically, basic RNN, LSTM, GRU, and transformer sequence-to-sequence
architectures are explored with an English-to-French translation task. We begin
with an exploratory process of generating a dataset for the task. Next we explore
sequence-to-sequence architectures, comparing the effects of various hyperparame-
ters and architecture designs on quality.
The dataset we use is a large set of English sentences with French translations
from the Tatoeba website. The original data is a raw set of paired examples with no
designated train, val, and test splits, so we create these during the EDA process.
7.7 Case Study 349
The raw format of the text contained in the Tatoeba dataset is a tab-separated En-
glish sentence followed by the French translation, with one pair per line. Counting
the number of lines gives us a total of 135, 842 English–French pairs. By selecting
a few random samples, as in Fig. 7.21, we can see that it contains punctuation, capi-
talization, as well as unicode characters. Unicode should come as no surprise when
considering a translation task; however, it must be considered when dealing with
any computational representation due to variations in libraries and their support for
unicode characters.
First, we inspect the sequence lengths in the dataset. We use spaCy for both En-
glish and French tokenization. The tokenizers can be applied by torchtext fields
when reading in the data, automatically applying the tokenizer. Fields in torchtext
are generic data types for a dataset. In our example, there are two types of fields,
a source field represented as “SRC” which will contain details on how the English
sentences should be processed, while a second field called the “TRG” contains the
target French data and its type handling. We can attach a tokenizer to each as fol-
lows.
350 7 Recurrent Neural Networks
1 def t o k e n i z e f r ( t e x t ) :
2 ”””
3 T o k e n i z e s F r e n c h t e x t from a s t r i n g i n t o a l i s t o f s t r i n g s
4 ”””
5 r e t u r n [ tok . t e x t for tok in spacy fr . tokenizer ( t e x t ) ]
6
7 def tokenize en ( t e x t ) :
8 ”””
9 T o k e n i z e s E n g l i s h t e x t from a s t r i n g i n t o a l i s t o f s t r i n g s
10 ”””
11 r e t u r n [ tok . t e x t for tok in spacy en . tokenizer ( t e x t ) ]
12
13 SRC = F i e l d ( t o k e n i z e = t o k e n i z e e n , i n i t t o k e n = ’< s o s > ’ ,
e o s t o k e n = ’< eos > ’ , l o w e r = T r u e )
14 TRG = F i e l d ( t o k e n i z e = t o k e n i z e f r , i n i t t o k e n = ’< s o s > ’ ,
e o s t o k e n = ’< eos > ’ , l o w e r = T r u e )
15
16 SRC . b u i l d v o c a b ( t r a i n d a t a , m i n f r e q = 0 )
17 TRG . b u i l d v o c a b ( t r a i n d a t a , m i n f r e q = 0 )
Torchtext can take a tokenizer of any type, as it is just a function that operates
on the text that is passed in. The tokenizers in spaCy are useful as they have stop
words, token exceptions, and various types of punctuation handling.
Another consideration when training sequence-based models is the length of the
examples. We plot a histogram of the sequence lengths in Fig. 7.22. The longer sen-
Fig. 7.22: Histogram of sentence lengths for both English and French. Notice that a
majority of the sentences are short, and there are very few long sentences
tences are likely to hold a more complex structure, and likely have longer range
dependencies. We would not expect to learn these examples, as they are under rep-
resented in the dataset. If we desire to learn translation for longer sentences, we
would have to collect more data, or intelligently break long examples into shorter
ones, where we have more data. Additionally, long examples can lead to memory
concerns with mini-batch training, as the batch size can be larger with shorter ex-
amples.
7.7 Case Study 351
For this case study, we remove longer examples by setting a threshold on the
length of our examples. We select a limit of 20 time steps on input or output se-
quence, which will allow for a maximum of 18 actual words in the sequences af-
ter incorporating the < sos > and < eos > tokens. This restriction means our max
length incorporates all of the sequence lengths that have significant data. The result-
ing length distribution is shown in Fig. 7.23.
20000
15000
10000
5000
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Example label
Fig. 7.23: Histogram of sentence lengths for both English and French after filtering
the max length to 18 (20 if we include < sos > and < eos > tokens)
After filtering the longer examples we create our training, validation, and testing
splits, without replacement using a shuffling index technique shown below.
1 n examples = len ( a l l d a t a )
2 i d x a r r a y = l i s t ( range ( n examples ) )
3 random . s h u f f l e ( i d x a r r a y )
4 t r a i n i n d e x s = i d x a r r a y [ : i n t ( 0 . 8 ∗ n e x a m p l e s ) ] # 80% t r a i n i n g
data
5 v a l i n d e x s = i d x a r r a y [ i n t (0.8∗ n examples ) : i n t (0.9∗ n examples )
] # 10% v a l i d a t i o n d a t a
6 t e s t i n d e x s = i d x a r r a y [ i n t ( 0 . 9 ∗ n e x a m p l e s ) : ] # 10% t e s t i n g
data
This technique should provide each split of the datasets with similar characteristic.
The final dataset allocates 80% for training, 10% for validation, and 10% for testing.
We save the data into files so that they can be used in other experiments if desired,
without having to repeat all the preprocessing. Inspecting the resulting data splits
shows a similar length distribution for each, as depicted in Fig. 7.24.
The vocabulary object offers many common NLP functions, such as indexed access
to the terms, simplified embedding creation, and frequency filtering.
We now load and tokenize the data splits. The overall vocabulary size is:
352 7 Recurrent Neural Networks
Number of examples
15000
12500
10000
7500
5000
2500
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Example label
(a)
Validation Sentence Lengths
2500 English
French
Number of examples
2000
1500
1000
500
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Example label
(b)
Testing Sentence Lengths
2500
English
French
2000
Number of examples
1500
1000
500
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Example label
(c)
Fig. 7.24: Histogram of sentence lengths for the (a) training data, (b) validation data,
and (c) testing data
5 SRC . b u i l d v o c a b ( t r a i n d a t a , m i n f r e q = 0 )
6 TRG . b u i l d v o c a b ( t r a i n d a t a , m i n f r e q = 0 )
7
7.7 Case Study 353
8 p r i n t ( ” E n g l i s h v o c a b u l a r y s i z e : ” , l e n ( SRC . v o c a b ) )
9 p r i n t ( ” F r e n c h v o c a b u l a r y s i z e : ” , l e n (TRG . v o c a b ) )
10
11 > E n g l i s h v o c a b u l a r y s i z e : 12227
12 > F r e n c h v o c a b u l a r y s i z e : 20876
The vocabulary frequencies are shown in Fig. 7.25. The distribution displays a “long
tail” effect, where a small subset of tokens have high counts, for example “.” that
occurs in almost all sentences, and other tokens that are seen just once, for ex-
ample “stitch.” In the most extreme case of a word only appearing once, training
relies solely on that single example to inform the model, likely leading to overfit-
ting. Additionally, the distribution for the softmax will assign some probability to
104
Word frequency
103
102
101
100
0 2000 4000 6000 8000 10000 12000
Vocabulary index (sorted by count length)
(a)
Unfiltered French Vocabulary
105
104
Word frequency
103
102
101
100
0 2000 4000 6000 8000 10000 12000
Vocabulary index (sorted by count length)
(b)
Fig. 7.25: Unfiltered word frequency for (a) English and (b) French. The counts
were sorted and placed on a log scale to capture the severity of the word representa-
tions in this dataset. As we can see, there are many words that are used rarely, while
a small subset is used frequently
these terms. As the infrequent words occupy the majority of the vocabulary, much
of the probability mass will be assigned to these terms in the early stages, slowing
learning. A common approach is to map infrequent words to the unknown token,
< unk >. This allows the model to ignore a likely invalid representation of an un-
354 7 Recurrent Neural Networks
When investigating the final vocabulary, we still notice that there is a long tail
distribution for the frequency of the words, shown in Fig. 7.26. This shouldn’t be too
much of a surprise, given that we chose a minimum frequency of 5. If the threshold
is too high, removing many words, then the model becomes too restricted in its
learning, with many values mapping to the unknown token.
105 105
104 104
Word frequency
Word frequency
103 103
102 102
101 101
0
10 100
0 2000 4000 6000 8000 10000 12000 0 2500 5000 7500 10000 12500 15000 17500 20000
Vocabulary index (sorted by count length) Vocabulary index (sorted by count length)
(a) (b)
Fig. 7.26: Term frequency graph for the filtered vocabulary of the (a) English and
(b) French training data
Figure 7.27 shows the top 50 terms for English and French in the training set. An
analysis of the list leads to some interesting questions about the data. For example,
one of the most common words shown in the vocabulary list is the word “n’t.” This
seems odd since there is no word “n’t,” in the English language. A deeper inspection
reveals that spaCy tokenization splits contractions in this way, leaving an isolated
token “n’t” whenever a contraction such as “don’t” or “can’t” appear. The same situ-
ation occurs when the contraction “I’m” is processed. This illustrates the importance
of iterative improvements on data, as preprocessing is a fundamental component of
the feature generation, as well as post-processing if results are computed on the final
output.
7.7 Case Study 355
Fig. 7.27: Frequency counts for the top 20 terms from the training set for (a) English
and (b) French
Now that the dataset is ready, we investigate models and their performance on the
training and validation sets. Specifically, we focus on various simple RNNs, LSTMs,
and GRUs. Each of these architectures is investigated with respect to learning rate,
depth, and bidirectionality. Each technique involves optimization of multiple hy-
perparameters to regularize the network, while also changing the training dynamics
of the network. To alleviate a full grid search over all possible hyperparameters,
we only tune learning rate to the introduced architecture. This does not completely
alleviate the need to tune other parameters, but it makes the problem tractable.
Each model we train utilizes the script shown in Fig. 7.28. Note GRU and RNN
configurations are not implemented in fairseq. We added these to the library for the
purposes of this comparison.
Each model is trained for a maximum of 100 epochs. We reduce the learning
rate when validation performance plateaus, and stop when learning plateaus. The
embedding dimension is fixed at 256 and dropout for the input and output are set
to 0.2. For simplicity, we fix the hidden size to 512 for all experiments (except
for bidirectional architectures). A bidirectional provides two hidden states to the
decoder, and therefore the decoder size must double. Some may argue that com-
356 7 Recurrent Neural Networks
1 p y t h o n t r a i n . py d a t a s e t s / en−f r \
2 −−a r c h { r n n t y p e } \
3 −−e n c o d e r −d r o p o u t −o u t 0 . 2 \
4 −−e n c o d e r −l a y e r s { n l a y e r s } \
5 −−e n c o d e r −h i d d e n −s i z e 512 \
6 −−e n c o d e r −embed−dim 256 \
7 −−d e c o d e r −l a y e r s { n l a y e r s } \
8 −−d e c o d e r −embed−dim 256 \
9 −−d e c o d e r −h i d d e n −s i z e 512 \
10 −−d e c o d e r − a t t e n t i o n F a l s e \
11 −−d e c o d e r −d r o p o u t −o u t 0 . 2 \
12 −−o p t i m i z e r adam −− l r { l r } \
13 −−l r −s h r i n k 0 . 5 −−max−e p o c h 100 \
14 −−s e e d 1 −−l o g −f o r m a t j s o n \
15 −−num−w o r k e r s 4 \
16 −−b a t c h −s i z e 512 \
17 −−w e i g h t −d e c a y 0
Fig. 7.28: Base training configuration for our fairseq model training. The rnn type,
number of layers, and learning rate (lr) can be controlled by inserting parameter
appropriately
parability between models would only be achieved if the models have the same
number of parameters. For example, LSTMs have roughly 4× the number of pa-
rameters as standard RNNs; however, for simplicity and clarity, we maintain a fixed
hidden representation. In the following figures, each model name takes the form,
{rnn type} {lr} {num layers} {metric}.
Validation Loss
5
0 10 20 30 40 50
Epoch
Fig. 7.29: Validation loss for a single layer RNN with different learning rates on
English–French translation
Next, we compare RNN, LSTM, and GRU architectures. We vary the learning rate
for each, as the dynamics are likely different for each architecture. The validation
results are shown below in Fig. 7.30.
Upon inspection, we notice that some configurations take much longer to con-
verge than others. In particular, with a learning rate of 0.0001, both the GRU and
LSTM architectures reach the maximum 100 epochs. Secondly, we see that the
LSTM and GRU architectures converge to lower losses, much faster, and with higher
learning rates than RNN architectures. The GRU appears to be the best performing
model here, but both the LSTM and GRU show similar convergence.
We now compare the effect of depth on each architecture. Here, we vary the depth
configuration in addition to the learning rate for each architecture. The depths ex-
plored are 1, 2, and 4-layers deep. Results are displayed in Fig. 7.31.
Now that we have many models, it becomes more difficult to draw general con-
clusions about their properties. If we compare the RNN models, we notice that many
of the configurations converge to a much higher validation loss than either the GRU
or LSTM architectures. We also observe that the deeper architectures tend to per-
form well with lower learning rates than their shallower counterparts. Additionally,
358 7 Recurrent Neural Networks
Istm_0.005_1_valid_loss
5 mn_0.0001_1_valid_loss
mn_0.0005_1_valid_loss
mn_0.001_1_valid_loss
4 mn_0.005_1_valid_loss
0 20 40 60 80 100
Epoch
Fig. 7.30: Comparison of single layer RNN, GRU, and LSTM networks on English–
French translation
both the LSTM and GRU architectures achieve their best models with a depth of 2
layers and a learning rate of 0.001.
Next, we look at the effects of bidirectional models. Many of the models perform
similarly. Figure 7.32b shows the perplexity of the models predictions (ppl) instead
of the validation loss. This value is 2loss , exaggerating the effects in the graph, which
can be useful when visibly inspecting curves.
Once again we see that the LSTM and GRU architectures outperform the RNN
architectures, with the GRU architecture performing slightly better.
So far, the best performing models have been the 2-layer LSTM and GRU models
and the single layer bidirectional LSTM and GRU models. Here, we combine the
two components to see if the benefits are complimentary. In this set of experiments
we remove the under-performing RNN models for clarity. The results are shown in
Fig. 7.33.
Fig. 7.31: Depth comparison for (a) RNN, (b) LSTM, and (c) GRU architectures
360 7 Recurrent Neural Networks
Fig. 7.32: Comparison of (a) validation loss and (b) ppl for single layer, bidirectional
RNN, GRU, and LSTM networks. Note that although the colors are similar, the top
two lines are RNN models (not GRU models)
This set of results shows that the 2-layer GRU architecture, with a learning rate
of 0.001, is the best model in the bidirectional comparison.
We now turn our attention to the transformer architecture, where attention is applied
directly to the input sequence without incorporating recurrent networks. Similar to
previous experiments, we fix the input and output dimensionality to 256, set 4 atten-
tion heads in both the encoder and decoder, and fix the fully connected layers size to
512. We explore a small selection of depths and vary the learning rates accordingly.
The results are shown in Fig. 7.34, with the 4-layer transformer architecture using a
learning rate of 0.0005 performing the best.
7.7 Case Study 361
7 bidirectional_gru_0.0001_2_valid_loss
bidirectional_gru_0.0005_2_valid_loss
bidirectional_gru_0.001_2_valid_loss
bidirectional_gru_0.005_2_valid_loss
6 bidirectional_Istm_0.0001_2_valid_loss
bidirectional_Istm_0.0005_2_valid_loss
bidirectional_Istm_0.001_2_valid_loss
Validation Loss
5 bidirectional_Istm_0.005_2_valid_loss
0 10 20 30 40
Epoch
Transformer Comparison
7
baseline_transformer_0.0001_4_valid_loss
baseline_transformer_0.0005_4_valid_loss
baseline_transformer_0.001_1_valid_loss
6 baseline_transformer_0.001_2_valid_loss
baseline_transformer_0.001_4_valid_loss
5
Validation Loss
1
0 5 10 15 20 25 30 35 40
Epoch
Fig. 7.34: Comparison of transformer architectures with different learning rates and
depths. Note the depth is the same for both the encoder and decoder
362 7 Recurrent Neural Networks
0 10 20 30 40
Epoch
Having explored many types of architectures for machine translation, we now com-
pare the outputs of each experiment. This set includes the best performing RNN
from the baseline experiments, the single-layer unidirectional and bidirectional
GRU, the 2-layer unidirectional and bidirectional GRU, and the 4-layer transformer
network. Comparing loss of these models on the validation set (Fig. 7.35) we see
that the 4-layer transformer network is our best performer.
7.7.4 Results
We now compare the models from each experiment on the test set (Table 7.1).
Table 7.1: NMT network performance on the test set. The best result is highlighted
When we sample outputs from the model (Fig. 7.36) we see that the results look
pretty good. Notice how the model may produce reasonable translations even though
it may not predict the target exactly.
In conclusion, we have shown that for our task it is almost always preferable to
use GRU or LSTM architectures over base RNNs. Additionally, we have shown that
the initial learning rate has a significant impact on a model’s quality, even when us-
ing adaptive learning rate methods. Furthermore, the learning rate needs to be tuned
for each configuration of the model given the dynamic nature of the deep networks.
Lastly, deeper networks are not always better. On this dataset, the 2-layer recurrent
architectures outperformed 4-layer counterparts. And a single-layer, bidirectional
GRU showed marginal improvements over the 2-layer counterpart on the final test-
ing set, even though it performed slightly worse on the validation loss comparison.
These results show the importance of tuning hyperparameters for not only the ap-
plication, but also the dataset. In real-world applications, it is recommended to tune
as many hyperparameters as possible to achieve the best result.
7.8 Discussion
The results from RNNs on many NLP tasks are quite impressive, achieving state-
of-the-art results in almost every area. Their effectiveness is remarkable given their
simplicity. However, in practice, real-world settings require additional considera-
tions, such as small datasets, lack of diversity in data, and generalization. Following
is a short discussion focusing on these concerns and common debates that arise.
All of the deep learning techniques that have been discussed so far come with the
risk of overfitting. Additionally, many of the academic tasks for various NLP tasks
are heavily focused on a particular problem with ample data that may not represent a
real-world task.5 The correlations between training and testing data allow some level
of overfitting to be advantageous to both the validation and testing sets; however, it
is arguable whether or not these correlations are just representative of the domain
itself. The difficulty is knowing whether or not the network is memorizing certain
sequences that are significant to lower the overall cost or learning correlations of un-
derlying semantic structure of the problem. Some of the symptoms of memorization
are illustrated in the need for decoding algorithms such as beam search and random
selection with temperature to produce variety in the output sequences.
In Ref. [Gre16], Grefenstette explored the question of whether or not recurrent
networks are capable of learning push-down automate, which is arguably the sim-
plest form of computation required for natural language. This work cites some of
the limitations of “simple RNNs” as:
• Non-adaptive capacity
• Target sequence modeling dominates training
• Gradient-starved encoder.
The suggestion, focused specifically on simple RNNs, was that RNNs are arguably
only capable of learning finite state machines.
In Liska et al. [LKB18], the authors studied the ability of RNNs to learn a com-
position structure, which would show an RNN’s ability to transfer learning from one
task to another. A small number of the RNNs in the experiment showed that it was
possible to learn compositional solutions without architectural constraints, although
many of the RNN attempts were not successful. The results achieved show that gra-
dient descent and evolutionary strategies may be a compelling direction for learning
compositional structures.
5This is not to say that academic benchmarks are not relevant, but rather to point out the importance
of domain and technological understanding for domain adaptation.
References 365
References
[Soc+13] Richard Socher et al. “Reasoning with neural tensor networks for
knowledge base completion”. In: Advances in neural information pro-
cessing systems. 2013, pp. 926–934.
[SVL14b] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. “Sequence to sequence
learning with neural networks”. In: Advances in neural information
processing systems. 2014, pp. 3104–3112.
[TSM15] Kai Sheng Tai, Richard Socher, and Christopher D Manning. “Im-
proved semantic representations from tree-structured long short-term
memory networks”. In: arXiv preprint arXiv:1503.00075 (2015).
[Tan+15] Ming Tan et al. “LSTM-based deep learning models for non-factoid
answer selection”. In: arXiv preprint arXiv:1511.04108 (2015).
[Vas+17b] Ashish Vaswani et al. “Attention is all you need”. In: Advances in
Neural Information Processing Systems. 2017, pp. 5998–6008.
[Ven+14] Subhashini Venugopalan et al. “Translating videos to natural lan-
guage using deep recurrent neural networks”. In: arXiv preprint
arXiv:1412.4729 (2014).
[VFJ15] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. “Pointer net-
works”. In: Advances in Neural Information Processing Systems.
2015, pp. 2692–2700.
[Vin+15b] Oriol Vinyals et al. “Show and tell: A neural image caption genera-
tor”. In: Proceedings of the IEEE conference on computer vision and
pattern recognition. 2015, pp. 3156–3164.
[Wan+16a] Cheng Wang et al. “Image captioning with deep bidirectional
LSTMs”. In: Proceedings of the 2016 ACM on Multimedia Confer-
ence. ACM. 2016, pp. 988–997.
[Wan+15b] Xin Wang et al. “Predicting polarities of tweets by composing word
embeddings with long short-term memory”. In: Proceedings of the
53rd Annual Meeting of the Association for Computational Linguis-
tics and the 7th International Joint Conference on Natural Language
Processing (Volume 1: Long Papers). Vol. 1. 2015, pp. 1343–1353.
[XMS16] Caiming Xiong, Stephen Merity, and Richard Socher. “Dynamic
memory networks for visual and textual question answering”. In: In-
ternational conference on machine learning. 2016, pp. 2397–2406.
[Yin+17] Wenpeng Yin et al. “Comparative study of CNN and RNN for natural
language processing”. In: arXiv preprint arXiv:1702.01923 (2017).
[Zil+16] Julian G. Zilly et al. “Recurrent Highway Networks”. In:
CoRRabs/1607.03474 (2016).
Chapter 8
Automatic Speech Recognition
8.1 Introduction
Automatic speech recognition (ASR) has grown tremendously in recent years, with
deep learning playing a key role. Simply put, ASR is the task of converting spoken
language into computer readable text (Fig. 8.1). It has quickly become ubiquitous
today as a useful way to interact with technology, significantly bridging in the gap
in human–computer interaction, making it more natural. Historically, ASR is tightly
coupled with computational linguistics, given its close connection with natural lan-
guage, and phonetics, given the variety of speech sounds that can be produced by
humans. This chapter introduces the fundamental concepts of speech recognition
with a focus on HMM-based methods.
Fig. 8.1: The focus of ASR is to convert a digitized speech signal into computer
readable text, referred to as the transcript
Simply put, ASR can be described as follows: given an input of audio samples X
from a recorded speech signal, apply a function f to map it to a sequence of words
W that represent the transcript of what was said.
W = f (X) (8.1)
However, finding such a function is quite difficult, and requires consecutive model-
ing tasks to produce the sequence of words.
The selection of acoustic features for ASR is a crucial step. Features extracted from
the acoustic signal are the fundamental components for any model building as well
as the most informative component for the artifacts in the acoustic signal. Thus, the
acoustic features must be descriptive enough to provide useful information about
the signal, as well as resilient enough to the many perturbations that can arise in the
acoustic environment.
Let us first begin with a quick overview of how humans produce speech. While a
full study of the anatomy of the human vocal system is beyond the scope of this
book, some knowledge of human speech production can be helpful. The physical
production of speech consists of changes in air pressure that produces compression
waves that our ears interpret in conjunction with our brain. Human speech is created
8.2 Acoustic Features 371
from the vocal tract and modulated with the tongue, teeth, and lips (often referred
to as articulators):
• Air is pushed up from the lungs and vibrates the vocal cords (producing quasi-
periodic sounds).
• The air flows into the pharynx, nasal, and oral cavities.
• Various articulators modulate the waves of air.
• Air escapes through the mouth and nose.
Human speech is usually limited to the range 85 Hz–8 kHz, while human hearing is
in the range 20 Hz–20 kHz.
The waves of air pressure produced are converted into a voltage via a microphone
and sampled with an analog-to-digital converter. The output of the recording process
is a 1-dimensional array of numbers representing the discrete samples from the dig-
ital conversion. The digitized signal has three main properties: sample rate, number
of channels, and precision (sometimes referred to as bit depth). The sample rate is
the frequency at which the analog signal is sampled (in Hertz). The number of chan-
nels refers to audio capture with multiple microphone sources. Single-channel audio
is referred to as monophonic or mono audio, while stereo refers to two-channel au-
dio. Additional channels such as stereo and multi-channel audio can be useful for
signal filtering in challenging acoustic environments [BW13]. The precision or bit
depth is the number of bits per sample, corresponding to the resolution of the infor-
mation.
Standard telephone audio has a sampling rate of 8 kHz and 16-bit precision. CD
quality is 44.1 kHz, 16-bit precision, while contemporary speech processing focuses
on 16 kHz or higher.
Sometimes bit rate is used to measure the overall quality of audio computed by:
The raw speech signal is high dimensional and difficult to model. Most ASR
systems rely on features extracted from the audio signal to reduce the dimensionality
and filter unwanted signals. Many of these features come from some form of spectral
analysis that converts the audio signal to a set of features that strengthen signals
that mimic the human ear. Many of these methods depend on computing a short
time Fourier transform (STFT) on the audio signal using FFT, filter banks, or some
combination of the two [PVZ13].
372 8 Automatic Speech Recognition
8.2.3 MFCC
Mel frequency cepstral coefficients (MFCC) [DM90] are the most commonly used
features for ASR. Their success relies upon their ability to perform similar types of
filtering that correlates to the human auditory system and their low dimensionality.
There are seven steps to computing the MFCC features [MBE10]. The overall
process is shown in Fig. 8.2. These steps are similar for most feature generation
techniques, with some variability in the types of filters that are used and the filter
banks applied. We discuss each step individually:
1. Pre-emphasis
2. Framing
3. Hamming windowing
4. Fast Fourier transform
5. Mel filter bank processing
6. Discrete cosine transform (DCT)
7. Delta energy and delta spectrum.
Fig. 8.2: Diagram of MFCC processing with a visual representation for various parts
of the process. All spectrograms and features are shown in log-space
8.2.3.1 Pre-emphasis
be lower. Pre-emphasis processing applies a filter to the input signal that emphasizes
the amplitudes of higher frequencies and lowers the amplitudes of lower frequency
bands. For example:
yt = xt − α xt−1 (8.3)
would make the output less dependent on a strong signal from the previous time
steps.
8.2.3.2 Framing
The acoustic signal is perpetually changing in speech. Modeling this changing signal
is done by treating small segments sampled from the audio as stationary. Framing
is the process of separating the samples from the raw audio into fixed length seg-
ments referred to as frames. These segments are converted to the frequency domain
with an FFT, yielding a representation of the strength of frequencies during each
frame. The segments signify the boundaries between the phonetic representations
of speech. The phonetic sounds associated with speech tend to be in the range of
5–100 ms, so the length of frames is usually chosen to account for this. Typically,
frames are in the range of 20 ms for most ASR systems, with a 10 ms overlap, yield-
ing a resolution of 10 ms for our frames.
8.2.3.3 Windowing
Windowing multiplies the samples by a scaling function. The purpose of this func-
tion is to smooth the potentially abrupt effects of framing that can cause sharp differ-
ences at the edges of frames. Applying windowing functions to the samples there-
fore tapers the changes to the segment to dampen signals near the edges of the frame
that may have harsh effects after the application of the FFT.
Many windowing functions can be applied to a signal. The most commonly used
for ASR are Hann windowing and Hamming windowing.
Hann window:
2π n πn
w(n) = 0.5 1 − cos = sin2
(8.4)
N −1 N −1
Hamming window:
2π n
w(n) = 0.54 − 0.46 cos (8.5)
N −1
A short-time Fourier transform (STFT) converts the 1-dimensional signal from the
time domain into the frequency domain by using the frames and applying a discrete
Fourier transform (DFT) to each. An illustration of the DFT conversion is shown in
Fig. 8.3. The fast Fourier transforms (FFT) is an efficient algorithm to compute the
DFT under suitable circumstances and is common for ASR.
Fig. 8.3: The desired effect of an FFT on an input signal (shown on the left) and the
normalized FFT output in the frequency domain (shown on the right)
|FFT(xi )|2
Sp = (8.7)
N
where N is the number of points considered for the FFT computation (typically 256
or 512).
8.2 Acoustic Features 375
Spectrogram of example.wav
8000
7000 5
6000
Frequency [Hz]
0
5000
4000 –5
3000
–10
2000
–15
1000
0 –20
0.5 1.0 1.5 2.0
Time [sec]
Most of the significant frequencies are in the lower portion of the frequency
spectrum, so the spectrogram is typically mapped into the log scale.
The features created from the STFT transformation of the audio aim to simulate
conversions made by the human auditory system processes. The Mel filter bank is a
set of bandpass filters that mimic the human auditory system. Rather than follow a
linear scale, these triangular filters act logarithmic at higher frequencies and linear
at lower frequencies, which is typical in speech signals. Figure 8.5 shows the Mel
filter bank. The filter bank usually has 40 filters.
The conversion between the Mel (m) and Hertz ( f ) domains can be accomplished
by:
f
m = 2595 log10 1 +
700 (8.8)
m
f = 700 10 2595 −1
Each of the filters produces an output that is the weighted sum of the spectral
frequencies that correspond to each filter. These values map the input frequencies
into the Mel scale.
The discrete cosine transform (DCT) maps the Mel scale features into the time do-
main. The DCT function is similar to a Fourier transform but uses only real numbers
(a Fourier transform produces complex numbers). It compresses the input data into
a set of cosine coefficients that describe the oscillations in the function. The output
of this conversion is referred to as the MFCC.
376 8 Automatic Speech Recognition
Mel Filterbank
0.005
Amplitude 0.004
0.003
0.002
0.001
0.000
0 500 1000 1500 2000 2500 3000 3500 4000
Frequency (Hz)
0.8
Amplitude
0.6
0.4
0.2
0.0
0 500 1000 1500 2000 2500 3000 3500 4000
Frequency (Hz)
Fig. 8.5: Mel filter bank shown with 16 filters. The filters are applied to the input
signal to produce the Mel-scale output
The delta energy (delta) and delta spectrum (also known as “delta delta” or “dou-
ble delta”) features provide information about the slope of the transition between
frames. The delta energy features are the difference between consecutive frames’
coefficients (the current and previous frames). The delta spectrum features are the
difference between consecutive delta energy features (the current and previous delta
energy features). The equations for computing the delta energy and delta spectrum
features are:
Many acoustic features have been proposed over the years, applying different filters
and transforms to highlight various aspects of the acoustic spectrum. Many of these
8.3 Phones 377
Various attempts have been tried to learn the feature representations directly, rather
than relying on engineered features, which may not be best for the overall task of re-
ducing WER. Some of the approaches include: supervised learning of features with
DNNs [Tüs+14], CNNs on raw speech for phone classification [PCD13], combined
CNN-DNN features [HWW15], or even unsupervised learning with RBMs [JH11].
Automatically learned features improve quality in specific scenarios but can also
be limiting across domains. Features produced with supervised training learn to
distinguish between the examples in the dataset and may be limited in unobserved
environments. With the introduction of end-to-end models for ASR, these features
are tuned during the end-to-end task alleviating the two-stage training process.
8.3 Phones
Following from NLP, the most logical linguistic representation for transforming
speech into a transcript may seem to be words, ultimately because a word-level tran-
script is the desired output and there is meaning attached at the word-level. Practi-
cally speaking, however, speech datasets tend to have few transcribed examples per
word, making word-level modeling difficult. A shared representation for words is
desirable, to obtain sufficient training data for the variety of words that are possible.
For example, phonemes can be used to phonetically discretize words in a particular
language. Swapping one phoneme with another changes the meaning of the word
(although this may not be the case for the same phonemes in another language). For
example, if the third phone in the word sweet [swit] is changed from [i] to [E], the
meaning of the whole word changes: sweat [swEt].
Phonemes, themselves, tend to be too strict to use practically due to the attach-
ment of meaning. Instead phones are used as a phonetic representation for the
linguistic units (with potentially multiple phones mapping to a single phoneme).
378 8 Automatic Speech Recognition
Phones do not map to any specific language, but rather, are absolute to speech
itself, distinguishing sounds that signify speech. Figure 8.6 shows the phone set
for English.
AA AY EH HH L OY T W
AE B ER IH M P TH Y
AH CH EY IY N R UH Z
AO D F JH NG S UW ZH
AW DH G K OW SH V
Fig. 8.6: English phone set, based on the ARPAbet symbols for ASR as used in the
CMU Sphinx framework. The phone set is made up of 39 phones
With phones, words are mapped to their phonetic counterpart by using a phonetic
dictionary similar to the one shown in Fig. 8.7. A phonetic entry should be present
for each word in the vocabulary (sometimes more than one entry if there are multiple
ways to pronounce a word). By using phones to represent words, the shared repre-
sentations can be learned from many examples across words, rather than modeling
the full words.
Fig. 8.7: Phonetic dictionary for supported words in an ASR system. Note: the stress
of the syllable is sometimes included adding an additional features to the phone
representations
If every word were pronounced with the same phones, then a mapping from the
audio to the set of phones to words would be a relatively straight-forward transfor-
mation. However, audio exists as a continuous stream, and a speech signal does not
necessarily have defined boundaries between the phone units or even words. The sig-
nal can take many forms in the audio stream and still map to the same interpretable
output. For example, the speaker’s pace, accent, cadence, and environment can all
play significant roles in how to map the audio stream into an output sequence. The
words spoken depend not only on the phone at any given moment, but also on the
states that have come before and after the context. This natural dynamic in speech
places a strong emphasis on the dependency of the surrounding context and phones.
Combining phone states is a common strategy to improve quality, rather than
relying on their canonical representations. Specifically, the transitions between
words can be more informative than single phone states. In order to model this,
8.4 Statistical Speech Recognition 379
Statistical ASR focuses on predicting the most probable word sequence given a
speech signal, via an audio file or input stream. Early approaches did not use a
probabilistic focus, aiming to optimize the output word sequence by applying tem-
plates for reserved words to the input acoustic features (this was historically used
for recognizing spoken digits). Dynamic time warping (DTW) was an early way to
expand this templating strategy by finding the “lowest constrained path” for the tem-
plates. This approach allowed for variations in the input time sequence and output
sequence; however, it was difficult to come up with appropriate constraints, such as
distance metrics, how to choose templates, and the lack of a statistical, probabilistic
foundation. These drawbacks made the DTW-templating approach challenging to
optimize.
A probabilistic approach was soon formed to map the acoustic signal to a word
sequence. Statistical sequence recognition introduced a focus on maximum posterior
probability estimation. Formally, this approach is a mapping from a sequence of
acoustic, speech features, X, to a sequence of words, W . The acoustic features are a
sequence of feature vectors of length T : X = {xt ∈ RD |t = 1, . . . , T }, and the word
sequence is defined as W = {wn ∈ V|n = 1, . . . , N}, having a length N, where V is the
380 8 Automatic Speech Recognition
Solving this quantity is the center of ASR. Traditional approaches factorize this
quantity, optimizing models to solve each component, whereas more recent end-to-
end deep learning methods focus on optimizing for this quantity directly.
Using Bayes’ theorem, statistical speech recognition is defined as:
P(X|W )P(W )
P(W |X) = (8.12)
P(X)
The quantity P(W ) represents the language model (the probability of a given
word sequence) and P(X|W ) represents the acoustic model. Because this equation
drives the maximization of the numerator to achieve the most likely word sequence,
the goal does not depend on P(X), and it can be removed:
Often, one of the most challenging components of speech recognition is the sig-
nificant difference between the number of steps in the input sequence compared to
the output sequence (T N). For example, extracted acoustic features may repre-
sent a 10 ms frame from the audio signal. A typical ten-word utterance could have
a duration of 3-s utterance, leading to an input sequence length of 300 and a target
output sequence of 10 [You96]. Thus, a single word can spread many frames and
take a variety of forms, as shown in Fig. 8.9. It is, therefore, sometimes beneficial to
split a word into sub-components that span fewer frames.
8.4 Statistical Speech Recognition 381
D A V I D
7000
6000
4000
3000
2000
1000
0
0.5 1.0 1.5 2.0
Time [sec]
The statistical definition in Eq. (8.13) can be augmented to incorporate the mapping
acoustic features to phones and then from phones to words:
where P(X|S) maps the acoustic features to phone states and P(S|W ) maps phones
to words (commonly referred to as the pronunciation model).
Equation (8.13) showed two factors P(X|W ) and P(W ). Each of these factors
are considered models and therefore have learnable parameters, ΘA and ΘL , for the
acoustic model and language model, respectively.
This model now depends on predicting the likelihood of observations X, with the
factor P(X|W, ΘA ). Solving this quantity requires a state-based modeling approach
(HMMs). If a discrete-state model is assumed, the probability of an observation can
be defined by introducing a state sequence S, where S = {st ∈ {s(i) , . . . s(Q) }|t =
1, . . . , T } into P(X|W ).
Equation 8.16 can be factorized further using the chain rule of probability to pro-
duce the framewise likelihood. For notational convenience, let x1:n = x1 , x2 , . . . , xn .
T
P(X|S) = ∏ P(xt |x1:t−1 , S) (8.17)
t=1
Using the conditional independence assumption, this quantity can be reduced to:
T
P(X|S) ≈ ∏ P(xt |st ) (8.18)
t=1
The conditional independence assumption limits the context that is considered for
prediction. We assume that any observation xt is only dependent on the current
state, st , and not on the history of observations x1:t−1 , as shown in Fig. 8.10. This
assumption reduces the computational complexity of the problem; however, it limits
the contextual information included in any decision. The conditional independence
assumption is often one of the biggest hurdles in ASR, due to the contextual nature
of speech. Thus, a variety of techniques are centered around providing “context
features” to improve quality.
The transitions between states st only depend on the previous state st−1 . The lexicon
model (discussed in the next section) provides the initial transition state probabili-
ties. These transitions can be self-loops, allowing the time dilation that is necessary
to allow elasticity in the frame-based prediction.
The HMM is optimized, learning π , a, and b(x) by training on the acoustic
observations X and the target sequence of phone states Y . An initial estimate for
P(s( j) |s(i) ) can be obtained from the lexicon model, P(S|W ). The forward-recursion
algorithm is used to score the current model parameters a, b(x), yielding parameters
to obtain P(X|S). The Viterbi algorithm is used to avoid computing the sum of all
paths and serves as an approximation of the forward algorithm:
Fig. 8.11: All possible state transitions to produce the 3-phone word, “cat” for an
8-frame utterance. The Viterbi path applied to the possible state transitions is shown
in red
Once again, using the conditional independence assumption, this quantity is ap-
proximated by:
P(st |s1:t−1 ,W ) = P(st |st−1 ,W ) (8.21)
The introduction of the conditional independence assumption is also the first-
order Markovian assumption, allowing us to implement the model as a first-order
HMM. The states of the model, st , are not directly observable; therefore, we are not
able to observe the transition from one phonetic unit to another; however, the obser-
vations xt do depend on the current state st . The HMM allows us to infer information
about the state sequence from the observations.
First, the word vocabulary V is converted into the state representations for each
term to create a word model.
The lexicon model can be used to determine the initial probability of each state
P(s1 ) by counting the occurrence rate for the beginning of each word. The transition
probabilities accumulate over the lexical version of the transcript targets for the
acoustic data.
A state-based word-sequence model can be created for each word in the vocabu-
lary, as shown in Fig. 8.12.
8.4 Statistical Speech Recognition 385
Fig. 8.12: Phone state model for the 3-phone word “cat” with transition probabilities
The language model P(W ) is typically an n-gram language model leveraging the
probabilistic chain rule. It is factorized by using the conditional independence as-
sumption except with an (m − 1)-th order Markov assumption, where m is the num-
ber of grams to be considered. It can be described as:
N
P(W ) = ∏ P(wn |ww1 :wn−1 )
n=1
(8.22)
N
≈ ∏ P(wn |wn−m−1:n−1 )
n=1
HMMs are robust models for training and decoding sequences. When training
the HMM models, we focused on training individual models for the state align-
ments, and then combine them into a single HMM for continuous speech recog-
nition. HMMs also allow the word sequence to be incorporated, creating a state
sequence based on words and apply the word sequence priors as well. Furthermore,
HMMs support compositionality; therefore, the time dilation, pronunciation, and
word sequences (grammar) are handled in the same model by composing the indi-
vidual components:
Unfortunately, many of the assumptions that are needed to optimize HMMs limit
their functionality due to [BM12]:
• HMM and DNN models are trained independently of each other and yet depend
on each other.
• A priori choices of statistical distributions rely on linguistic information from
handcrafted pronunciation dictionaries. These are subject to human error.
• The first-order Markov assumption often referred to as the conditional indepen-
dence assumption (states are only dependent on their previous state) forces strict
386 8 Automatic Speech Recognition
The decoding process for an HMM-based ASR model finds the optimal word se-
quence, combining the various models. The process decodes a state sequence from
the acoustic features initially and then decodes to the optimal word sequence from
the state sequence. Phonetic decoding has traditionally relied on interpreting the
HMMs probability lattice constructed for each word from the phonetic lexicons ac-
cording to the acoustic features. Decoding can be done using the Viterbi algorithm
on the HMM output lattice, but this is expensive for large vocabulary tasks. Viterbi
decoding performs an exact search efficiently, making it infeasible for a large vo-
cabulary task. Beam search is often used instead to reduce the computation. The
decoding process uses backtracking to keep track of the word sequence produced.
Fig. 8.13: (a) HMM state representation, (b) phone state transitions for the word
“data,” and (c) grammar state model. This figure is adapted from [MPR08]
8.5 Error Metrics 387
During prediction, decoding the HMM typically relies on using weighted au-
tomata and transducers. In a simple case, weighted finite state acceptors (WFSA),
the automata are composed of a set of states (initial, intermediate, and final), a set
of transitions between states with a label and weight, and final weights for each fi-
nal state. The weights express the probability, or cost, of each transition. You can
express HMMs in the form of finite state automata. In this approach, a transition
connects each state. WFSA accept or deny possible decoding paths depending on
the states and the possible transitions. The topology could represent a word, the pos-
sible word pronunciation(s), or the probabilities of the states in the path to result in
this word, (Fig. 8.13). Decoding, therefore, depends on combining the state models
from the HMM with the pronunciation, dictionary, and n-gram language models that
must be combined in some way.
Usually, weighted finite state transducers (WFST) are used to represent the dif-
ferent levels of state transition in the decoding phase [MPR08]. WFSTs transduce
an input sequence to an output sequence. WFSTs add an output label, which can be
used to tie different levels of the decoding relationships together, such as phones and
words. A WFSA is a WFST without the output label. The WFST representation al-
lows models to be combined and optimized jointly via its structural properties with
efficient algorithms: compositionality, determinism, and minimization. The compo-
sition property allows for different types of WFSTs to be constructed independently
and composed together, such as combining a lexicon (phones to words) WFST and a
probabilistic grammar. Determinism forces unique initial states, where no two tran-
sitions leaving a state share the same input label. Minimization combines redundant
states and can be thought of as suffix sharing. Thus, the whole decoding algorithm
for a DNN-HMM hybrid model can be represented by WFSTs via four transducers:
• HMM: mapping HMM states to CD phones
• Context-dependency: mapping CD phones to phones
• Pronunciation lexicon: mapping phones to words
• Word-level grammar: mapping words to words.
In Kaldi, for example, these transducers are referred to as H, C, L, and G, re-
spectively. Compositionality allows a composition between L and G into a single
transducer, L G, that maps phone sequences to a word sequence. Practically, the
composition of these transducers may grow too large, so the conversion usually
takes the form: HCLG, where
The most commonly used metric for speech recognition is word error rate (WER).
WER measures the edit distance between the prediction and the target by consid-
ering the number of insertions, deletions, and substitutions, using the Levenshtein
distance measure.
388 8 Automatic Speech Recognition
GMMs were a popular choice because they are capable of modeling P(xt |st ) di-
rectly. Additionally, they provide a probabilistic interpretation of the input, mod-
eling the distribution under each state. However, the Gaussian distribution at each
state is a strong assumption itself. In practice, the features may be strongly non-
Gaussian. DNNs showed significant improvements over GMMs with their ability
8.6 DNN/HMM Hybrid Model 389
1 i m p o r t L e v e n s h t e i n a s Lev
2
3 d e f wer ( s1 , s 2 ) :
4 ”””
5 Computes t h e Word E r r o r Rate , d e f i n e d a s t h e e d i t d i s t a n c e
between t h e
6 two p r o v i d e d s e n t e n c e s a f t e r t o k e n i z i n g t o words .
7 Arguments :
8 s1 ( s t r i n g ) : s p a c e −s e p a r a t e d s e n t e n c e
9 s2 ( s t r i n g ) : s p a c e −s e p a r a t e d s e n t e n c e
10 ”””
11
12 # b u i l d mapping o f words t o i n t e g e r s
13 b = s e t ( s1 . s p l i t ( ) + s2 . s p l i t ( ) )
14 word2char = d i c t ( z ip ( b , range ( len ( b ) ) ) )
15
16 # map t h e words t o a c h a r a r r a y ( L e v e n s h t e i n p a c k a g e s o n l y
accepts
17 # strings )
18 w1 = [ c h r ( w o r d 2 c h a r [w ] ) f o r w i n s 1 . s p l i t ( ) ]
19 w2 = [ c h r ( w o r d 2 c h a r [w ] ) f o r w i n s 2 . s p l i t ( ) ]
20 w e r l e v = Lev . d i s t a n c e ( ’ ’ . j o i n ( w1 ) , ’ ’ . j o i n ( w2 ) )
21 w e r i n s t = f l o a t ( w e r l e v ) / l e n ( s 1 . s p l i t ( ) ) ∗ 100
22 r e t u r n ’WER: { 0 : . 2 f } ’ . f o r m a t ( w e r i n s t )
23
24 d e f c e r ( s1 , s 2 ) :
25 ”””
26 Computes t h e C h a r a c t e r E r r o r Rate , d e f i n e d a s t h e e d i t
distance .
27 Arguments :
28 s 1 ( s t r i n g ) : s p a c e −s e p a r a t e d s e n t e n c e
29 s 2 ( s t r i n g ) : s p a c e −s e p a r a t e d s e n t e n c e
30 ”””
31 s1 , s2 , = s 1 . r e p l a c e ( ’ ’ , ’ ’ ) , s 2 . r e p l a c e ( ’ ’ , ’ ’ )
32 c e r i n s t = f l o a t ( Lev . d i s t a n c e ( s1 , s 2 ) ) / l e n ( s 1 ) ∗ 100
33 r e t u r n ’CER : { 0 : . 2 f } ’ . f o r m a t ( c e r i n s t )
34
to learn non-linear functions. The DNN cannot provide the conditional likelihood
directly. The framewise posterior distribution is used to turn the probabilistic model
of P(xt |st ) into a classification problem P(st |xt ) using the pseudo-likelihood trick as
an approximation of the joint probability. The application of the pseudo-likelihood
is referred to as the “hybrid-approach.”
T T
P(st |xt )
∏ P(xt |st ) ∝ ∏ (8.27)
t=1 t=1 p(st )
390 8 Automatic Speech Recognition
1 p r e d i c t i o n = ’ t h e c a t s a t on t h e mat ’
2 t a r g e t = ’ t h e c a t s a t on t h e mat ’
3 p r i n t ( ’ P r e d i c t i o n : ’ + p r e d i c t i o n , ’ \ n Ta r g e t : ’ + t a r g e t )
4 p r i n t ( wer ( p r e d i c t i o n , t a r g e t ) )
5 print ( cer ( prediction , t arget ) )
6
7 > P r e d i c t i o n : t h e c a t s a t on t h e mat
8 > Ta r g e t : t h e c a t s a t on t h e mat
9 > WER: 0 . 0 0
10 > CER : 0 . 0 0
Fig. 8.15: An exact match between the prediction and the target yields a WER and
CER of 0
1 p r e d i c t i o n = ’ t h e c a t s a t on t h e mat ’
2 t a r g e t = ’ t h e c a t s a t on t h e h a t ’
3 p r i n t ( ’ P r e d i c t i o n : ’ + p r e d i c t i o n , ’ \ n Ta r g e t : ’ + t a r g e t )
4 p r i n t ( wer ( p r e d i c t i o n , t a r g e t ) )
5 print ( cer ( prediction , target ) )
6
7 > P r e d i c t i o n : t h e c a t s a t on t h e mat
8 > Ta r g e t : t h e c a t s a t on t h e h a t
9 > WER: 1 6 . 6 7
10 > CER : 5 . 8 8
Fig. 8.16: Changing one character of the predicted word yields a larger increase in
WER because the entire word is wrong, albeit phonetically similar. The change in
CER is much smaller by comparison because there are more characters than words;
thus a single character change has less effect
1 p r e d i c t i o n = ’ c a t mat ’
2 t a r g e t = ’ t h e c a t s a t on t h e mat ’
3 p r i n t ( ’ P r e d i c t i o n : ’ + p r e d i c t i o n , ’ \ n Ta r g e t : ’ + t a r g e t )
4 p r i n t ( wer ( p r e d i c t i o n , t a r g e t ) )
5 print ( cer ( prediction , t arget ) )
6
7 > P r e d i c t i o n : c a t mat
8 > Ta r g e t : t h e c a t s a t on t h e h a t
9 > WER: 2 0 0 . 0 0
10 > CER : 1 8 3 . 3 3
Fig. 8.17: WER and CER are typically not treated as percentages, because they can
exceed 100%. The loss or insertion of large sections can greatly increase the WER
and CER
8.7 Case Study 391
The numerator is a DNN classifier, trained with a set of input features as the in-
put xt and target state st . In a simple case, if we consider 1-state per phone, then
the number of classifier categories will be len(q). The denominator P(st ) is the
prior probability of the state st . Note that training the framewise model requires
framewise alignments with xt as the input and st as the target, as shown in Fig. 8.18.
This alignment is usually created by leveraging a weaker HMM/GMM alignment
system or through human-created labels. The quality and quantity of the alignment
labels are typically the most significant limitations with the hybrid-approach.
Fig. 8.18: In order to apply the DNN classifier, a framewise target must exist. A
constrained alignment is computed using an existing classifier to align the acoustic
features and the known sequence of states
Classifier construction requires the selection of target states (words, phones, tri-
phone states). Selection of the states, Q, can make a significant difference in the
quality and complexity of the task. First, it must support the recognition task to
obtain the alignments. Second, it must be practical for the classification task. For
example, phones may be easier to train a classifier for, however, getting framewise
labels for the training data and a decoding scheme may be much more difficult to
obtain. Alternatively, word-based states are straight-forward to create but are harder
to get framewise alignments and train a classifier.
In this case study, the main focus is on training ASR systems using open source
frameworks. We begin by training a traditional ASR engine and then move towards
the more advanced models in the frameworks, ending with a TDNN model.
392 8 Automatic Speech Recognition
In this case study we focus on building ASR models for the Common Voice1
dataset released by Mozilla. Common Voice is a 500 h speech corpus of recorded
speech from text. It is composed of crowdsourced speakers recoding one utterance
per example. These recordings are then peer-reviewed to assess the quality of the
transcript-recording pair. Depending on the number of positive and negative votes
that each utterance receives, it is labeled as either valid, invalid, or other. The valid
category contains samples that have had at minimum two reviews, and the majority
confirms the audio matches the text. The invalid category similarly has had at least
two reviews, with the majority confirming the audio does not match the text. The
other category contains all the files with less than two votes or with no majority con-
sensus. Each of the sub-groups, valid and other, is further split into train, test, and
dev (validation). The “cv-valid-train” dataset contains a total of 239.81 h of audio
in total. Overall, the dataset is complex, containing a variety of accents, recording
environments, ages, and genders.
Kaldi is one of the most widely used toolkits for ASR, developed mainly for re-
searchers and professional use. It is developed primarily by Johns Hopkins Univer-
sity and built entirely in C++ with shell scripts tying various components of the
library together. The design focuses on providing a flexible toolkit that can be mod-
ified and extended to the task. Collections of scripts referred to as “recipes” are used
to connect the components to perform training and inference.
CMU Sphinx is an ASR toolkit that developed at Carnegie Mellon University.
It also relies on HMM-based speech recognition and n-gram language models for
ASR. There have been various releases of the Sphinx toolkit with Sphinx 4 being the
most current. A version of Sphinx called PocketSphinx is more suitable for embed-
ded systems (typically it comes with some losses in quality due to the restrictions of
the hardware).
8.7.3 Sphinx
In this section, we train a Sphinx ASR model on the Common Voice dataset. This
framework relies on a variety of packages, mainly based on C++. In light of this,
much of the work in this ASR introduction chapter focuses on the scripts and asso-
ciated concepts, namely the data preparation. Like many frameworks, once the data
is appropriately formatted, the framework is relatively straight-forward.
1 https://voice.mozilla.org/en/data.
8.7 Case Study 393
The data preparation required for Sphinx is the most important step. Sphinx is con-
figured to look in specific places for certain things and expects consistency between
files and file names. The conventional structure is a top level directory with the same
name as the dataset. This name is used as the file name for the subsequent files. In-
side this directory, there are two directories “wav” and “etc.” The “wav” directory
contains all training and testing audio files in wav form. The “etc” directory contains
all configuration and transcript files. Figure 8.19 shows the file structure.
1 / common voice
2 etc /
3 common voice . d i c
4 common voice . f i l l e r
5 common voice . i d n g r a m
6 common voice . lm
7 common voice . lm . b i n
8 common voice . phone
9 common voice . v o c a b
10 common voice test . f i l e i d s
11 common voice test . t r a n s c r i p t i o n
12 common voice train . f i l e i d s
13 common voice train . t r a n s c r i p t i o n
14 wav /
15 t r a i n s a m p l e 0 0 0 0 0 0 . wav
16 t e s t s a m p l e 0 0 0 0 0 0 . wav
17 t r a i n s a m p l e 0 0 0 0 0 1 . wav
18 t e s t s a m p l e 0 0 0 0 0 1 . wav
19 ...
20
Common Voice initially comes packaged with “mp3” files. These will be con-
verted to “wav” by using the “SoX” tool.2 The processing script is shown in
Fig. 8.20.
After we create the “wav” files, we create a list of all files that should be used for
training, and separately, validation (referred to as testing for the Sphinx framework).
The file list contains the “.fileids” files. The “.fileids” contain a single file name per
line without the file extension. This is illustrated in Fig. 8.21.
Next, we create the transcript files. The transcript files have one utterance tran-
script per line with the “fileid” specified at the end. A sample of a transcript file is
shown in Fig. 8.22.
2 http://sox.sourceforge.net/.
394 8 Automatic Speech Recognition
Fig. 8.20: Converts mp3 files to wav files, using the sox library. The function is
parallelized to increase the speed of the conversion
1 train sample000000
2 train sample000001
3 train sample000002
4 ...
5
Once the transcript files are created, we turn our attention to the phonetic units
used. In this example, the same phones are used as illustrated in Fig. 8.6, with one
additional phone <SIL> to symbolize the silent token.
The next step is to create the phonetic dictionary. We create a list of all words in
the training dataset transcripts. The script in Fig. 8.23 shows a simple way of doing
this.
Next, we create a phonetic dictionary (lexicon) using the word and phone lists.
Creating the lexicon model typically requires linguistic expertise or existing mod-
els to create mappings for these words, as the phonetic representation should match
8.7 Case Study 395
1 import c o l l e c t i o n s
2 import os
3
4 counter = c o l l e c t i o n s . Counter ( )
5 w i t h open ( c s v f i l e ) a s c s v f i l e :
6 r e a d e r = csv . DictReader ( c s v f i l e )
7 f o r row i n r e a d e r :
8 t r a n s = row [ ’ t e x t ’ ]
9 c o u n t e r += c o l l e c t i o n s . C o u n t e r ( t r a n s . s p l i t ( ) )
10
11 w i t h open ( o s . p a t h . j o i n ( e t c d i r , ’ common voice . words ’ ) , ’w’ ) a s f :
12 for item in counter :
13 f . w r i t e ( i t e m . l o w e r ( ) + ’\ n ’ )
14
Fig. 8.23: This script creates a file, “common voice.words” that contains one word
per line from the training data. Note: each word is only represented once in this file
the pronunciation. To ease this dependency, CMU Lextool3 is used to create our
phonetic dictionary, and saved as “common voice.dic”. Note: there is some extra
processing required here to ensure that there are no additional phones added to the
representation than those specified in our “.phone” file. Additionally, in this exam-
ple, all phones and transcripts are represented as lower case. The phonetic dictionary
needs to match as well. A sample from the phonetic dictionary is shown in Fig. 8.24
1 a ah
2 a (2) ey
3 monk m ah ng k
4 dressed d r eh s t
5 in ih n
6 black b l ae k
7 came k ey m
8 to t uw
9 ...
10
Our final step is to create a language model. Most language models follow the
ARPA format, representing the n-grams and its associated probabilities once per
line, with section delimiters for increases in the number of grams. We create a 3-
gram language model using the CMUCLMTK, a language modeling toolkit from
CMU. The script counts the different n-grams and computes the probabilities of
3 http://www.speech.cs.cmu.edu/tools/lextool.html.
396 8 Automatic Speech Recognition
each. The script is shown in Fig. 8.25 and a sample of the language model is shown
in Fig. 8.26.
1 # C r e a t e vocab f i l e
2 t e x t 2 w f r e q < e t c / common voice train . t r a n s c r i p t i o n | wfreq2vocab >
e t c / common voice . v o c a b
3
4 # C r e a t e n−gram c o u n t from t r a i n i n g t r a n s c r i p t f i l e
5 t e x t 2 i d n g r a m −v o c a b e t c / common voice . v o c a b −i d n g r a m e t c /
common voice . i d n g r a m < e t c / c o m m o n v o i c e t r a i n . t r a n s c r i p t i o n
6
7 # C r e a t e l a n g u a g e model from n−grams
8 i d n g r a m 2 l m −v o c a b t y p e 0 −i d n g r a m e t c / common voice . i d n g r a m −v o c a b
e t c / common voice . v o c a b −a r p a e t c / common voice . lm
9
10 # C o n v e r t l a n g u a g e model t o b i n a r y ( c o m p r e s s i o n )
11 s p h i n x l m c o n v e r t − i e t c / common voice . lm −o e t c / common voice . lm .
DMP
12
With the preprocessing complete, we are ready to train the ASR models.
The model training process for Sphinx is straight-forward, with everything set up to
follow the configuration file. To generate the configuration file, we run:
1 s p h i n x t r a i n − t common voice s e t u p
With the setup config, the Sphinx model can be trained by running:
1 s p h i n x t r a i n run
The training function runs a series of scripts that check the configuration and
setup. It then performs a series of transforms on the data to produce the features,
and then trains a series of models. The Sphinx framework achieves a WER of 39.824
and CER of 24.828 on the Common Voice validation set.
8.7.4 Kaldi
In this section, we train a series of Kaldi models to train a high-quality ASR model
on the Common Voice dataset. Training a high-quality model requires intermediate
models to align the acoustic feature frames to the phonetic states. The code and
explanations are adapted from the Kaldi tutorial.
8.7 Case Study 397
1 \data\
2 ngram 1=8005
3 ngram 2=31528
4 ngram 3=49969
5
6 \1−grams :
7 −6.8775 </s> 0.0000
8 −0.9757 <s> −4.8721
9 −1.6598 a −4.5631
10 −5.0370 a a r o n −1.2761
11 −4.5116 a b a n d o n −1.7707
12 −3.9910 a b a n d o n e d −2.2851
13 ...
14
15 \2−grams :
16
17 −1.9149 <s> a −3.2624
18 −4.1178 <s> a b i g a i l 0.0280
19 −2.8197 <s> a b o u t −2.4474
20 −4.0634 <s> abraham 0 . 0 4 8 3
21 −2.9228 <s> a b s o l u t e l y −1.8134
22 ...
23
24 \3−grams :
25 −0.9673 <s> a boy
26 −1.6977 <s> a breeze
27 −2.6800 <s> a bunch
28 −1.5866 <s> a card
29 −2.2998 <s> a citation
30
Fig. 8.26: Sample from the “common voice.lm” file in the standard ARPA format
The data preparation in Kaldi is similar to the Sphinx preparation, requiring tran-
scription and audio ID files, shown in Fig. 8.27. Kaldi has a set of scripts in place to
automate the construction of these files, to reduce the manual work required in the
Sphinx setup.
Prepare a mapping from the “.wav” files to the audio path. We create an utterance
ID for each of the files. This utterance ID is used to tie the file to the different
representations in the training pipeline. In Kaldi, this is treated as a simple text file,
with the “.scp” extension.
Secondly, a file mapping the utterance ID to the utterance transcript is created
(Fig. 8.28).4 This file will be used to create the utterance.
A corpus file contains all the utterances from the dataset. It is used to compute
the word-level decoding graphs for the system.
4If there are additional labels like speaker and gender, these can also be used in the process.
Common Voice does not have these labels, so each utterance is treated independently.
398 8 Automatic Speech Recognition
1 d a d 4 4 2 f o u r f o u r two
2 j u l y 1 2 5 one two f i v e
3 july 6 8 3 six eight three
4 # and s o on . . .
These files allow the rest of the required files to be generated, such as the “lexi-
con.txt” file, which contains all the words from the dictionary with the phone tran-
script. Additionally, there are non-silence and silence phone files that provide ways
to handle non-speech audio (Fig. 8.29).
1 ! SIL s i l
2 <UNK> s p n
3 e i g h t ey t
4 f i v e f ay v
5 f o u r f ao r
6 n i n e n ay n
7 one hh w ah n
8 one w ah n
9 s e v e n s eh v ah n
10 six s ih k s
11 three th r iy
12 two t uw
13 z e r o z i h r ow
14 z e r o z i y r ow
15
Once these files are prepared, Kaldi scripts can be used to create the ARPA lan-
guage model and vocabulary. With Sphinx, all the terms in the dictionary needed a
lexical entry to be entered manually (we leveraged premade dictionaries and addi-
tional inference dictionaries to accomplish this). The CMU dictionary is also used in
this case, except in Kaldi, a pre-trained model is used to estimate the pronunciations
8.7 Case Study 399
of OOV words.5 Once the dictionary is prepared, the lexical model is built, with
the phonetic representation of each word in the dataset. An FST is then constructed
from the transcripts and lexicon model and used to train the model.
The next step in the data preprocessing is to produce MFCCs for all of the train-
ing data. A file for each utterance is saved individually to reduce feature generation
for various experiments. During this process, we also create two smaller datasets: a
dataset of the 10 k shortest utterances and a dataset of the 20 k shortest utterances.
These are used to build the earlier models. Once features are extracted, we can train
our models.
Much of the model training is scripted in Kaldi and is straight-forward after the
data preparation is completed. The first model that is trained is an HMM-GMM
model. It is trained for 40 epochs through the 10 k shortest utterances. This model
is then used to align the 20 k utterances. After each model is trained, we rebuild the
decoding graph and apply it to the test set. This model yields a WER of 52.06 on
the Validation (Fig. 8.30).
Fig. 8.30: Train the monophone “mono” model with alignments on the 10 k shortest
utterances subset from the training data
Next, we use the 20 k alignments to train a new model incorporating the delta
and double delta features. This model will also leverage triphones in the training
process. The previous process will be performed again, with a separate training
script, producing a model that achieves a WER of 25.06 (Fig. 8.31).
The third model that is trained is an LDA+MLLT model. This model will be
used to compute better alignments using the learnings of the previous model on the
20 k dataset. So far, we have been using the 13-dimensional MFCC features. In this
5 Note: It is possible to add specific words to the lexicon by exiting the lexicon-iv.txt file.
400 8 Automatic Speech Recognition
Fig. 8.31: Train another monophone “tri1” model with alignments from the “mono”
model on the 20 k training subset
model, multiple frames are considered in a single input t to provide more context
at each state. The added input dimensionality increases the computational require-
ments of the classifier, so we use linear discriminant analysis (LDA) to reduce the
dimensionality of the features. Additionally, a maximum likelihood linear transform
(MLLT) to further decorrelate the features and make them “orthogonal” to be better
modeled by diagonal-covariance Gaussians [Rat+13]. The resulting model yields a
WER of 21.69 (Fig. 8.32).
The next model that is trained is the speaker adapted model referred to as
“LDA+MLLT+SAT”. The 20 k dataset is aligned again using the previous model,
and using the same architecture as the previous model with the additional speaker
adapted features. Because our data doesn’t include speaker tags, we would not ex-
pect to get gains in this area, and we do not see any. The resulting model yields a
WER of 22.25 (Fig. 8.33).
We now apply the alignments computed with the previous model to the entire
training dataset. We train another LDA+MLLT+SAT model on the new alignments.
The resulting model gives a WER of 17.85 (Fig. 8.34).
The final model is a TDNN model [PPK15]. The TDNN model is an 8 layer
DNN, with batch normalization. This model requires a GPU to train due to the
depth and need for parallelization (Fig. 8.35).
The final model, after integrating the 8-layer DNN, achieves a WER of 4.82 on
the validation set.
8.7 Case Study 401
1 s t e p s / a l i g n s i . s h −−n j 10 −−cmd ” $ t r a i n c m d ” \
2 d a t a / t r a i n 2 0 k d a t a / l a n g exp / t r i 1 exp / t r i 1 a l i t r a i n 2 0 k
3
4 s t e p s / t r a i n l d a m l l t . s h −−cmd ” $ t r a i n c m d ” \
5 −−s p l i c e −o p t s ”−− l e f t −c o n t e x t =3 −−r i g h t −c o n t e x t =3” 2500 15000 \
6 d a t a / t r a i n 2 0 k d a t a / l a n g exp / t r i 1 a l i t r a i n 2 0 k exp / t r i 2 b
7
8 # d e c o d e u s i n g t h e LDA+MLLT model
9 u t i l s / mkgraph . s h d a t a / l a n g t e s t exp / t r i 2 b exp / t r i 2 b / g r a p h
10 (
11 f o r t e s t s e t i n v a l i d d e v ; do
12 s t e p s / d e c o d e . s h −−n j 20 −−cmd ” $decode cmd ” exp / t r i 2 b /
graph \
13 d a t a / $ t e s t s e t exp / t r i 2 b / d e c o d e $ t e s t s e t
14 done
15 )&
16
Fig. 8.32: Train the LDA+MLLT “tri2b” model with alignments from the “tri1”
model
1 # A l i g n u t t s u s i n g t h e t r i 2 b model
2 s t e p s / a l i g n s i . s h −−n j 10 −−cmd ” $ t r a i n c m d ” −−use−g r a p h s t r u e \
3 d a t a / t r a i n 2 0 k d a t a / l a n g exp / t r i 2 b exp / t r i 2 b a l i t r a i n 2 0 k
4
5 s t e p s / t r a i n s a t . s h −−cmd ” $ t r a i n c m d ” 2500 15000 \
6 d a t a / t r a i n 2 0 k d a t a / l a n g exp / t r i 2 b a l i t r a i n 2 0 k exp / t r i 3 b
7
8 # d e c o d e u s i n g t h e t r i 3 b model
9 (
10 u t i l s / mkgraph . s h d a t a / l a n g t e s t exp / t r i 3 b exp / t r i 3 b / g r a p h
11 f o r t e s t s e t i n v a l i d d e v ; do
12 s t e p s / d e c o d e f m l l r . s h −−n j 10 −−cmd ” $decode cmd ” \
13 exp / t r i 3 b / g r a p h d a t a / $ t e s t s e t exp / t r i 3 b /
decode $testset
14 done
15 )&
16
Fig. 8.33: Train the LDA+MLLT+SAT “tri3b” model with alignments from the
“tri2b” model
8.7.5 Results
The results achieved throughout this case study are summarized in Table 8.1. The
best Kaldi and Sphinx model are then evaluated on the test set. These results are
shown in Table 8.2. We see that the addition of deep learning in the final Kaldi
TDNN model shows significant quality improvements over the traditional learning
algorithms for the acoustic model.
402 8 Automatic Speech Recognition
1 # A l i g n u t t s i n t h e f u l l t r a i n i n g s e t u s i n g t h e t r i 3 b model
2 s t e p s / a l i g n f m l l r . s h −−n j 20 −−cmd ” $ t r a i n c m d ” \
3 data / v a l i d t r a i n data / lang \
4 exp / t r i 3 b exp / t r i 3 b a l i v a l i d t r a i n
5
6 # t r a i n a n o t h e r LDA+MLLT+SAT s y s t e m on t h e e n t i r e t r a i n i n g s e t
7 s t e p s / t r a i n s a t . s h −−cmd ” $ t r a i n c m d ” 4200 40000 \
8 data / v a l i d t r a i n data / lang \
9 exp / t r i 3 b a l i v a l i d t r a i n exp / t r i 4 b
10
11 # d e c o d e u s i n g t h e t r i 4 b model
12 (
13 u t i l s / mkgraph . s h d a t a / l a n g t e s t exp / t r i 4 b exp / t r i 4 b / g r a p h
14 f o r t e s t s e t i n v a l i d d e v ; do
15 s t e p s / d e c o d e f m l l r . s h −−n j 20 −−cmd ” $decode cmd ” \
16 exp / t r i 4 b / g r a p h d a t a / $ t e s t s e t \
17 exp / t r i 4 b / d e c o d e $ t e s t s e t
18 done
19 )&
20
Fig. 8.34: Train the LDA+MLLT+SAT “tri4b” model with alignments from the
“tri3b” model
1 l o c a l / c h a i n / r u n t d n n . s h −−s t a g e 0
2
Fig. 8.35: Script: Train the TDNN model using the “tri4b” model
Some other interesting problems readers and practitioners can try on their own in-
clude:
1. How are additional words added to the vocabulary?
2. Evaluate the real-time factor (RTF) for this system.
Table 8.1: Speech recognition performance on Common Voice validation set. Best
result is shaded
Approach WER
Sphinx 39.82
Kaldi monophone (10 k sample) 52.06
Kaldi triphone (with delta and double delta, 20 k sample) 25.06
Kaldi LDA+MLLT (20 k sample) 21.69
Kaldi LDA+MLLT+SAT (20 k sample) 22.25
Kaldi LDA+MLLT+SAT (all data) 17.85
Kaldi TDNN (all data) 4.82
References 403
Table 8.2: Speech recognition performance on Common Voice test set. Best result
is shaded
Approach WER
Sphinx 53.85
Kaldi TDNN 4.44
References
9.1 Introduction
In deep learning networks, as we have seen in the previous chapters, there are good
architectures for handling spatial and temporal data using various forms of convolu-
tional and recurrent networks, respectively. When the data has certain dependencies
such as out-of-order access, long-term dependencies, unordered access, most stan-
dard architectures discussed are not suitable. Let us consider a specific example
from the bAbI dataset where there are stories/facts presented, a question is asked,
and the answer needs to be inferred from the stories. As shown in Fig. 9.1, it requires
out of order access and long-term dependencies to find the right answer.
Deep learning architectures have made significant progress in the last decade in
capturing implicit knowledge as features in various NLP tasks. Many tasks such
as question answering or summarization require storage of explicit knowledge so
that it can be used in the tasks. For example, in the bAbI dataset, information about
Mary, Sandra, John, football, and its location is captured for answering the question
such as “where is the football?” Recurrent networks such as LSTM and GRU cannot
capture such information over very long sequences. Attention mechanisms, mem-
ory augmented networks, and some combinations of the two are currently the top
techniques which address many of the issues discussed above. In this chapter, we
will discuss in detail many popular techniques of attention mechanisms and memory
networks that have been successfully employed in speech and text.
Though the attention mechanism has become very popular in NLP and speech
in recent times after Bahdanau et al. proposed their research, it had been previously
introduced in some forms in neural architectures. Larochelle and Hinton highlight
the usefulness of “fixation points” to improve performance in the image recognition
tasks [LH10]. Denil et al. also proposed a similar attention model for object tracking
In a much more general way, attention is a concept very well known in human psy-
chology, where humans who are limited by processing bottlenecks have a selective
focus on certain part of the information and ignore the rest of the visible informa-
tion. Mapping the same concept of human psychology to sequence data such as text
streams or audio streams, when we focus on certain parts of sequences or regions
and blur the remaining ones during the learning, the process is called an attention
mechanism. Attention was introduced in Chap. 7 while introducing recurrent net-
works and sequence modeling. Since many techniques using attention are either
9.2 Attention Mechanism 409
related to or are used in memory augmented networks, we will cover some of the
modern techniques that have broad application.
Let us consider a translation from English to French for a sentence “I like coffee”
which in French is ‘J’aime le café.” We will use a machine translation use case
with the sequence-to-sequence models to highlight the need for the attention mech-
anism. Let us consider a simple RNN with encoder–decoder network as shown in
Fig. 9.2. We see in the above neural machine translation that the entire sentence
ci = ∑ ai, j s j (9.1)
j
ai = align(hi , s j ) (9.2)
= softmax(score(hi , s j )) (9.3)
ai j are called the alignment weights. The equation above captures how every in-
put element can influence the output element at a given position. The predefined
function score is called the attention score function and there are many variants
of these that will be defined in the next section.
• The source side context vector ci and the hidden state hi are combined using con-
catenation [ci ; hi ] and the non-linear tanh operation to give the attention hidden
vector h̃i :
h̃i = tanh(Wc [ci ; hi ]) (9.4)
where the weights Wc are learned in the training process.
• The attention hidden vector h̃i is passed through a softmax function to generate
the probability distribution given by:
– Bahdanau et al. use bidirectional LSTM layers in the encoders, and con-
catenate the hidden states.
– Bahdanau et al. use the previous state, i.e., hi−1 , and the computational
path is hi−1 −→ ai −→ ci −→ h̃i as compared to Luong et al. which has
hi −→ ai −→ ci −→ h̃i .
9.2 Attention Mechanism 411
– Bahdanau et al. use the linear combination of previous state and encoder
states in the scoring function given by score(s j , hi ) = va tanh(Wa s j +
Ua hi ).
– Luong et al. have the input-feeding mechanism where the attentional hid-
den vectors h̃i are concatenated with the target input.
Fig. 9.3: Step-by-step computation process for soft attention in the encoder–decoder
network
Table 9.1 gives different ways attention score functions can be computed to give
different flavors of attention.
• The multiplicative and additive score functions generally give similar re-
sults, but multiplicative score functions are faster in both computation and
space-efficiency using efficient matrix multiplication techniques.
412 9 Attention and Memory Augmented Networks
• Additive attention performs much better when the input dimension is large.
The scaled dot-product method defined above has been used to mitigate
that issue in the general dot product.
The only difference between soft attention and hard attention is that in hard at-
tention it picks one of the encoder states rather than a weighted average over all the
inputs as in soft attention. The hard attention is given by:
Thus the difference between hard attention and soft attention is based on the search
when the context is computed.
Hard attention uses the arg max function which is not a continuous func-
tion, not differentiable and hence cannot be used in standard backpropagation
methods. Techniques such as reinforcement learning to select the discrete part
and Monte Carlo based sampling are often used. Another technique is to use
the Gaussian trick given in the next section.
The soft attention methods such as the Bahdanau’s research are also referred to
as global attention, as each decoder state takes “all” of the encoder inputs while
computing the context vector. The process of iterating over all the inputs can be both
9.2 Attention Mechanism 413
computationally expensive and many times impractical when the sequence length is
large.
Luong et al. introduced local attention which is a combination of the soft at-
tention and the hard attention to overcome these issues [Luo+15]. One of the ways
local attention can be achieved is to have a small window of the encoder hidden
states used for computing context. This is called predictive alignment and it re-
stores differentiability.
At any decoder state for time i, the network generates an aligned position pi ,
and a window of size D around either side of the hidden state of the position, i.e.,
[pi − D, pi + D] is used to compute the context vector c. The position pi is a scalar
computed using a sigmoid function on the current decoder hidden state hi and using
the sentence length S as given by:
where W p and v p are the model parameters to be learned to predict the position
and S is the length of the sequence and pi ∈ [0, S]. The difficulty is in how to fo-
cus around the location pi without using the non-differentiable arg max. One way
to focus the alignment near pi , a Gaussian distribution is centered around pi with
standard deviation σ = D2 given by:
(s − pt )2
ai = align(s j , hi ) exp − (9.8)
2σ 2
9.2.6 Self-Attention
where Wa and va are weight matrices and vectors learned on the training data. The
final sentence vector c is computed by:
c = Ha (9.11)
Instead of just using the single vector va , several hops of attention are performed by
using a matrix V which captures multiple relationships existing in the sentences and
allows us to extract an attention matrix A as:
414 9 Attention and Memory Augmented Networks
Fig. 9.4: Step-by-step computation process for local attention in the encoder-
decoder network
A = so f tmax(Vatanh(Wa H )) (9.12)
C = AH (9.13)
To encourage diversity and penalize redundancy in the attention vectors, we use the
following orthogonality constraint as a regularization technique:
Key-value attention by Daniluk et al. is another variant which splits the hidden layer
into key-value where the keys are used for attention distribution and the values for
context representation [Dan+17]. The hidden vector h j is split into a key k j and a
value v j : [k j ; v j ] = h j . The attention vector ai of length L is given by:
Vaswani et al. in their work propose transformer network using multi-head self-
attention without any recurrent networks to achieve state-of-the-art results in ma-
chine translation [Vas+17c]. We will describe the multi-head self-attention in
a step-by-step manner in this section and as illustrated in Fig. 9.5. The source
input with word1 , word2 , . . . , wordn words is first mapped to an embedding layer to
get vectors for the words x1 , x2 , . . . , xn . There are three matrices WQ , WK , and WV
called the query, key, and value weight matrices that are trained during the training
process. The word embedding vectors are multiplied by the matrices WQ , WK , and
WV to get the query, key, and value vectors, respectively, for each word given by
q, k, and v. Next is to calculate a score for each word for every other word in the
sentence by using the dot product of that query vector q and the key vectors k for
every word. This scoring captures a single interaction of the word with every other
word. For instance, for the first word:
score1 = q1 k1 + q2 k2 + · · · + qn kn
√
This is then divided by the length of the key vector dk and a softmax is computed
to get weights between 0 and 1. This score is then multiplied with all the value
vectors to get the weighted value vectors. This gives attention or focus to specific
words in the sentence rather than every word. Then the value vectors are summed to
416 9 Attention and Memory Augmented Networks
compute the output attention vector given for that word given by:
Now, instead of this step-by-step computation, the whole thing can be computed
by taking the sentence representation as embedding matrix of all word vectors X
and multiplying it with the respective weight matrices WQ , WK , and WV to get the
matrices for all the words as Q, K, and V and then using the equation to compute
the attention:
(QK )
attention(Q, K, V) = Z = softmax √ V (9.17)
dk
Instead of using just one attention as computed above, they use multi-head atten-
tion where there are many such attention matrices computed for the input and can
be represented by Z0 , Z1 , . . . , Zm . These matrices are concatenated and multiplied
by another weight matrix WZ to get the final attention Z.
Yang et al. used hierarchical attention for document classification tasks showing the
advantage of having attention mechanisms at sentence level for the context and the
word level for importance [Yan+16]. As shown in Fig. 9.6, the overall idea is to
have word level encoding using bidirectional GRUs, word level attention, sentence
level encoding, and sentence level attention hierarchically. We will briefly explain
each of these components.
Let us consider input as a set of documents, each document has a maximum L
sentences, and each sentence has a maximum T words such that wit represents a tth
word in the ith sentence in a document. The sentences with all the words go through
an embedding matrix We that converts them to a vector xi j = We wi j . It then goes
through bidirectional GRU as:
vector uw is initialized randomly and then learned in the training process. The intu-
ition behind the context vector uw according to the authors is that it captures a fixed
query like “what is the informative word” in the sentence.
exp(uit uw )
αit = (9.22)
∑ exp(uit uw )
t
Given L of the si sentence vector, the document hidden vectors are computed similar
to word vectors using bidirectional GRUs.
ui = tanh(Ws hi + bs ) (9.26)
exp(ui us )
αi = (9.27)
∑ exp(uTi us )
i
v = ∑ αi hi (9.28)
i
The document vector v goes through the softmax for classification and negative
log-likelihood of label to prediction is used for training.
Many NLP and NLU research have used attention mechanisms for tasks such as sen-
tence embedding, language modeling, machine translation, syntactic constituency
parsing, document classification, sentiment classification, summarization, and di-
alog systems to name a few. Lin et al.’s self-attention using LSTM for sentence
embedding showed significant improvements over other embeddings on a variety
of tasks such as sentiment classification and textual entailment [Lin+17]. Daniluk
et al. applied attention mechanisms to language modeling and showed compara-
ble results to memory augmented networks [Dan+17]. Neural machine transla-
tion implementations discussed in the chapter has achieved the state-of-the-art re-
sults [BCB14b, LPM15, Vas+17c]. Vinyals et al. showed that attention mechanisms
for syntactic constituency parsing could not only attain state-of-the-art results but
also improve on speed [Vin+15a]. Yang et al.’s research showed that using hierar-
chical attention can outperform many CNN and LSTM based networks by a large
margin [Yan+16]. Wang et al. showed that attention-based LSTM could achieve
state-of-the-art results in the aspect-level sentiment classification [Wan+16b]. Rush
et al. showed how local attention methods could give significant improvements in
the text summarization task [RCW15].
Chorowski et al. introduced how attention mechanisms can achieve better
normalization for smoother alignments and using previous alignments for gen-
erating features in speech recognition [Cho+15b]. Bahdanau et al. used end-
to-end attention-based networks for large vocabulary speech recognition prob-
lems [Bah+16b]. Listen, attend, and spell (LAS), an attention-based model, has
9.3 Memory Augmented Networks 419
Next, we will describe some well-known memory augmented networks that have
been very effective in NLP and speech research.
Memory networks (MemNN) by Weston et al. were motivated by the ability to store
information coming from stories or knowledge base facts so that various questions
pertaining to these can be easily answered [WCB14]. Memory networks have been
extended in many ways for various other applications but performing question and
answer on the stories or facts can be considered its basic application that we will
focus on our narrative.
Memory networks consists of memory m indexed by mi and has four components
as shown in Fig. 9.7
1. Input Feature Map I: This component converts the incoming data to the internal
feature representation. This component can do any task-specific preprocessing
such as converting the text to embeddings or POS representations, to name a
few. Given an input x, the goal is to convert to an internal feature representation
I(x).
2. Generalization G: This component uses the representation of the input from
above and updates the memory by using any transformation if necessary. The
transformation can be as simple as using the representation as is or making a
coreference resolution, to complex reasoning based on the tasks. This transfor-
mation is given by:
mH(x) = I(x) (9.29)
In general the updating of the memories mi for the new input is given by H(·),
which is a general function that can be used to do various things from simplest
such as finding a slot index in the memory to the complex part of finding the
slot if full or forget certain memory slot. Once the slot index is detected, the G
stores the input I(x) in that location:
3. Output O: This is the “read” part from the memory where necessary inference
to deduce relevant parts of memories for generating the response happens. This
can be represented as computing the output features given the new input and
memory as o = O(I(x), m).
4. Response R: This component converts the output from memory into a repre-
sentation that the outside world can understand. The decoding of the output
features to give the final response can be given as r = R(o).
In the paper the input component stores the sentence as is, for both the stories and
questions. The memory write or generalization is also basic writing to the next slot,
i.e., mN = x, N = N + 1. Most of the work is done in the output O, R part of the
network.
The output module finds the closest match for the input using k memories that
support the fact and a scoring function
The sO is the scoring function that matches the input question or the input fact/story
sentence to all the existing memory slots for the best match. In the simplest case,
they choose k = 2 for the output inferencing. This can be represented as:
The input [x, mO1 , mO2 ] is given to the response component which generates a single
word with highest ranking given by:
9.3 Memory Augmented Networks 421
where W is the set of all words in the vocabulary and sR is the scoring function to
match words to the inputs. In the paper the scoring functions sO and sR have same
form and can be written as:
where f¯, f¯ , r̄ are other choices apart from the true label, i.e., it adds a margin loss if
the score of the wrong choices is greater than the ground truth minus γ .
The scoring function o1 and o2 given in Eqs. 9.32 and 9.33 can be computa-
tionally expensive when the memory storage is large. The paper uses a cou-
ple of tricks such as hashing the words and clustering word embeddings in a
cluster k. The clustering approach gives a nice trade-off between speed and
accuracy with the cluster size k choice.
Let us take a simple example with two supporting facts dataset from the bAbI al-
ready in the memory slots given in the table below. When the question “Where is the
football?” is asked, the input after going through k = 2, x = “Where is the football?”
is matched to everything in memory and the slot mO1 = “Mary got the football
there” and using this, i.e., [x, mO1 ], it will perform another similarity search and find
mO2 = “Mary went back to the garden” giving rise to the new output [x, mO1 , mO2 ].
The R component uses [x, mO1 , mO2 ] input to generate an output response r = “gar-
den.”
To overcome the issues of memory networks such as the need for each of the
component to be trained in a supervised manner, issues of training hard atten-
tion, to name a few, Sukhbaatar et al. proposed end-to-end memory networks or
MemN2N. MemN2N overcomes many of the disadvantages of MemNN by having
soft attention while reading from the memory, performing multiple lookups or hops
on memory, and training end-to-end with backpropagation with minimal supervi-
sion [Suk+15].
The MemN2N takes three inputs; (a) the story/facts/sentences x1 , x2 , . . . , xn , (b) the
query/question q, and (c) the answer/label a. We will walk through different com-
ponents and interactions of MemN2N architecture next considering only one layer
of memory and the controller.
The input sentences, for instance, xi is the ith sentence with words wi j as given by
xi = xi1 , xi2 , . . . , xin , are converted into a memory representation m1 , m2 , . . . , mn of
dimension d using embedding matrix A of dimension d × |V |, where |V | is the size
of the vocabulary. The operation is given by:
mi = ∑ Axi j (9.37)
j
The paper discusses different ways of combining word embeddings for all the words
in the sentence, for example, by performing a sum operation on all the word embed-
dings to get a sentence embedding. Similarly, the query or the question sentence is
mapped to a vector of dimension d using embedding matrix B of dimension d × |V |.
9.3 Memory Augmented Networks 423
The query representation u from the embedding matrix B for the controller is then
matched with every memory index mi using the dot product for similarity and soft-
max for choosing the state. The operation can be given by:
exp(uT mi )
pi = (9.38)
∑ j exp(uT m j )
Each input sentences xi are also mapped to the controller as vectors ci of dimen-
sion d using a third embedding matrix C of dimension d × |V |. The output is then
combined using the softmax outputs pi and the vector ci as:
o = ∑ pi ci (9.39)
i
The output vector o and input query with embeddings u are combined and then
passed through a final weight matrix W and a softmax to produce the label:
exp(W(o + u))
â = (9.40)
∑ j exp(W(o + u))
The true label a and the predicted label â are used to then train the networks includ-
ing the embeddings A, B, C, and W using cross-entropy loss and stochastic gradient
descent. A single layer MemN2N with complete flow from input sentences, query,
and answer is shown in Fig. 9.8.
The single-layered MemN2N is then extended to multiple layers as shown in Fig. 9.9
in the following way:
• Each layer has its own memory embedding matrix A for input and controller/out-
put embedding matrix C.
• Each layer input K + 1 combines the output of current layer ok and its input uk
using:
uk+1 = ok + uk (9.41)
• The top layer uses the output with softmax function in a similar way to generate
a label â.
424 9 Attention and Memory Augmented Networks
• The final output â is similarly compared to the actual label a, and the entire
network is trained using cross-entropy loss and stochastic gradient descent.
Since many of the tasks such as QA require temporal context, i.e., an en-
tity was at some place before going to another place, the paper modifies the
memory vector to encode the temporal context using a temporal matrix. For
example, the input memory mapping can be written as:
Graves et al. proposed a memory augmented network called neural Turing machines
(NTM) for performing complex tasks that were repetitive and needing information
over longer periods [GWD14b]. As shown in Fig. 9.10, it has a neural network com-
ponent called controller which interacts with the outside world and the inner mem-
9.3 Memory Augmented Networks 425
ory for all its operations. Drawing inspiration from Turing machines, the controller
interacts with the memory using the read heads and the write heads. Since mem-
ory read or write can be seen as discrete and non-continuous operations, it cannot
be differentiated, and thus most gradient-based algorithms cannot be used as is.
One of the most important concepts introduced in the research was to overcome
this issue by using blurry operations in both read and write that interact with all
the memory elements with varying degrees. By using these blurry operations, all
reading and writing can be continuous, differentiable, and learned effectively using
gradient-based algorithms such as stochastic gradient descent.
Let us consider memory M to be a two-dimensional matrix (N × M) with N rows
corresponding to the memory and M columns for each row where values get stored.
Next, we will discuss different operations in NTM.
Attention is used to move the read (and write) heads in NTM. The attention mech-
anism can be written as a length-N normalized weight vector wt , reading contents
from the memory Mt at a given time t. Individual elements of this weight vector
will be referred to as wt (i).
426 9 Attention and Memory Augmented Networks
The read head will return a length-M read vector rt which is a linear combination
of the memory’s rows scaled by the weight vector as given by:
M
rt ← ∑ wt (i)Mt (i) (9.45)
i=1
Writing in NTM can be seen as two distinct steps: erasing the memory content and
then adding new content. The erasing operation is done through a length-M erase
vector et in addition to the weight vector wt to specify which elements in the row
should be completely erased, left unchanged, or some changes carried out. Thus the
9.3 Memory Augmented Networks 427
weight vector wt gives us a row to attend and the erase vector et erases the elements
in that row giving the update:
After the erase state, i.e., Mt−1 converted to Mterased , the write head uses a length-M
add vector at to complete the writing as given by:
Since both erase and write operations are differentiable, entire write operation is
differentiable.
The weights used in reading and writing are computed based on two addressing
mechanisms: (a) content-based addressing and (b) location-based addressing.
The idea behind content-based addressing is to take information generated from the
controller, even if it is partial, and find an exact match in the memory. In certain
tasks, especially variable-based operations, it is imperative to find the location of
the variables for tasks such as iterations and jumps. In such cases, location-based
addressing is very useful.
The weights are computed in different stages and passed on to the next stage.
We will walk through every step in the process of computing the weights as given
in Fig. 9.11. The first stage known as content addressing takes the two inputs: a key
vector kt of length-M and a scalar key strength βt . The key vector kt is compared
to every vector Mt (i) using a similarity measure K[·, ·]. The key strength, βt , acts
428 9 Attention and Memory Augmented Networks
Location-based addressing is performed in the next three stages. The second stage,
called interpolation gt ∈ (0, 1), takes a scalar parameter from the controller head
which is used to combine the content weight from the previous step wtc and previous
time step’s weight vector wt−1 to generate the gated weighting wtg given by:
The next stage is a convolutional shift which works to shift attention to other rows. It
takes a shift vector st from the controller head as input and the previous interpolated
output wtg . The shift vector can have various values such as +1 to shift forward
one row, 0 to stay as is, and −1 to shift backward one row. The operation is a shift
modulo N so that attention shift of the bottom moves the head to the top and vice
versa. The convolution shift is given by w̃t and the operation is:
N−1
w̃t (i) ← ∑ wt ( j)g st (i − j) (9.50)
j=0
The final stage is sharpening, which prevents the previous convolution shifted
weights from blurring using another parameter γ ≥ 1 from the controller head. The
final output of the weight vector wt is given by:
w̃t (i)γt
wt (i) ← (9.51)
∑ j w̃t ( j)γt
Thus the address of reading and writing is computed by the above operations and
all of the parts are differentiable and hence can be learned by gradient-based al-
gorithms. The controller network has many choices such as type of neural network,
number of read heads, number of write heads, etc. The paper used both feed-forward
and LSTM based recurrent neural network for the controller.
The controller network receives input vector xt ∈ RX at every time step and gen-
erates an output yt ∈ RY . It also receives as input rather R read vectors the previ-
t−1 ∈ R
ous time step as rt−1 1 , . . . , rR from the memory matrix M N×W via the read
t−1
heads. The input and the read vectors are concatenated as single controller input
1 , . . . , rR ]. The controller uses a neural network such as LSTM.
xcont = [xt ; rt−1 t−1
Location selection happens using weight vectors that are non-negative and sum to
1. The complete set of “allowed” weighting over N locations in the memory is given
by a non-negative orthant and constraints as:
-
N
ΔN = α ∈ RN αi ∈ [0, 1], ∑ αi ≤ 1 (9.52)
i=1
The read operation is carried out using R read weights {wtr,1 , . . . , wtr,R } ∈ ΔN , thus
giving read vectors {rt1 , . . . , rtR } by equation:
The read vectors get appended to the controller input at the next time step.
The write operation is carried out by write weighting wtw ∈ RN together with
write vector vt ∈ RW and the erase vector et ∈ [0, 1]W both emitted by the controller
to modify the memory as:
430 9 Attention and Memory Augmented Networks
The weightings from controller outputs are parameterized over the memory rows
with three forms of attention mechanisms: content-based, memory allocation, and
temporal order. The controller interpolates among these three mechanisms using
scalar gates.
Similar to NTM, selective attention uses a partial key vector kt of length-W and
a scalar key strength βt . The key vector kt is compared to every vector Mt [i] us-
ing a similarity measure K[·, ·] to find the closest to the key normally using cosine
similarity as given by:
The controller uses an allocation gate (gta ∈ [0, 1]) to interpolate between writing to
the newly allocated location in the memory (at ) or an existing location found by
content (ctw ) with gtw ∈ [0, 1] being the write gate:
A temporal link matrix (Lt [i, j] ∈ RN×N ) represents the degree to which location
i was the location after location j. The matrix gets updated using the precedence
weight vector pt as given by:
The controller can use the temporal link matrix to retrieve the write before (bti ) or
r,i
after (fti ) the last read location (wt−1 ) allowing the forward and backward movement
in time given by the following equations:
Kumar et al. proposed dynamic memory networks (DMN), where many tasks in
NLP can be formulated as a triplet of facts–question–answer and end-to-end learn-
ing can happen effectively [Kum+16]. We will describe the components of DMN
as shown in Fig. 9.13. We will use the small example given in Fig. 9.13 to explain
each step.
The input module takes the stories/facts, etc. as sentences in raw form, transforms
them into a distributed representation using embeddings such as GloVe from the
memory module, and encodes it using a recurrent network such as GRU. The input
can be a single sentence or list of sentences concatenated together—say of TI words
given by w1 , . . . , wTI . Every sentence is converted by adding an end-of-sentence to-
ken and then the words are concatenated. Each end-of-sentence generates a hidden
state corresponding to that sentence like ht = GRU(L(wt ), ht−1 ), where wt is the
word wt index at time t and L is the embedding matrix. The outputs of this input
432 9 Attention and Memory Augmented Networks
module are length Tc fact sequences of hidden state of each sentence and ct is the
fact at step t. In the simplest case where each sentence output is encoded as a fact
Tc is equal to the number of sentences.
The question module is similar to the input module, where the question sen-
tence of TQ words is converted to an embedding vector, and given to the recur-
rent network. The GRU based recurrent network is used to model it given by
qt = GRU(L(wtQ ), qt−1 ), where L is the embedding matrix. The hidden state, which
is the final state at the end of the question, was given by q = qTQ . The word embed-
ding matrix L is shared across both the input and the question module.
The hidden states of the input module across all the sentences and the question
modules output are the inputs for the episodic memory module. Episodic memory
has an attention mechanism to focus on the states from the input and a recurrent
network which updates its episodic memory. The episodic memory updates are part
of an iterative process. In each iteration the attention mechanism attends over the
input module’s hidden states mapping to the fact representation c, question q, and
the past memory mi−1 to produce an episode ei . The episode is then used along
with previous memory mi−1 to update episodic memory mi = GRU(ei , mi−1 ). The
GRU is initialized with a question as the state, i.e., m0 = q. The iterative nature
9.3 Memory Augmented Networks 433
of the episodic memory helps in focusing on different sections of the inputs and
thus has the transitive nature required for inferencing. The number of passes TM is a
hyperparameter and the final episodic memory mTM is given to the answer module.
The attention mechanism has a feature generation part and a scoring part using
the gating mechanism. The gating happens through a function G that takes as input
a candidate fact ct , previous memory mi−1 , and the question q to compute a scalar
gt which acts as a gate:
gti = G(ct , mi−1 , q) (9.64)
The feature vector z(c, m, q) which feeds into the scoring function G above using
different similarities between the input facts, previous memory, and the question as
given by:
z(c, m, q) = [c ◦ m; c ◦ q; |c − m|; c − m] (9.65)
where ◦ is the element-wise product between the vectors. The scoring function G is
the standard two-layer feed-forward network where
where weights W(1) , W(2) and biases b1 , b2 are learned through the training process.
The episode at iteration i uses the GRU with sequences c1 , . . . , cTC weighted by the
gate gi and the final hidden state is used to update as given by:
ei = hiTC (9.68)
Either maximum iteration is set or a supervised symbol to mark the end of phase
token is passed to stop the iteration.
The answer module can either be triggered at the end of every episodic memory
iteration or the final one based on the task. It is again modeled as a GRU with an
input question, last hidden state at−1 , and previous prediction yt−1 . The initial state
a0 is initialized to the last memory as a0 = mTM . Thus the updates can be written as:
yt = so f tmax(Wa at ) (9.69)
9.3.5.5 Training
The end-to-end training is done in a supervised manner where the answer generated
by the answer module is compared to the real labeled answer, and a cross-entropy
loss is propagated back using stochastic gradient descent. To give a concrete exam-
434 9 Attention and Memory Augmented Networks
ple, let us consider the story with sentences s1 , . . . , s6 as inputs to the input module
and question q Where is the football? to be passed to the question module as shown
in Fig. 9.13. In the first pass of the episodic memory let us assume it will try to
attend to the word football from the question, all the facts c coming as hidden states
from the input modules and will score all the facts from input where football ap-
pears and give maximum to facts such as Mary got the football there. In the next
iteration, it will take the output from this episodic state and try to focus on the next
part Mary and thus select all statements such as Mary moved to the bathroom, Mary
got the football there, Mary went back to the kitchen, and Mary went back to the
garden. From these let us assume it will select the last sentence Mary went back
to the garden. The selection of the right sentences to focus happens in an end-to-
end manner using backpropagation, where the actual label from the answer module
garden compares the generated output to propagate the errors back.
Grefenstette et al. explore learning interactions between the controller and memory
using the traditional data structures such as stacks, queues, and deques. They pro-
duce superior generalization when compared with RNNs [Gre+15]. In the next few
sections, we will explore the basic working of neural stack architecture and then
generalize it to others.
The neural stack is a differentiable structure which allows storing the vectors
through push operations, and retrieving the vectors through pop operations anal-
ogous to the stack data structure as shown in Fig. 9.14.
The entire stack content at a given time t is denoted by matrix Vt , each row
corresponding to memory address i contains a vector vt of size m such that it is in
the space Rm . Associated with each index in the matrix is a strength vector giving
the weight associated with that index of content and is given by st . The push signal
is given by a scalar dt ∈ (0, 1) and the pop signal is given by a scalar ut ∈ (0, 1). The
value read from the stack is given by rt ∈ Rm .
The necessary operations for the neural stack are given by the following three
equations for Vt , st , and rt :
Vt−1 [i] if 1 ≤ i < t
Vt [i] = (9.71)
vt if i = t, Vt [i] = vt for all i ≤ t
9.3 Memory Augmented Networks 435
Equation 9.71 captures the updates to the stack as an ever-growing list-like structure
of a neural stack where every old index gets the value from the previous time step
and the new vector is pushed on the top:
⎧
⎪
⎨max 0, s [i] − max 0, u − t−1 s [ j]
t−1 t ∑ t−1 if 1 ≤ i < t
st [i] = j=i+1 (9.72)
⎪
⎩
dt if i = t
Equation 9.72 captures the weight updates, where the case i = t means that we di-
rectly pass the push weight dt ∈ (0, 1). Removing a entry from stack doesn’t remove
it physically but sets the strength value at the index 0. Each of the strengths lower
down the stacks changes based on the following calculation, subtract the pop signal
strength ut and the relative sum above that index i + 1 and below index at value t − 1
and cap it by finding maximum between that value and 0. Then subtract it with the
current value at the index st+1 and cap it by finding maximum between that value
and 0.
We look at Fig. 9.14 at time t = 3 and lowest index i = 1 assuming a with previous
value of 0.7, it will become max(0, 0.7− max(0, 0.9− 0.5)) = 0.3. Similarly, we can
plug the same value for t = 3 and next index i = 2 with previous value of 0.7, it will
become max(0, 0.9 − max(0, 0.9 − 0)) = 0. Finally, at t = 3, the top index i = 3 will
have the dt value of 0.9:
t t
rt = ∑ min st [i], max 0, 1 − ∑ st [ j] ·Vt [i] (9.73)
i=1 j=i+1
Equation 9.73 can be seen as the state that the network sees at time t. It is a com-
bination of the index vector and its strength, where the strengths are constrained to
sum to 1.
Again when we look at Fig. 9.14 at time t = 3, we see that everything is normal
combinations except the strength of index 1 is changed from 0.3 to 0.1 because
substituting we get min(0.3, max(0, 1 − 0.9)) = 0.1.
Fig. 9.14: Neural stack states with respect to time and operations of push and pop
436 9 Attention and Memory Augmented Networks
The gradual extension of the neural stack from above as a recurrent network and the
controller actions are shown in Fig. 9.15a, b. The entire architecture marked with
dotted lines is a recurrent network with inputs (a) previous recurrent state Ht−1 and
(b) current input it ; and outputs (a) next recurrent state Ht and (b) ot . The previous
recurrent state Ht−1 consists of three parts: (a) the previous state vector from RNN
ht−1 , (b) the previous stack read rt , and (c) the state of the stack from previous state
(Vt−1 , st ). In the implementation, all of the vectors except h0 , which is randomly
initialized, all are set to 0 to start with.
Fig. 9.15: Neural stack with recurrent network and controller. (a) Neural stack as
recurrent network. (b) Neural stack recurrent network with controller
9.3 Memory Augmented Networks 437
The current input it is concatenated with previous read of the stack rt−1 to the
controller which has its own previous state ht−1 generating next state ht and the
output ot . The output ot results in the push signal scalar dt , the pop signal scalar ut ,
and the value vector vt that go as input signals to the neural stack and the output
signal ot for the whole. The equations are:
The Neural DeQue works similarly to a neural stack, but has the ability to take the
input signals of push, pop, and value for both the top and the bottom sides of the
list.
Henaff et al. designed a highly parallel architecture with a long dynamic memory
which performs well on many NLU tasks known as recurrent entity networks (Ent-
Net) [Hen+16]. The idea is to have blocks of memory cells, where each cell can
store information about an entity in the sentence so that many entities correspond-
ing to names, locations, and others have information content in the cells. We will
discuss the core components of EntNet in Fig. 9.16.
yi is the single word answer. The input encoding layer transforms the sequence of
words into a fixed length vector. This can be done as the authors describe using
BOW representation and end states of RNN. They chose a simple representation
given by using a set of vectors {f1 , . . . , fk } with the input embeddings of the words
{e1 , . . . , ek } for a given input at given time t:
st = ∑ fi ◦ ei (9.80)
i
As shown in Fig. 9.16b the input encoded sentences flow into blocks of memory
cells and the whole network is a form of gated recurrent unit (GRU) with hidden
states in these blocks, which concatenated together give the total hidden state of the
network. The total blocks h1 , . . . , hm are of the order 5–20 and each block h j has
20–100 units. Each block j is given a hidden state h j ∈ Rd and a key w j ∈ Rd .
The role of a block is to capture information about an entity with the facts. This is
accomplished by associating the weights of the key vectors with the embeddings of
entities of interest so that the model learns information about the entities occurring
within the text. A generic j block with weight w j and hidden state h j is given by:
htj ← ht−1
j + g j ◦ h̃ j
t
(new memory) (9.83)
htj
htj ← (reset memory) (9.84)
htj
where g j is the gate that decides how much of the memory will be updated, φ is
the activation function like ReLU, htj is the new memory that is combining the older
timestamp with current, and the normalization in the last step helps in forgetting
the previous information. The matrices P ∈ Rd×d , Q ∈ Rd×d , R ∈ Rd×d are shared
across all blocks.
9.3 Memory Augmented Networks 439
Fig. 9.16: EntNet. (a) A single block in the EntNet. (b) Recurrent entity networks
(EntNet) with multiple blocks
The output module when presented with question q creates a probability distribution
over all the hidden states and the entire equations can be written as:
p j = softmax(q h j ) (9.85)
u = ∑ p jh j (9.86)
j
y = Rφ (q + Hu) (9.87)
440 9 Attention and Memory Augmented Networks
The matrices R ∈ R|V |×d and H ∈ Rd×d are again trained with the rest of the pa-
rameters. The function φ adds the non-linearity and can be an activation like ReLU.
The entire network is trained using backpropagation. The entities can be extracted
as part of preprocessing, and the key vectors can be specifically tied to the embed-
dings of the entities existing in the stories such as {Mary, Sandra, John, bathroom,
bedroom, kitchen, garden, football} in the bAbI example.
Most memory networks have been successfully applied to complex NLU tasks
such as question answering and semantic role labeling [WCB14, Suk+15, Gra+16,
Hen+16]. Sukhbaatar et al. applied end-to-end memory networks to outperform tra-
ditional RNNs by increasing memory hops in the language modeling task [Suk+15].
Kumar et al. have interestingly converted most NLP tasks from syntactic to seman-
tic tasks in a question-answering framework and applied dynamic memory networks
successfully [Kum+16]. Grefenstette et al. showed significant performance gains,
obtained using memory networks such as neural stacks, queues, and deques in trans-
duction tasks such as inversion transduction grammars (ITG) used in machine trans-
lation [Gre+15].
In this section, we explore two NLP topics: attention-based NMT and memory net-
works for question and answering. Each topic follows the same format as used in
previous chapters, and provides exercises in the end.
In this portion of the case study, we compare attention mechanisms on the English-
to-French translation task introduced in Chap. 7. The dataset used is composed of
translation pairs from the Tatoeba website. This is the same dataset used in the
Chap. 7 case study.
9.4 Case Study 441
For the EDA process, we refer the readers to Sect. 7.7.2 for the steps that were used
to create the dataset splits.
The dataset summary is shown below.
1 T r a i n i n g s e t s i z e : 107885
2 V a l i d a t i o n s e t s i z e : 13486
3 T e s t i n g s e t s i z e : 13486
4 S i z e o f E n g l i s h v o c a b u l a r y : 4755
5 S i z e o f F r e n c h v o c a b u l a r y : 6450
When we first explored neural machine translation, we used the fairseq library which
leverages PyTorch. To the best of our knowledge, there is not a single library that
supports all the different attention mechanisms. Therefore, we combine a collection
of libraries to compare the attention approaches. Specifically, we use PyTorch as
the deep learning framework, AllenNLP for most attention mechanism implemen-
tations, spaCy for tokenization, and torchtext for the data loader. The code contained
here extends some of the original work in the PyTorch tutorials with additional func-
tionality and comparisons.
We compare five different attention mechanisms, training for 100 epochs. For each
attention mechanism the model that performs the best on the validation data is cho-
sen to run on the testing data. The models trained are 4-layer bidirectional GRU
encoders with a single unidirectional GRU decoder. The encoder and decoder both
have a hidden size of 512 with the encoding and decoding embeddings having a size
of 256. The models are trained with cross-entropy loss and SGD, with a batch size
of 512. The initial learning rate is 0.01 for the encoder and 0.05 for the decoder, and
momentum is applied to both with a value of 0.9. A learning rate schedule is used
to reduce the learning rate when the validation loss hasn’t improved for 5 epochs.
To regularize our model, we add dropout to both the encoder and decoder with a
probability of 0.1 and the norms of the gradients are clipped at 10.
We incorporate a batch implementation of the model to leverage the parallel com-
putation of the GPUs. The architecture is the same for each of the models except for
the Bahdanau model, which required introducing a weight matrix for the bidirec-
tional output of the encoder in the attention mechanism.
We define the different components of our networks as follows:
1 c l a s s E n c o d e r ( nn . Module ) :
2 def init ( s e l f , i n p u t d i m , emb dim , e n c h i d d i m ,
dec hid dim ,
442 9 Attention and Memory Augmented Networks
3 d r o p o u t , n u m l a y e r s =1 , b i d i r e c t i o n a l = F a l s e ) :
4 super () . init ()
5 s e l f . input dim = input dim
6 s e l f . emb dim = emb dim
7 s e l f . enc hid dim = enc hid dim
8 s e l f . dec hid dim = dec hid dim
9 s e l f . num layers = num layers
10 self . bidirectional = bidirectional
11
22 if self . bidirectional :
23 hidden = torch . tanh ( s e l f . fc ( torch . cat ( ( hidden [ −2 ,: ,:] ,
h i d d e n [ − 1 , : , : ] ) , dim = 1 ) ) )
24
25 i f n o t s e l f . b i d i r e c t i o n a l and s e l f . n u m l a y e r s > 1 :
26 hidden = hidden [ −1 ,: ,:]
27
28 r e t u r n outputs , hidden
1 c l a s s D e c o d e r ( nn . Module ) :
2 def init ( s e l f , o u t p u t d i m , emb dim , e n c h i d d i m ,
dec hid dim , dropout ,
3 attention , b i d i r e c t i o n a l i n p u t =False ) :
4 super () . init ()
5 s e l f . emb dim = emb dim
6 s e l f . enc hid dim = enc hid dim
7 s e l f . dec hid dim = dec hid dim
8 s e l f . output dim = output dim
9 s e l f . dropout = dropout
10 self . attention = attention
11 self . bidirectional input = bidirectional input
12
15 if bidirectional input :
16 s e l f . r n n = nn .GRU ( ( e n c h i d d i m ∗ 2 ) + emb dim ,
dec hid dim )
17 s e l f . o u t = nn . L i n e a r ( ( e n c h i d d i m ∗ 2 ) + d e c h i d d i m +
emb dim , o u t p u t d i m )
18 else :
19 s e l f . r n n = nn .GRU ( ( e n c h i d d i m ) + emb dim , d e c h i d d i m
)
20 s e l f . o u t = nn . L i n e a r ( ( e n c h i d d i m ) + d e c h i d d i m +
emb dim , o u t p u t d i m )
9.4 Case Study 443
21
22 s e l f . d r o p o u t = nn . D r o p o u t ( d r o p o u t )
23
29 # R e p e a t h i d d e n s t a t e f o r a t t e n t i o n on b i d i r e c t i o n a l
outputs
30 i f h i d d e n . s i z e ( −1) ! = e n c o d e r o u t p u t s . s i z e ( −1) :
31 a t t n = s e l f . a t t e n t i o n ( hidden . r e p e a t (1 , 2) ,
e n c o d e r o u t p u t s . permute (1 , 0 , 2) )
32 else :
33 a t t n = s e l f . a t t e n t i o n ( hidden , e n c o d e r o u t p u t s . permute
(1 , 0 , 2) )
34
35 a = a t t n . unsqueeze (1)
36
37 e n c o d e r o u t p u t s = e n c o d e r o u t p u t s . permute (1 , 0 , 2)
38
39 w e i g h t e d = t o r c h . bmm( a , e n c o d e r o u t p u t s )
40 weighted = weighted . permute (1 , 0 , 2)
41
42 r n n i n p u t = t o r c h . c a t ( ( embedded , w e i g h t e d ) , dim = 2 )
43
46 embedded = embedded . s q u e e z e ( 0 )
47 output = output . squeeze (0)
48 weighted = weighted . squeeze (0)
49
50 o u t p u t = s e l f . o u t ( t o r c h . c a t ( ( o u t p u t , w e i g h t e d , embedded ) ,
dim = 1 ) )
51
1 c l a s s Seq2Seq ( nn . Module ) :
2 def init ( s e l f , encoder , decoder , device ) :
3 super () . init ()
4 s e l f . encoder = encoder
5 s e l f . decoder = decoder
6 s e l f . device = device
7
13 o u t p u t s = t o r c h . z e r o s ( ma x l e n , b a t c h s i z e , t r g v o c a b s i z e )
. to ( s e l f . device )
14
6 a t t n = D o t P r o d u c t A t t e n t i o n ( ) # Changed f o r e a c h t y p e o f model
7 e n c = E n c o d e r ( INPUT DIM ,
8 ENC EMB DIM ,
9 ENC HID DIM ,
10 DEC HID DIM ,
11 ENC DROPOUT ,
12 n u m l a y e r s =ENC NUM LAYERS ,
13 b i d i r e c t i o n a l =ENC BIDIRECTIONAL )
14 d e c = D e c o d e r ( OUTPUT DIM ,
15 DEC EMB DIM ,
16 ENC HID DIM ,
17 DEC HID DIM ,
18 DEC DROPOUT ,
19 attn ,
20 b i d i r e c t i o n a l i n p u t =ENC BIDIRECTIONAL )
21
0 20 40 60 80
(a)
4.0
3.5
3.0
2.5
0 20 40 60 80
(b)
Fig. 9.17: (a) Training and (b) validation losses for each attention model
The Bahdanau attention employs a fully connected layer to combine the concate-
nated outputs of the bidirectional layer, rather than duplicating the hidden state.
Incorporating this requires slight alterations to accommodate the changes in tensor
sizes.
1 c l a s s B a h d a n a u E n c o d e r ( nn . Module ) :
2 def init ( s e l f , i n p u t d i m , emb dim , e n c h i d d i m ,
dec hid dim , dropout ) :
3 super () . init ()
446 9 Attention and Memory Augmented Networks
200
150
100
50
0
0 20 40 60 80
(a)
Attention Comparison: Validation PPL
bahdanau_val_ppl
bilinear_val_ppl
175 cosine_val_ppl
dot_val_ppl
150 linear_val_ppl
125
100
75
50
25
0 20 40 60 80
(b)
Fig. 9.18: (a) Training and (b) validation PPL for each attention model
shorter
should
<eos>
<sos>
<eos>
<sos>
than
you
see
you
try
to
m
it
.
i
vous je
essayer plus
0.6 0.6
de petit
le que
0.4 0.4
voir vous
. 0.2 . 0.2
<eos> <eos>
0.0 0.0
(a) (b)
shorter
<eos>
<sos>
than
you
m
.
i
should
<eos>
<sos>
je 0.17
you
see
try
to
it
suis 0.16
tu
0.15
plus 0.15
devrais
0.14
jeune 0.14
essayer
0.13
que 0.13
de
0.12
le vous 0.12
0.11
voir vous
0.11
0.10
. .
0.09 0.10
<eos> <eos>
0.08 0.09
(a) (b)
15
Minor alterations are made to the decoder to handle the difference between the hid-
den size and the encoder output.
1 c l a s s B a h d a n a u A t t e n t i o n ( nn . Module ) :
2 def init ( self , enc hid dim , dec hid dim ) :
3 super () . init ()
4
shorter
<eos>
<sos>
than
you
should
<eos>
<sos>
.
i
you
see
1.0
try
to
it
.
1.0 je
tu
suis 0.8
devrais 0.8
plus
essayer
0.6
0.6 petit
de
que
le 0.4
0.4
vous
voir
0.2 . 0.2
.
<eos> <eos>
0.0 0.0
(a) (a)
shorter
<eos>
<sos>
than
you
should
<eos>
m
<sos>
.
i
you
see
try
to
it
je
.
0.7 0.5
tu
suis
0.6
devrais 0.4
plus
essayer 0.5
petit 0.3
de 0.4
que
le 0.3 0.2
vous
voir
0.2
. . 0.1
0.1
<eos> <eos>
0.0 0.0
(a) (b)
Fig. 9.22: Linear attention examples. Note how the model was unable to learn a
useful mapping from the attention mechanism while still being able to translate
some examples
8 s e l f . a t t n = nn . L i n e a r ( ( e n c h i d d i m ∗ 2 ) + d e c h i d d i m ,
dec hid dim )
9 s e l f . v = nn . P a r a m e t e r ( t o r c h . r a n d ( d e c h i d d i m ) )
10
17 e n c o d e r o u t p u t s = e n c o d e r o u t p u t s . permute (1 , 0 , 2)
18
21
22 v = s e l f . v . r e p e a t ( b a t c h s i z e , 1) . unsqueeze (1)
23
24 a t t e n t i o n = t o r c h . bmm( v , e n e r g y ) . s q u e e z e ( 1 )
25 r e t u r n F . s o f t m a x ( a t t e n t i o n , dim = 1 )
shorter
<eos>
<sos>
than
you
should
<eos>
m
<sos>
.
i
you
see
try
to
it
je
.
vous
suis 0.8
devriez 0.8
plus
essayer 0.6
0.6 petite
de
que
le 0.4
0.4
vous
voir
0.2 . 0.2
.
<eos> <eos>
0.0 0.0
(a) (b)
9.4.2.4 Results
The best performing model for each of the attention mechanisms was run on the
testing set to produce the following results in Table 9.2.
Table 9.2: Test results for attention models. Best results are shown in bold
Bilinear attention performed the best in this experiment. We can see from the
attention alignments in Fig. 9.21 that the attention output is strongly correlated with
the input. Moreover, the strength of attention is highly confident throughout the pre-
diction sequence, ever so slightly losing confidence towards the end of the sequence.
450 9 Attention and Memory Augmented Networks
To help the reader familiarize themselves with the attention and memory networks,
we will apply the concepts of this chapter to the question-answering task with the
bAbI dataset. The bAbI is a collection of 20 simple QA tasks with limited vocab-
ulary. For each task, there is a set of 1000 training and 1000 stories, test questions
and answers as well as an extended training set with 10,000 samples. Despite its
simplicity, bAbI effectively captures the complexities of memory and long-range
dependencies in question answering. For this case study, we will focus on tasks 1–
3, consisting of questions where up to three supporting facts from the stories provide
information to support the answer.
We will implement several architectures with Keras and TensorFlow for this case
study. Keras provides a useful example recurrent neural network architecture for
the question-answering task that will serve as our baseline. We will contrast per-
formance with several of the memory network-based architectures discussed in this
chapter, including a differentiable neural computer model from DeepMind. Rather
than providing full coverage of each architecture here, we direct the reader to the
notebooks accompanying this chapter for full implementation details.
Our first step is to download the bAbI dataset and to extract the training and test sets
for our analysis. We will focus on the extended dataset with 10,000 training samples
and 1000 test samples. Let’s first take a quick look at the samples for tasks QA1,
QA2, and QA3:
QA1 Story: Mary moved to the bathroom. John went to the hallway.
QA1 Query: Where is Mary?
QA1 Answer: bathroom
QA2 Story: Mary moved to the bathroom. Sandra journeyed to the bedroom. Mary got
the football there. John went to the kitchen. Mary went back to the kitchen.
Mary went back to the garden.
QA2 Query: Where is the football?
QA2 Answer: garden
QA3 Story: Mary moved to the bathroom. Sandra journeyed to the bedroom. Mary got
the football there. John went back to the bedroom. Mary journeyed to the
office. John journeyed to the office. John took the milk. Daniel went back
to the kitchen. John moved to the bedroom. Daniel went back to the hall-
way. Daniel took the apple. John left the milk there. John travelled to the
9.4 Case Study 451
kitchen. Sandra went back to the bathroom. Daniel journeyed to the bath-
room. John journeyed to the bathroom. Mary journeyed to the bathroom.
Sandra went back to the garden. Sandra went to the office. Daniel went to
the garden. Sandra went back to the hallway. Daniel journeyed to the office.
Mary dropped the football. John moved to the bedroom.
QA3 Query: Where was the football before the bathroom?
QA3 Answer: office
Analysis of the datasets shows the increasing complexity and long-range memory
that is required when progressing from task QA1 to QA3. The distribution of story
Task train stories test stories min(story size) max(story size) query size vocab size
QA1 10,000 1000 12 68 4 21
QA2 10,000 1000 12 552 5 35
QA3 10,000 1000 22 1875 8 36
lengths and question lengths (in terms of the number of tokens) can be seen in
Fig. 9.24.
The average length of the stories increases substantially from tasks QA1 to QA3,
which makes it significantly more difficult. Remember that for task QA3, there are
only three support facts and most of the story is considered “noise.” We will see how
well different architectures are able to learn to identify the relevant facts from this
noise.
We use the Keras LSTM architecture example to serve as our baseline. This archi-
tecture consists of the following:
1. The tokens of each story and question are mapped to embeddings (that are not
shared between them).
2. The stories and questions are encoded using separate LSTMs.
3. The encoded vectors for the story and question are concatenated.
4. These concatenated vectors are used as an input to a DNN whose output is a
softmax over the vocabulary.
5. The entire network is trained to minimize the error between the softmax output
and the answer.
The Keras model that implements this architecture is:
1 RNN = r e c u r r e n t . LSTM
2
3 s e n t e n c e = l a y e r s . I n p u t ( shape =( s t o r y m a x l e n , ) , dtype = ’ i n t 3 2 ’ )
4 e n c o d e d s e n t e n c e = l a y e r s . Embedding ( v o c a b s i z e ,
EMBED HIDDEN SIZE ) ( s e n t e n c e )
452 9 Attention and Memory Augmented Networks
1000
0
2000
1000
0
2000
1000
0
0 250 500 750 1000 1250 1500 1750
5000
0
10000
5000
0
10000
5000
0
4 5 6 7 8
Fig. 9.24: Distributions of story and question lengths in bAbI tasks 1–3
5 e n c o d e d s e n t e n c e = Dropout ( 0 . 3 ) ( e n c o d e d s e n t e n c e )
6 e n c o d e d s e n t e n c e = RNN( SENT HIDDEN SIZE ,
7 return sequences=False ) ( encoded sentence )
8
15 merged = l a y e r s . c o n c a t e n a t e ( [ e n c o d e d s e n t e n c e , e n c o d e d q u e s t i o n
])
16 merged = D r o p o u t ( 0 . 3 ) ( merged )
17 p r e d s = l a y e r s . Dense ( v o c a b s i z e , a c t i v a t i o n = ’ s o f t m a x ’ ) ( merged )
9.4 Case Study 453
18
19 model = Model ( [ s e n t e n c e , q u e s t i o n ] , p r e d s )
20 model . c o m p i l e ( o p t i m i z e r = ’ adam ’ , l o s s = ’ c a t e g o r i c a l c r o s s e n t r o p y ’
, m e t r i c s =[ ’ a c c u r a c y ’ ] )
We train this model using the extended bAbI training sets with 50-dim embeddings,
100-dim encodings, batch size of 32, and the Adam optimizer for 100 epochs. The
performance on tasks QA1, QA2, and QA3 is given in Table 9.3. As seen in the
results, the longer the stories, the worse the performance of the LSTM model due to
the increased “noise” in the data.
Memory networks offer the opportunity to store long-term information and thereby
improve performance, especially on longer sequences such as task QA3. Memory
networks are able to store supporting facts as memory vectors which are queried
and used for prediction. In the original form by Weston, the memory vectors are
learned via direct supervision with hard attention and supervision is required at each
layer of the network. This requires significant effort. To overcome this need, end-
to-end memory networks as proposed by Sukhbaatar use soft attention in place of
supervision that can be learned during training via backpropagation. This end-to-end
architecture takes the following steps:
1. Each story sentence and query are mapped to separate embedding representa-
tions.
2. The query embedding is compared with the embedding of each sentence in the
memory, and a softmax function is used to generate a probability distribution
analogous to a soft attention mechanism.
3. These probabilities are used to select the most relevant sentence in memory using
a separate set of sentence embeddings.
4. The resulting vector is concatenated with the query embedding and used as input
to an LSTM layer followed by a dense layer with a softmax output.
5. The entire network is trained to minimize the error between the softmax output
and the answer.
Note that this is termed a 1-hop or single-layered MemN2N, since we query the
memory only once. As described earlier, memory layers can be stacked to improve
454 9 Attention and Memory Augmented Networks
performance, especially where multiple facts are relevant and necessary to predict
the answer. The Keras implementation of the architecture is given below.
1 input sequence = Input ( ( story maxlen , ) )
2 i n p u t e n c o d e d m = Embedding ( i n p u t d i m = v o c a b s i z e ,
3 o u t p u t d i m =EMBED HIDDEN SIZE ) (
input sequence )
4 input encoded m = Dropout ( 0 . 3 ) ( input encoded m )
5
6 i n p u t e n c o d e d c = Embedding ( i n p u t d i m = v o c a b s i z e ,
7 output dim=query maxlen ) (
input sequence )
8 i n p u t e n c o d e d c = Dropout ( 0 . 3 ) ( i n p u t e n c o d e d c )
9
16 match = d o t ( [ i n p u t e n c o d e d m , q u e s t i o n e n c o d e d ] , a x e s = ( 2 , 2 ) )
17 match = A c t i v a t i o n ( ’ s o f t m a x ’ ) ( match )
18
19 r e s p o n s e = add ( [ match , i n p u t e n c o d e d c ] )
20 r e s p o n s e = Permute ( ( 2 , 1) ) ( r e s p o n s e )
21
We train this single-layered model using the extended bAbI training sets with 50-
dim embeddings, batch size of 32, and the adam optimizer for 100 epochs. The
performance on tasks QA1, QA2, and QA3 is given in Table 9.4. In comparison to
the baseline LSTM, the MemN2N model did significantly better for all three tasks,
and especially for QA1.
9.4 Case Study 455
As discussed earlier, dynamic memory networks take memory networks one step
further and encode memories using a GRU layer. An episodic memory layer is the
key to dynamic memory networks, with its attention mechanisms for feature gener-
ation and scoring. Episodic memory is composed of two nested GRUs, where the
inner GRU generates the episodes and the outer GRU generates the memory vector
from the sequence of episodes. DMNs follow the following steps:
1. The input story sentences and query are encoded using GRUs and passed to the
episodic memory module.
2. Episodes are generated by attending over these encodings to form a memory
such that sentence encodings with low attention scores are ignored.
3. Episodes along with previous memory states are used to update the episodic
memory.
4. The query and memory states serve as inputs to the GRU within the answer
module which is used to predict the output.
5. The entire network is trained to minimize the error between the GRU output and
answer.
A TensorFlow implementation of the episodic memory module for a dynamic mem-
ory network is provided below. Note that EpisodicMemoryModule depends on a
soft attention GRU implementation, which is included in the case study code.
1 c l a s s EpisodicMemoryModule ( L a y e r ) :
2
3 # a t t e n t i o n network
4 s e l f . l 1 = Dense ( u n i t s =emb dim , b a t c h s i z e = b a t c h s i z e ,
a c t i v a t i o n = ’ tanh ’ )
5 s e l f . l 2 = Dense ( u n i t s =1 , b a t c h s i z e = b a t c h s i z e ,
a c t i v a t i o n =None )
6
7 # Episode network
8 s e l f . episode GRU = SoftAttnGRU ( u n i t s = u n i t s ,
9 r e t u r n s e q u e n c e s =False ,
10 batch size=batch size )
11
12 # Memory g e n e r a t i n g n e t w o r k
13 s e l f . memory net = Dense ( u n i t s = u n i t s , a c t i v a t i o n = ’ r e l u ’ )
14
24 episode = K. c o n c a t e n a t e ( [ f a c t s , a t t e n t i o n s ] , a x i s
=2)
25 e p i s o d e = s e l f . episode GRU ( e p i s o d e )
26
29 r e t u r n K . c o n c a t e n a t e ( [ memory , q u e s t i o n ] , a x i s = 1 )
We train a DMN model using the extended bAbI training sets with 50-dim GloVe
embeddings, batch size of 50, 100 hidden units, 3 memory steps, and the adam
optimizer for just 20 epochs. The performance on tasks QA1, QA2, and QA3 is
given in Table 9.5. In comparison with earlier architectures, we can see that dynamic
memory networks perform better than MemN2N and LSTM networks for all three
tasks, reaching perfect prediction on task QA1.
For this case study, we will apply the TensorFlow-DNC implementation developed
by DeepMind to the bAbI extended datasets. The DNC module for this implemen-
tation is given by:
1 DNCState = c o l l e c t i o n s . n a m e d t u p l e ( ’ DNCState ’ , ( ’ a c c e s s o u t p u t ’
,
2 ’ access state ’ ,
3 ’
controller state ’))
4 c l a s s DNC( s n t . RNNCore ) :
5 # modules
6 s e l f . c o n t r o l l e r = s n t . LSTM( ∗ ∗ c o n t r o l l e r c o n f i g )
7 s e l f . a c c e s s = a c c e s s . MemoryAccess ( ∗ ∗ a c c e s s c o n f i g )
8
9 # output
10 prev access output = prev state . access output
11 prev access state = prev state . access state
12 prev controller state = prev state . controller state
13
We train a DNC model using the extended bAbI training sets with 50-dim GloVe
embeddings, hidden size of 256, memory size of 256 × 64, 4 read heads, 1 write
head, batch size of 1, and the RMSprop optimizer with gradient clipping for 20,000
iterations. The performance on tasks QA1, QA2, and QA3 is given in Table 9.6. It
may not be surprising to see that the DNC model outperforms all previous models,
given the increased complexity. The trade-off between accuracy and training time
458 9 Attention and Memory Augmented Networks
should be carefully weighed when choosing which architecture is most suitable for
the task. For simple tasks, a single LSTM implementation may be all that is required.
DNCs with their scalable memory are a better choice when complex knowledge is
required for task prediction.
7 d e f g e t c a n d i d a t e ( s e l f , s t a t e j , k e y j , i n p u t s , U, V, W,
U bias ) :
8 key V = t f . matmul ( k e y j , V)
9 s t a t e U = t f . matmul ( s t a t e j , U) + U b i a s
10 i n p u t s W = t f . matmul ( i n p u t s , W)
11 r e t u r n s e l f . a c t i v a t i o n ( s t a t e U + i n p u t s W + key V )
12
20 key j ,
21 inputs ,
22 U, V, W, U b i a s )
23 s t a t e j n e x t = s t a t e j + t f . expand dims ( g a t e j ,
−1) ∗ c a n d i d a t e j
24 s t a t e j n e x t n o r m = t f . norm ( t e n s o r = s t a t e j n e x t ,
25 ord= ’ e u c l i d e a n ’ ,
26 a x i s =−1,
27 keep dims =True )
28 s t a t e j n e x t n o r m = t f . where ( t f . g r e a t e r (
state j next norm , 0.0) ,
29 state j next norm ,
30 tf . ones like (
state j next norm ) )
31 state j next = state j next / state j next norm
32 n e x t s t a t e s . append ( s t a t e j n e x t )
33 s t a t e n e x t = t f . c o n c a t ( n e x t s t a t e s , a x i s =1)
34 return state next , state next
We train an EntNet using the extended bAbI training set with 100-dim embeddings,
20 blocks, batch size of 32, and the ADAM optimizer with gradient clipping for
200 epochs. The performance on tasks QA1, QA2, and QA3 is given in Table 9.7.
The performance of our implementation on bAbI tasks QA1, QA2, and QA3 ex-
ceeds all previous architectures. Note that with proper hyperparameter tuning, the
performance of EntNet and the previous architectures can be improved on the bAbI
tasks.
The readers and practitioners can consider extending the case study to the following
problems in order to expand their knowledge:
1. Memory and complexity can be limited when using the same embedding matrix
for both the encoder and decoder. What would need to change to address this
problem?
2. Tune and increase the number of epochs for the baseline LSTM model during
training. Does adding dropout help?
460 9 Attention and Memory Augmented Networks
3. Add a second and third hop to the end-to-end memory network and see if perfor-
mance improves on bAbI tasks QA2 and QA3.
4. How does restricting the size of the memory representation affect performance?
5. Is there a significant effect by using a different similarity scoring function instead
of the softmax within the memory controller of a MemN2N network?
6. Explore the architectures in this case study on bAbI tasks 3-20. Does the simple
baseline LSTM outperform on certain tasks?
References
[BCB14b] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. “Neural Ma-
chine Translation by Jointly Learning to Align and Translate”. In:
CoRR abs/1409.0473 (2014).
[Bah+16b] Dzmitry Bahdanau et al. “End-to-end attention-based large vocabu-
lary speech recognition”. In: 2016 IEEE International Conference on
Acoustics, Speech and Signal Processing ICASSP 2016, Shanghai,
China, March 20–25, 2016. 2016, pp. 4945–4949.
[Cha+16a] William Chan et al. “Listen, attend and spell: A neural network for
large vocabulary conversational speech recognition”. In: 2016 IEEE
International Conference on Acoustics, Speech and Signal Processing
ICASSP 2016, Shanghai, China, March 20–25, 2016. 2016, pp. 4960–
4964.
[Cho+15b] Jan Chorowski et al. “Attention-Based Models for Speech Recogni-
tion”. In: Advances in Neural Information Processing Systems 28: An-
nual Conference on Neural Information Processing Systems 2015, De-
cember 7–12, 2015, Montreal, Quebec, Canada. 2015, pp. 577–585.
[Dan+17] Michal Daniluk et al. “Frustratingly Short Attention Spans in Neural
Language Modeling”. In: CoRR abs/1702.04521 (2017).
[DGS92] Sreerupa Das, C. Lee Giles, and Guo-Zheng Sun. “Using Prior Knowl-
edge in a {NNPDA} to Learn Context-Free Languages”. In: Advances
in Neural Information Processing Systems 5, [NIPS Conference, Den-
ver, Colorado, USA, November 30 - December 3, 1992]. 1992, pp. 65–
72.
[Den+12] M. Denil et al. “Learning where to Attend with Deep Architectures for
Image Tracking”. In: Neural Computation (2012).
[GWD14b] Alex Graves, Greg Wayne, and Ivo Danihelka. “Neural Turing Ma-
chines”. In: CoRR abs/1410.5401 (2014).
[Gra+16] Alex Graves et al. “Hybrid computing using a neural network with
dynamic external memory”. In: Nature 538.7626 (Oct. 2016), pp. 471–
476.
[Gre+15] Edward Grefenstette et al. “Learning to Transduce with Unbounded
Memory”. In: Advances in Neural Information Processing Systems 28:
Annual Conference on Neural Information Processing Systems 2015,
References 461
10.1 Introduction
Fig. 10.1: Traditional machine learning system on two different sources and target
10.2 Transfer Learning: Definition, Scenarios, and Categorization 465
10.2.1 Definition
In order to define transfer learning precisely, we will first define a couple of concepts
as given by Pan and Yang, i.e., domains and tasks [PY10]. A domain D = (X, P(X))
is defined in terms of (a) the feature space X and (b) the marginal probability dis-
tribution P(X), where X represents the training data samples X = x1 , x2 . . . xn ∈ X.
For example, in the task of sentiment analysis with binary classification, the X cor-
responds to a bag-of-words representation and xi corresponds to the ith term in the
corpus. Thus, when either the feature spaces or the marginal probability distribution
are different for two systems, we say that the domains do not match.
A task T = (Y, f (·)) is defined in terms of (a) a label space Y and (b) an objective
prediction function f (·) that is not directly observed but learned from the input and
label pairs (xi , yi ). The label space consists of a set of all actual labels, for example,
true and false for binary classification. The objective prediction function f (·) is
used to predict the label given the data and can be interpreted in probabilistic view
as f (·) ≈ p(y|x).
Given a source domain DS , source task TS , target domain DT , and target task
TT , transfer learning can be defined as the process of learning the target predictive
function fT (·) = P(YT |XT ) in the target domain DT using the knowledge from the
source domain DS and the source task TS , such that DS = DT or TS = TT .
466 10 Transfer Learning: Scenarios, Self-Taught Learning, and Multitask Learning
Based on the different components of domain and task for both source and target,
there are four different transfer learning scenarios that are listed below:
Based on “how to transfer” and “what to transfer” between the source and target,
transfer learning can be further categorized into many different types of which many
have become an independent field for research and applications. In this section,
we will not cover many traditional machine learning based categorizations already
defined in the survey by Pan and Yang [PY10]. Instead, we will cover only those
categories that have been explored or made an impact in the deep learning area.
Based on the label availability and task similarities between the source and target,
there can be various sub-categories of transfer learning, as shown in Fig. 10.3.
When the source labels are unavailable, but a large volume of source data exists
and few to large numbers of target data exist, then the category of learning is known
as self-taught learning. Many real-world applications in speech and text, where the
cost or the effort of labeling poses constraints, and the large volume of data can be
used to learn and transfer to specific tasks with labels, this technique has been very
successful. Employing some form of unsupervised learning on the source to capture
features that can help transfer knowledge to the target is the core assumption made
in these learning systems.
When the goal is not only to do well on the target tasks but somehow learn jointly
and do well in both source and target, where the tasks are slightly different, the
form of transfer learning is called multitask learning. The core assumption made is
10.3 Self-Taught Learning 467
that sharing information among related tasks, which should have some similarities,
improves the overall generalization.
Related to multitask learning, where the tasks between source and target differ,
domain adaptation is a form of learning where the domains (i.e., either the feature
space or marginal distribution in data) are different between source and the target.
The core principle is to learn a domain-invariant representation from the source that
can be transferred to the target with a different domain in an effective manner.
In domain adaptation, the domains differ and small to large labeled data is avail-
able in source. Domain adaptation can be zero-shot, one-shot, and few-shot learn-
ing, based on the available number of labeled data (0, 1, n).
Fig. 10.3: Transfer learning categories based on labeled data, tasks, and domains for
source and target
Self-taught learning, as shown in Fig. 10.4, consists of two distinct steps: (a) learn-
ing features in an unsupervised manner from the unlabeled source dataset and (b)
tuning these learned features with a classifier on the target dataset which has labels.
468 10 Transfer Learning: Scenarios, Self-Taught Learning, and Multitask Learning
Fig. 10.4: Self-taught learning using pre-training and fine-tuning steps. (a) Using
unlabeled source dataset to learn features. (b) Using the labeled target data to fine-
tune the features with a classifier
10.3.1 Techniques
In this section we will summarize various approaches and then discuss specific al-
gorithms or techniques that have been successful in NLP and speech.
The input to Algorithm 1 is the source unlabeled dataset of size n; the (S) in
the subscript denotes the source. The first part of learning proceeds in an unsuper-
vised way from the source, as shown in Algorithm 1. This has many similarities
to feature or dimensionality reduction and manifold learning in traditional machine
learning. This process generally employs linear and non-linear techniques to find a
latent representation of the input that has a smaller dimension than the input. In deep
learning, the train in the algorithm above corresponds to many unsupervised tech-
niques such as PCA or ICA layers, restricted Boltzmann machines, autoencoders,
10.3 Self-Taught Learning 469
Algorithm 2: SupervisedFineTuning
Data: Training Dataset (x1(T ) , y2 ), (x2(T ) , y2 ), ..(xm(T ) , yn ) such that xi(T ) ∈ Rd and
yi ∈ {+1, −1}, Trained Layers h1 , h2 , · · · , hL , Training layer start k
Result: Weight matrix Wl ∈ Rd and bl ∈ R for each layer l
begin
appendClassifierLayer(hL+1 )
for l = k to L do
Wl , bl = train((x1(T ) , y1 ), (x2(T ) , y2 ), ..(xm(T ) , yn ))
return Wl , bl for each layer l
10.3.2 Theory
In their seminal work, Erhan et al. give interesting theoretical and empirical insights
around unsupervised pre-training and fine-tuning [Erh+10]. They use various archi-
tectures such as feed-forward neural networks, deep belief networks, and stacked
denoising autoencoders on different datasets to empirically verify the various theo-
retical conclusions in a step-by-step, controlled manner.
They show that pre-training not only gives a good starting condition but captures
complex dependencies among the parameters as well. The research also shows that
unsupervised pre-training can be a form of regularization that guides the weights
towards a better basin of attraction of minima. The regularization obtained from
the pre-training process influences the starting point in supervised learning, and the
effect does not disappear with more data in comparison with standard regulariza-
470 10 Transfer Learning: Scenarios, Self-Taught Learning, and Multitask Learning
tion techniques such as L1 /L2 . Unsurprisingly, the research concludes that in small
training data settings, unsupervised pre-training has many advantages. It also shows
that, in some cases, the order of training examples impacts the results, but pre-
training reduces the variance even in such cases. The experiments and results that
show unsupervised pre-training as a general variance reducing technique and even
optimization technique for better training are insightful.
Using unsupervised techniques for word embeddings from a large corpus of data
and employing it for various supervised tasks has been the most basic application in
NLP. Since this was discussed at length in Chap. 5, we will focus more on other
NLP tasks. Dai and Le show that unsupervised feature learning using sequence
autoencoders or language model-based systems and then using supervised train-
ing achieves great results in text classification tasks on various datasets, such as
IMDB, DBpedia, and 20 Newsgroup [DL15]. The sequence autoencoder uses an
LSTM encoder–decoder to capture the dependencies in an unsupervised manner.
The weights from the LSTM are used to initialize the LSTM with softmax classi-
fier in a supervised setting. The unsupervised autoencoder training shows superior
results across all datasets, and the generality of the technique gives it an edge for all
sequence-to-sequence problems.
Ramachandran et al. show that the LSTM encoder pre-trained for language
modeling can be used very effectively without fine-tuning in sentiment classifica-
tion [RLL17]. Deing et al. show that TopicRNN, an architecture using RNN for
local syntactic dependencies and topic modeling for global semantic latent repre-
sentations, can be a very effective feature extractor [Die+16]. TopicRNN achieves
nearly state-of-the-art results on sentiment classification task. Turian et al. show that
learning features in an unsupervised manner from multiple embeddings and apply-
ing it to various supervised NLP tasks, such as chunking and NER, can give nearly
state-of-the-art results [TRB10].
Very early, Dahl et al. showed in their research that unsupervised pre-training gives
a great initialization for the weights, and using labeled fine-tuning on deep belief
networks further improves results in automatic speech recognition task [Dah+12].
Hinton et al. show unsupervised pre-training for learning layer by layer in RBMs
and then fine-tuning with labeled examples not only reduces overfitting but reduces
time to learn on labeled examples [Hin+12]. Lee et al. show that unsupervised fea-
ture learning done on large dataset can learn phonemes that can help various audio
classification tasks using deep convolutional networks [Lee+09].
10.4 Multitask Learning 471
10.4.1 Techniques
The two general ways of handling multitask learning in deep learning are through
hard or soft parameter sharing as shown in Fig. 10.5. Hard parameter sharing is
one of the oldest techniques in NNs with a single model, where the hidden layers
share the common weights, and task-specific weights are learned at the output lay-
ers [Car93]. The most important benefit of hard parameter sharing is the prevention
of overfitting by enforcing more generalization across tasks. Soft parameter sharing,
on the other hand, has individual models with separate parameters per tasks, and a
constraint is put to make the parameters across tasks more similar. Regularization
techniques are often used in soft parameter sharing for enforcing the constraints.
In the next section, we will go through selected deep learning networks that have
proven useful for multitask learning.
One of the earliest deep learning networks for multitask learning was intro-
duced by Long and Wang, and was known as the multilinear relationship network
(MRN) [LW15]. MRN showed state-of-the-art performance in different tasks in im-
age recognition. The MRN, as shown in Fig. 10.6, is a modification of the AlexNet
architecture that was discussed in Chap. 6. The first few layers are convolutional,
and a fully connected layer learns the transferable features, while the rest of the
fully connected layers closer to the output learn task-specific features. If there are
T tasks with training data Xt , Yt t=1
T
, where Xt = xt1 , · · · , xtN and Yt = yt1 , · · · , ytN , Nt
number of training examples and labels of the tth task with D-dimensional feature
space and C-cardinality label space, network parameters of t task in the lth layer are
472 10 Transfer Learning: Scenarios, Self-Taught Learning, and Multitask Learning
Fig. 10.5: Two generic methods of multitask learning. (a) Hard parameter sharing
in the hidden layers. (b) Soft parameter sharing across the hidden layers for various
tasks
given by Wt,l ∈ RD1 ×D2 , where Dl1 and Dl2 are the dimensions of matrix Wt,l and
l l
parameter tensor Wl = [W1,l ; · · · ; WT,l ] ∈ RD1 ×D2 ×T . The fully connected layers
l l
where J(·) is the cross-entropy loss function and ft (xtn ) is the conditional probability
that the network assigns for the data point xtn to the label ytn . MRN has tensor normal
priors over the parameter tensors in the fully connected task-specific layers similar
to Bayesian models that acted as regularization on task related learning.
The maximum a posteriori (MAP) estimation of network parameters W =
Wl : l ∈ L for task-specific layers L = f c7, f c8 given the training data is:
with assumptions made that the prior P(Wl ) and the parameter tensors Wl for each
layer are independent of the other layers.
The maximum likelihood estimation (MLE) part P(Y|X, W) is modeled to learn
the transferable features in the lower layers, and all the parameters for the layers
(conv1 − f c6) are shared. The task-specific layers ( f c7, f c8) are not shared to avoid
negative transfer. The prior part p(W) is defined as the tensor normal distribution
and is given as:
p(W) = TNDl ×Dl ×T (O, Σl1 , Σl2 , Σl3 ) (10.4)
1 2
10.4 Multitask Learning 473
where Σl1 , Σl2 , and Σl3 are the modes of covariance matrices. In the tensor prior, the
row covariance matrix Σl1 ∈ RD1 ×D1 learns the relationships between the features,
l l
the column covariance matrix Σl2 ∈ RD2 ×D2 learns the relationship between classes,
l l
and the covariance matrix Σl3 ∈ RT ×T learns the relationships between tasks in the
lth layer parameters Wl = W1,l ; · · · ; WT,l . The empirical error given in Eq. 10.1 is
integrated with the prior given in Eq. 10.4 into the MAP estimation given in Eq. 10.3
and following the process of taking a negative logarithm, the equation to optimize is:
T Nt
min
ft |t=1
T ,Σl |K
∑ ∑ J( ft (xtn ), ytn )
k k=1 t=1 n=1
K
1 Dl
+ ∑
2 l∈L
vec(W l T
) ( Σ l
1:K ) −1
vec(W l
) − ∑ l ln(|Σl
k |) (10.5)
k=1 Dk
Fig. 10.6: A multilinear relationship network, in which the first few layers learn the
shared features, and the final layers learn task-specific features with tensor normal
priors
things such that t ≤ T at the layer l and then recursively move in a top-down
manner to the next layer l − 1 and so on. The grouping of the tasks is done
by associating a concept of “affinity” which is the probability of concurrently
observing simple or difficult examples from the training data for the pair of
tasks.
3. Final Model Training: The last step is to train the final model after the thin
model initialization and the recursive widening process.
Fig. 10.7: An iterative process showing how the network is widened at a layer on a
specific iteration to group the tasks
Fig. 10.8: A cross-stitch network trying to learn a latent representation that is useful
for two tasks
NLP tasks generally can be considered to be in the pipeline of hierarchy, where one
task can be useful and used as input to the next task. Søgaard and Golberg show that
supervised multitasking at different layers using bidirectional RNN architecture so
that the low-level tasks feed into high-level tasks can achieve great results [SG16].
Hashimoto et al. extend the idea by creating a single end-to-end deep learning net-
work, where the network has the growing depth to accomplish linguistic hierarchies
from syntactic and semantic representations, as shown in Fig. 10.10 [Has+16]. It
has been shown that a single end-to-end network with this architecture can achieve
state-of-the-art results in different tasks such as chunking, dependency parsing, se-
mantic relatedness, and textual entailment.
A given sentence s of length l has wt words. For each word, there is skip-gram
word embedding and character embedding. The word representation xi is done by
concatenating both word and n-gram character embeddings which are learned us-
ing skip-gram with negative sampling for words. The character n-grams are used to
give morphological features for the tasks. The first task is of POS tagging and is
performed using bidirectional LSTM with embedded inputs and softmax for clas-
sifying the tags. The POS tags are learnable embeddings which is used in the next
chunking layer. The label embedding for POS tagging (and many other tasks) is
given by:
10.4 Multitask Learning 477
C
ytpos = ∑ p(yt1 = j|ht1 )l( j) (10.7)
j=1
where C is the number of POS tags, p() is the probability that jth POS tag is as-
signed to the wth token, and l is the label embedding for jth POS tag. The second
task is chunking, which uses bidirectional LSTM and takes the hidden state from
POS bidirectional LSTM, the hidden state of its LSTM, embedded token and la-
bel embedding from POS tagging. The third task is dependency parsing with inputs
from hidden states from chunking layer, previous hidden state from dependency
parsing, embedded token and label embeddings of POS layer and the chunking layer.
POS tagging layer and the chunking layer with hidden states are useful in generating
low-level features that are useful for many tasks as known from traditional feature
engineering in NLP. The fourth task is dependency parsing, again using bidirectional
LSTM with inputs as hidden LSTM states, embedded tokens, and label embedding
from POS tagging and chunking layer. The next two tasks are semantically related
as compared to syntactic tasks in the previous layers. The semantic relatedness task
is to compare two sentences and give a real-valued output for a measure of their
relatedness. The sentence level representation is obtained via max pooling of the
hidden states of the LSTM and is given by:
hrelat
s = max(hrelat
1 , hrelat
2 , · · · , hrelat
L ) (10.8)
The relatedness for two sentences (s, s ) is given by:
d1 (s, s ) = [|hrelat
s − hrelat
s
|; hrelat
s hrelat
s
] (10.9)
478 10 Transfer Learning: Scenarios, Self-Taught Learning, and Multitask Learning
The values of d1 (s, s ) are given to softmax layer with maxout hidden layer to
give a relatedness score.
The last task is that of textual entailment, which again takes two sentences and
gives one of the categories of entailment, contradiction, or neutrality. The label em-
beddings from the relatedness task along with distance measure similar to Eq. 10.9
in the relatedness derived from LSTM layer feed into a softmax classifier for classi-
fication.
When a network is sequentially trained for one task and then trained on another
task, it generally “forgets” or has bad performance on the first. This phenomenon
is called catastrophic interference or catastrophic forgetting. The training for
each layer is similar with a loss function that takes into account (a) a measure for
classification loss for the layer using predictions and label, (b) L2-norm of its weight
vectors, and (c) a regularization term for parameters of previous tasks if they are
inputs. Joint learning gives the framework robustness from catastrophic interference
according to the authors. An example is for chunking layer, with inputs from POS
tagging given by weights and bias θPOS and one after POS layer with current epoch
given by θPOS , weights of chunking layer WCHK , and probability p(yCHK t = α |hCHK
t )
of assigning correct label α to wt in the sentence is:
2
J2 (θCHK ) = − ∑ ∑ log p(yCHK
t = α |hCHK
t ) + λ WCHK 2 + δ θPOS − θPOS
s t
(10.10)
Ruder et al. recently proposed a general deep learning architecture, known as sluice
networks, that combines concepts from many previous types of research such as
hard parameter sharing, cross-stitch networks, block-sparse regularization, and NLP
linguistic hierarchical multitask learning [Rud17]. The sluice network for the main
task A and an auxiliary task B consists of the shared input layer, three hidden layers
per task, and two task-specific output layers as shown in Fig. 10.11. Each hidden
layer for the task is an RNN divided into two subspaces, for example, task A and
layer 1 has GA,1,1 and GA,1,2 , which allows them to learn task-specific and shared
representations effectively. The output of hidden layers flows through α parameters
to the new layer, which carries out linear combinations of the inputs to weigh the
importance of sharing and task-specific learning. By making the subspaces each
have their weights and controlling how they share, sluice networks have an adaptive
way of learning in multitask settings only things that are useful. The final recurrent
hidden layers pass the information to β parameters which try to combine all the
things the layers have learned. Ruder et al. empirically show how main tasks, such
as NER and SRL, can benefit from auxiliary tasks such as POS and improve on
errors by a significant value.
Ruder et al. cast the entire learning as a matrix regularization problem.
If there are M tasks that are loosely related with M non-overlapping datasets
10.4 Multitask Learning 479
Fig. 10.11: Sluice networks for multitask learning across loosely connected tasks
The loss functions Li are cross-entropy loss functions, and the weights λi deter-
mine the importance of the task i during the training. If Gm,k,1 and Gm,k,2 are the two
subspaces for each layer, the inductive bias is given by the orthogonality constraints:
M K
∑ ∑ Gm,k,1 T Gm,k,2 F
2
Ω= (10.12)
m=1 k=1
The matrix regularization is carried out by updating the α parameters with sim-
ilarity to Misra et al.’s cross-stitch units [Mis+16]. For the two tasks (A, B) and k
layers for one subspace, the extension to cross-stitch linear combination looks like:
⎡ ⎤ ⎡ ⎤⎡ ⎤
h̃A1,k αA1 A1 · · · αA1 B2 hA1,k
⎢ .. ⎥ ⎢ .. .. .. ⎥ ⎢ .. ⎥
⎣ . ⎦=⎣ . . . ⎦⎣ . ⎦ (10.13)
h̃B1,k α A1 B2 · · · αB2 B2 hB1,k
where hA1,k is the output of first subspace for task A in the layer k, and h̃A1,k is
the linear combination of that first subspace and task A. The input to layer k + 1
is the concatenation of the two, given as hA,k = [h̃A1,k , h̃A2,k ]. The hierarchical rela-
tionship between the low-level tasks and the high-level tasks is learned using the
skip-connections between the layers with the β parameters. This acts as a mixture
model and can be written as:
480 10 Transfer Learning: Scenarios, Self-Taught Learning, and Multitask Learning
⎡ ⎤
βA,1
h̃TA = ⎣ · · · ⎦ hA,1 T · · · hA,k T (10.14)
βA,k
where hA,k is the output of layer k for task A, and h̃A,t is the linear combination of
all layer outputs that gets fed to a softmax classifier.
10.4.2 Theory
Caruana in his early research on MTL and then Ruder in his work have summarized
different reasons why and when multitask learning works and is effective [Car97,
Rud17].
1. Implicit Data Augmentation—When the constraint is limited data per task,
by jointly learning different tasks which are similar, the total training data size
increases. As learning theory suggests, the more the training data, the better the
model quality.
2. Attention Focusing—When the constraint is noisy data per task, by jointly
learning different tasks, focus on relevant features that are useful across tasks to
get more attention. This joint learning, in general, helps as an implicit feature
selection mechanism.
3. Eavesdropping—When the training data is limited, the features that may be
needed for a particular task may not be in the data. By having multiple datasets
for multiple tasks, features can eavesdrop, i.e., the features learned for a separate
task can be used for the task in question and help in the generalization of that
specific task.
4. Representation Bias—Multitask learning enforces a representation that gener-
alizes across the tasks and thus forces better generalization.
5. Regularization—Multitask learning is also considered as a regularization tech-
nique through inductive bias, which theoretically and empirically is known to
improve model quality.
In his work, Rei shows that using language modeling as an auxiliary task along with
sequence labeling tasks such as POS tagging, chunking, and named entity detection
for the main task can improve the results significantly over the benchmarks [Rei17].
Fang and Cohn illustrate the advantage of cross-lingual multitask joint learning for
POS tagging in a low-resource language [FC17]. Yang et al. show that a deep hi-
erarchical neural network with cross-lingual multitask learning can achieve state-
of-the-art results in various sequence tagging tasks, such as NER, POS tagging,
and chunking [YSC16]. Duong et al. use cross-lingual multitask learning to achieve
10.4 Multitask Learning 481
Watanabe et al. highlight how multiple tasks associated with speech recognition can
be performed in a hybrid end-to-end deep learning framework [Wat+17]. This archi-
tecture combines two main architectures, CTC loss and attention-based sequence-
to-sequence, to give results that are comparable with previous HMM-deep learning
based methods. Watanabe et al. again highlight how multiple tasks such as auto-
matic speech recognition (ASR) and language identification/classification across
ten languages can be performed at the same time using end-to-end deep learning
with multitask learning [WHH17]. Watanabe et al. highlight how multitask learning
on ASR and speaker identification can improve the total performance significantly
compared to separately trained models [Wat+18].
In this case study, we explore how multitask learning can be applied to some com-
mon NLP tasks such as POS tagging, chunking, and named entity recognition. The
overall performance depends on many choices such as sequence-to-sequence archi-
tecture, embeddings, and sharing techniques.
We will try to answer whether the low-level tasks such as POS tagging can bene-
fit the high-level tasks such as chunking? What would be the impact of joint learning
with closely related tasks and loosely related tasks? Is there an impact of connec-
tivity and sharing on learning? Is there a negative transfer and how that impacts the
learning? Do the neural architecture and embedding choices impact the multitask
learning? We will use CoNLL-2003 English dataset which has annotations at token
levels for each of the tasks in our experiments. CoNLL-2003 dataset already has the
standard splits of train, validation, and test. We will use accuracy on the test set as
our performance metric for the case study.
• Exploratory data analysis
• Multitask learning experiments and analysis
We will describe the main open source tools and libraries we have used below for
our case study:
• PyTorch: We use http://github.com/pytorch/pytorch as our deep learning toolkit
in this case study.
• GloVe: We use https://nlp.stanford.edu/projects/glove/ for our pre-trained em-
beddings in the experiments. https://github.com/SeanNaren/nlp multi task
learning pytorch/ for multitask learning experiments.
10.5 Case Study 483
The raw data for training, validation, and testing have a columnar format with
annotations for each token as given in Table 10.1.
Basic analysis for total articles, sentences, and tokens for each dataset is given
in Table 10.2. The tags follow the “inside–outside–beginning” (IOB) scheme for
chunking and NER.
NER categories and number of tokens for each are given in Table 10.3.
We base our model on Søgaard and Golberg’s research using bidirectional RNNs
for encoder and decoder networks in “joint learning” mode. We explore “joint
learning” in two different configurations: (a) shared layers between all the tasks that
are connected to three different softmax layers (POS, chunk, and NER) and (b) each
RNN is in different layer and the hidden layer of the lower layer flows into the next
higher layer as shown in Fig. 10.12.
Fig. 10.12: Bidirectional LSTM configured for multitask learning with cascading
layered architecture
We highlight the code below for the JointModel class where all the configu-
rations of (a) individual learning, (b) joint with shared layers, and (c) joint with
cascading are defined.
1 # i n i t i a l i z a t i o n of the graph
2 def forward ( self , input , ∗ hidden ) :
3 i f s e l f . t r a i n m o d e == ’ J o i n t ’ :
4 # when t h e number o f l a y e r s i s same , h i d d e n l a y e r s
are shared
5 # and c o n n e c t e d t o d i f f e r e n t o u t p u t s
6 i f s e l f . n l a y e r s 1 == s e l f . n l a y e r s 2 == s e l f . n l a y e r s 3
:
7 l o g i t s , shared hidden = s e l f . rnn ( input , hidden
[0])
8 outputs pos = self . linear1 ( logits )
9 outputs chunk = s e l f . linear2 ( l o g i t s )
10.5 Case Study 485
Chunk Acc
Models POS Acc %
%
POS single task 86.33 –
Chunk single task – 84.69
MTL joint shared 83.91 85.23
MTL joint separate 86.88 85.78
Chunk Acc
Models POS Acc %
%
POS single task 86.56 –
Chunk single task – 86.88
MTL joint shared 84.53 88.20
MTL joint separate 87.34 87.11
Chunk Acc
Models POS Acc %
%
POS single task 90.55 –
Chunk single task – 88.05
MTL joint shared 89.84 88.12
MTL joint separate 90.86 87.73
Chunk Acc
Models POS Acc %
%
POS single task 92.42 –
Chunk single task – 89.69
MTL joint shared 91.72 89.53
MTL joint separate 92.34 89.61
• Tables 10.4 and 10.5 show that joint multitask learning with separate
LSTM layers as compared to shared layer between both improve the per-
formance for both combinations, i.e., POS tagging and chunking and POS
tagging and NER.
• Table 10.6 shows that when all three tasks are combined joint MTL with
shared as well as with separate layers, the results deteriorate except for
chunking. These results are in contrast with Tables 10.4 and 10.5 and show
that when there is a mix of tasks which are not all related strongly, the
“negative transfer” comes into the picture.
• The experiment results in Table 10.7 use bidirectional LSTM and show
similar performance as LSTM models in Table 10.4, indicating that just
by adding architectural complexity by itself does not change the multitask
behavior at least in this case.
• Introducing pre-trained embeddings using GloVe vectors shows a huge in-
crease of around 4% in performance of single tasks for both POS tagging
and chunking as shown in Table 10.8. The marginal improvements in MTL
are similar to without GloVe.
• Experiment 6 as given in Table 10.9 shows that when both bidirectional
LSTM and pre-trained GloVe vectors are used, not only the individual tasks
improve but the behavior of multitask learning is different as that of the ba-
sic first experiment in Table 10.4. The shared and separate layers both show
worse performance than single tasks here. Somehow, better the individual
task performance is, the impact of multitask learning diminishes.
• Experiment 7, where we combine bidirectional LSTM and pre-trained
GloVe for POS tagging and NER results as given in Table 10.10, shows
very different results than experiment 2 as given in Table 10.5. The joint
multitask learning using shared shows performance boost for both tasks
which has not been seen in the previous experiments.
• Experiment 8, results given in Table 10.11, where we combine all tasks
with bidirectional LSTM and GloVe, shows different performance as com-
pared to the experiment 3 as given in Table 10.6. The POS tagging and
chunking show improvements with shared but NER shows deterioration
in performance. Except for chunking, all others show worse performance
with separate layers as compared to experiment 3.
References 489
Some of the extensions and extra ideas for researchers to try are given below:
1. What is the impact of using different pre-trained embeddings such as word2vec?
2. What is the impact of adding more layers to RNN for both shared and separate?
Does that change the MTL behavior?
3. We tried MTL with LSTM but not with GRU or even base RNN, is there a
significant difference in the performance of MTL with the choice of recurrent
networks?
4. What is the impact of hyperparameters like the number of hidden units, batch
size, and epochs on the MTL?
5. If we add more tasks such as language models, sentiment classification, seman-
tic role labeling, to name a few in the mix, what would be the performance
impact on MTL?
6. Use the same dataset with other research like cross-stitch networks, sluice net-
works, and others to get comparative analysis across the methods.
References
[JZ07] Jing Jiang and Chengxiang Zhai. “Instance weighting for domain adap-
tation in NLP”. In: In ACL 2007. 2007, pp. 264–271.
[Joh+16] Melvin Johnson et al. “Google’s Multilingual Neural Machine
Translation System: Enabling Zero-Shot Translation”. In: CoRR
abs/1611.04558 (2016).
[KC17] Arzoo Katiyar and Claire Cardie. “Going out on a limb: Joint Extrac-
tion of Entity Mentions and Relations without Dependency Trees”. In:
Proceedings of the 55th Annual Meeting of the Association for Compu-
tational Linguistics. 2017, pp. 917–928.
[Lee+09] Honglak Lee et al. “Unsupervised feature learning for audio classifica-
tion using convolutional deep belief networks”. In: Advances in Neural
Information Processing Systems 22: 23rd Annual Conference on Neu-
ral Information Processing Systems. 2009, pp. 1096–1104.
[Liu+15] Xiaodong Liu et al. “Representation Learning Using Multi-Task Deep
Neural Networks for Semantic Classification and Information Re-
trieval”. In: NAACL HLT 2015, The 2015 Conference of the North
American Chapter of the Association for Computational Linguistics.
[LW15] Mingsheng Long and Jianmin Wang. “Learning Multiple Tasks with
Deep Relationship Networks”. In: CoRR abs/1506.02117 (2015).
[Lu+16] Yongxi Lu et al. “Fully-adaptive Feature Sharing in Multi-Task Net-
works with Applications in Person Attribute Classification”. In: CoRR
abs/1611.05377 (2016).
[Luo+17] Bingfeng Luo et al. “Learning to Predict Charges for Criminal Cases
with Legal Basis”. In: Proceedings of the 2017 Conference on Empir-
ical Methods in Natural Language Processing, EMNLP 2017, 2017,
pp. 2727–2736.
[Luo+15] Minh-Thang Luong et al. “Multi-task Sequence to Sequence Learn-
ing”. In: CoRR abs/1511.06114 (2015).
[Mis+16] Ishan Misra et al. “Cross-stitch Networks for Multi-task Learning”. In:
CoRR abs/1604.03539 (2016).
[NC17] Jan Niehues and Eunah Cho. “Exploiting Linguistic Resources for Neu-
ral Machine Translation Using Multi-task Learning”. In: Proceedings
of the Second Conference on Machine Translation. Association for
Computational Linguistics, 2017, pp. 80–89.
[PY10] Sinno Jialin Pan and Qiang Yang. “A Survey on Transfer Learning”. In:
IEEE Trans. on Knowl. and Data Eng. 22.10 (Oct. 2010), pp. 1345–
1359.
[Pan+08] Sinno Jialin Pan et al. “Transfer Learning for WiFi-based Indoor Lo-
calization”. In: 2008.
[Rai+07] Rajat Raina et al. “Self-taught Learning: Transfer Learning from Unla-
beled Data”. In: Proceedings of the 24th International Conference on
Machine Learning. ICML ’07. 2007, pp. 759–766.
492 10 Transfer Learning: Scenarios, Self-Taught Learning, and Multitask Learning
[YM17] Bishan Yang and Tom M. Mitchell. “A Joint Sequential and Rela-
tional Model for Frame-Semantic Parsing”. In: Proceedings of the 2017
Conference on Empirical Methods in Natural Language Processing
EMNLP 2017. 2017, pp. 1247–1256.
[YSC16] Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. “Multi-
Task Cross-Lingual Sequence Tagging from Scratch”. In: CoRR
abs/1603.06270 (2016).
[ZK16] Barret Zoph and Kevin Knight. “Multi-Source Neural Translation”. In:
CoRR abs/1601.00710 (2016).
Chapter 11
Transfer Learning: Domain Adaptation
11.1 Introduction
Domain adaptation is a form of transfer learning, in which the task remains the
same, but there is a domain shift or a distribution change between the source and
the target. As an example, consider a model that has learned to classify reviews on
electronic products for positive and negative sentiments, and is used for classifying
the reviews for hotel rooms or movies. The task of sentiment analysis remains the
same, but the domain (electronics and hotel rooms) has changed. The application
of the model to a separate domain poses many problems because of the change
between the training data and the unseen testing data, typically known as domain
shift. For example, sentences containing phrases such as “loud and clear” will be
mostly considered positive in electronics, whereas negative in hotel room reviews.
Similarly, usage of keywords such as “lengthy” or “boring” which may be prevalent
in domains such as book reviews might be completely absent in domains such as
kitchen equipment reviews.
As discussed in the last chapter, the central idea behind domain adaptation is to
learn from the source dataset (labeled and unlabeled) so that the learning can be used
on a target dataset with a different domain mapping. To learn the domain shift be-
tween the source and the target, traditional techniques that are employed fall under
two broad categories: instance-based and feature-based. In instance-based, the dis-
crepancy between the source and the target domain is reduced by reweighting source
samples and learning models from the reweighed ones [BM10]. In feature-based, a
common shared space or a joint representation is learned between the source and
the target where the distributions match [GGS13]. In recent times, deep learning
architectures have been successfully implemented for domain adaptation in various
applications especially in the field of computer vision [Csu17]. In this chapter, we
discuss at length some of the techniques using deep learning in domain adaptation
and their applications in text and speech. Next, we discuss techniques in zero-shot,
one-shot, and few-shot learning that have gained popularity in the domain adapta-
tion field. We perform a detailed case study using many techniques discussed in the
chapter to give the readers practical aspects of domain adaptation at the end.
In this chapter, we will use the notations that are similar to the research papers
they cite for easy mapping to the references.
11.1.1 Techniques
In this section, we will highlight some of the well-known techniques that can be
very effective and generic enough for solving domain adaptation problem in text
and speech.
One of the earliest works in domain adaptation comes from Glorot et al. in the area
of sentiment classification [GBB11b]. The source domain contains a large number
of sentiments on Amazon reviews, while the target is completely different products
with small labeled data. In this work, the researchers use stacked denoising autoen-
coders (SDA) on the source and target data combined to learn the features as shown
in Fig. 11.1 as the first step. Then, a linear SVM is trained on the features extracted
by the encoder part of the autoencoder and is used to predict unseen target data of
different domains. The researchers report state-of-the-art results for sentiment clas-
sification across the domains.
Variants, such as stacked marginalized denoising autoencoders (mSDA),
which have better optimal solutions and faster training times, have also been
employed very successfully in classification tasks, such as sentiment classifica-
tion [Che+12]. To explain the method, let us assume that for source S and target T ,
we have sample source data DS = {x1 , · · · , xnS } ∈ Rd and labels Ls = {y1 , · · · , ynS }
and the target sample data DT = {xns+1 , · · · , xn } ∈ Rd and no labels. The goal is
to learn the classifier h ∈ H with labeled source training data DS to predict on the
unlabeled target data DT .
The basic building block in this work is the one-layered denoising autoencoder.
The input for this is the entire set of the source and the target data, i.e., D = DS ∪
DT = {x1 , · · · , xn } and it is corrupted by removal of feature with probability p ≥
0. For example, if the representation of the vector is a bag-of-words vector, some
values can be flipped from 1 to 0. Let us consider x̃i as the corrupted version of xi .
Instead of using the two-level encoder–decoder, a single mapping W : Rd → Rd is
used that minimizes the squared reconstruction loss given by:
1 n
∑ xi − Wx̃i
2n i=1
(11.1)
11.1 Introduction 497
Fig. 11.1: Stacked denoising autoencoders for learning features and SVM as a clas-
sifier
If we repeat this m times, the variance gets lowered and the solution for W can
be obtained from:
m n
1
Lsquared (W) =
2mn ∑ ∑ xi − Wx̃i, j (11.2)
j=1 i=1
where x̃i, j represents the jth corrupted version of the original xi input.
In matrix notation, with inputs X = [x1 , · · · , xn ] ∈ Rd×n and its m-times repeated
X̄ and corrupted being X̃. The loss equation can be written as:
1
Lsquared (W) = tr (X̄ − WX̃) (X̄ − WX̃) (11.3)
2mn
The off-diagonal entries in matrix [x̃i x̃i ] are uncorrupted if two features α and β
both survive the corruption. This has a probability of (1 − p)2 . For the diagonal, it
holds with probability 1 − p. If we define a vector q = [1 − p, · · · , 1 − p, 1] ∈ Rd+1 ,
where qα represents the probability of the feature α surviving the corruption, then
the scatter matrix of the original uncorrupted input can be represented as S = XX
and the expectation of matrix Q can be written as:
Sα ,β qα qβ if α = β
E[Q]α ,β = (11.7)
Sα ,β qα if α = β
Very similar to traditional machine learning, research by Chopra et al. uses source
and target with different domains to be mixed in different proportions to learn inter-
mediate representations [CBG13]. This work is known as deep learning for domain
adaptation by interpolating between domains (DLID). The researchers use convolu-
tional layers with pooling and predictive sparse decomposition method to learn the
non-linear features in an unsupervised way. The predictive sparse decomposition
method is similar to sparse coding models but with fast and smooth approxima-
tor [KRL10]. The labeled data is passed through the same transformation to get
features, concatenate them, and use classifier such as logistic regression to get a
joint model. In this way, the model learns useful features in an unsupervised manner
11.1 Introduction 499
from both source and target. These features can be employed for domain transfer on
the target alone. Figure 11.2a, b show schematically how this process works.
Fig. 11.2: DLID method. (a) Unsupervised latent representation learning. The top
circles show the intermediate path between the source and the target, where the filled
circles are the intermediate representations, and the empty circles are the source/tar-
get representations. (b) Unsupervised features and labels with classifier for learning
model
Let S be the source domain with data samples DS , T be the target domain with
DT as the data samples, and p ∈ [1, 2, · · · , P] be the index over P datasets. The mix
between source and target is done in such a way that at p = 1, DS = DT and from
then onwards the number of source samples decreases and that of target increases
in exact same proportion. For each dataset p ∈ [2, · · · , P − 1], DS , the number of
samples goes down and DT goes up incrementally for next p. Each dataset D p as an
input to a non-linear feature extractor FWp with weights Wp trained in an unsuper-
500 11 Transfer Learning: Domain Adaptation
That representation and the label Z i ,Y i are passed to the classifier or regressor for
the task and uses standard loss functions to optimize. The unseen data goes through
the same path, and predictions from the classifier are used for obtaining the class
and probability.
The deep domain confusion (DDC) architecture, shown in Fig. 11.3, is proposed by
Tzeng et al. and is one of the popular discrepancy-based domain adaptation frame-
works [Tze+14]. The researchers introduce a domain adaptation layer and a confu-
sion loss to learn a representation that is semantically meaningful and provides do-
main invariance. A Siamese convolutional network-based architecture is proposed,
where the main goal is to learn a representation that minimizes the distribution dis-
tance between the source and the target domain. The representation can be used
as features along with the source-labeled dataset to minimize the classification loss
and is applied directly to unlabeled target data. In this work, the task of minimizing
the distribution distance is done using maximum-mean discrepancy (MMD), which
computes the distance on a representation φ ()˙ for both source and target as:
1 1
|Xs | xs∑ ∑ φ (xt )
MMD(Xs , Xt ) = φ (xs ) − (11.9)
∈Xs |Xt | xt ∈Xt
The representation learned from this is used in the loss function as a regularizer
with the regularization hyperparameter λ also acting as the amount of confusion
between the source and the target domain:
where LC (XL , y) is the classification loss from the labeled data XL , y is the label
or the ground truth, and MMD(Xs , Xt ) is the maximum-mean discrepancy (MMD)
between the source XS and the target Xt . The hyperparameter λ controls the amount
of confusion between the source and the target domain. The researchers use the
standard AlexNet and modify it to have an additional lower -dimensional bottleneck
layer “fc adapt.” The lower-dimensional layer acts as a regularizer and prevents from
overfitting on the source distribution. The MMD loss discussed above is added on
top of this layer so that it learns the representation useful for both source and the
target.
11.1 Introduction 501
Fig. 11.3: Deep domain confusion network (DDCN) for domain adaptation
Long et al. propose the deep adaptation network (DAN) as shown in Fig. 11.4, which
is a modified AlexNet, where the discrepancy loss happens at the last fully con-
nected layers [LW15]. If the null hypothesis is that the samples are drawn from
the same distribution, and the alternate hypothesis is that they come from two dif-
ferent distributions, maximum-mean discrepancies (MMD) is one of the statisti-
cal approaches [Sej+12]. The multiple kernel variant of MMD (MK-MMD) mea-
sures the reproducing kernel Hilbert space (RKHS) distance between the mean em-
beddings of two distributions (source and target) with a characteristic kernel k. If
Hk is the reproducing kernel Hilbert space endowed with a characteristic kernel k,
the mean embedding of distribution p in Hk is a unique element μk (p), such that
Ex∼p f (x) = f (x), μk (p)Hk . The squared distance for any layer l, kernel k between
source (S), and target (T ) is given by:
1 na
min
Θ
∑ J(Θ(xai ), yai )
na i=1
(11.13)
where J is the cross-entropy loss function and Θ (xai ) is the conditional probability
of assigning the data point xai a label yai . By adding the MK-MMD-based multi-layer
adaptation regularizer to the above risk, we get a loss similar to the DDC loss that
can be expressed as:
l2
1 na
min
Θ
∑
na i=1
J(Θ(xai ), yai ) + λ ∑ dk2 (DlS , DlT ) (11.14)
l=l
1
where λ > 0 is a regularization constant and l1 = 6 and l2 = 8 for the DAN setup.
Many techniques employ domain-invariant representation using the source and the
target data as a way to learn a common representation that can help in domain adap-
tation.
CORrelation ALignment (CORAL) is a technique to align the second-order
statistics (covariances) of the source and target using a linear transformation. Sun
11.1 Introduction 503
and Saenko extend the framework to learn the non-linear transformation that aligns
the correlations of the layers, known as Deep CORAL [SS16]. Deep CORAL ex-
tends AlexNet and has the second-order statistics loss computed at the last layer,
i.e., a fully connected layer before the output. If DS = {xi }, x ∈ Rd are the source
domain training data of size nS , and DT = {ui }, u ∈ Rd are the unlabeled target data
of size nT , Di,S j indicates jth dimension of the ith source data instance, Di,T j indicates
jth dimension of the ith target data instance, CS is the source feature covariance
matrix, and CT is the target covariance matrix, then the CORAL loss is measured as
the distance between the covariances:
1
CORAL = |CS −CT |2F (11.15)
4d 2
where | · |2F represents the squared matrix Frobenius norm.
The covariance matrices for the source and the target are given by:
1 1
CS = DS DS − (1 DS )(1 DS ) (11.16)
(nS − 1) nS
1 1
CT = DT DT − (1 DT )(1 DT ) (11.17)
(nT − 1) nT
where 1 is the column vector. The joint training that reduces the classification loss
lCLASS and the CORAL loss is given by:
t
l = lCLASS + ∑ λiCORAL (11.18)
i=1
where t is the number of layers, and λ is used to balance between classification and
domain adaptation, aiming at learning a representation common between the source
and the target (Fig. 11.5).
There are other domain-invariant representations that have been successfully em-
ployed in various works. Pan et al. use domain-invariant representation via transfer
component analysis that uses maximum-mean discrepancies (MMD) and tries to
reduce the distance between the two domains in the subspace [Pan+11]. Zellinger
et al. propose a new distance function—central moment discrepancy (CMD)—to
match the higher-order central moments of probability distributions [Zel+17]. They
show the generality of their techniques in domain adaptation across object recogni-
tion and sentiment classification tasks (Fig. 11.5).
The research by Tzeng et al. on deep domain confusion has the disadvantage that
it needs both large labeled data in the source domain and sparsely labeled data in
the target domain. Tzeng et al. in their work propose domain confusion loss over
both labeled and unlabeled data to learn the invariant representation across domains
504 11 Transfer Learning: Domain Adaptation
and tasks [Tze+15]. The transfer learning between the source and the domain is
achieved by a) maximizing the domain confusion by making marginal distributions
between source and target as similar to each other as possible; and b) transfer of
correlation between classes learned on source examples to target examples. The
completely labeled source data (xS , yS ) and the sparsely labeled target data (xT , yT )
are used to produce a classifier θC that operates on feature representation f (x; θrepr )
parameterized by representation parameters θrepr and has good accuracy in classi-
fying the target samples:
The parameters θD and θrepr are learned iteratively by the following objectives:
where λ and ν are the hyperparameters that control the domain confusion and the
soft label influence during the optimization.
Ganin et al. employ an interesting technique of “gradient reversal” layer for domain
shift adaptation through domain-adversarial neural network (DANN) [Gan+16b].
The process is generic to all neural networks and can be easily trained using standard
stochastic gradient methods. They show state-of-the-art results in different domains
of computer vision and sentiment classification.
Let S = {(xi , yi )}ni=1 ∼ (DS )n ; T = {xi }Ni=n+1 ∼ (DXT )n be the source and tar-
get data drawn from DS and DT as distribution; N = n + n is the total number
of samples. DXT is the marginal distribution of DT over the input space X, and
Y = 0, 1, · · · , L − 1 is the set of labels. The network has three important layers: (a)
the feature generation layers which learn features from the inputs with parameters.
The hidden layer G f : X → RD parameterized by matrix–vector pair θ f = (W, b):
(b) the label prediction layer Gy : RD → [0, 1]L parameterized by matrix–vector pair
θy = (V, c):
Gy (G f (x); θy ) = softmax(Vx + c) (11.27)
506 11 Transfer Learning: Domain Adaptation
and (c) the domain classification layer Gd : RD → [0, 1] is a logistic regressor pa-
rameterized by vector–scalar pair θd = (u, z) that predicts whether the example is
from the source or the target domain. Figure 11.6 shows the training across the three
different layers.
The prediction loss for (xi , yi ) can be written as:
The domain loss (xi , di ), where di is the domain, can be written as:
The total training loss for a single layer network can be written as:
n
1 n i 1 1 N
Ltotal (θ f , θy , θd ) = ∑ Ly (θ f , θy ) − λ ∑ Lid (θ f , θd ) + ∑ Lid (θ f , θd )
n i=1 n i=1 n i=n+1
(11.30)
The hyperparameter λ controls the trade-off between the losses. The parameters
are obtained by solving the equations:
The gradient updates are very similar to standard stochastic gradient descent with
∂ Li
a learning rate μ except the reversal with λ ∂ θ d . The gradient reversal layer has no
f
parameter and its forward pass is the identity function, while the backward pass is
the gradient from subsequent layer multiplied by −1:
∂ Liy ∂ Lid
θ f ←− θ f − μ −λ (11.33)
∂θf ∂θf
∂ Liy
θy ←− θy − μ (11.34)
∂ θy
∂ Lid
θd ←− θd − μ (11.35)
∂ θd
tied between source and the target, and has a GAN loss for computing adversarial
loss [Tze+17].
Let us assume we have source data Xs and labels Ys drawn from source distribu-
tion ps (x, y), and target data Xt from a target distribution pt (x, y) with no labels. The
goal is to learn a target mapping Mt and a classifier Ct that can classify K categories.
In adversarial methods, the goal is to minimize the distribution distance between
the source and target mapping Ms (Xs ) and Mt (Xt ), so that the source classification
model Cs can be used directly on the target so that C = Cs = Ct . The standard super-
vised loss can be written as:
K
min Lclass (Xs ,Ys ) = −E(xs ,ys )∼(Xs ,Ys ) ∑ 1[k=ys ] logC(Ms (xs )) (11.36)
Ms ,C
k=1
The domain discriminator D classifies if the data is from source or target, and D
is optimized using LadvD :
min LadvM (Xs , Xt , D) = −Exs ∼Xt [log D(Mt (xt ))] (11.38)
Ms ,Mt
The training happens in phases, as shown in Fig. 11.7. The process begins with
Ladvclass over Ms and C, using the labeled data Xs and labels Ys . We can then perform
adversarial adaptation by optimizing LadvD , LadvM .
508 11 Transfer Learning: Domain Adaptation
Liu and Tuzel propose a coupled generative adversarial network (CoGAN) for learn-
ing joint distribution between two domains and show it to be very successful in
computer vision [LQH16]. As discussed in Chap. 4, GANs consist of generative
and discriminative models. The generative model is used to generate synthetic data
resembling real data, while the discriminative model is used to distinguish between
the two. Formally, a random vector z is input to the generative model that outputs
g(z) that has the same support as the input x. The discriminative model outputs
f (x) = 1 if drawn from real x ∼ pX and f (x) = 0 if drawn from synthetic or gen-
erated x ∼ pG . Thus, GANs can be seen as a minimax two-player game solving
through optimization:
In CoGAN, as shown in Fig. 11.8, there are two GANs for two different domains.
Generative models try to decode from higher-level features to lower-level features
as opposed to discriminative models. If x1 and x2 are two inputs drawn from the
marginal distribution of first (x1 ∼ pX1 ) and second (x2 ∼ pX2 ), respectively, then
generative models GAN1 and GAN2 map a random vector z to examples having the
same support as x1 and x2 . The distribution of g1 (z) and g2 (z) is pG1 and pG2 . When
11.1 Introduction 509
where k is the shared layers, and θgi and θgi are the parameters of gi1 and gi2 , re-
1 2
spectively. This constraint enables the first layers that decode high-level features to
decode it in the same way for both generators g1 and g2 .
Discriminative models map the input to a probability, estimating the likelihood
that the input is from the data distribution. If f1i and f2i correspond to the layers of
discriminative networks for two GANs with n1 and n2 layers, it can be written as:
(n1 ) (n1 −1) (2) (1)
f1 (x1 ) = f1 ( f1 (· · · f1 ( f1 (x1 )))) (11.43)
(n2 ) (n2 −1) (2) (1)
f2 (x1 ) = f2 ( f2 (· · · f2 ( f2 (x2 )))) (11.44)
f1i f2i
where and are the layers in the corresponding f1 and f2 with layers n1 and n2 ,
respectively. The discriminative models work in contrast to the generative models
and extract low-level features in the first layers and high-level features in the last
layers. To ensure the data has the same high-level features, we share the last layers
using:
θ (n1 −i) = θ (n2 −i) f or i = 0, 1, . . . (l − 1) (11.45)
f1 f2
where l is the shared layers and θ f i and θ f i are the parameters of f1i and f2i , re-
1 2
spectively. It can be shown that learning in CoGAN corresponds to a constrained
minimax game given by:
max min V (g1 , g2 , f1 , f2 ) = Ex1 ∼pX1 [− log( f1 )(x1 )] + Ez∼pZ [− log(1 − f1 (g1 (z)))]
g1 ,g2 f1 , f2
where
LGAN (G, DY , X,Y ) = Ey∼pdata (y) [log DY (y)] + Ex∼pdata (x) [log(1 − DY (G(x)))]
(11.49)
11.1 Introduction 511
Similarly
min max LGAN (F, DX ,Y, X) (11.50)
F DX
LGAN (F, DX ,Y, X) = Ex∼pdata (x) [log DX (x)] + Ey∼pdata (y) [log(1 − DX (F(y)))]
(11.51)
The cycle-consistency loss is about bringing the original data x from the trans-
lation x → G(x) → F(G(x)) ≈ x for x domain and y from the translation
y → F(y) → G(F(y)) ≈ y captured as:
Ltotal (G, F, DX , DY ) = LGAN (G, DY , X,Y ) + LGAN (F, DX ,Y, X) + λ Lcyc (G, F)
(11.53)
Fig. 11.9: Cycle generative adversarial networks (CycleGAN) with forward cycle-
consistency and backward cycle-consistency
CycleGAN does not need the data pairs in the domains to match. It can learn
the underlying relationship and help transfer between the domains.
Domain separation networks by Bousmalis et al. have private encoders for learning
individual domains, shared encoders for learning common representations across
domains, shared decoder for effective generalization using reconstruction loss, and
a classifier using shared representations for robustness [Bou+16].
The source domain Ds has Ns labeled data Xs = xsi , ysi and the target domain
Dt has Nt unlabeled data Xt = xti . Let E p (x; θ p ) be the function that maps input
x to a hidden representation h p for a representation that is private for the domain.
512 11 Transfer Learning: Domain Adaptation
Let Ec (x; θc ) be the function that maps input x to a hidden representation hc that
is common across the source and the target. Let D(h; θd ) be the decoding function
that maps the hidden representation h to the original reconstruction x̂. Reconstruc-
tion can be given by x̂ = D(Ec (x) + E p (x)). Let G(h; θg ) be the classifier func-
tion that maps the hidden representation h to predictions ŷ given by ŷ = G(Ec (x)).
Figure 11.10 captures the entire process of DSN.
The total loss can be written as:
where the hyperparameters α , β , γ control the weight of each loss term. The classi-
fication loss is the standard negative log-likelihood given by:
Ns
Lclass = − ∑ ySi · log(ŷsi ) (11.55)
i=0
The difference loss, as the name suggests, is applied to both domains and is meant
to capture different aspects of inputs for the private and shared encoders. Let Hsc and
Htc be the rows of matrices which are common between the source and target hidden
layers. Let Hsp and Htp be the rows of matrices which are private to the source and
the target hidden layers. The difference loss is given by:
Ldifference = Hsc Hsp 2F + Htc Htp 2F (11.57)
The maximum-mean discrepancy (MMD) loss can also be used instead of the
DANN described above.
Domain separation networks capture explicitly and jointly both the private and
shared components of the domain representations making it less vulnerable to
noise that is correlated with the shared distributions.
11.1 Introduction 513
11.1.2 Theory
We will describe two topics that have been studied in the last couple of years to
give a formal mapping to domain adaptation that is applicable in the deep learn-
ing area. One is the generalization of most domain adaptation networks by Tzeng
et al. [Tze+17], and another is the optimization transport theory for giving a theo-
retical foundation to domain adaptation [RHS17].
Tzeng et al. present a generalized Siamese architecture which captures most im-
plementations in domain adaptations using deep learning as shown in Fig. 11.11
[Tze+17]. The architecture has two streams, the source input which is labeled, and
the target input which is unlabeled. The training is done with a combination of
classification loss with either discrepancy-based loss or adversarial loss. The classi-
fication loss is computed only using the labeled source data. The discrepancy loss is
computed based on the domain shift between the source and the target. The adver-
sarial loss tries to capture latent features using the adversarial objective with respect
to the domain discriminator. This study helps to put all the architectures seen as var-
ious extensions of the general architecture with changes to how classification loss,
discrepancy loss, and adversarial loss are computed.
The setup can be generalized to drawing source-labeled samples (Xs ,Ys ) from a
distribution ps (x, y) and unlabeled target samples Xt from a distribution pt (x, y). The
goal is to learn from source examples and a classifier Cs a representation mapping
514 11 Transfer Learning: Domain Adaptation
Ms , and also have a target mapping Mt with a classifier Ct at prediction time that
learns to classify unseen examples in k categories.
The goal of most adversarial methods is to minimize the distance between the
distributions of Ms (Xs ) and Mt (Xt ), which implicitly means that in most cases the
source and target classifiers can be same C = Cs = Ct . Source classification can be
given in a generic loss optimization form as:
K
min Lclass (XS ,YS ) = −E(xs ,ys )∼(XS ,YS ) ∑ 1[k=ys ] logC(MS (xs )) (11.59)
MS ,C
k=1
A domain discriminator D which classifies whether the data is drawn from the
source or the target can be written as:
Fig. 11.11: Siamese networks for generalizing the domain adaptation implementa-
tion
With the source and target mapping constraints given by ψ (MS , MT ), a discriminator
D that can distinguish between them can be captured as an adversarial objective
LadvM :
Various techniques described in domain adaptation can now be understood with this
general framework.
The gradient reversal process can be written in terms of optimizing the discrimi-
nator loss directly as LadvM = −LadvD .
When using GANs there are two losses: the discriminator loss and the generator
loss. The discriminator loss LadvD remains the same, while generator loss can be
written as:
11.1 Introduction 515
min LadvM (XS , XT , D) = −Ext ∼XT [log D(MT (xt ))] (11.62)
D
The domain confusion loss can be written as minimizing the cross-entropy loss
given by:
1 1
min LadvM (XS , XT , D) = − ∑ Exd ∼XD log D(Md (xd )) + log(1−D(Md (xd )))
D
d∈s,t 2 2
(11.63)
In the last few years, optimal transport theory has come into prominence from var-
ious statistical, optimization, and machine learning perspectives. Optimal transport
can be seen as a way of measuring the transport of data between two different dis-
tributions that are based on the geometry of the data points in the two and has a
cost function related to transportation [Mon81]. This transport mechanism maps
very well to domain adaptation, where the source and target domains can be seen
as two different distributions, and optimal transport explains from both theory and
optimization the mapping. The Wasserstein distance in optimal transport which is
used to measure the distance between two distributions can also be used as a min-
imization objective or regularization function in the overall loss function. Optimal
transport has been used to give a good generalization bound to deep domain adap-
tation frameworks [RHS17].
Glorot et al. in the very early days of deep learning showed how stacked autoen-
coders with sparse rectifier units could learn feature-level representations that could
perform domain adaptation on sentiment analysis very effectively [GBB11c].
Nguyen and Grishman employ word embeddings along with word clustering
features to show that domain adaptation in relation extraction can be very effec-
tive [NG14]. Nguyen et al. further explore the use of word embeddings and tree
kernels to generate a semantic representation for relation extraction and improve-
ments over feature based methods [NPG15]. Nguyen and Grishman show how basic
CNNs with word embeddings, position embeddings, and entity type embeddings as
input can learn effective representation that gives a good domain adaptation method
for event detection [NG15a]. Fu et al. show the effectiveness of domain adaptation
for relation extraction using domain-adversarial neural networks (DANN) [Fu+17].
They use word embeddings, position embeddings, entity type embeddings, chunk-
ing, and dependency path embeddings. They use CNNs and DANN with a gradient
reversal layer to effectively learn the relationship extraction with cross-domain fea-
tures.
516 11 Transfer Learning: Domain Adaptation
Zhou et al. use a novel bi-transferring deep neural networks to transfer source
examples into the target and vice versa for achieving close to the state-of-the-art re-
sults in sentiment classification [Zho+16]. Zhang et al. use the mapping between the
keywords to the source and target and employ it in adversarial training for domain
adaptation in classification [ZBJ17]. Ziser and Reichart show how pivot features
(common features that are present in source and target) along with autoencoders can
learn representation that is very effective in domain adaptation for sentiment clas-
sification [ZR17]. Ziser and Reichart further extend the research to a pivot-based
language model in a structure-aware manner that can be employed for various clas-
sification and sequence-to-sequence-based tasks for improved results [ZR18]. Yu
and Ziang combine the ideas of structural correspondence learning, pivot-based fea-
tures, and joint-task learning for effective domain adaptation in sentiment classifi-
cation [YJ16].
Falavigna et al. show how deep neural networks and automatic quality estimation
(QE) can be used for domain adaptation [Fal+17]. They use a two-step process in
which first manually labeled transcripts are used for evaluating WER on the data for
different quality. Then adaptation is made on unseen data according to WER scores
by the QE component to show significant improvements in performance.
Hosseini-Asl et al. extend the CycleGAN concepts to have multiple discrim-
inators (MD-CycleGAN) for unsupervised non-parallel speech domain adapta-
tion [Hos+18]. They use multiple discriminator-enabled CycleGAN to learn fre-
quency variations in spectrograms between the domains. They use different gender
speech ASR in training and testing to evaluate the domain adaptation aspect of the
framework and report a good performance by using the MD-CycleGAN architecture
on unseen domains.
Adapting to different speakers with different accents is one of the open research
problems in speech recognition. Wang et al. in their work do a detailed analysis
treating this as a domain adaptation problem with different frameworks to give im-
portant insights [Wan+18a]. They use three different speaker adaptation methods,
such as linear transformation (LIN), learning hidden unit contribution (LHUC), and
Kullback–Leibler divergence (KLD) on a i-vector based DNN acoustic model. They
show that based on the accents using one of the methods, ASR performance can be
significantly improved for not only medium-to-heavy accents but also for slight-
accent speakers. Sun et al. use domain-adversarial training for solving accented
speech in ASR [Sun+18a]. Employing domain adversarial training in the learning
objective from unlabeled target domain with different accents to separate source and
target while using labeled source domain for classification, they show a significant
drop in error rates for unseen accents.
Improving ASR quality in the presence of noise by improving the robustness
of the models can also be approached from the domain adaptation view based
11.2 Zero-Shot, One-Shot, and Few-Shot Learning 517
on how the noise in target domain or unseen data is different from the source
domains. Serdyuk et al. use GANs for domain adaptation in unseen noisy target
datasets [Ser+16]. The model has the encoder, decoder, and the recognizer with a
hidden representation in between that is used to perform dual tasks of improving
the recognition and minimizing domain discrimination. They show their method to
be better at generalization when the target domain has more noise categories than
the ones used in the source training data. Sun et al. use adversarial data augmenta-
tion using fast gradient sign method (FSGM) to show significant improvements in
the robustness of acoustic models [Sun+18b]. Meng et al. use domain separation
networks (DSN) for domain adaptation between source and targets for robustness
on target data with different noise levels [Men+17]. The shared components learn
the domain invariance between the source and the target domains. The private com-
ponents are orthogonal with the shared ones and learn to increase domain invari-
ance. They show a significant decrease in the WER over baseline with an unadapted
acoustic model with their approach.
The extremes of domain adaptation or transfer learning problem are when there are
limited training examples to match the test example. The best example is the facial
recognition problem from computer vision, where there is exactly 1 training exam-
ple for each person, and when someone appears, the need is to match the existing or
classify it as a new unseen. Based on a number of training examples corresponding
to the unseen example we get at prediction time, there are different flavors such as
zero-shot learning, one-shot learning, and few-shot learning. In the next sections,
we will discuss each of them and techniques that have been popular to address them.
on similar queries if the cost of parts for performing specific functions is normally
used.
11.2.1.1 Techniques
We will illustrate a general method and variations that have been successful in com-
puter vision and language/speech understanding/recognition tasks [XSA17].
The approach is to measure the similarity between source and target domains. In
computer vision, for example, one way is to map the label space to a vector space
based on side information, such as attributes capturing the picture. The attributes
can be meta-level or image-level features, such as “presence of specific color,” “size
of the object,” and others. The vector representation can be a one-hot vector of these
attributes. The source data features are embedded in the source feature space. The
next step is to find the compatibility between the source feature space, as shown in
Fig. 11.12, using a compatibility function.
8. Direct Attribute Prediction: Another technique uses attributes associated with the
class to be learned directly, given by:
M
p(acm |x)
f (x) = arg max ∏ (11.73)
c m=1 p(acm )
where M is the total number of attributes, acm is the mth attribute of class c, and
p(acm |x) is that attribute probability associated with the given data x.
The general problem in one-shot learning is to learn from a dataset where there is
one example for a class. The same general form is used for the similarity function
in representation between the training examples, so that during the prediction the
similarity function is used to find the closest available example in the training data.
11.2.2.1 Techniques
The Siamese network-based architectures with variations are generally the common
way to learn similarities in these frameworks. The network parameters are learned
through pairwise learning from the training dataset, as shown in Fig. 11.13. One
variation is that, instead of the fully connected layers going to a softmax layer,
features or encoding of the input can be used for similarity; the resulting are called
matching networks. One way to learn the parameters of the network is during train-
ing time to minimize the difference when inputs are similar and maximize when
dissimilar while during prediction to use the learned representation to compute sim-
ilarity with existing training samples. If xi and x j are two examples from the training
data, the similarity function can be the difference between the two predictions in the
Siamese networks given by:
Another way to learn the parameters is through the triplet loss function by
Schroff et al. [FSP15]. The idea is to pick an anchor data xA for which a positive
xP and a negative xN sample are used to learn parameters of the network, so that the
difference between anchor data and positive data is maximized, and the difference
between anchor data and the negative is minimized:
In Eq. 11.75, the parameter α is similar to the margin in SVMs. The training
data is used to generate the triplets, and stochastic gradient method can be used for
learning the parameters with this loss function.
11.2.3.1 Techniques
The deep learning techniques for few-shot learning can be described as either data-
based approaches or model-based approaches. In the data-based approach, some
form of augmenting the training data in different forms is the general process uti-
lized to increase the number of similar samples.
In contrast, the model- or parameter-based approach enforces regularization
in some form to prevent overfitting of the model from the limited training sam-
ples. Donghyun et al. use the interesting idea of correlating activations from in-
put data to form “groups” of similar neurons or parameters per layer in the source
data [Yoo+18]. The hyperparameter “number of groups” per layer is chosen using k-
means clustering algorithm, and k is further learned using reinforcement techniques.
522 11 Transfer Learning: Domain Adaptation
Once they are trained on source dataset, these groups of neurons are fine-tuned on
the target domain using group-wise backpropagation. As the number of parame-
ters increases, with small training data for each category, optimization algorithms
such as SGD are not effective. Mengye et al. propose a meta-learning approach for
solving this in two steps: (a) a teacher model learns from a large amount of data to
capture the parameter space, and (b) then guides the actual pupil or a classifier to
learn using the parameter manifold giving excellent results [Ren+18].
11.2.4 Theory
Most of the applications of zero-shot, one-shot, and few-shot learning have been
in computer vision. Only recently have there been applications in NLP and speech.
Pushp and Srivastav employ zero-shot learning in classification for text categoriza-
tion [PS17]. The source dataset is the news headlines crawled from the web, and
the categories are the search engine. The target test data is the UCI news and the
tweets categorization dataset. They employ different neural architectures based on
how and what one feeds to the LSTM networks. The trained model is then applied to
the dataset which has not seen the relationships before (UCI news and tweets) to get
very impressive results, showing the effectiveness of zero-shot learning methods.
Levy et al. employ zero-shot learning in relation extraction by learning to answer
questions from a corpus [LS17]. Yogatama et al. try to explore RNNs as genera-
tive models and empirically show the promise of generative learning in a zero-shot
11.3 Case Study 523
learning setting [Yog+17]. The relationship is learned by posing questions and hav-
ing sentences in the answers that map to an entity, where the relationship is men-
tioned from slot-filling datasets such as WikiReading. They show that even on the
unseen relationship, the zero-shot learning shows enough promise as a methodol-
ogy. Mitchell et al. employ zero-shot learning using explanations about the labels or
categories to learn the embedding space using constraints and show good results on
email categorization [MSL18].
Dagan et al. propose a zero-shot learning framework for the event extrac-
tion problem using event ontologies and small, manually annotated labeled
datasets [Dag+18]. They show transferability to even unseen types and additionally
report results close to the state of the art.
Yan et al. address the difficult short text classification problem by using few-shot
learning [YZC18]. They use the Siamese CNNs to learn the encoding that distin-
guishes complex or informal sentences. Different structures and topics are learned
using the few-shot learning method and shown to generalize and have better accu-
racies than many traditional and deep learning methods.
Ma et al. have proposed a neural architecture for both few-shot and zero-shot
learning on fine-grained named entity typing, i.e., detecting not only the entity from
the sentence, but also the type (for example, “John is talking using his phone”
not only identifies “John” as the entity, but can also decode that “John” is the
speaker [MCG16]). They use prototypical and hierarchical information to learn label
embeddings and give a huge performance boost for the classification. In their work,
Yazdani and Henderson use zero-shot learning for spoken language understanding,
where they assign label actions with attributes and values from the utterance output
of ASR dialogs [YH15]. They build a semantic space between the words and the la-
bels, so that it can form a representation layer that predicts unseen words and labels
in very effective way.
Rojas-Barahona et al. have shown the success of deep learning and zero-shot
learning in semantic decoding of spoken dialog systems. They use deep learning
for learning features jointly from known and unknown categories [Roj+18]. They
then use unsupervised learning to tune the weights, further using risk minimization
to achieve zero-shot learning when tested on unseen data with slot pairs not known
in the training set. Keren et al. use one-shot learning with Siamese networks to
compute the similarity between the single exemplars from the source data to the
unseen examples from the target data in the spoken term detection problem in the
audio domain [Ker+18].
We will go through a detailed case study to explore and understand different things
discussed in the chapter from a practical point of view. We chose the Amazon prod-
uct review dataset published in the research by Blitzer et al. [BDP07] for the senti-
ment classification task. The dataset has reviews for various product domains such
524 11 Transfer Learning: Domain Adaptation
as books, DVDs, kitchen, and electronics. All the domains have 2000 labeled ex-
amples with binary labels (positive and negative) based on the reviews. Kitchen
and electronics domain has large number of unlabeled examples as well. In our ex-
periments we have not used the unlabeled examples but treated many labeled as
unlabeled when required.
We chose two different cases: (1) the source domain is kitchen and the target
domain is electronics and (2) the source domain is books and the target domain is
kitchen for our experiments. We divided all the datasets into training and testing
with 1600 and 400 examples, respectively. The validation data is chosen from the
training dataset either as a percent or a stratified sample. Though the goal is not to
replicate papers or fine-tune each method to get best results, we have done some
parameter tuning and kept most parameters standard or constant to see a relative
impact.
We will describe the main open source tools and libraries we have used below for
our case study. There are some open source packages for specific algorithms which
we have either used, adapted, or extended that are mentioned in the notebook itself:
• Keras (www.keras.io)
• TensorFlow (https://www.tensorflow.org/)
• Pandas (https://pandas.pydata.org/)
• scikit-learn (http://scikit-learn.org/)
• Matplotlib (https://matplotlib.org/)
Similar to the other case studies we will be performing some basic EDA to under-
stand the data and some of its characteristics. The plots shown in Fig. 11.14a, b
show the word distribution bar charts across entire corpus of source and target for
the sentiments. It clearly shows that going from the domain kitchen to electronics
may not be that different as going from books to kitchen reviews.
The plots shown in Fig. 11.15a–c illustrate the word cloud for the positive senti-
ment data across books, kitchen, and electronics reviews. Just visually exploring
some high frequency words, the similarity between the word cloud of kitchen–
electronics as also the differences between books–kitchen is very evident. The plots
shown in Fig. 11.16a–c depicting the word cloud for the negative sentiment data
across books, kitchen, and electronics reviews also illustrate the same characteris-
tics.
11.3 Case Study 525
We next describe in detail all the experiments we carried out with transfer learning
techniques in the form of training process, model, algorithms, and changes. Again,
the goal was not to get the best tuned models for each but understand practically how
each technique with its biases and processes performs on some of these complex
Quartile Quartiles
80
120 Square (Books) Source (Kitchen)
Target (Kitchen) 70 Target (Electronics)
Number of Words in Text
60 40
30
40
20
20
10
0 0
Q1 Q2 Q3 Q1 Q2 Q3
(a) (b)
Fig. 11.14: Word distribution comparisons in quartiles of 25, 50, and 75%. (a) Books
and kitchen comparison. (b) Kitchen and electronics comparison
Fig. 11.15: Word cloud for positive sentiments from (a) books, (b) kitchen, and (c)
electronics data, respectively
Fig. 11.16: Word cloud for the negative sentiments from (a) books, (b) kitchen, and
(c) electronics data, respectively
526 11 Transfer Learning: Domain Adaptation
real-world tasks. We carry the experiments for both books–kitchen and kitchen–
electronics as our source–target domains. We use classification accuracy as a metrics
to see the performance as the test data was having equal number of positive and
negative sentiments.
11.3.3.1 Preprocessing
We perform some basic preprocessing on the raw data for carrying out the sentiment
classification tasks. The data is parsed from XML based documents, tokenized into
words with basic stop words removed and some basic padding of sequences to give a
constant maximum length representation for each. We create a vocabulary of words
by finding all the words from source and the target sealed with maximum size of
15000 for most experiments. For some of the vector–space models we use bag-of-
words representation with n-grams of size 2 and maximum features of size 10000.
11.3.3.2 Experiments
We will use Kim’s CNN model shown in Fig. 11.17 as our classifier model in most
experiments. We used the standard GloVe embeddings with 100 dimensions which
were trained on 6 billion words. We will list the name of the experiments and the
purpose behind it below:
1. Train Source + Test Target: The goal is to understand the transfer learning
loss that happens when you train on the source data and test only on the target
data due to domain change. This as we discussed can happen incrementally over
time or due to completely different environment where the model is deployed.
This gives a basic worst case analysis for our experiments.
2. Train Target + Test Target: This experiment gives us the best case analysis
for the model which has not seen the source data but completely trained on the
target training data and predicted on the target test data.
3. Pre-trained Embeddings Source + Train Target: We try to understand the
impact of unsupervised pre-trained embeddings on the learning process. The
embedding layer is frozen and non-trainable in this experiment. We train the
model on the target domain using unsupervised embeddings and test on the
target test set.
4. Pre-trained Embeddings Source + Train Target + Fine-tune Target: We
train the model on the target domain using unsupervised embeddings but fine-
tune the embeddings layer with the target train data.
5. Pre-trained Embeddings + Train Source + Fine-tune Source and Target:
This can be the best case of pre-training and fine-tuning where you get the
advantage of learning embeddings from unsupervised, train on the source, fine-
tune on the target, and thus have more examples for learning useful representa-
tion across.
11.3 Case Study 527
6. Stacked Autoencoders and DNNs: We use the DNNs with stacked autoen-
coders for the unsupervised latent feature representation learning. Train the
model on the source domain in unsupervised way. Fine-tune the model with
new classification layers on the target training data and test on the target data.
7. Stacked Autoencoders and CNNs: Goal is to understand the impact of la-
tent representation learned from unseen source data on the target domain using
CNNs for autoencoders as shown in Fig. 11.18a, b.
A sample code that shows how the autoencoder is constructed:
1 i n p u t l a y e r = I n p u t ( shape =(300 , 300) )
2 # e n c o d i n g l a y e r s t o form t h e b o t t l e n e c k
3 e n c o d e d h 1 = Dense ( 1 2 8 , a c t i v a t i o n = ’ t a n h ’ ) ( i n p u t i =
layer )
4 e n c o d e d h 2 = Dense ( 6 4 , a c t i v a t i o n = ’ t a n h ’ ) ( e n c o d e d h 1 )
5 e n c o d e d h 3 = Dense ( 3 2 , a c t i v a t i o n = ’ t a n h ’ ) ( e n c o d e d h 2 )
6 e n c o d e d h 4 = Dense ( 1 6 , a c t i v a t i o n = ’ t a n h ’ ) ( e n c o d e d h 3 )
7 e n c o d e d h 5 = Dense ( 8 , a c t i v a t i o n = ’ t a n h ’ ) ( e n c o d e d h 4 )
8 # l a t e n t or coding l a y e r
9 l a t e n t = Dense ( 2 , a c t i v a t i o n = ’ t a n h ’ ) ( e n c o d e d h 5 )
10 # decoding l a y e r s
11 d e c o d e r h 1 = Dense ( 8 , a c t i v a t i o n = ’ t a n h ’ ) ( l a t e n t )
12 d e c o d e r h 2 = Dense ( 1 6 , a c t i v a t i o n = ’ t a n h ’ ) ( d e c o d e r h 1 )
528 11 Transfer Learning: Domain Adaptation
Fig. 11.18: Unsupervised training from source using (a) DNN and (b) CNN autoen-
coders and further trained/tested on target with classification layer in Keras
13 d e c o d e r h 3 = Dense ( 3 2 , a c t i v a t i o n = ’ t a n h ’ ) ( d e c o d e r h 2 )
14 d e c o d e r h 4 = Dense ( 6 4 , a c t i v a t i o n = ’ t a n h ’ ) ( d e c o d e r h 3 )
15 d e c o d e r h 5 = Dense ( 1 2 8 , a c t i v a t i o n = ’ t a n h ’ ) ( d e c o d e r h 4 )
16 # output layer
17 o u t p u t l a y e r = Dense ( 3 0 0 , a c t i v a t i o n = ’ t a n h ’ ) ( d e c o d e r h 5 )
18 # a u t o e n c o d e r u s i n g deep n e u r a l networks
19 a u t o e n c o d e r = Model ( i n p u t l a y e r , o u t p u t l a y e r )
20 a u t o e n c o d e r . summary ( )
21 a u t o e n c o d e r . c o m p i l e ( ’ a d a d e l t a ’ , ’ mse ’ )
22
12 # classification layer
13 c l a s s i f i c a t i o n m o d e l . add ( Dense ( 2 , a c t i v a t i o n = ’ s o f t m a x ’ ) )
14 c l a s s i f i c a t i o n m o d e l . compile ( o p t i m i z e r = ’ rmsprop ’ ,
15 loss=’
categorical crossentropy ’ ,
16 m e t r i c s =[ ’ a c c u r a c y ’ ] )
17
Table 11.1: Domain adaptation experiments on two different datasets for analyzing
impact of source–target domain shift
Some of the observations and analysis of the results are given below:
1. Books to kitchen has higher domain transfer loss, with train accuracy
(78.25) and test accuracy (69.00) it is 9.25, as compared to kitchen to
electronics, with train accuracy (83.75) and test accuracy (78.00) it is
5.75. The word cloud and the data distribution confirm that reviews writ-
ten for books are very different as compared to kitchen and electronics.
2. Using the pre-trained embeddings has an impact and the incremental im-
provement seen going from just the frozen embeddings to the embeddings
trained on source and target justifies the transfer learning.
3. One of the best results is seen for both books–kitchen and kitchen–
electronics is when the pre-trained embeddings are used, trained end-to-
end first on source and then on the target. Thus, the advantage of learning
unsupervised and fine-tuning to adapt to the domain shift is very evident.
4. Stacked autoencoders with CNNs show better results than with plain
DNNs, proving the effectiveness of autoencoders in capturing the latent
features and layered CNNs in capturing the signals for classification.
5. Most of the statistical techniques such as CORAL, CMD, and MMD don’t
show good performance
6. Adversarial methods such as DANN show a lot of promise with just the
shallow networks.
Some other interesting problems readers and practitioners can attempt on their own
include:
1. What will be the impact of combining source and target training data together
and test on unseen target test?
2. What will be the impact of using labeled and unlabeled data from source and
target to learn embeddings and then with various techniques? Do sentiment based
embeddings give better results than general embeddings?
3. What will be the impact of different embedding techniques learned in Chap. 5 on
the experiments?
4. What will be the impact of different deep learning frameworks for classification
that we learned in Chap. 6 on the experiments?
5. What will we see with other domain adaptation techniques such as CycleGAN
or CoGAN?
6. What will be the transfer loss and improvements on other source–target such as
DVD–kitchen?
7. Which of these techniques can be employed for speech recognition transfer learn-
ing problems?
References 531
References
[BDP07] John Blitzer, Mark Dredze, and Fernando Pereira. “Biographies, Bol-
lywood, boomboxes and blenders: Domain adaptation for sentiment
classification”. In: ACL. 2007, pp. 187–205.
[Bou+16] Konstantinos Bousmalis et al. “Domain Separation Networks”. In: Ad-
vances in Neural Information Processing Systems 29. Ed. by D. D. Lee
et al. 2016, pp. 343–351.
[BM10] Lorenzo Bruzzone and Mattia Marconcini. “Domain Adaptation Prob-
lems: A DASVM Classification Technique and a Circular Validation
Strategy”. In: IEEE Trans. Pattern Anal. Mach. Intell. 32.5 (May
2010), pp. 770–787.
[Che+12] Minmin Chen et al. “Marginalized Denoising Autoencoders for Do-
main Adaptation”. In: Proceedings of the 29th International Confer-
ence on International Conference on Machine Learning. ICML’12.
2012, pp. 1627–1634.
[CBG13] Sumit Chopra, Suhrid Balakrishnan, and Raghuraman Gopalan.
“DLID: Deep learning for domain adaptation by interpolating between
domains”. In: in ICML Workshop on Challenges in Representation
Learning. 2013.
[Csu17] Gabriela Csurka, ed. Domain Adaptation in Computer Vision Ap-
plications. Advances in Computer Vision and Pattern Recognition.
Springer, 2017.
[Dag+18] Ido Dagan et al. “Zero-Shot Transfer Learning for Event Extraction”.
In: Proceedings of the 56th Annual Meeting of the Association for
Computational Linguistics, ACL 2018. 2018, pp. 2160–2170.
[Fal+17] Daniele Falavigna et al. “DNN adaptation by automatic quality esti-
mation of ASR hypotheses”. In: Computer Speech & Language 46
(2017), pp. 585–604.
[FFFP06] Li Fei-Fei, Rob Fergus, and Pietro Perona. “One-Shot Learning of Ob-
ject Categories”. In: IEEE Trans. Pattern Anal. Mach. Intell. 28.4 (Apr.
2006), pp. 594–611.
[FSP15] Dmitry Kalenichenko Florian Schroff and James Philbin. “FaceNet: A
unified embedding for face recognition and clustering”. In: 2015 IEEE
Conference on Computer Vision and Pattern Recognition, CVPR 2015,
2015, pp. 815–823.
[Fu+17] Lisheng Fu et al. “Domain Adaptation for Relation Extraction with
Domain Adversarial Neural Network”. In: Proceedings of the Eighth
International Joint Conference on Natural Language Processing, IJC-
NLP. 2017, pp. 425–429.
[Gan+16b] Yaroslav Ganin et al. “Domain-adversarial Training of Neural Net-
works”. In: J. Mach. Learn. Res. 17.1 (Jan. 2016), pp. 2096–2030.
[GBB11b] Xavier Glorot, Antoine Bordes, and Yoshua Bengio. “Domain Adap-
tation for Large-Scale Sentiment Classification: A Deep Learning Ap-
proach”. In: Proceedings of the 28th International Conference on Ma-
532 11 Transfer Learning: Domain Adaptation
[NG14] Thien Huu Nguyen and Ralph Grishman. “Employing Word Repre-
sentations and Regularization for Domain Adaptation of Relation Ex-
traction”. In: Proceedings of the 52nd Annual Meeting of the Associa-
tion for Computational Linguistics, ACL. 2014, pp. 68–74.
[NG15a] Thien Huu Nguyen and Ralph Grishman. “Event Detection and Do-
main Adaptation with Convolutional Neural Networks”. In: Proceed-
ings of the 7th International Joint Conference on Natural Language
Processing of the Asian Federation of Natural Language Processing.
2015, pp. 365–371.
[NPG15] Thien Huu Nguyen, Barbara Plank, and Ralph Grishman. “Semantic
Representations for Domain Adaptation: A Case Study on the Tree
Kernel-based Method for Relation Extraction”. In: Proceedings of the
53rd Annual Meeting of the Association for Computational Linguis-
tics and the 7th International Joint Conference on Natural Language
Processing of the Asian Federation of Natural Language Processing,
ACL. 2015, pp. 635–644.
[Pal+09] Mark Palatucci et al. “Zero-shot Learning with Semantic Output
Codes”. In: NIPS. Curran Associates, Inc., 2009, pp. 1410–1418.
[Pan+11] Sinno Jialin Pan et al. “Domain Adaptation via Transfer Component
Analysis”. In: IEEE Trans. Neural Networks 22.2 (2011), pp. 199–
210.
[PS17] Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. “Train
Once, Test Anywhere: Zero-Shot Learning for Text Classification”.
In: CoRR abs/1712.05972 (2017).
[RHS17] Ievgen Redko, Amaury Habrard, and Marc Sebban. “Theoretical
Analysis of Domain Adaptation with Optimal Transport”. In: Machine
Learning and Knowledge Discovery in Databases - European Con-
ference, ECML PKDD 2017, Skopje, Macedonia, September 18–22,
2017, Proceedings, Part II. 2017, pp. 737–753.
[Ren+18] Mengye Ren et al. “Meta-Learning for Semi-Supervised Few-Shot
Classification”. In: CoRR abs/1803.00676 (2018).
[Roj+18] Lina Maria Rojas-Barahona et al. “Nearly Zero-Shot Learning
for Semantic Decoding in Spoken Dialogue Systems”. In: CoRR
abs/1806.05484 (2018).
[Sej+12] Dino Sejdinovic et al. “Equivalence of distance-based and RKHS-
based statistics in hypothesis testing”. In: CoRR abs/1207.6076
(2012).
[Ser+16] Dmitriy Serdyuk et al. “Invariant Representations for Noisy Speech
Recognition”. In: CoRR abs/1612.01928 (2016).
[SS16] Baochen Sun and Kate Saenko. “Deep CORAL: Correlation Align-
ment for Deep Domain Adaptation”. In: ECCV Workshops (3). Vol.
9915. Lecture Notes in Computer Science. 2016, pp. 443–450.
[Sun+18a] Sining Sun et al. “Domain Adversarial Training for Accented Speech
Recognition”. In: CoRR abs/1806.02786 (2018).
534 11 Transfer Learning: Domain Adaptation
[Zho+16] Guangyou Zhou et al. “Bi-Transferring Deep Neural Networks for Do-
main Adaptation”. In: Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics, ACL. 2016.
[Zhu+17] Jun-Yan Zhu et al. “Unpaired Image-to-Image Translation us-
ing Cycle-Consistent Adversarial Networks”. In: Computer Vision
(ICCV), 2017 IEEE International Conference on. 2017.
[ZR17] Yftah Ziser and Roi Reichart. “Neural Structural Correspondence
Learning for Domain Adaptation”. In: Proceedings of the 21st Con-
ference on Computational Natural Language Learning (CoNLL 2017).
2017, pp. 400–410.
[ZR18] Yftah Ziser and Roi Reichart. “Pivot Based Language Modeling for
Improved Neural Domain Adaptation”. In: Proceedings of the 2018
Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, NAACL-
HLT. 2018, pp. 1241–1251.
Chapter 12
End-to-End Speech Recognition
12.1 Introduction
into an acoustic model, lexicon model, and language model by using Bayes’ the-
orem. This approach relies heavily on the use of the conditional independence as-
sumption and separate optimization procedures for the different models.
Deep learning was first incorporated into the statistical framework by replacing
the Gaussian mixture models to predict phonetic states based on the observations.
One of the drawbacks of this approach is that the DNN/HMM hybrid models rely
on training each component separately. As seen previously in other scenarios, the
separate training process can lead to sub-optimal results, due to the lack of error
propagation between models. In ASR, these drawbacks tend to manifest themselves
as sensitivity to noise and speaker variation. Applying deep learning for end-to-
end ASR allows the model to learn from the data instead of relying on heavily
engineered features, allowing the models to learn from the data directly. Thus, there
have been some approaches to train ASR models in an end-to-end fashion. End-
to-end methods instead try to optimize the quantity P(W |X) directly, rather than
separating it.
With end-to-end modeling, the input–target pair need only be the speech utter-
ance and the linguistic representation of the transcript. Many representations are
possible: phones, triphones, characters, character n-grams, or words. Given that
ASR focuses on producing word representations from the speech signal, words are
the more obvious choice; however, there are some drawbacks. The vocabulary size
requires large output layers, as well as examples of each word in training, leading
to much lower accuracies than other representations. More recently, end-to-end ap-
proaches have moved towards using characters, character n-grams, and some word
models as well given enough data. These data pairs can be easier to produce, al-
leviating the requirements for linguistic knowledge when creating phonetic dictio-
naries. Jointly optimizing the feature extraction and sequential components together
provides numerous benefits, specifically: lower complexity, faster processing, and
higher quality.
The key component to accomplishing end-to-end ASR requires a method of re-
placing the HMM to model the temporal structure of speech. The most common
methods are CTC and attention. In this chapter, the components of traditional ASR
are substituted with end-to-end training and decoding techniques. We begin by in-
troducing CTC, a method for training unaligned sequences. Next, we explore some
architectures and techniques that have been used to train end-to-end models. We
then review attention and how to apply it to ASR networks and some of the architec-
tures that have been trained with these techniques. Following attention, we discuss
multitask networks trained with both CTC and attention. We explore common de-
coding techniques for CTC and attention during inference, incorporating language
models to improve prediction quality. Lastly, we discuss embedding and unsuper-
vised techniques and then end with a case study, incorporating both a CTC and an
attention network.
DL-HMM models rely on an alignment of the linguistic units to the audio signal to
train the DNN to classify as phonemes, senones, or triphone states (plainly stated,
this sequence of acoustic features should yield this phone). Manually obtaining these
alignments can be prohibitively expensive for large datasets. Ideally, an alignment
would not be necessary for an utterance-transcript pair. Connectionist temporal clas-
sification [Gra+06] was introduced to provide a method of training RNNs to “label
unsegmented sequences directly,” rather than the multistep process in the hybrid
mode.
Given an acoustic input X = [x1 , x2 , . . . , xT ] of acoustic features with the desired
output sequence Y = [y1 , y2 , . . . , yU ]. An accurate alignment of X to Y is unknown,
and there is additional variability in the ratio of the lengths of X and Y , typically with
T U (consider the case where there is a period of silence in the audio, yielding a
shorter transcript).
How can possible alignments be constructed for a (X,Y ) pair? A simple theo-
retical alignment, as shown in Fig. 12.1, illustrates a potential method where each
input xt has an output assigned to it, with repeated outputs combined into a single
prediction.
This alignment approach has two problems: First, in speech recognition, the input
may have periods of silence that do not directly align with the output assigned.
Second, we can never have repeated characters (such as the two “l”s in “hello”),
because they are collapsed together into a single prediction.
The CTC algorithm alleviates the issues of this naive approach by introducing a
blank token that acts as a null delimiter. This token is removed after collapsing re-
12.2 Connectionist Temporal Classification (CTC) 539
Fig. 12.1: Naive alignment for an input X and with a length of 6 and an output
Y = [c, a,t]. Example from [Han17]
peated predictions, allowing repeated sequences and periods of “silence.” Thus, the
blank token is not included in the loss computation or decoding; however, it allows
a direct alignment between the input and the output without forcing a classification
to the output vocabulary. Note that the blank token is separate from the space token
that is used to denote the separation of words. Figure 12.2 shows an example of the
CTC alignment.
Fig. 12.2: CTC alignment for an input X and output Y = [h, e, l, l, o, , w, o, r, l, d].
Note that the blank token is represented by “ε ” and the space character is represented
by the underscore “ ”
With this output representation, there is a 1:1 alignment between the lengths of
the input sequence and output sequence. Furthermore, the introduction of the blank
token implies that there can be many predicted alignments that lead to the same
output. For example:
[h, e, l, ε , l, o, ε ] → “hello”
[h, ε , e, l, ε , l, o] → “hello”
Because any token in the output can have an ε before or after, we can imagine
the desired output sequence having an ε before and after each label.
Y = [ε , y1 , ε , y2 , . . . , ε , yU ]
Multiple paths/alignments can yield a correct solution, and therefore, all correct
solutions must be considered. The CTC algorithm itself is “alignment-free”; how-
540 12 End-to-End Speech Recognition
Fig. 12.3: Valid CTC paths for the target sequence, Y = [a, b]. Notice that the blank
token, ε , is removed from the final sequence. Therefore, there are two possible initial
states, ε and ja, and two possible final states, ε and b. Additionally, to achieve final
output, the transition from epsilon must be to itself or the next token in the sequence,
while the transition from a could be to itself, ε , or b
All paths are considered mutually exclusive, so we sum the probability of all
alignments, giving the conditional probability for a single utterance (X,Y ):
T
P(Y |X) = ∑ ∏ P(αt |X) (12.3)
A∈AX,Y t=1
where AX,Y is the set of valid alignments. Dynamic programming is used to improve
the computation of the CTC loss function. By supplying blank tokens around each
label in the sequence, the paths can be easily comparable and merged when they
reach the same output at the same time step.
Combining everything gives the loss function for CTC.
T
LCTC (X,Y ) = − log ∑ ∏ P(at |X) (12.4)
a∈AX,Y t=1
12.2 Connectionist Temporal Classification (CTC) 541
The gradient for backpropagation can be computed for each time step from the
probabilities at each frame.
CTC assumes conditional independence between each time step in that the output
at each time step is independent of the previous time steps. Although this property
allows for frame-wise gradient propagation, it limits the ability to learn sequential
dependencies. Using a language model (Sect. 12.5.2) alleviates some of the issues,
by providing a word or n-gram context.
CTC was initially successful on the TIMIT [ZSG90] phoneme recognition task
[GMH13]. Various architectures, trained with CTC, were explored yielding state-of-
the-art performance on the task. The architecture mapped Mel filter-bank features
to the phonetic sequence with a single end-to-end network. The authors explored
unidirectional and bidirectional RNNs. A stacked, bidirectional LSTM architecture
provided the best results. Bidirectional RNNs seemed to allow the network to lever-
age the context of the whole utterance, rather than the forward only context.
The authors used two regularization techniques in the training of this network:
weight noise and early stopping. Weight noise adds Gaussian noise to the weights
during training to reduce overfitting to specific sequences. These regularization tech-
niques turned out to be crucial to the training of the network.
Fig. 12.4: RNN model used in the original Deep Speech paper. The architecture
incorporates a single bidirectional LSTM layer after three fully connected layers
that lean features on the input spectrogram
Given the size of the dataset and computational requirements of the architecture,
multiple GPUs were needed to facilitate training. The Deep Speech work was pivotal
in overcoming many engineering challenges, such as how to train on large datasets.
Many contributions of the paper focused on scaling the training of the architecture
on multiple GPUs. Two types of parallelism were used to train the models across
multiple GPUs: data parallelism and model parallelism. Data parallelism focuses on
retaining a copy of the architecture on each GPU, splitting a large training batch
12.2 Connectionist Temporal Classification (CTC) 543
across the separate GPUs, performing the forward and backward propagation steps
on the separate data, and finally aggregating the gradient updates for all of the mod-
els. Data parallelism provides near linear scaling with the number of GPUs (it may
impact the convergence rate, due to the effective batch size). The second type of
parallelism is model parallelism. Model parallelism focuses on splitting the model’s
layers and distributing the layers across the set of available GPUs. Incorporating
model parallelism can be difficult when working with recurrent neural networks,
due to their sequential nature. In the Deep Speech architecture, the authors achieved
model parallelism by splitting the model in half along the time dimension. These de-
cisions allowed the authors to train on 5000 h of audio and achieve state-of-the-art
results on two noisy speech benchmarks.
In Deep Speech 2 (DS2) [Amo+16], a follow-on paper to Deep Speech, the authors
extended the original architecture to perform character-based, end-to-end speech
recognition. The authors validated the modeling techniques on both English and
Mandarin Chinese transcription. The Deep Speech 2 modifications introduced many
improvements to the original architecture, as well as engineering optimizations
achieving 7× speedup over the original Deep Speech implementation. Figure 12.5
shows the updates architecture.
Fig. 12.5: The Deep Speech 2 architecture incorporated convolutional layers that
learned features from utterance spectrograms and significantly increased the depth
544 12 End-to-End Speech Recognition
The main difference in the Deep Speech and Deep Speech 2 architectures is the
increase in depth. In the Deep Speech 2 work, many different architectures were
explored, varying the number of convolutional layers between 1 and 3 and the num-
ber of recurrent layers from 1 to 7. The optimal DS2 architecture for English tran-
scription included 11 layers (3 convolutional, 7 bidirectional recurrent, and 1 fully
connected layer). Batch normalization is incorporated after each layer (apart from
the fully connected layer), and gradient clipping was also included to improve con-
vergence. The overall architecture contained approximately 35 million parameters.
With the incorporation of an n-gram language model, this leads to a 43.4% relative
improvement in WER over the already competitive DS1 architecture.
Other key components in the improvements of Deep Speech 2 were training tech-
niques and further increasing the dataset size. Training can be unstable in the early
stage of CTC models. The authors use a training curriculum to improve the stability
of the model when training. By first selecting the shorter utterances, the model can
benefit from smaller gradient updates in the earlier part of the first epoch. Addition-
ally, the authors increased the size of the dataset to 12, 000 h in Deep Speech 2. They
note that scaling the data decreases the WER by 40% for each factor of 10 increase
in the training set size.
12.2.4 Wav2Letter
Fig. 12.6: Wav2Letter architecture for recognition on a raw waveform. The first
layer is not included when training on MFCCs instead of the raw waveform. The
convolutional parameters are organized as (kw, dw, dim ratio), where kw is the ker-
nel width, dw is the kernel stride, and dim ration is the number of input dimensions
to the number of output dimensions
on Wav2Letter++ [Pra+18], the authors improved the speed of the ASR system,
achieving linear scaling for training times (up to 64 GPUs).
12.2.5.1 Gram-CTC
Gram-CTC [Liu+17] extended the CTC algorithm to address the fixed alphabet and
fixed target decomposition. This approach focused on learning to predict the char-
acter n-grams rather than single characters, allowing the model to output multiple
546 12 End-to-End Speech Recognition
characters at a given time step. Using character n-grams can mildly alleviate the ef-
fects of the frame independence assumption, due to the need to learn multiple labels
together.
The work also experimented with automatically learning the character n-grams
(referred to as “grams”) during the training process, leveraging the forward–
backward algorithm. Although it is feasible for both the grams and transcription to
be learned jointly, the model needs to learn the alignment and decomposition of the
target in tandem, and the training becomes unstable. Multitask learning is used to
combat this instability by jointly optimizing CTC as well as Gram-CTC. Overall,
the incorporation of grams resulted in improvements across multiple datasets, even
when the grams were manually selected.
The RNN transducer [Gra12] extends CTC by assuming a local and monotonic
alignment between the input and output sequences. This approach alleviates the
conditional independence assumption of CTC by incorporating two RNN layers
that model the dependencies between outputs at different time steps.
where ut signifies the output time step aligned to the input time step t. T is the length
of the alignment sequence including the number of blank labels predicted. Note that
y1:u is the sequence of predictions excluding blanks up to time step u. The RNN
incorporates the full history of the non-blank labels into the CTC prediction at the
next time step. Training the RNN-T model requires using the forward–backward
algorithm to compute the gradients (similar to the CTC computation). In online
speech recognition, a unidirectional RNN can be used to model the dependencies
between time steps in the forward direction.
12.3 Seq-to-Seq
Instead, the attention-based approach is used and can model the probability of
the output sequence directly:
U
P(Y |X) = ∏ P(yu |y1:u−1 , X) (12.6)
u=1
ht = Encoder(X)
ContentAttention(qu−1 , ht )
aut =
LocationAttention({au−1 }t=1
T ,q
u−1 , ht )
(12.7)
T
cu = ∑ aut ht
t=1
P(yu |y1:u−1 , X) = Decoder(cu , qu−1 , yu−1 )
Content-based attention learns a weight vector g and two linear layers,W and V
(without bias parameters), to weigh the previous prediction and the encoder hidden
state ht . This is represented as follows:
{ft }t=1
T
= K ∗ au−1 (12.10)
where ∗ represents the one-dimensional convolution operation over the time axis
t with convolutional matrix K. A linear layer U is also learned to map the output
548 12 End-to-End Speech Recognition
The listen, attend, and spell (LAS) network [Cha+16b] used a pyramid BiLSTM to
encode the input sequence, referred to as the listener. The decoder was an attention-
based RNN to predict characters.
The drawback for seq-to-seq models is that they tend to be more difficult to train
(more so than CTC) and slower during inference. The decoder cannot predict until
the attention mechanism has weighed all of the previous hidden states for each new
time step. Some techniques have been introduced to deal with this, such as window-
ing mechanisms to reduce the number of time steps considered during decoding and
label smoothing, which prevent overconfidence in predictions.
One of the other difficulties for seq-to-seq models is that they cannot be used in a
full online streaming fashion. The entire context must be encoded before decoding
can begin.
In [VDO+16], the Wav2Text architecture used a CNN-RNN model with at-
tention to predict character-based transcripts directly on the raw waveform. The
encoder is a convolutional architecture combined with two bidirectional LSTMs,
12.4 Multitask Learning 549
and the decoder is a single bidirectional LSTM. The convolutional layers are used
mainly to reduce the dimensionality of the input. Due to the additional complex-
ity of attention and the utilization of the raw waveform, the network was trained via
transfer learning. Initially, only the lower encoder layers predict the spectral features
(MFCC and log Mel-scale spectrogram) as the target from the raw input waveform.
The network is then trained with these features through the attention-based encoder–
decoder with CTC to produce a transcript.
Many of the drawbacks of attention and CTC led to multitask learning approaches.
Attention usually performs better in end-to-end scenarios; however, it typically has
difficulties converging and tends to suffer in noisy environments. CTC, on the other
hand, usually yields lower quality due to the conditional independence assumption,
but is more stable. The trade-offs between CTC and attention make their combina-
tion highly valuable via multitask learning. ESPnet [KHW17, Xia+18] was trained
to do just this: jointly optimizing an attention-based encoder–decoder model with
CTC and attention.
The training loss for ESPnet is a multi-objective loss (MOL) defined as:
∗
LMOL = λ log Pctc (C|X) + (1 − λ ) log Patt (C|X) (12.13)
550 12 End-to-End Speech Recognition
Fig. 12.8: Attention-based end-to-end ASR model from [Cha+16b], using a pyra-
mid LSTMs in the encoder
where λ is the weight for each loss function and 0 ≤ λ ≤ 1. Pctc is the CTC objective
∗ is the attention objective.
and Patt
The ESPnet architecture uses a 4-layer bidirectional LSTM encoder and a 1-layer
LSTM decoder. To reduce the number of time steps of the output, the top two layers
of the encoder read every second state, which reduces the length of the output h by
a factor of 4.
The decoding process can be extended by providing a prior of the language in the
form of a language model. These language models can be trained on large volumes
of text data to accurately bias predicted transcripts to particular domains.
12.5.1.1 N-gram
In the Deep Speech 2 paper, the authors experimented with n-gram language mod-
els. Although the RNN layers included in the architecture learn an implicit language
model, it tends to err on homophones and spelling of certain words. Therefore, an n-
gram language model was trained using the KenLM [Hea+13] toolkit on the Com-
mon Crawl Repository,1 using the 400,000 most frequent words from 250 million
lines of text.
1 http://commoncrawl.org.
552 12 End-to-End Speech Recognition
RNN language models have surfaced various times in this book. The application of
RNN language models relies on utilizing the likelihood of the next word to predict
the most likely sequence of words given the previous word.
These models can be incorporated as an additional score during the beam decod-
ing in the same way as the n-gram language model or as a rescoring of the top n
hypotheses.
Word-based models suffer from the OOV issue, but they have successfully beaten
phoneme-based CTC models when trained on very large datasets (125 kh) [SLS16].
This limitation has prompted research on incorporating character-based prediction
when encountering OOV terms [Li+17].
Decoding a CTC network (a deep learning network trained with CTC) refers to find-
ing the most probable output for the classifier at inference time, similar in spirit to
HMM decoding. Mathematically, the decoding process is described by the function
h(x):
h(x) = argmax P(l|x) (12.15)
l∈L≤T
h(x) = B(π ∗ )
π ∗ = argmax p(π |x) (12.16)
π ∈N t
12.5 End-to-End Decoding 553
The beam search algorithm can be extended with an n-gram language model. A
simple approach is to rescore the word sequence each time an end-of-word (space)
token is reached. However, this relies on the model to predict full words with no
misspellings.
A better approach is to use prefix search decoding, which incorporates the sub-
word level information during the decoding process, utilizing the prefixes of the
language model. Converting a word-level language model to a “label-level” or
character-based model is accomplished by representing the output sequence as the
concatenation of the longest completed word sequence and the remaining word pre-
fix, denoted as w and p, respectively. The function for computing the probability of
the next label given the current sequence becomes:
This approach also requires length normalization, to prevent a bias towards se-
quences with fewer transitions.
Attention decoding already produces the most probable sequence given the previous
predictions. Therefore, as seen previously, greedy decoding could be applied here,
yielding the most probable character at each time step. However, it likely would not
yield the most probable sequence Ĉ.
A beam search can also be applied to attention models during the decoding process.
Because the previous time step is provided as an input to the next prediction, the top
n most probably paths at each time step can be retained at each time step. The beam
search begins by first considering the start of sentence symbol, < s >.
Shallow fusion (used originally for NMT) combines the scores of the LM and ASR
models during the decoding [Gul+15]. This type of language model decoding incor-
porates an external language model during the beam search to incorporate word or
character probabilities into consideration. Shallow fusion can be used with a word
or character-based language models to determine the probability of a particular se-
quence.
Y ∗ = argmax log P(Y |X) + λ PLM (Y ) (12.20)
Y
Character language models are helpful for rescoring hypotheses before a word
boundary is reached or as a rescoring mechanism for character-based languages,
such as Japanese and Mandarin Chinese. Additionally, character-based language
models can predict unseen character sequences, which a word-based model would
not allow.
556 12 End-to-End Speech Recognition
Fig. 12.10: Beam search decoding example with a beam size of 2 on a three-
character alphabet {a, b, c}. With attention decoding, the previous time step is incor-
porated into the next character prediction. Therefore the probabilities are dependent
on the path chosen. The best paths at each time step are highlighted, with the darker
one being the top prediction. Note how the greedy decoding of this example would
yield a sub-optimal result
Shallow fusion has been incorporated into RNN-T models, allowing the CTC
training to alleviate the frame independence while also incorporating the language
model bias into the prediction [He+18].
When incorporating neural language models into end-to-end ASR, it is quickly ap-
parent that the two could be optimized together, leveraging the acoustic information
as well as the linguistic information from large text corpora. The two most popular
techniques for jointly training the acoustic and language models are deep fusion and
cold fusion.
Deep fusion [Gul+15] on the other hand incorporates the LM into the acous-
tic model (specifically an encoder–decoder model), creating a combined network.
12.5 End-to-End Decoding 557
gt = σ (vTg stLM + bg )
stDF = [ct ; st ; gt stLM ] (12.21)
P(yt |h,Y1:(t−1) ) = softmax(WDF stDF + bDF )
where ct is the context vector, h is the output of the encoder, and vg , bg , bDF , and
WDF are all learned during the continued training phase.
Cold fusion [Sri+17] extends the idea of deep fusion, incorporating the LM into
the training procedure. However, in cold fusion the acoustic model is trained from
scratch incorporating the pre-trained LM.
stLM = DNN(dtLM )
stED = WED [dt ; ct ] + bED
gt = σ (Wg [stED ; stLM ] + bg )
(12.22)
t = [st ; gt ◦ st ]
sCF ED LM
rCF CF
t = DNN(st )
P(yt |h,Y1:(t−1) ) = softmax(WCF rCF
t + bCF )
Because cold fusion incorporates the LM into the training process from the be-
ginning, retraining is required if there are changes in the LM. The original paper
introduces a means for switching language models by using LM logits instead of
the LM hidden states; however, this does increase the number of learned parameters
and computation.
12.5.5.1 Rescoring
Rescoring relies on a two-step method. The first step is to produce a set of complete
hypotheses from the attention decoder. The second step is to rescore these hypothe-
ses based on the CTC and attention probabilities (the forward algorithm is used to
get the CTC probabilities).
One-pass decoding, on the other hand, focuses on computing the probabilities of the
partial hypotheses as characters are generated.
A language model can also be incorporated into the decoding process [HCW18]
by adding an additional language modeling term to the decoder:
Ĉ = argmax{λ log PCTC (C|X) + (1 − λ ) log PAT T (C|X) + γ log PLM (C)} (12.24)
C∈U ∗
P(h, . . . |X) =
0
∑ P(h; v|X) (12.27)
v∈(U <EOS>)+
The amount of unsupervised data that is available can be orders of magnitude higher
than the amount of paired speech-text parallel corpora. Thus, unsupervised speech
recognition and acoustic embeddings for audio processing are promising areas of
research.
One of the earliest works for embeddings in speech was [BH14]. In this work, the
authors used a form of Siamese network to train acoustic word embeddings where
similar sounding words (acoustically similar) are clustered near each other in the
embedding space. In this fashion “words are nearby if they sound alike.” By mod-
eling words directly, the paradigm of speech recognition shifts away from trying to
model states in the traditional HMM.
This network is trained in two parts: initially, a CNN classification model is
trained to classify spoken words in a fixed segment of audio (2 s). Second, this net-
work is fixed and incorporated in a word embedding network. The word embedding
network is trained to align the embedding of the correct word with the acoustic
embedding while separating wrong words. To reduce the size of the input embed-
ding space from all words by using bag-of-letter-n-grams, only the top 50,000 letter
n-grams are used to reduce the size of the input embedding space (bag-of-letter-n-
grams). The architecture diagram is shown in Fig. 12.11.
Fig. 12.11: Acoustic embedding model trained with a triplet ranking loss to align
acoustic vectors and word vectors from subword units
560 12 End-to-End Speech Recognition
12.6.2 Unspeech
In Unspeech [MB18], the authors used a Siamese network to train embeddings used
with acoustic models for speaker adaptation, utterance clustering, and speaker com-
parison. This work relies on the assumption that similar areas of speech are likely
to have the same speaker. The contexts for true and false examples of speakers are
taken from neighboring contexts windows in the same utterance or from separate
files. This idea is similar to the concept of negative sampling. This network, there-
fore, does not expect similar words to be in the same embedding space, but rather
the same speaker. The architecture is shown in Fig. 12.12.
One of the drawbacks to the CNN approach is that it requires fixed-length audio
segments. Audio Word2Vec [Chu+16] used a sequence-to-sequence autoencoder to
learn a fixed representation for variable-length spoken words. Because the learned
representation is the input itself, it can be learned in a completely unsupervised
way, hence the reference to word2Vec. The resulting model is, therefore, able en-
code acoustic samples for use in a query-by-example system for words. Training
the model does not require supervision; however, creating the word embeddings
requires knowledge of word boundaries in the embedding process.
Audio Word2Vec was extended in [WLL18] to utterance level by learning a seg-
mentation method as well. The method is an example of a segmental sequence-to-
sequence autoencoder (SSAE). The SSAE learns segmentation gate to determine
the word boundaries in the utterance and a sequence-to-sequence autoencoder that
learns an encoding for each segment. Some guidance is needed to keep the autoen-
coder from splitting the utterance into too many embeddings; however, learning an
appropriate estimate is not differentiable. Reinforcement learning is used to estimate
this quantity, due to the non-differentiability of learning a discrete variable.
12.7 Case Study 561
Fig. 12.12: Unspeech embeddings are trained using a Siamese CNN network
(VGG16A), to compute embedding vectors. The dot product of the two vectors is
computed, and a logistic loss is used to optimize a binary classification task, of
whether the context window was a true or false context window of the target
In this case study, we continue to focus on building ASR models on the Mozilla
Common Voice dataset.2 In this chapter, we focus specifically on a Deep Speech
2 model that trains an end-to-end network with CTC and a hybrid attention-CTC
model.
Since the release of the Deep Speech 2 paper, there have been multiple open sourced
implementations of the architecture, with the most common difference being the
deep learning framework used. The most popular are the TensorFlow implemen-
tation by Mozilla,3 the PaddlePaddle implementation,4 and the PyTorch version.5
Each has a variety of benefits and drawbacks, some of which are the deep learning
2 https://voice.mozilla.org/en/data.
3 https://github.com/mozilla/DeepSpeech.
4 https://github.com/PaddlePaddle/DeepSpeech.
5 https://github.com/SeanNaren/deepspeech.pytorch.
562 12 End-to-End Speech Recognition
The data preparation requires either a directory structure or manifest file. In the first
approach, a dataset directory is structured as follows (Figs. 12.13, 12.14, and 12.15).
There is no additional need for phonetic dictionaries for character-based mod-
els; the data is processed into a spectrogram and then converted to a tensor at data
loading time.
In this implementation, one can also use a “manifest” file to define the datasets
used. The manifest is similar to the Kaldi and Sphinx structures, containing a list of
the examples in each dataset split. Manifest files can be useful for filtering longer
files when using variable-length RNNs.
First, we train a base model given the default configuration. The resulting model has
two convolutional layers and five bidirectional GRU layers, yielding approximately
41 million learnable parameters. We enable the augmentation step during training
as well, which applies small changes to the tempo and gain to reduce overfitting.
6 https://github.com/espnet/espnet.
12.7 Case Study 563
1 / common voice
2 / train
3 txt /
4 train sample000000 . txt
5 train sample000001 . txt
6 ...
7 wav /
8 train s a m p l e 0 0 0 0 0 0 . wav
9 train s a m p l e 0 0 0 0 0 1 . wav
10 ...
11 / val
12 txt /
13 ...
14 wav /
15 ...
16 / test
17 txt /
18 ...
19 wav /
20 ...
21
1 / p a t h / t o / t r a i n s a m p l e 0 0 0 0 0 0 . wav , / p a t h / t o / t r a i n s a m p l e 0 0 0 0 0 0 . t x t
2 / p a t h / t o / t r a i n s a m p l e 0 0 0 0 0 1 . wav , / p a t h / t o / t r a i n s a m p l e 0 0 0 0 0 1 . t x t
3 ...
4
Fig. 12.14: Manifest structure for the training set for Deep Speech 2
1 p y t h o n t r a i n . py −−t r a i n −m a n i f e s t d a t a / t r a i n m a n i f e s t . c s v −−v a l −
manifest data / v a l m a n i f e s t . csv
2
We train all models on a GPU,7 with early stopping based on the WER of the
validation set. In our case, the model began diverging after about 15 epochs, as
shown in Fig. 12.16 and achieves its best validation WER of 23.470. Once the model
is trained we evaluate the best model on the test set, where we achieve an average
WER of 22.611 and CER of 7.757, using greedy decoding. A few samples of the
greedy decoding of the trained model are shown in Fig. 12.17.
7 Although it is possible to train this model on a CPU, it is unrealistic due to the computationally
intensive nature of the convolutional and recurrent layers.
564 12 End-to-End Speech Recognition
Fig. 12.16: Training curve of Deep Speech 2 with the default configuration
1 Ref : i u n d e r s t a n d s h e e p t h e y ’ r e no l o n g e r a p r o b l e m and t h e y c a n
be good f r i e n d s
2 Hyp : i u n d e r s t a n d s h e e t h e y ’ r e no l o n g e r y p r o b l e m and t h e y c a n be
good f r i e n d s
3 WER: 0 . 2 1 4 CER : 0 . 0 2 7
4
5 Ref : a s he l o o k e d a t t h e s t o n e s he f e l t r e l i e v e d f o r some r e a s o n
6 Hyp : a s h e l o o k e d a t t h e s t o n e s he f e l t r e l i e v e d f o r som a s o n
7 WER: 0 . 3 3 3 CER : 0 . 0 5 1
8
Fig. 12.17: Output from the base Deep Speech 2 model. Note how many of the
mistakes seem phonetic and create nonlogical words, such as shee and ashe
8 https://github.com/parlance/ctcdecode.
12.7 Case Study 565
implementation. One thing to note about this language model is that it incorporates
a character FST as well. The FST acts as a spell checker, enforcing the production
of words.
Decoding schemes can be applied to improve the error rates of the predictions.
These results are summarized in Table 12.1.
The KenLM toolkit [Hea+13] is used to train an n-gram language model. The
language model is created from transcripts of the training corpus to provide com-
parable results to previous case studies. In practice, language models are usually
trained on very large training corpora such as the previously mentioned, Common
Crawl (Fig. 12.18).9
Fig. 12.18: Train a 2-gram language model with KenLM on the training transcripts.
The first command creates an ARPA language model from the transcripts, and the
second command creates a binary trie-structure from the language model used in
the decoding phase
We determine the best language model for the system by evaluating them on the
validation set, and the best model is chosen to apply to the testing set. Table 12.1
summarizes the WER and CER for different language models.
Table 12.1: Validation results for different decoding methods. The best results are
in bold
After applying the language model with the default beam size (beamwidth = 10),
we see that our best model is the 4-gram model. Now, we can increase the size of
the beam to evaluate the impact on the predictions. The results are summarized in
Table 12.2.
9 http://commoncrawl.org.
566 12 End-to-End Speech Recognition
Table 12.2: Validation results for different beam sizes. The best results are in bold
1 Ref : i u n d e r s t a n d s h e e p t h e y ’ r e no l o n g e r a p r o b l e m and t h e y c a n
be good f r i e n d s
2 Hyp : i u n d e r s t a n d s h e e p t h e y ’ r e no l o n g e r a p r o b l e m and t h e y c a n
be good f r i e n d s
3 WER: 0 . 0 CER : 0 . 0
4
5 Ref : a s he l o o k e d a t t h e s t o n e s he f e l t r e l i e v e d f o r some r e a s o n
6 Hyp : a s he l o o k e d a t t h e s t o n e s he f e l t r e l i e v e d f o r some a s
7 WER: 0 . 0 8 3 CER : 0 . 0 6 8
8
9
Fig. 12.19: Test output with language model decoding. Note many of the phonetic
mistakes are corrected when incorporating the language model during the decoding;
however, it can also cause different mistakes. In the second example, greedy decod-
ing output ason instead of reason, but after the application of the language model,
the hypothesis reduced this to as, reducing the WER and increasing the CER for this
example
The computation time increases linearly with the beam size. In practice, it is best
to choose a beam size that is a good trade-off between performance and quality.
After applying our best LM (4-gram) with a beam size of 512 to the test set, we
achieve a WER of 5.587 and CER of 3.232. Some examples of the decoded output
are in Fig. 12.19.
12.7.4 ESPnet
10 https://github.com/espnet/espnet.
12.7 Case Study 567
similar to Kaldi, with Chainer and PyTorch backends. In this portion of the case
study, a hybrid CTC–attention architecture is trained on the Common Voice dataset,
using the ESPnet toolkit.
The data preparation is very similar to Kaldi, with a reliance on Kaldi for some of the
preprocessing. The main difference is the lack of phonetic lexicons and dictionaries
required in Kaldi. We generate MFCC features and store them in a JSON format.
This format contains the target transcript, the tokenized transcript, location of the
features, and some additional information for various components of the training.
An example of the formatted training data is shown in Fig. 12.20.
After extracting the features and creating the input file, the network is ready to
train.
The model training procedure also follows the Kaldi scripts to some degree; how-
ever, once the features are extracted, we run the training scripts.
The model trained is a 4-layer bidirectional LSTM encoder and a 1-layer uni-
directional LSTM decoder. We train this model with Adadelta for 20 epochs on a
single GPU. The full list of training arguments is shown in Fig. 12.21.
During the training procedure the losses for both CTC and attention can be mon-
itored to ensure that there is consistency in the convergence. The overall loss for
training and validation is the weighted sum of the two components. We also notice
that the validation loss trends with the training data loss until the final epoch. In this
example, we set a hard stop for computational reasons on the number of epochs run.
To obtain our best model, we would continue training until the validation consis-
tently diverges from the training loss, and choose the model that performs best on
the validation data (Fig. 12.22).
The accuracy curves, shown in Fig. 12.23, display the network performance
as training progresses. The first two epochs show significant gains in the early
stages, with modest improvements as training progresses. Our best model in training
achieves a WER of 12.07 on the validation data.
We can inspect the output attention weights during the decoding process by plot-
ting the weight of each time step during the decoding. Visualizing attention, as be-
fore, shows what portion of the input is attended to during inference. This is shown
in Fig. 12.24. We notice that the output generally correlates with the input audio file,
yielding an aligned output that is capable of segmenting the audio, as well as deal-
ing with the offsets in time. During the early stages, we notice some breaks in the
attention alignments to the input, whereas in the latter case, the attention alignments
appear seamlessly aligned to the input.
568 12 End-to-End Speech Recognition
1 {
2 ” utts ”: {
3 ” cv−v a l i d −dev−sample −000000”: {
4 ” input ”: [
5 {
6 ” f e a t ”: ”. valid dev / d e l t a f a l s e / f e a t s . 1 . ark
:27” ,
7 ” name ” : ” i n p u t 1 ” ,
8 ” shape ” : [
9 502 ,
10 83
11 ]
12 }
13 ],
14 ” output ”: [
15 {
16 ” name ” : ” t a r g e t 1 ” ,
17 ” shape ” : [
18 55 ,
19 31
20 ],
21 ” t e x t ” : ”BE CAREFUL WITH YOUR PROGNOSTICATIONS
SAID THE STRANGER” ,
22 ” t o k e n ” : ”B E <s p a c e > C A R E F U L <s p a c e > W I T
H <s p a c e > Y O U R <s p a c e > P R O G N O S T I C A T I O N S <
s p a c e > S A I D <s p a c e > T H E <s p a c e > S T R A N G E R” ,
23 ” t o k e n i d ” : ”5 8 3 6 4 21 8 9 24 15 3 26 12 23 11
3 28 18 24 21 3 19 21 18 10 17 18 22 23 12 6 4 23 12 18 17 22
3 22 4 12 7 3 23 11 8 3 22 23 21 4 17 10 8 21”
24 }
25 ],
26 ” u t t 2 s p k ” : ” cv−v a l i d −dev−sample −000000”
27 },
28 ...
29 }
30
Fig. 12.23: Training and validation accuracy curves for the model
Our base model achieves a WER of 12.34 and CER of 6.25 on the testing set with
greedy decoding (beam size of 1). When incorporating a beam search of 20 (ESPnet
default selection) into the predictions on the testing set, we reduce the WER to 11.56
and CER to 5.80. We leave tuning the beam size and incorporating a language model
as an exercise. Note the significant improvement when we added this to the Deep
Speech 2 architecture.
570 12 End-to-End Speech Recognition
10
Decoder Index
15
20
25
30
35
0 25 50 75 100 125 150 175 200
Encoder Index
(a)
0
10
Decoder Index
15
20
25
30
35
0 25 50 75 100 125 150 175 200
Encoder Index
(b)
Fig. 12.24: Attention weights for a single file on the input audio after the (a) 1st
epoch and (b) after the 20th epoch
12.7.5 Results
We now provide a summary of the techniques evaluated in this case study. The
testing results are displayed in Table 12.3.
References 571
Table 12.3: End-to-end speech recognition performance on Common Voice test set
(best result highlighted)
Approach WER
Deep Speech 2 (no decoding) 22.83
Deep Speech 2 (4-gram LM, beam size of 512) 5.59
ESPnet (no decoding) 12.34
ESPnet (no LM, beam size of 20) 11.56
Kaldi TDNN (Chap. 8) 4.44
Overall, with a CTC–attention model, we get faster, more stable convergence and
a lower WER for the base acoustic model compared to the Deep Speech 2 baseline
(WER of 22.83).
Although this result is not better than the one achieved with Kaldi in Chap. 8 case
study, the results between the Deep Speech 2 (with a language model) and Kaldi
models are comparable, even without a lexicon model. The training procedure is
more straight-forward than the training steps required for the traditional approaches
to ASR, such as removing the requirement of iterative training and aligning. Ad-
ditional benefits can also be gained from the inclusion of a language model during
decoding to provide compelling results without significant linguistics resources.
Some other interesting problems readers and practitioners can try on their own in-
clude:
1. What changes are required to train a Deep Speech 2 model on a new language?
2. What would be the effects of training a language model on more data?
3. Would the incorporation of the testing transcripts improve the results on the val-
idation data? What about the testing data?
4. Does the incorporation of the testing transcripts in the language model corrupt
the validity of the results?
5. How could an RNN language model be incorporated into the decoding process
for Deep Speech 2? For ESPnet?
6. Perform a grid search for the beam size on the ESPnet model.
References
[BCB14a] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. “Neural ma-
chine translation by jointly learning to align and translate”. In: arXiv
preprint arXiv:1409.0473 (2014).
[Bah+16c] Dzmitry Bahdanau et al. “End-to-end attention-based large vocabu-
lary speech recognition”. In: Acoustics, Speech and Signal Processing
(ICASSP), 2016 IEEE International Conference on. IEEE. 2016, pp.
4945–4949.
[BH14] Samy Bengio and Georg Heigold. “Word embeddings for speech
recognition”. In: Fifteenth Annual Conference of the International
Speech Communication Association. 2014.
[Cha+16b] William Chan et al. “Listen, attend and spell: A neural network for
large vocabulary conversational speech recognition”. In: Acoustics,
Speech and Signal Processing (ICASSP), 2016 IEEE International
Conference on. IEEE. 2016, pp. 4960–4964.
[Cho+14] Kyunghyun Cho et al. “Learning phrase representations using RNN
encoder-decoder for statistical machine translation”. In: arXiv preprint
arXiv:1406.1078 (2014).
[Cho+15c] Jan K Chorowski et al. “Attention-based models for speech recogni-
tion”. In: Advances in neural information processing systems. 2015,
pp. 577–585.
[Chu+16] Y.-A. Chung et al. “Audio Word2Vec: Unsupervised Learning of Audio
Segment Representations using Sequence-to-sequence Autoencoder”.
In: ArXiv e-prints (Mar 2016).
[CPS16] Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve.
“Wav2letter: an end-to-end ConvNet-based speech recognition
system”. In: arXiv preprint arXiv:1609.03193 (2016).
[Gra12] Alex Graves. “Sequence transduction with recurrent neural networks”.
In: arXiv preprint arXiv:1211.3711 (2012).
[GMH13] Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. “Speech
recognition with deep recurrent neural networks”. In: Acoustics, speech
and signal processing (ICASSP), 2013 IEEE international conference
on. IEEE. 2013, pp. 6645–6649.
[Gra+06] Alex Graves et al. “Connectionist temporal classification: labelling un-
segmented sequence data with recurrent neural networks”. In: Proceed-
ings of the 23rd international conference on Machine learning. ACM.
2006, pp. 369–376.
[Gul+15] Caglar Gulcehre et al. “On using monolingual corpora in neural ma-
chine translation”. In: arXiv preprint arXiv:1503.03535 (2015).
[Han17] Awni Hannun. “Sequence Modeling with CTC”. In: Distill. (2017).
[Han+14a] Awni Hannun et al. “Deep speech: Scaling up end-to-end speech recog-
nition”. In: arXiv preprint arXiv:1412.5567 (2014).
[Han+14b] Awni Y Hannun et al. “First-pass large vocabulary continuous speech
recognition using bi-directional recurrent DNNs”. In: arXiv preprint
arXiv:1408.2873 (2014).
References 573
13.1 Introduction
In this chapter, we investigate deep reinforcement learning for text and speech ap-
plications. Reinforcement learning is a branch of machine learning that deals with
how agents learn a set of actions that can maximize expected cumulative reward. In
past research, reinforcement learning has focused on game play. Recent advances
in deep learning have opened up reinforcement learning to wider applications for
real-world problems, and the field of deep reinforcement learning was spawned. In
the first part of this chapter, we introduce the fundamental concepts of reinforce-
ment learning and their extension through the use of deep neural networks. In the
latter part of the chapter, we investigate several popular deep reinforcement learning
algorithms and their application to text and speech NLP tasks.
13.2 RL Fundamentals
Reinforcement learning (RL) is one of the most active fields of research in artificial
intelligence. While supervised learning requires us to provide labeled, independent
and identically distributed data, reinforcement learning requires us to only specify
a desired reward. Furthermore, it can learn sequential decision making tasks that
involve delayed rewards, especially those that occur far distant in the future.
A reinforcement learning agent interacts with its environment in discrete time
steps. At each time t, the agent in state st chooses an action at from the set of
available actions and transitions to a new state st+1 and receives reward rt+1 . The
goal of the agent is to learn the best set of actions, termed a policy, in order to
generate the highest overall cumulative reward (Fig. 13.1). The agent can (possibly
randomly) choose any action available to it. Any one set of actions that an agent
takes from start to finish is termed an episode. As we will see below, we can use
Markov decision processes to capture the episodic dynamics of a reinforcement
learning problem.
such that:
∑ ∑ p(s , r|s, a) = 1, for all s ∈ S, a ∈ A(s) (13.3)
s ∈S r∈R
Thus, the next state s depends on the current state s and the decision makerś action
a. But given s and a, it is conditionally independent of all previous states and actions;
in other words, the state transitions of an MDP satisfies the Markov property.
Markov decision processes are an extension of Markov chains where the dif-
ference is the addition of a set of actions (allowing choice) and rewards (giving
motivation). Conversely, if only one action exists for each state and all rewards are
equal, a Markov decision process reduces to a Markov chain.
We define rt as the reward we receive at time t. We can define the return as the sum
of the sequence of future rewards:
Normally, we include a time discount factor γ ∈ (0, 1), and the future cumulative
reward can be expressed as:
∞
Gt = ∑ γ k rt+k+1 (13.5)
k=0
With this definition, we can define the concept of a value function of a state s as the
expected cumulative return:
The value function for any particular state is not unique. It depends on the set of
actions we take going forward in the future. We define a set of future actions known
as a policy π :
a = π (s) (13.7)
Then the value function associated with this policy is unique:
Note that while this policy-associated value function is unique, the actual value can
be stochastic under a non-deterministic policy (e.g., one where we sample from a
distribution of possible actions defined by the policy):
In addition to finding the value function of a particular state, we can also define a
value function for a particular action given a state. This is known as the action-value
function or Q function:
Like the value function, the Q function is uniquely specified for a particular policy
π of actions. The expectation takes into account the randomness in future actions
according to the policy, as well as the randomness of the returned state from the
environment. Note that:
Vπ (s) = Ea∼π [Qπ (s, a)] (13.13)
The advantage function for a policy π measures the importance of an action by
finding the difference between the state-value and state-action-value functions:
Because the value function V measures the value of state s following policy π while
the Q function measures the value of following action a from state s, the advantage
function measures the benefit or loss of following a particular action from state s.
What this equation tells us is that the state value function associated with policy π is
the expectation of the reward received at the next state and its discounted state value
function. Similarly, the Bellman equation for the Q function is given by:
) *
Qπ (s, a) = Es ,a r + γ Qπ (s , a )|st = s, at = a (13.16)
The importance of the Bellman equations is that they let us express values of states
as values of other states. This means that if we know the value of st+1 , we can
very easily calculate the value of st . This opens the door to iterative approaches for
calculating the value for each state, since if we know the value of the next state, we
13.2 RL Fundamentals 579
can calculate the value of the current state. Sound familiar? This is similar to the
notion of backpropagation.
13.2.4 Optimality
The goal of any reinforcement learning problem is to find the optimal decisions that
lead to highest expected cumulative reward. Reinforcement methods fall under one
of several main categories depending on how they optimize the policy π for:
1. for the expected reward:
∞
max E
π
∑γ k
rt+k+1 (13.17)
k=0
The Bellman equation can be applied to the optimal state-value function v∗ to give
us the Bellman Optimality Equation which is independent of any chosen policy:
V∗ (s) = max Es r + γ V∗ (s ) (13.22)
a
Given a policy π , we can determine the state-value function vπ for this policy. Using
the Bellman equation above, it is possible to start with an approximation for vπ and
iteratively update an estimate vk until it converges to vπ as k → ∞:
The above is an expected update since it is based on the expectation over all possible
next states and actions (Fig. 13.2).
Consider the next action a for state s that is not from policy π . The value of taking
this action is given by the action-value function:
If we compare the value of taking this action to our policy π , we can decide if we
should adopt a new policy that takes action a. This leads to the policy improvement
theorem, which states that for any two deterministic policies π and π , if:
it must be that:
Vπ (s) ≥ Vπ (s) (13.29)
When we find that a new policy π is better, we can take its value Vπ and use it to find
a better policy. Here, E denotes policy iteration and I denotes policy improvement.
This iterative process is called policy iteration, where we cycle between policy
evaluation (π → Vπ ) and policy improvement (Vπ → π ) until we find the optimal
policy π∗ :
13.2 RL Fundamentals 581
E I E I E I E
π0 −→ Vπ0 −→ π1 −→ Vπ1 −→ π2 −→ . . . −→ π∗ −→ V∗ (13.30)
There is a potentially serious drawback with policy iteration in that the evaluation of
a policy π is computationally expensive as it requires iterative calculation over every
state in the MDP. Instead of waiting for convergence as k → ∞, we can approximate
vπ by performing a single update iteration (Vπ ≈ Vk+1 ):
13.2.5.4 Bootstrapping
and methods that bootstrap over longer sequences of successor states will have bet-
ter convergence properties.
13.2.5.5 Asynchronous DP
Dynamic programming methods operate over the entire set of states of a finite MDP.
Where the set of states is large, DP is intractable, as every state must necessarily
be updated before one sweep is completed. Asynchronous dynamic programming
methods do not wait for all states to be updated, but instead update a subset of states
during each sweep. Such methods will converge as long as all states are eventually
updated. Asynchronous DP methods are very useful in that they can run in an online
manner, concurrently as an agent is experiencing the states of the MDP. As such, the
agent’s experience can be considered in choosing the subset of states to update. This
is similar to the concept of beamsearch.
Unlike dynamic programming methods that require complete knowledge of the en-
vironment, Monte Carlo (MC) methods learn from a set of agent experiences. These
episodic experiences are actual or simulated sequences of actions, states, and re-
wards from the interaction of the agent with the environment. MC methods require
no prior knowledge but can still yield optimal policies by simply using averaged
sample reward for each state and action.
Consider a set of episodes E, where each occurrence of state s ∈ E is called a
visit. To estimate vπ (s), we can follow each of the visits all the way to the end of the
episode to calculate return G, and then average them to generate an update:
where α is the learning rate (Fig. 13.3). It is noteworthy that, with Monte Carlo
methods, estimates for each state are independent of each other. It does not use
bootstrapping. As such, Monte Carlo methods permit us to focus on a subset of
relevant states to improve results.
MC methods can be used to estimate state-action values as well as state values.
Instead of following visits to a state s, we can follow from an action a taken at a visit
to state s, and average accordingly. Unfortunately, however, it may be that certain
state-action pairs may never be visited. For deterministic policies, only one action is
taken from any state and therefore only one state-action pair will be estimated. The
value of all actions from each state must be estimated for policy improvement.
13.2 RL Fundamentals 583
Off-policy methods are based on two separate policies: a target policy that will be
optimized and another exploratory policy that is used to generate behavior (termed
the behavior policy). Off-policy Monte Carlo methods typically use the notion of
importance sampling, which is a technique for estimating expectations of one dis-
tribution given samples from another. The key idea is to sample values more fre-
quently that have greater impact on the expectation by shifting the probability mass.
Note that the target policy and behavior policies can be unrelated, with either or
both deterministic or stochastic.
Temporal difference (TD) learning seeks to combine the best of both worlds of
dynamic programming and Monte Carlo methods. In similar fashion to dynamic
programming, it uses bootstrapping to update estimates without waiting until the
end of the episode. Concurrently, it can learn from experience without an explicit
model of the environment like Monte Carlo methods. The simplest TD learning
method is one-step TD, also known as TD(0). It is based on updating the state value
function by (Fig. 13.4):
This n-step return is an approximation of the full return where the last term is an
estimate of the remaining returns beyond n-steps. While one-step TD can update
once the successor state is computed, n-step TD must wait until after n-steps of the
episode before updating. As a tradeoff, n-step TD provides better estimates for state
value functions with better convergence properties than one-step TD.
13.2.7.1 SARSA
Action-value methods are advantageous in model free formulations as they can op-
erate on current states without access to the model of the environment. This is in
contrast to state value functions which require a model since they require knowledge
of future states and possible actions to be evaluated. We can apply the temporal dif-
ference method to estimate the action-value function by considering the transitions
from one state-action pair to the next state-action pair:
Note that this update can be applied only to transitions from non-terminal states,
since Q(st+1 , at+1 ) = 0 at terminal states. Because this update depends on the tuple
(st , at , rt+1 , st+1 , at+1 ), it is called SARSA. It is a fully online, on-policy method that
asymptotically converges to the optimal policy and action-value function.
586 13 Deep Reinforcement Learning for Text and Speech
Policy gradient methods seek to optimize the policy directly without having to learn
the state or action value function. In particular, these model-free methods use a
parametric representation for a stochastic policy π (a|s; θ ) with parameters θ and
seek to optimize expected return:
θ ← θ + α ∇θ Eπ [Gt ] (13.41)
Note that this formula evaluates the expectation prior to calculating the gradient,
which requires us to know the transitional probability distribution of π (a|s; θ ). For
analytical tractability, we can make use of the Policy Gradient Theorem, given by:
#
∇θ Eπ [Gt ] = ∇θ pθ (x)Gt (τ )dx (13.42)
x∼π
#
= pθ (x)∇θ log pθ (x)Gt (x)dx (13.43)
x∼π
= Eπ [∇θ log π (at |st ; θ )Gt ] (13.44)
Thus, we can update our policy without calculating the transition probability distri-
bution of actions and states or requiring a model.
Policy gradient methods are useful for both continuous and discrete action
spaces. A popular method, known as REINFORCE, applies stochastic gradient de-
scent such that only a single sequence is used for training at each step to estimate
13.2 RL Fundamentals 587
13.2.9 Q-Learning
Q-learning is based on the notion that if the optimal Q-function is available, the
optimal policy can be directly found by the relation:
Therefore, these methods try to learn the optimal Q-function directly by always
choosing the best action from any state, without needing to consider the policy being
followed. Q-learning is an off-policy TD method that updates the action-state value
function by:
⎡ ⎤
⎢ ⎥
⎢ ⎥
Q(st , at ) ← Q(st , at ) + α ⎢rt+1 + γ max Q(st+1 , a ) −Q(st , at )⎥ (13.48)
⎣ a ⎦
expected future reward
This equation is very similar to SARSA, except it estimates future expected reward
by maximizing over future actions. In effect, Q-learning uses a greedy update to
iterate toward the optimal Q-function, and has been shown to converge in the limit
to Q∗ .
588 13 Deep Reinforcement Learning for Text and Speech
13.2.10 Actor-Critic
Actor-critic methods, like policy gradient methods, are based on estimating a para-
metric policy. What makes actor-critic methods different is that they also learn a
parametric function which is used to evaluate action sequences and assist in learn-
ing. The actor is the policy being optimized, while the critic is value function and
can be thought of as a parametric estimate of the baseline reward in the policy gra-
dient update equation above:
⎡ ⎡ ⎤⎤
⎢ ⎢ ⎥⎥
θ ← θ + α Eπ ⎣∇θ log π (at |st ; θ ) ⎣Qπ (st , at ) −Vπ (st )⎦⎦ (13.49)
actor critic
where Aπ (st , at ) = Qπ (st , at )−Vπ (st ). Similar to the REINFORCE algorithm, actor-
critic methods can use stochastic gradient descent to sample a single sequence. In
this instance, the Advantage function takes the form:
During learning, the actor provides sample states st and st+1 for the critic to esti-
mate the value function. The actor then uses this estimate to calculate the advantage
function used to update the policy parameters θ .
Since actor-critic methods rely on current samples to train the critic (as an on-
policy model), they suffer from the fact that estimates by the actor and critic are cor-
related. This can be alleviated by moving to off-policy training where samples are
accumulated and stored in a memory buffer. This buffer is then randomly batched-
sampled to train the critic. This is called experience replay, a sample efficient tech-
13.2 RL Fundamentals 589
nique since individual samples can be used multiple times during training. In gen-
eral, batch training with Actor-critic models can yield low variance estimates, but
they will be biased with a poor critic estimator. This is in contrast to policy gradient
models which may have high bias but are unbiased.
A way to reduce variance with online training is to use multiple threads that act
in parallel together as a batch to train the model. Each thread uses a single sample
and calculates an update using the advantage function. When all threads have fin-
ished calculating their update, they are batched together to update the model. This is
known as the synchronous advantage actor-critic model, or A2C. As an algorithm,
A2C is highly efficient and does not require memory buffer. Furthermore, it can
leverage modern multi-core processors very effectively to accelerate computation.
Rather than waiting for all threads to finish calculating an update, we can update the
model asynchronously. As soon as a thread calculates an update, it can broadcast
the update to other threads which immediately apply it in their calculation. This is
known as asynchronous advantage actor-critic or A3C and has received tremendous
attention and unprecedented success due to its light computational footprint and
quick training times.
Deep policy gradient methods train a deep neural network to learn the optimal pol-
icy. This can be accomplished with a seq2seq model where the output state of the
decoder is used to represent the state of a model. The agent is thus modeled as the
deep neural network (seq2seq model), where the output layer predicts a discrete ac-
tion taken by this agent (Fig. 13.5). Policy gradient methods such as REINFORCE
can be applied by choosing actions according to the deep neural network during
training to generate sequences. The reward is observed at the end of the sequence
or when an end-of-sequence (EOS) symbol is predicted. This reward can be a per-
formance metric evaluated on the difference between the generated sequence and
ground-truth sequence.
Unfortunately, the algorithm must wait until the end of a sequence to update,
causing high variance and making it slow to converge. Furthermore, at the start
of training when the deep neural network is initialized randomly, early predicted
actions might lead the model astray. Recent work suggest pre-training the policy
gradient model using cross-entropy loss before switching over to the REINFORCE
algorithm, which is a concept known as a warm start.
Instead of learning an estimate of the policy directly, we can use deep neural net-
works to approximate the action value function from which we can determine an
optimal policy. These methods are commonly known as deep Q-learning, where we
learn to estimate a Q-function Q(s, a; θ ) with parameters θ by minimizing the loss
function: 2
1
L(θ ) = E r + γ max Q(s , a ; θ ) − Q(s, a; θ ) (13.52)
2 a
Unfortunately, the update rule has convergence issues and can be rather unstable,
which limits the use of deep Q-learning models by themselves.
13.3.3.1 DQN
The deep Q-network (DQN) algorithm is a deep Q-learning model that utilizes ex-
perience replay and target networks to overcome instability (Fig. 13.6) [Mni+13].
Some have attributed the launch of the field of deep reinforcement learning to the in-
13.3 Deep Reinforcement Learning Algorithms 593
A target network is an extra copy of the deep Q-network. Its weights θtarget are
periodically copied over from the original Q-network but remain fixed during all
other times. This target network is used to compute the temporal difference during
the update:
⎡ ⎤
⎢ ⎥
⎢ ⎥
θ ← θ + α ⎢r + γ max Q(s , a ; θtarget ) −Q(s, a; θ )⎥ ∇θ Q(s, a; θ ) (13.54)
⎣ a ⎦
target network
Together, experience replay and a target network effectively smooth out learning
and avoid parameter oscillations or divergence. Typically a finite memory buffer of
length M is used for experience replay, such that only the most recent M transi-
tions are stored and sampled. Furthermore, experiences are uniformly sampled from
the buffer, regardless of significance. More recently, prioritized experience replay
has been proposed [Sch+15a], where more significant transitions are sampled more
frequently based on TD error and importance sampling.
594 13 Deep Reinforcement Learning for Text and Speech
DQN methods suffer from the problem in that they fundamentally tend to overesti-
mate Q-values. To see this, consider that the following relation holds:
In this expression, it can be seen that the target network is used twice; first to choose
the next best action, and then to estimate the Q value of this action. As a result,
there is a tendency to overestimate Q-values. Double Deep Q-Learning networks
overcome this by using two separate target networks: one to select the next best
action, and the other to estimate Q-values given the action selected.
Rather than introducing another target network, Double Deep Q-Networks
(DDQN) uses the current Q-network to select the next best action and the target
network to estimate its Q-value. The DDQN loss function can be written as:
2
1
L(θ ) = E r + γ Q(s , arg max Q(s , a ; θ ); θtarget ) − Q(s, a; θ ) (13.57)
2 a
13.3 Deep Reinforcement Learning Algorithms 595
DDQN alleviates the need for a third network as used in Double Deep Q-Learning
to resolve the overestimation problem.
DQN and DDQN methods are useful when the action space is small. In NLP appli-
cations, however, the action space can be equal to the size of the vocabulary, even
though only a small subset might be feasible at any one time. Estimating the Q-value
of each action in such a large space can be prohibitively expensive and slow to con-
verge. Consider the fact that in some states, the choice of action may have little to
no effect, while in other states, choice of action might be life-or-death.
The dueling network method uses a single network to simultaneously predict
both a state value function and advantage function that are aggregated to estimate
the Q-function. By doing so, it avoids the need to estimate the value of each ac-
tion choice. In one possible design, the dueling network is based on a Q-network
architecture with CNN lower layers, followed by two separate fully connected layer
streams whose outputs are summed together to estimate the Q-value.
We have seen that the addition of a separate target network to deep Q-learning meth-
ods can help overcome high variance and overestimation. Recall in DDQN that we
use the current network to select an action, and the target network to evaluate the
action. In effect, the current network serves as the actor and the target network as
the critic, with the caveat that the two networks are identical in architecture and the
weights of the target network are periodically synchronized to the current network.
This need not be the case, as a different network can be trained to estimate the
value function and act as a critic. Since deep neural networks tend to be unstable
estimators of the state value function, deep actor-critic methods usually focus on
estimating and maximizing the advantage function.
Instead of the advantage function defined as the difference between the state
value function and Q-function, we can use the TD error:
This value network method is known as deep advantage actor-critic (Fig. 13.7). In
this case, only a single Q network is necessary, though for stability reasons it is best
trained with experience replay and a target network similar to DQN.
For large-scale domains, labeled training data is often the largest constraint to
performance, as it can be prohibitively expensive to obtain accurately labeled data.
Distant supervision is one method that seeks to alleviate this need by leveraging an
external knowledge graph to automatically align text to extract entities or relations
[Min+09b]. However, extraction generated in this manner is not directly labeled
and can be incomplete. This is where reinforcement learning can be helpful.
For entity extraction tasks, external information can be used to resolve ambiguities
and boost accuracy by querying similar documents and comparing extracted entities.
This is a sequential task that can be addressed with a reinforcement learning agent
where we model the extraction task as a Markov decision process.
Figure 13.8 is an example of the architecture proposed by K. Narasimhan et
al. [NYB16] based on a DQN agent. In this model, the states are real-valued vectors
that encode the matches, context, and confidence of extracted entities from the target
and query documents. The actions are to accept, reject, or reconcile the entities of
two documents and query the next document. The reward function is selected to
maximize the final extraction accuracy:
To minimize the number of queries, a negative reward is added to each step. Since
this model is based on a continuous state space, the DQN algorithm can be trained
to approximate the Q-function, where the parameters of the DQN are learned using
stochastic gradient descent with experience replay and a target network to reduce
variance.
Consider a deep learning network for the task of relation extraction. This network is
regarded as the DRL agent, whose role is to take as input a sequence of words in a
sentence, and whose output are the extracted relations. If the sentences are regarded
as states and the relations as actions, we can learn an optimal policy to perform rela-
tion extraction. The process of extracting relations from a bag of sentences becomes
an episode.
Figure 13.9 depicts a deep policy gradient approach to this relation extraction
task. The reward function is given by the accuracy of the predicted relations in
a bag in comparison with a set of gold labels. The REINFORCE algorithm has
been applied [Zen+18] to optimize the policy of this model by defining the reward
function of a state si to be:
R(si ) = γ n−i rn (13.61)
where n is the number of sentences in the bag and rn is either +1 or −1. The objective
function for the policy gradient method is:
The task of extracting action sequences from text is challenging in that they usually
are highly affected by context. Traditional methods depend on a set of templates
which do not generalize well to natural language. Sequence labeling methods do
not perform well, since there is only a subset of sequences that can be considered
meaningful actions. The action extractor can be modeled as a DRL agent, where the
states are regarded as word sequences, and actions are the set of labels associated
with the word sequence. This agent can learn an optimal labeling policy by training
a DQN model. Figure 13.10 shows the architecture proposed by Feng et al. [FZK18]
called EASDRL that is based on first extracting action names and then the action
targets. To do so, this architecture defines two Q-functions associated with separate
CNN networks for modeling the action name Q(s, a) and action target Q(ŝ, a), and is
trained using a variant of experience replay that weighs positive-reward transitions
higher.
Deep learning for text classification has mainly focused on learning representations
of words and sentences that can effectively capture semantic context and structure.
Current methods, however, are unable to automatically learn and optimize structure,
since they are trained explicitly using supervised input or treebank annotations. In
contrast, DRL can be used to build hierarchical-structure sentence representations
without the need for annotations.
Figure 13.12 shows an architecture that consists of three components: a policy
network, a representation model, and a classification network [ZHZ18]. The policy
network is based on a stochastic policy whose states are vectors representations
of both word level and phrase level structure. These vectors are the output of the
representation model which consists of a two-level hierarchical LSTM that connects
a sequence of words to form a phrase and a sequence of phrases to form a sentence
representation. The actions of the policy network label whether a word is inside or
at the end of a phrase. Whereas the policy network focuses on building sentence
representations that capture structure, the classification network takes the output
from the representation model and uses it to perform the classification task.
To jointly train the policy and classification networks, the hierarchical LSTM is
first initialized and pre-trained using the cross-entropy loss of the classifier network,
given by:
K
L=− ∑ ∑ p(y, X) log P(y|X) (13.65)
X∈D y=1
602 13 Deep Reinforcement Learning for Text and Speech
where p and P are the target and predicted distributions, respectively. The parame-
ters for the representation model and classifier networks are then held constant and
the policy network is pre-trained using the REINFORCE algorithm. After the warm
start, all three networks are jointly trained until convergence.
A DQN agent has been successfully applied to train a dialogue system [ZE16,
GGL18] that unifies state tracking and dialogue policy and treats both as actions
available to the RL agent. The architecture learns an optimal policy that generates a
verbal response or updates the current dialogue state. Figure 13.13 depicts the DQN
model, which uses an LSTM network to generate a dialogue state representation.
The output of the LSTM serves as input to a set of policy networks in the form
of multilayer perceptron networks representing each possible action. The output of
these networks represents the action-state value functions for each action.
Due to the high-dimensional state and action spaces, a large number of labeled
dialogues are typically required to train dialogue systems. To overcome this need for
training data, a two-stage deep RL method has been proposed [Fat+16] that uses an
actor-critic architecture where the policy network is first supervised-trained using a
small number of high-quality dialogues via categorical cross-entropy to bootstrap
learning. The value network can then be trained using the deep advantage actor-
critic method.
One of the recent breakthroughs in neural machine translation has been the use of
seq2seq models. As noted above, teacher-forcing is the primary method to train
these networks. These models exhibit exposure bias during prediction time. Fur-
thermore, decoders cannot generate target sequences of interest with specific objec-
tives. This is especially so if beam-search is employed, which tends to focus more
on short-term rewards, a concept termed myopic bias. Machine translation is most
often evaluated based on the discrete BLEU measure, which creates a train-test mis-
match.
Deep RL models have been proposed to overcome some of these shortfalls. A
deep PG model [Li+16] based on the REINFORCE training algorithm can address
the non-differentiable nature of the BLEU metric. However, REINFORCE suffers
from the inability to learn policies in large action spaces as is the case with language
translation.
More recently, an actor-critic model has been proposed by using a decoding
scheme that incorporates longer-term rewards through a value function estimate
[Bah+16a]. In this model, the main sequence prediction model is the actor/agent
and the value function acts as a critic. The current sequence prediction output is the
state, and candidate tokens are actions of the agent. The critic is implemented by a
separate RNN and is trained on the ground-truth output using temporal difference
methods, with a target critic used to reduce variance.
Deep neural networks have significantly improved the performance of speech recog-
nition systems nowadays. When they are used as part of a hybrid system together
with GMMs or HMMs, alignment of the acoustic model is a necessity during train-
ing. This can be avoided when deep neural networks are used in end-to-end systems
606 13 Deep Reinforcement Learning for Text and Speech
that learn transcriptions by directly maximizing the likelihood of the input data
[YL18]. Such systems, while currently leading state-of-the-art performance, still
suffer from a variety of limitations.
Drawing from the experiences with text, researchers and practitioners have begun
to apply deep reinforcement learning methods to speech and audio, including tasks
such as automatic speech recognition, speech enhancement, and noise suppression.
In the near future, we expect to see wider adoption of deep RL techniques in other
aspects of speech, including applications in speaker diarization, speaker tone detec-
tion, and stress analysis.
The task of automatic speech recognition (ASR) is in many ways similar to machine
translation. ASR most often uses CTC maximum likelihood learning while measur-
ing performance with a discrete measure like word-error rate (WER). As a result,
train-test mismatch is a problem. Furthermore, as a sequence prediction task, ASR
suffers from exposure bias since it will be trained on ground-truth labels that are not
available at prediction time.
A deep RL approach using policy gradients has been shown to be effective
in [ZXS17] overcoming these limitations (Fig. 13.15). In this approach, the ASR
model is regarded as the agent, and training samples as the environment. The policy
πθ (y|x) is parameterized by θ , the actions are considered to be generated transcrip-
tions, and the model state is the hidden data representation. The reward function is
taken to be WER. The policy gradient is updated by the rule:
Machine learning speech enhancement methods have been in existence for quite a
while. Enhancement techniques usually fall under four subtasks: voice-activity de-
tection, signal-to-noise estimation, noise suppression, and signal amplification. The
first two provide statistics on the target speech signal while the latter two use these
statistics to extract the target signal. This can be naturally thought of as a sequen-
tial task. A deep RL method based on policy gradients has been proposed [TSN17]
for the task of speech enhancement with an architecture that is based on using an
LSTM network to model a filter whose parameters θ are determined by a learned
policy πθ . In this model, the filter is the agent, the state is a set of filter parameters,
and actions are increases or decreases in a filter parameter. The reward function
measures the mean-square error between the filter output and a ground-truth clean
13.6 Case Study 607
signal sequence. This policy gradient model, trained using the REINFORCE algo-
rithm, can improve signal-to-noise ratio with no algorithmic changes to the baseline
speech-enhancement process. Furthermore, by incorporating a deep reinforcement
agent, the filter can adjust to changing underlying conditions through dynamic pa-
rameter adaptation.
In this case study, we will apply the deep reinforcement learning concepts of this
chapter to the task of text summarization. We will use the Cornell NewsRoom Sum-
marization dataset. The goal here is to show readers how we can use deep reinforce-
ment learning algorithms to train an agent that can learn to generate summaries of
these articles. For the case study, we will focus on deep policy gradient and double
deep Q-network agents.
learning applications such as neural networks. It is used for both research and
production at Google.
• RLSeq2Seq is an open-source library which implements various RL techniques
for text summarization using sequence-to-sequence models.
• pyrouge is a python interface to the perl-based ROUGE-1.5.5 package that com-
putes ROUGE scores of text summaries.
The Cornell Newsroom dataset consists of 1.3 million articles and summaries writ-
ten by news authors and editors from 38 major publications between 1998 and 2017.
The dataset is split into train, dev, and test sets of 1.1 m, 100 k, and 100 k samples.
A sample of the dataset is provided below:
Story: Coinciding with Mary Shelley’s birthday week, this Scott family affair produced by
Ridley for director son Luke is another runout for the old story about scientists who cre-
ate new life only to see it lurch bloodily away from them. Frosty risk assessor Kate Mara’s
investigations into the mishandling of the eponymous hybrid intelligence (The Witch’s still-
eerie Anya Taylor-Joy) permits Scott Jr a good hour of existential unease: is it the placid
Morgan or her intemperate human overseers (Toby Jones, Michelle Yeoh, Paul Giamatti)
who pose the greater threat to this shadowy corporation’s safe operation? Alas, once that
question is resolved, the film turns into a passably schlocky runaround, bound for a guess-
able last-minute twist that has an obvious precedent in the Scott canon. The capable cast
yank us through the chicanery, making welcome gestures towards a number of science-
fiction ideas, but cranked-up Frankenstein isn’t one of the film’s smarter or more original
ones.
13.6 Case Study 609
Summary: Ridley and son Luke turn in a passable sci-fi thriller, but the horror turns to
shlock as the film heads for a predictable twist ending
For our case study, we will use subsets of 10,000/1000/1000 articles and sum-
maries from the Cornell Newsroom dataset for our training, validation, and test sets,
respectively. We will tokenize and map these data sets using 100-dim embeddings
generated with word2vec. For memory considerations, we limit our vocabulary to
50,000 words.
Our first task is to train a deep policy gradient agent that can produce summaries
of the articles. Before we do so, we pre-train the seq2seq model using maximum
likelihood loss, an encoder and decoder layer size of 256, batch size of 20, and
adagrad with gradient clipping for 10 epochs (Fig. 13.16). After pre-training, we
evaluate this model on test set to find the results shown in Table 13.1.
Seq2seq: at 90-years old this tortoise has never moved better despite a horrific rat attack
that caused legs
Reference: a 90-year old tortoise was given wheels after a rat attack caused her to lose her
front legs
Seq2seq: a city employee in baquba the capital of diyala province vividly described
his ambivalence
Reference: iraqis want nothing more than to have u.s. soldiers leave iraq but there is
nothing they can less afford
Seq2seq: google reported weaker than expected results thursday for its latest quarter
Reference: the tech giant ’s shares rose after it reported a smaller than expected rise in
sales for its latest quarter
In comparison with the reference summaries, the generated summaries are fair
but leave some room for improvement.
Let’s apply a deep policy gradient algorithm to improve our summaries (Fig. 13.17).
We set our reward function to ROUGE-L F1 score and switch from MLE loss to RL
610 13 Deep Reinforcement Learning for Text and Speech
loss. We continue training for 8 epochs, after which we evaluate the RL-trained
model on the test set to find the results shown in Table 13.2.
DPG: conservative pundit glenn beck says the obama administration is using churches
and other faith based groups to promote its climate change agenda
Reference: glenn beck says obama uses churches on climate change green house
DPG: the zoo in georgia ’s capital has reopened three months after a devastating flood
that killed more than half of its 600 animals including about 20 tigers lions and jaguars
Reference: a georgia zoo that had half its animals killed during floods in june has reopened
13.6 Case Study 611
13.6.3.3 DDQN
Let’s see if we can improve on the results above using a double deep Q-learning
agent. We start as before by pre-training the seq2seq language model using maxi-
mum likelihood loss for 10 epochs. We then train the double deep-Q network for 8
epochs using a batch size of 20, replay buffer of 5000 samples and updating the tar-
get network every 500 iterations. For better results, we will first pre-train the DDQN
agent with a fixed actor for a single epoch. When we then evaluate the resulting
model on the test set, we find the results in shown in Table 13.3.
DDQN: the commander of us forces in the middle east said that the refusal to follow orders
occurred during the battle for the recently liberated town of manbij syria
Reference: a top us general said tuesday that isis fighters defied their leader ’s orders to
fight to the death in a recent battle instead retreating to the north
DDQN: an online discussion of the washington area rental market featuring post columnist
sara gebhardt
Reference: welcome to apartment life an online discussion of the washington area rental
market featuring post columnist sara gebhardt
612 13 Deep Reinforcement Learning for Text and Speech
DDQN: albania has become the largest producer of outdoor grown cannabis in europe
Reference: albania has become the largest producer of outdoor grown cannabis in europe
The DDQN agent outperforms the DPG agent for the chosen parameters. There
are a myriad of possibilities to improve results further—we could use scheduled
or prioritized sampling, intermediate rewards, and/or some form of attention at the
encoder or decoder.
1. How would you combine a DQN agent for the task of text classification when
using a seq2seq model with soft attention?
2. Does it make sense to use two separate target networks for the double DQN
agent? Why or why not?
3. What kind of deep neural networks would we use for the Q-learning model?
Why would or would not CNNs be appropriate?
References
[Bah+16a] Dzmitry Bahdanau et al. “An Actor-Critic Algorithm for Sequence Pre-
diction.” In: CoRR abs/1607.07086 (2016).
[Fat+16] Mehdi Fatemi et al. “Policy Networks with Two-Stage Training for
Dialogue Systems.” In: CoRR abs/1606.03152 (2016).
[FZK18] Wenfeng Feng, Hankz Hankui Zhuo, and Subbarao Kambhampati.
“Extracting Action Sequences from Texts Based on Deep Reinforce-
ment Learning.” In: IJCAI. ijcai.org, 2018, pp. 4064–4070.
[Fen+17] Yuntian Feng et al. “Joint Extraction of Entities and Relations Using
Reinforcement Learning and Deep Learning.” In: Comp. Int. and Neu-
rosc. 2017 (2017), 7643065:1–7643065:11.
[GGL18] Jianfeng Gao, Michel Galley, and Lihong Li. “Neural Approaches to
Conversational AI.” In: CoRR abs/1809.08267 (2018).
[GHS16] Tomas Gogar, Ondrej Hubácek, and Jan Sedivý. “Deep Neural Net-
works for Web Page Information Extraction.” In: AIAI. Vol. 475.
Springer, 2016, pp. 154–163.
[HGS15] Hado van Hasselt, Arthur Guez, and David Silver “Deep Reinforce-
ment Learning with Double Q-learning.” In: CoRR abs/1509.06461
(2015).
[Ken+18] Yaser Keneshloo et al. “Deep Reinforcement Learning For Se quence
to Sequence Models.” In: CoRR abs/1805.09461 (2018).
References 613
[LL17] Gyoung Ho Lee and Kong Joo Lee. “Automatic Text Summarization
Using Reinforcement Learning with Embedding Features.” In: IJC-
NLP(2). Asian Federation of Natural Language Processing, 2017, pp.
193–197.
[Li+16] Jiwei Li et al. “Deep Reinforcement Learning for Dialogue Gener
ation”. In: CoRR abs/1606.01541 (2016).
[Min+09b] Mike Mintz et al. “Distant supervision for relation extraction without
labeled data.” In: ACL/IJCNLP. The Association for Computer Linguis-
tics, 2009, pp. 1003–1011.
[Mni+13] Volodymyr Mnih et al. “Playing Atari with Deep Reinforcement Learn-
ing.” In: CoRR abs/1312.5602 (2013).
[NYB16] Karthik Narasimhan, Adam Yala, and Regina Barzilay “Improving In-
formation Extraction by Acquiring External Evidence with Reinforce-
ment Learning.” In: CoRR abs/1603.07954 (2016).
[PXS17b] Romain Paulus, Caiming Xiong, and Richard Socher. “A Deep
Reinforced Model for Abstractive Summarization.” In: CoRR
abs/1705.04304 (2017).
[Qi+14] Yanjun Qi et al. “Deep Learning for Character-Based Information Ex-
traction.” In: ECIR. Vol. 8416. Springer, 2014, pp. 668–674.
[Sch+15a] Tom Schaul et al. “Prioritized Experience Replay.” In: CoRR
abs/1511.05952 (2015).
[SLM17] Abigail See, Peter J. Liu, and Christopher D. Manning. “Get To The
Point: Summarization with Pointer-Generator Networks.” In: CoRR
abs/1704.04368 (2017).
[TSN17] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. “Sequence-to-
Sequence ASR Optimization via Reinforcement Learning.” In: CoRR
abs/1710.10774 (2017).
[YL18] Dong Yu and Jinyu Li. “Recent Progresses in Deep Learn-
ing based Acoustic Models (Updated).” In: CoRR (2018).
http://arxiv.org/abs/1804.09298
[Zen+18] Xiangrong Zeng et al. “Large Scaled Relation Extraction With Rein-
forcement Learning.” In: AAAI AAAI Press, 2018.
[ZHZ18] Tianyang Zhang, Minlie Huang, and Li Zhao. “Learning Structured
Representation for Text Classification via Reinforcement Learning.”
In: AAAI. AAAI Press, 2018.
[ZE16] Tiancheng Zhao and Maxine Eskénazi. “Towards End-to-End Learning
for Dialog State Tracking and Management using Deep Reinforcement
Learning.” In: SIGDIAL Conference. The Association for Computer
Linguistics, 2016, pp. 1–10.
[ZXS17] Yingbo Zhou, Caiming Xiong, and Richard Socher. “Improving
End-to-End Speech Recognition with Policy Learning.” In: CoRR
abs/1712.07101 (2017).
[Çe+18b] Asli Çelikyilmaz et al. “Deep Communicating Agents for Abstractive
Summarization.” In: NAACL-HLT. Association for Computational Lin-
guistics, 2018, pp. 1662–1675.
Future Outlook
Predicting the future of AI is no more possible today than it has been in years past.
Furthermore, the farther into the future we project, the greater the uncertainty. In
general, some things may go exactly as expected (improvements in computational
speed), some expectations may have slight variability (the dominant deep learning
architectures), and others are maverick innovations that are unlikely to be predicted
(the intersection of big data, computational speed, and emergence of deep learning
all at the same time). At the conclusion of this book, we would like to provide our
predictions based on the current trajectories, trends, and usefulness of the research
we’ve discussed. We reject all claims to be considered soothsayers or even reliable
parties in these projections. We attempt to only provide considerations for the reader
at the conclusion of these topics and suggest areas of awareness over upcoming
years.
Given the success of many end-to-end approaches in both NLP and speech, we
expect that more will move towards these architectures. One of the areas where
these approaches lack robustness is in the tuning to particular environments, for
example, the usefulness of a lexicon model in the ASR hybrid architecture or in
the adaptation of language models to new domains. This is an area that must be
addressed for deep learning to make a significant impact in domains where training
data is costly or unavailable.
Transition to AI-Centric
One of the simplest projections is that more companies will shift to or center around
an AI-Centric strategy. Many of the leading tech companies—for example, Google,
Facebook, and Twitter—have moved in this direction, and this trend will likely
continue into many other large and mid-sized companies. This shift will introduce
machine learning into every level of software development and with it the need for
tools and processes to ensure reliability and generality. Some have coined the term
“Software 2.01 ” in light of this shift. Transitioning to this state will require increased
rigor around data, interpretability of models, an increased focus on model security,
and resiliency to adversarial scenarios.
Specialized Hardware
We expect the focus of machine learning to shift. Deep learning has seen the largest
improvements with supervised data; however, the costs associated with creating
large, labeled datasets are often prohibitively expensive. In many scenarios, large
unlabeled sources exist that can be used by unsupervised algorithms, and we expect
a greater concentration of algorithms in this area, as seen in the progression of word
embeddings and language models.
Explainable AI
Though end-to-end deep learning techniques are powerful and can result in impres-
sive performance metrics such as accuracy, they suffer from interpretability. Many
applications in the financial world (such as loan applications or conduct surveil-
lance) or in healthcare (like predicting disease) need models and predictions to be
explainable. There has been a shift in the industry towards explainable AI (XAI).
Many techniques such as Local Interpretable Model-agnostic Explanation (LIME),
Deep Learning Important FeaTures (DeepLIFT), SHapley Additive exPlanation
(SHAP) to name a few have been very promising in providing model-agnostic expla-
1 https://medium.com/@karpathy/software-2-0-a64152b37c35.
Future Outlook 617
Democratization of AI
AI and deep learning are used by a still very small but rapidly growing group of
researchers, educators, experts, and practitioners. To make them accessible to the
masses through applications, tools, or education, there needs to be a change in at-
titude, policies, investment, and research, especially from top companies and uni-
versities. This phenomenon is called the “democratization of AI.” Many companies
such as Google, Microsoft, and Facebook, as well as many universities such as MIT,
Stanford, and Oxford are contributing to software tools, libraries, datasets, courses,
etc. that are freely available on the web. The positive trend in this direction will play
a huge role in transforming lives through AI.
NLP Trends
fail cases, or improve the robustness of models is becoming a trend in deep learning
research. Moving towards under-resourced languages and using deep learning tech-
niques such as transfer learning is another area that many researchers are focusing
especially in tasks such as machine translation.
One of the most curious areas of development is in the area of reinforcement
learning. Instead of collecting data, training a model, putting it into production, and
testing the result, an agent could be created to interact with the environment (real or
synthetic) and learn based on its experience. Overall, we see the progression moving
from supervised to unsupervised to reinforcement techniques.
Speech Trends
Many of the end-to-end deep learning techniques are able to outperform traditional
hybrid HMM-based models with less tuning and linguistic expertise. These models
perform very well in scenarios where training data is widely available, typically in
general speech recognition tasks. However, they tend to struggle when context is
crucial to prediction. Additionally, the continued pursuit of fusing speech and NLP
is a direction likely to continue, with end-to-end learning taking the lead. Recent
advancements are focusing on incorporating domain information into the decoding
procedure via language model fusion for contextualized recognition.
Other areas where speech recognition still struggles are with acoustic environ-
ments and speaker-specific differences such as accents. Leveraging generated data
from speech-to-text systems is gaining traction, providing simulated environments
and speakers for improved robustness. We expect the incorporation of speech-to-text
systems, similar to GAN workflows, to continue to improve, and will potentially be
incorporated more fully into reinforcement workflows.
Closing Remarks
We hope that the readers have found the information in this book both informative
and helpful. Deep learning has heavily impacted NLP and speech in the past few
years, and the trend seems to be gaining speed. We have hopefully enabled the
readers to understand both fundamental and advanced techniques that deep learning
offers, while also showing how to practically apply them.
Index