Professional Documents
Culture Documents
Deep Learning Glossary
Deep Learning Glossary
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
The boundary between what is Deep
Learning vs. “general” Machine Learning
terminology is quite fuzzy. I am trying to
keep the glossary speci c to Deep
Learning, but these decisions are
somewhat arbitrary. For example, I am not
including “cross-validation” here because
it’s a generic technique uses all across
Machine Learning. However, I’ve decided to
include terms such as softmax or word2vec
because they are often associated with
Deep Learning even though they are not
Deep Learning techniques.
Activation Function
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Adadelta
Adagrad
Adam
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Adam: A Method for Stochastic
Optimization
An overview of gradient descent
optimization algorithms
Af ne Layer
Attention Mechanism
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Attention Mechanisms are inspired by
human visual attention, the ability to focus
on speci c parts of an image. Attention
mechanisms can be incorporated in both
Language Processing and Image
Recognition architectures to help the
network learn what to “focus” on when
making predictions.
Alexnet
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
introduced in ImageNet Classi cation with
Deep Convolutional Neural Networks.
Autoencoder
Average-Pooling
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Average-Pooling is a pooling technique
used in Convolutional Neural Networks for
Image Recognition. It works by sliding a
window over patches of features, such as
pixels, and taking the average of all values
within the window. It compresses the input
representation into a lower-dimensional
representation.
Backpropagation
Backpropagation is an algorithm to
ef ciently calculate the gradients in a
Neural Network, or more generally, a
feedforward computational graph. It boils
down to applying the chain rule of
differentiation starting from the network
output and propagating the gradients
backward. The rst uses of
backpropagation go back to Vapnik in the
1960’s, but Learning representations by
back-propagating errors is often cited as
the source.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Calculus on Computational Graphs:
Backpropagation
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Backpropagation Through Time: What
It Does and How to Do It
Batch Normalization
Bidirectional RNN
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
A Bidirectional Recurrent Neural Network
is a type of Neural Network that contains
two RNNs going into different directions.
The forward RNN reads the input sequence
from start to end, while the backward RNN
reads it from end to start. The two RNNs
are stacked on top of each others and their
states are typically combined by appending
the two vectors. Bidirectional RNNs are
often used in Natural Language problems,
where we want to take the context from
both before and after a word into account
before making a prediction.
Caffe
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
popular and performant for vision tasks
and CNN models.
Channel
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Input data to Deep Learning models can
have multiple channels. The canonical
examples are images, which have red,
green and blue color channels. A image can
be represented as a 3-dimensional Tensor
with the dimensions corresponding to
channel, height, and width. Natural
Language data can also have multiple
channels, in the form of different types of
embeddings for example.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Stanford CS231n class – Convolutional
Neural Networks for Visual
Recognition
Understanding Convolutional Neural
Networks for NLP
Deep Dream
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
A technique invented by Google that tries
to distill the knowledge captured by a deep
Convolutional Neural Network. The
technique can generate new images, or
transform existing images and give them a
dreamlike avor, especially when applied
recursively.
Dropout
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
use in CNNs, but have since been applied to
other layers, including input embeddings or
recurrent networks.
Embedding
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
task, such as Sentiment Analysis. Often, the
input layer of a network is initialized with
pre-trained embeddings, which are then
ne-tuned to the task at hand.
Fine-Tuning
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
another task (such as an unsupervised
training task), and then updating these
parameters based on the task at hand. For
example, NLP architecture often use pre-
trained word embeddings like word2vec,
and these word embeddings are then
updated during training based for a speci c
task like Sentiment Analysis.
Gradient Clipping
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
On the dif culty of training recurrent
neural networks
GloVe
GoogleLeNet
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
improve the utilization of the computing
resources inside the network.
GRU
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
RNN with Python and Theano
Highway Layer
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
ICML
ILSVRC
Inception Module
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Inception Modules are used in
Convolutional Neural Networks to allow for
more ef cient computation and deeper
Networks trough a dimensionality
reduction with stacked 1×1 convolutions.
Keras
LSTM
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
an RNN we help to the network to
ef ciently propagate gradients and learn
long-range dependencies.
Max-Pooling
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Pooling layers help to reduce the
dimensionality of a representation by
keeping only the most salient information,
and in the case of image inputs, they
provide basic invariance to translation (the
same maximum values will be selected even
if the image is shifted by a few pixels).
Pooling layers are typically inserted
between successive convolutional layers.
MNIST
Momentum
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Momentum is an extension to the Gradient
Descent Algorithm that accelerates or
damps the parameter updates. In practice,
including a momentum term in the
gradient descent updates leads to better
convergence rates in Deep Networks.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
See Categorical Cross Entropy Loss.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Neural Turing Machine (NTM)
Nonlinearity
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
softmax over a large number of possible
classes is prohibitively expensive. Using
NCE, we can reduce the problem to binary
classi cation problem by training the
classi er to discriminate between samples
from the “real” distribution and an
arti cially generated noise distribution.
Pooling
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
learn a representation of the data in an
unsupervised manner. An RBN consists of
visible and hidden layer, and connections
between binary neurons in each of these
layers. RBNs can be ef ciently trained using
Contrastive Divergence, an approximation
of gradient descent.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
the output a sentiment classi cation of the
sentence (N-to-1). An input could be a
single image, and the output could be a
sequence of words corresponding to the
description of an image (1-to-N). At each
time step, an RNN calculates a new hidden
state (“memory”) based on the current
input and the previous hidden state. The
“recurrent” stems from the facts that at
each step the same parameters are used
and the network performs the same
calculations based on different inputs.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Just like RNNs, Recursive Neural Networks
can be trained end-to-end using
backpropagation. While it is possible to
learn the tree structure as part of the
optimization problem, Recursive Neural
Networks are often applied to problem that
already have a prede ned structure, like a
parse tree in Natural Language Processing.
ReLU
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
most commonly used activation function in
Convolutional Neural Networks. There exist
several variations of ReLUs, such as Leaky
ReLUs, Parametric ReLU (PReLU) or a
smoother softplus approximation.
ResNet
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Highway Layers, but they are data-
independent and don’t introduce additional
parameters or training complexity. ResNets
achieved a 3.57% error rate on the
ImageNet test set.
RMSProp
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Seq2Seq
SGD
Softmax
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Network used for classi cation. It
normalizes the scores by exponentiating
and dividing by a normalization constant. If
we are dealing with a large number of
classes, a large vocabulary in Machine
Translation for example, the normalization
constant is expensive to compute. There
exist various alternatives to make the
computation more ef cient, including
Hierarchical Softmax or using a sampling-
based loss such as NCE.
TensorFlow
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Theano
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
counter this problem is to use activation
functions like ReLUs that do not suffer
from small gradients, or use architectures
like LSTMs that explicitly combat vanishing
gradients. The opposite of this problem is
called the exploding gradient problem.
VGG
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
word2vec
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
Proudly powered by WordPress
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD