Professional Documents
Culture Documents
TensorFlow 1.x Deep Learning Cookbook - Over 90 Unique Recipes To Solve Artificial-Intelligence Driven Problems With Python
TensorFlow 1.x Deep Learning Cookbook - Over 90 Unique Recipes To Solve Artificial-Intelligence Driven Problems With Python
TensorFlow 1.x
Deep Learning
Cookbook
Over 90 unique recipes to solve artiĆcial-intelligence
driven problems with Python
TensorFlow 1.x Deep Learning
Cookbook
0WFSVOJRVFSFDJQFTUPTPMWFBSUJ`DJBMJOUFMMJHFODFESJWFO
QSPCMFNTXJUI1ZUIPO
Antonio Gulli
Amita Kapoor
BIRMINGHAM - MUMBAI
TensorFlow 1.x Deep Learning Cookbook
Copyright a 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the authors, nor Packt Publishing, and its
dealers and distributors will be held liable for any damages caused or alleged to be caused
directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
XXXQBDLUQVCDPN
Credits
Authors Copy Editors
Antonio Gulli Safis Editing
Amita Kapoor Vikrant Phadkay
Reviewers
Nick McClure Project Coordinator
Narotam Singh Manthan Patel
Corrado Zoccolo
I thank every reader of this book for your attention and for the trust. I am humbled by the
number of comments received on LinkedIn and Facebook: You, the reader, provided
immense help in making this book better. I would also like to thank various people for
providing support during the process of writing the book. In no order: Susana, Ewa,
Ignacy, Dawid, Max, Jarek, Jerzy, Nina, Laura, Antonella, Eric, Ettore, Francesco, Liubov,
Marco, Fabio, Giacomo, Saskia, Christina, Wieland, and Yossi. I am very grateful to my
coauthor, Amita, for her valuable comments and suggestions. I am extremely thankful to
the reviewers of this book, Eric Brewer, Corrado Zoccolo, and Sujit Pal, for going through
the entire book content. Special thanks to my manager, Eyal, for supporting me during the
writing process and for the trust constantly offered. Part of this book has been written in
Charlotte Menora (IUUQCJTUSPDIBSMPUUFQM), a pub in Warsaw, where I found
myself writing pages after work. This is an inspirational place, which I definitively
recommend if you are visiting Poland. Modern and cool as the city of Warsaw is these
days. Last and not the least, I am grateful to the entire editorial team of Packt, especially
Tushar Gupta and Tejas Limkar for all the support, constant reminders regarding the
schedule, and continuous motivation. Thanks for your patience.
Amita Kapoor is an associate professor in the Department of Electronics, SRCASW,
University of Delhi. She has been actively teaching neural networks for the last 20 years. She
did her master's in electronics in 1996 and PhD in 2011. During her PhD, she was awarded
the prestigious DAAD fellowship to pursue a part of her research work in Karlsruhe
Institute of Technology, Karlsruhe, Germany. She had been awarded the best presentation
award at International Conference Photonics 2008 for her paper. She is a member of
professional bodies such as OSA (Optical Society of America), IEEE (Institute of Electrical
and Electronics Engineers), INNS (International Neural Network Society), and ISBS (Indian
Society for Buddhist Studies). Amita has more than 40 publications in international journals
and conferences to her credit. Her present research areas include machine learning, artificial
intelligence, neural networks, robotics, Buddhism (philosophy and psychology) and ethics
in AI.
This book is an attempt to summarize what all I had learned in the field of deep neural
networks. I have presented it in a manner that readers find easy to understand and apply,
and so the prime motivation of this book comes from you, the readers. I thank every reader
of this book for being consistently present at the back of my mind, especially when I felt
lazy. I would also like to thank professor Parongama Sen, University of Calcutta, for
introducing me to the subject in 1994, and my friends Nirjara Jain and Shubha
Swaminathan for the hours spent in college library discussing Asimov, his stories, and the
future that neural networks behold for our society. I am very grateful to my coauthor,
Antonio Guili, for his valuable comments and suggestions and the reviewers of this book,
Narotam Singh and Nick McClure, for painstakingly going through the entire content and
rechecking the codes. Last and not the least, I am grateful to the entire editorial team of
Packt, especially Tushar Gupta and Tejas Limkar for all the support, constant reminders
regarding schedule, and continuous motivation.
About the Reviewers
Narotam Singh has been with India Meteorological Department, Ministry of Earth Sciences,
India, since 1996. He has been actively involved with various technical programs and
training of officers of GoI in the field of information technology and communication. He did
his post-graduation in the field of electronics in 1996 and both diploma and post-graduate
diploma in the field of computer engineering in 1994 and 1997 respectively. He is currently
working in the enigmatic field of neural networks, machine learning, and deep learning.
Nick McClure is currently a senior data scientist at PayScale Inc in Seattle, Washington,
USA. Previously, he worked at Zillow and Caesarbs Entertainment. He has degrees in
applied mathematics from the University of Montana and the College of Saint Benedict and
Saint Johnbs University. Nick has also authored TensorFlow Machine Learning Cookbook by
Packt Publishing.
He has a passion for learning and advocating for analytics, machine learning, and artificial
intelligence. Nick occasionally puts his thoughts and musings on his blog, GSPNEBUBPSH,
or through his Twitter account at !OGNDDMVSF.
Corrado Zoccolo is a Senior Staff Software Engineer at Google, with 10+ years of experience
in distributed indexing and information retrieval systems.
I would like to show my gratitude to the many people who helped me succeed in my studies
and my carreer, in particular: my wife Ermelinda, that supported me over the many years I
dedicated to computer science; Prof. Marco Vanneschi who introduced me to the beautiful
world of Distributed Systems; my first manager at Google, Peter Dickman, who set me on
the right track for my career, and all my colleagues from which I keep learning every day.
www.PacktPub.com
For support files and downloads related to your book, please visit XXX1BDLU1VCDPN. Did
you know that Packt offers eBook versions of every book published, with PDF and ePub
files available? You can upgrade to the eBook version at XXX1BDLU1VCDPNand as a print
book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
TFSWJDF!QBDLUQVCDPN for more details. At XXX1BDLU1VCDPN, you can also read a
collection of free technical articles, sign up for a range of free newsletters and receive
exclusive discounts and offers on Packt books and eBooks.
IUUQTXXXQBDLUQVCDPNNBQU
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt
books and video courses, as well as industry-leading tools to help you plan your personal
development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial
process. To help us improve, please leave us an honest review on this book's Amazon page
at XXXBNB[PODPNEQ.
If you'd like to join our team of regular reviewers, you can email us at
DVTUPNFSSFWJFXT!QBDLUQVCDPN. We award our regular reviewers with free eBooks and
videos in exchange for their valuable feedback. Help us be relentless in improving our
products!
This book is dedicated to my son Lorenzo - for your growing passion for algorithms - to my
son Leonardo - for your curiosity to explore the world -, and to my daughter Aurora - for
your love. A special thought to Elio and Maria, my father and my mother.
"OUPOJP(VMMJ
This book is dedicated to the loving memory of my mother Late Smt Swarnlata Kapoor,
who instilled in me the value of education, and motivated me to continue learning despite
odds, and my father Mr Anil Mohan Kapoor for all his love and support.
"NJUB,BQPPS
Table of Contents
Preface 1
Chapter 1: TensorFlow - An Introduction 9
Introduction 10
Installing TensorFlow 12
Getting ready 12
How to do it... 13
How it works... 15
There's more... 16
Hello world in TensorFlow 17
How to do it... 17
How it works... 19
Understanding the TensorFlow program structure 20
How to do it... 20
How it works... 21
There's more... 22
Working with constants, variables, and placeholders 23
How to do it... 24
How it works... 28
There's more... 29
Performing matrix manipulations using TensorFlow 30
How to do it... 30
How it works... 32
There's more... 32
Using a data flow graph 32
How to do it... 32
Migrating from 0.x to 1.x 34
How to do it... 34
There's more... 35
Using XLA to enhance computational performance 35
Getting ready 36
How to do it... 37
Invoking CPU/GPU devices 38
How to do it... 38
How it works... 41
Table of Contents
[ ii ]
Table of Contents
Getting ready 85
How to do it... 86
How it works... 91
There's more... 92
See also 92
Single layer perceptron 92
Getting ready 93
How to do it... 93
There's more... 95
Calculating gradients of backpropagation algorithm 95
Getting ready 96
How to do it... 97
How it works... 100
There's more... 100
See also 100
MNIST classifier using MLP 100
Getting ready 100
How to do it... 101
How it works... 103
Function approximation using MLP-predicting Boston house prices 104
Getting ready 104
How to do it... 104
How it works... 108
There's more... 109
Tuning hyperparameters 110
How to do it... 110
There's more... 112
See also 112
Higher-level APIs-Keras 112
How to do it... 113
There's more... 114
See also 114
Chapter 4: Convolutional Neural Networks 115
Introduction 115
Local receptive fields 116
Shared weights and bias 116
A mathematical example 116
ConvNets in TensorFlow 117
[ iii ]
Table of Contents
[ iv ]
Table of Contents
[v]
Table of Contents
[ vi ]
Table of Contents
[ vii ]
Table of Contents
[ viii ]
Table of Contents
[ ix ]
Table of Contents
[x]
Table of Contents
[ xi ]
Table of Contents
[ xii ]
Preface
In this book, you will learn how to efficiently use TensorFlow, Google's open source
framework for deep learning. You will implement different deep learning networks such as
Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Deep Q-
learning Networks (DQNs), and Generative Adversarial Networks (GANs) with easy to
follow independent recipes. You will learn how to make Keras as backend with
TensorFlow.
You will understand how to implement different deep neural architectures to carry out
complex tasks at work. You will learn the performance of different DNNs on some
popularly used data sets such as MNIST, CIFAR-10, Youtube8m, and more. You will not
only learn about the different mobile and embedded platforms supported by TensorFlow
but also how to set up cloud platforms for deep learning applications. Get a sneak peek of
TPU architecture and how they will affect DNN future.
By the end of this book, you will be an expert in implementing deep learning techniques in
growing real-world applications and research areas such as reinforcement learning, GANs,
autoencoders and more.
$IBQUFS , Regression, talks about regression and its applications. We will discuss the
concepts involved in regression, understand how it is different from clustering and
classification. We will learn about different types of loss functions possible and how to
implement them in Tensorflow. We learn how to implement L1 and L2 regularizations. We
will discuss the gradient descent algorithm, learn how to optimize it and implement it in
Tensorflow. We will briefly learn about cross-entropy function and its implementation.
Preface
$IBQUFS , Neural Networks - Perceptron, covers artificial neural networks and explains why
it can do the wonderful tasks as claimed recently by DNN. We will learn about different
choices of activation functions. We will use all this to build a simple perceptron and use it
for function modeling. We will learn about regularization of data before training. We will
also learn to build a multilayer perceptron (MLP) layer by layer. We will see how the auto-
differentiator of TensorFlow makes our work easier.
$IBQUFS, CNN in Action, covers some wow success stories of CNN like face recognition.
We will write a recipe for using CNNs for sentiment analysis. We will discuss pre-tuning
CNN and learn how to implement transfer learning. We will learn how to use VGG16 for
transfer learning. We will learn classification of images with VGGNet, ResNet, Inception
and Xception. We will Generate music using dilated ConvNets, Wavenet and Nsynth. We
will also learn how to do Visual Q & A. We will learn how to classify videos.
$IBQUFS, Recurrent Neural Networks, discusses Recurrent Neural networks. We will learn
the basic unit of RNNs the RNN cell. We will learn about word embeddings and time
sequencing. We will briefly discuss LSTM networks. We will learn about seq2seq RNNs. We
will learn how RNNs can be employed for machine translation, generating text, and
predicting future values
[2]
Preface
$IBQUFS, TensorFlow Mobile Computation, covers TensorFlow mobile. We will learn about
different applications of mobile deep learning. We will learn how to use Tensorflow with
Android studio on Windows platform. We will learn how to use Tensorflow along with
XCode to make IOS based apps. We will learn how to optimize the Tensorflow graphs for
mobile devices. We will also learn how to transform Tensorflow graphs for mobile devices.
$IBQUFS, Distributed TensorFlow and Cloud Deep Learning, explains the cloud
environment, dockers, containers, and how to use them. We learn how to work with distributed
Tensorflow with multiple GPUs, and multiple servers. We learn how to setup AWS for deep
learning. We learn how to setup Google cloud for deep learning applications. We learn how
to setup Microsoft Azure cloud for deep learning applications. We learn about other
available cloud services
[3]
Preface
"QQFOEJY", Learning to Learn with AutoML (Meta-Learning), briefly talks about AutoML and
Siamese Networks.
"QQFOEJY#, TensorFlow Processing Unit, covers Tensor Processing Unit, its basic
architecture, and how it will affect DNN's future.
Sections
In this book, you will find several headings that appear frequently (Getting ready, How to
do itd, How it worksd, There's mored, and See also). To give clear instructions on how to
complete a recipe, we use these sections as follows.
Getting ready
This section tells you what to expect in the recipe, and describes how to set up any software
or any preliminary settings required for the recipe.
[4]
Preface
How to do it`
This section contains the steps required to follow the recipe.
How it works`
This section usually consists of a detailed explanation of what happened in the previous
section.
There's more`
This section consists of additional information about the recipe in order to make the reader
more knowledgeable about the recipe.
See also
This section provides helpful links to other useful information for the recipe.
Conventions
In this book, you will find a number of text styles that distinguish between different kinds
of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions,
pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Create a
new user for JIRA in the database and grant the user access to the KJSBEC database we just
created using the following command:"
New terms and important words are shown in bold. Words that you see on the screen, for
example, in menus or dialog boxes, appear in the text like this: "Select System info from the
Administration panel."
[5]
Preface
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this
book-what you liked or disliked. Reader feedback is important for us as it helps us develop
titles that you will really get the most out of. To send us general feedback, simply e-mail
GFFECBDL!QBDLUQVCDPN, and mention the book's title in the subject of your message. If
there is a topic that you have expertise in and you are interested in either writing or
contributing to a book, see our author guide at XXXQBDLUQVCDPNBVUIPST .
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you
to get the most from your purchase.
1. Log in or register to our website using your e-mail address and password.
2. Hover the mouse pointer on the SUPPORT tab at the top.
3. Click on Code Downloads & Errata.
4. Enter the name of the book in the Search box.
5. Select the book for which you're looking to download the code files.
6. Choose from the drop-down menu where you purchased this book from.
7. Click on Code Download.
[6]
Preface
You can also download the code files by clicking on the Code Files button on the book's
webpage at the Packt Publishing website. This page can be accessed by entering the book's
name in the Search box. Please note that you need to be logged in to your Packt account.
Once the file is downloaded, please make sure that you unzip or extract the folder using the
latest version of:
The code bundle for the book is also hosted on GitHub at IUUQTHJUIVCDPN
1BDLU1VCMJTIJOH5FOTPS'MPXY%FFQ-FBSOJOH$PPLCPPL. We also have other code
bundles from our rich catalog of books and videos available at IUUQTHJUIVCDPN
1BDLU1VCMJTIJOH. Check them out!
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do
happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-
we would be grateful if you could report this to us. By doing so, you can save other readers
from frustration and help us improve subsequent versions of this book. If you find any
errata, please report them by visiting IUUQXXXQBDLUQVCDPNTVCNJUFSSBUB, selecting
your book, clicking on the Errata Submission Form link, and entering the details of your
errata. Once your errata are verified, your submission will be accepted and the errata will
be uploaded to our website or added to any list of existing errata under the Errata section of
that title. To view the previously submitted errata, go to IUUQTXXXQBDLUQVCDPN
CPPLTDPOUFOUTVQQPSU and enter the name of the book in the search field. The required
information will appear under the Errata section.
Piracy
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At
Packt, we take the protection of our copyright and licenses very seriously. If you come
across any illegal copies of our works in any form on the Internet, please provide us with
the location address or website name immediately so that we can pursue a remedy. Please
contact us at DPQZSJHIU!QBDLUQVCDPN with a link to the suspected pirated material. We
appreciate your help in protecting our authors and our ability to bring you valuable
content.
[7]
Preface
Questions
If you have a problem with any aspect of this book, you can contact us at
RVFTUJPOT!QBDLUQVCDPN, and we will do our best to address the problem.
[8]
TensorFlow - An Introduction
1
Anyone who has ever tried to write code for neural networks in Python using only NumPy,
knows how cumbersome it is. Writing code for a simple one-layer feedforward network
requires more than 40 lines, made more difficult as you add the number of layers both in
terms of writing code and execution time.
TensorFlow makes it all easier and faster reducing the time between the implementation of
an idea and deployment. In this book, you will learn how to unravel the power of
TensorFlow to implement deep neural networks.
Installing TensorFlow
Hello world in TensorFlow
Understanding the TensorFlow program structure
Working with constants, variables, and placeholders
Performing matrix manipulations using TensorFlow
Using a data flow graph
Migrating from 0.x to 1.x
Using XLA to enhance computational performance
Invoking CPU/GPU devices
TensorFlow for deep learning
Different Python packages required for DNN-based problems
TensorFlow - An Introduction Chapter 1
Introduction
TensorFlow is a powerful open source software library developed by the Google Brain team
for deep neural networks (DNNs). It was first made available under the Apache 2.x License
in November 2015; as of today, its GitHub repository (IUUQTHJUIVCDPNUFOTPSGMPX
UFOTPSGMPX) has more than 17,000 commits, with roughly 845 contributors in just two years.
This by itself is a measure of both the popularity and performance of TensorFlow. The
following graph shows a comparison of the popular deep learning frameworks where it is
clearly visible that TensorFlow is the leader among them:
6JG(KIWTGKUDCUGFQPFCVCVCMGPHTQOVJG)KVJWDTGRQUKVQTKGUQHGCEJCUQP,WN['CEJ$WDDNGJCUCNGIGPF(TCOGYQTMEQPVTKDWVQTU
Let's first learn what exactly TensorFlow is and why is it so popular among DNN
researchers and engineers. TensorFlow, the open source deep learning library allows one to
deploy deep neural networks computation on one or more CPU, GPUs in a server, desktop
or mobile using the single TensorFlow API. You might ask, there are so many other deep
learning libraries such as Torch, Theano, Caffe, and MxNet; what makes TensorFlow
special? Most of other deep learning libraries like TensorFlow have auto-differentiation,
many are open source, most support CPU/GPU options, have pretrained models, and
support commonly used NN architectures like recurrent neural networks (RNNs),
convolutional neural networks (CNNs), and deep belief networks (DBNs). So, what
more is there in TensorFlow? Let's list them for you:
[ 10 ]
TensorFlow - An Introduction Chapter 1
It works with all the cool languages. TensorFlow works with Python, C++, Java,
R, and Go.
TensorFlow works on multiple platforms, even mobile and distributed.
It is supported by all cloud providers--AWS, Google, and Azure.
Keras, a high-level neural network API, has been integrated with TensorFlow.
It has better computational graph visualizations because it is native while the
equivalent in Torch/Theano is not nearly as cool to look at.
TensorFlow allows model deployment and ease of use in production.
TensorFlow has very good community support.
TensorFlow is more than a software library; it is a suite of software that includes
TensorFlow, TensorBoard, and TensorServing.
The list is long, and the possibilities in which one can use TensorFlow are even greater. This
book aims to provide you with an understanding of TensorFlow as applied to deep learning
models such that you can adapt them to your dataset with ease and develop useful
applications. Each chapter contains a set of recipes which deal with the technical issues, the
dependencies, the actual code, and its understanding. We have built the recipes one on
another such that, by the end of each chapter, you have a fully functional deep learning
model.
[ 11 ]
TensorFlow - An Introduction Chapter 1
Installing TensorFlow
In this recipe, you will learn how to do a fresh installation of TensorFlow 1.3 on different
OSes (Linux, Mac, and Windows). We will find out about the necessary requirements to
install TensorFlow. TensorFlow can be installed using native pip, Anaconda, virtualenv,
and Docker on Ubuntu and macOS. For Windows OS, one can use native pip or Anaconda.
As Anaconda works on all the three OSes and provides an easy way to not only install
but also to maintain different project environments on the same system, we will concentrate
on installing TensorFlow using Anaconda in this book. More details about Anaconda and
managing its environment can be read from IUUQTDPOEBJPEPDTVTFSHVJEFJOEFY
IUNM.
The code in this book has been tested on the following platforms:
Windows 10, Anaconda 3, Python 3.5, TensorFlow GPU, CUDA toolkit 8.0,
cuDNN v5.1, NVDIAc GTX 1070
Windows 10/ Ubuntu 14.04/ Ubuntu 16.04/macOS Sierra, Anaconda3, Python 3.5,
TensorFlow (CPU)
Getting ready
The prerequisite for TensorFlow installation is that the system has Python 2.5 or higher
installed. The recipes in this book have been designed for Python 3.5 (the Anaconda 3
distribution). To get ready for the installation of TensorFlow, first ensure that you have
Anaconda installed. You can download and install Anaconda for Windows/macOS or Linux
from IUUQTXXXDPOUJOVVNJPEPXOMPBET.
After installation, you can verify the installation using the following command in your
terminal window:
DPOEBWFSTJPO
Once Anaconda is installed, we move to the next step, deciding whether to install
TensorFlow CPU or GPU. While almost all computer machines support TensorFlow CPU,
TensorFlow GPU can be installed only if the machine has an NVDIAc GPU card with
CUDA compute capability 3.0 or higher (minimum NVDIAc GTX 650 for desktop PCs).
[ 12 ]
TensorFlow - An Introduction Chapter 1
CPU versus GPU: Central Processing Unit (CPU) consists of a few cores
(4-8) optimized for sequential serial processing. A Graphical Processing
Unit (GPU) on the other hand has a massively parallel architecture
consisting of thousands of smaller, more efficient cores (roughly in 1,000s)
designed to handle multiple tasks simultaneously.
For TensorFlow GPU, it is imperative that CUDA toolkit 7.0 or greater is installed, proper
NVDIAc drivers are installed, and cuDNN v3 or greater is installed. On Windows,
additionally, certain DLL files are needed; one can either download the required DLL files
or install Visual Studio C++. One more thing to remember is that cuDNN files are installed
in a different directory. One needs to ensure that directory is in the system path. One can
also alternatively copy the relevant files in CUDA library in the respective folders.
How to do it...
We proceed with the recipe as follows:
1. Create a conda environment using the following at command line (If you are
using Windows it will be better to do it as Administrator in the command line):
DPOEBDSFBUFOUFOTPSGMPXQZUIPO
8JOEPXT
BDUJWBUFUFOTPSGMPX
.BD046CVOUV
TPVSDFBDUJWBUFUFOTPSGMPX
8JOEPXT
UFOTPSGMPX$
.BD046CVOUV
UFOTPSGMPX
4. Next, depending on the TensorFlow version you want to install inside your
conda environment, enter the following command:
8JOEPXT
$167FSTJPOPOMZ
UFOTPSGMPX$ QJQJOTUBMMJHOPSFJOTUBMMFEVQHSBEF
IUUQTTUPSBHFHPPHMFBQJTDPNUFOTPSGMPXXJOEPXTDQVUFOTPSGMPX
[ 13 ]
TensorFlow - An Introduction Chapter 1
DSDQDQNXJO@BNEXIM
(167FSTJPO
UFOTPSGMPX$ QJQJOTUBMMJHOPSFJOTUBMMFEVQHSBEF
IUUQTTUPSBHFHPPHMFBQJTDPNUFOTPSGMPXXJOEPXTHQVUFOTPSGMPX@HQ
VDSDQDQNXJO@BNEXIM
.BD04
$16POMZ7FSTJPO
UFOTPSGMPXQJQJOTUBMMJHOPSFJOTUBMMFEVQHSBEF
IUUQTTUPSBHFHPPHMFBQJTDPNUFOTPSGMPXNBDDQVUFOTPSGMPXD
SQZOPOFBOZXIM
(16WFSTJPO
UFOTPSGMPXQJQJOTUBMMJHOPSFJOTUBMMFEVQHSBEF
IUUQTTUPSBHFHPPHMFBQJTDPNUFOTPSGMPXNBDHQVUFOTPSGMPX@HQV
DSQZOPOFBOZXIM
6CVOUV
$16POMZ7FSTJPO
UFOTPSGMPXQJQJOTUBMMJHOPSFJOTUBMMFEVQHSBEF
IUUQTTUPSBHFHPPHMFBQJTDPNUFOTPSGMPXMJOVYDQVUFOTPSGMPX
DSDQDQNMJOVY@Y@XIM
(167FSTJPO
UFOTPSGMPXQJQJOTUBMMJHOPSFJOTUBMMFEVQHSBEF
IUUQTTUPSBHFHPPHMFBQJTDPNUFOTPSGMPXMJOVYHQVUFOTPSGMPX@HQV
DSDQDQNMJOVY@Y@XIM
JNQPSUUFOTPSGMPXBTUG
NFTTBHFUGDPOTUBOU
8FMDPNFUPUIFFYDJUJOHXPSMEPG%FFQ/FVSBM
/FUXPSLT
XJUIUG4FTTJPO
BTTFTT
QSJOU
TFTTSVO
NFTTBHFEFDPEF
[ 14 ]
TensorFlow - An Introduction Chapter 1
How it works...
TensorFlow is distributed by Google using the wheels standard. It is a ZIP format archive
with the XIM extension. Python 3.6, the default Python interpreter in Anaconda 3, does not
have wheels installed. At the time of writing the book, wheel support for Python 3.6 exists
only for Linux/Ubuntu. Therefore, while creating the TensorFlow environment, we specify
Python 3.5. This installs pip, python, and wheel along with a few other packages in the
conda environment named UFOTPSGMPX.
[ 15 ]
TensorFlow - An Introduction Chapter 1
Once the conda environment is created, the environment is activated using the TPVSDF
BDUJWBUFBDUJWBUF command. In the activated environment, use the QJQJOTUBMM
command with appropriate TensorFlow-API URL to install the required TensorFlow.
Although there exists an Anaconda command to install TensorFlow CPU using conda forge
TensorFlow documentation recommends using QJQJOTUBMM. After installing TensorFlow
in the conda environment, we can deactivate it. Now you are ready to execute your first
TensorFlow program.
When the program runs, you may see a few warning (W) messages, some information (I)
messages, and lastly the output of your code:
8FMDPNFUPUIFFYDJUJOHXPSMEPG%FFQ/FVSBM/FUXPSLT
Congratulations for successfully installing and executing your first TensorFlow code! We
will go through the code in more depth in the next recipe.
There's more...
Additionally, you can also install Jupyter notebook:
DPOEBJOTUBMMDBOBDPOEBJQZUIPO
2. Install OC@DPOEB@LFSOFMT:
DPOEBJOTUBMMDIBOOFMDPOEBGPSHFOC@DPOEB@LFSOFMT
KVQZUFSOPUFCPPL
If you already have TensorFlow installed on your system, you can use QJQJOTUBMM
VQHSBEFUFOTPSGMPX to upgrade it.
[ 16 ]
TensorFlow - An Introduction Chapter 1
How to do it...
1. Import UFOTPSGMPX this imports the TensorFlow library and allows you to use
its wonderful features.
JNQPSUUFOTPSGMPXBTUG
NFTTBHFUGDPOTUBOU
8FMDPNFUPUIFFYDJUJOHXPSMEPG%FFQ/FVSBM
/FUXPSLT
3. To execute the graph element, we need to define the Session using XJUI and run
the session using SVO:
XJUIUG4FTTJPO
BTTFTT
QSJOU
TFTTSVO
NFTTBHFEFDPEF
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF44&JOTUSVDUJPOTCVU
UIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF44&JOTUSVDUJPOT
CVUUIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF44&JOTUSVDUJPOT
[ 17 ]
TensorFlow - An Introduction Chapter 1
CVUUIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF44&JOTUSVDUJPOT
CVUUIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF44&JOTUSVDUJPOT
CVUUIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF"79JOTUSVDUJPOTCVU
UIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF"79JOTUSVDUJPOT
CVUUIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5IF5FOTPS'MPXMJCSBSZXBTO UDPNQJMFEUPVTF'."JOTUSVDUJPOTCVU
UIFTFBSFBWBJMBCMFPOZPVSNBDIJOFBOEDPVMETQFFEVQ$16
DPNQVUBUJPOT
5. If you are working with TensorFlow GPU, you also get a list of informative
messages (I) giving details of the devices used:
'PVOEEFWJDFXJUIQSPQFSUJFT
OBNF(F'PSDF(59
NBKPSNJOPSNFNPSZ$MPDL3BUF
()[
QDJ#VT*%
5PUBMNFNPSZ(J#
'SFFNFNPSZ(J#
%."
:
$SFBUJOH5FOTPS'MPXEFWJDF
HQV
EFWJDFOBNF(F'PSDF
(59QDJCVTJE
8FMDPNFUPUIFFYDJUJOHXPSMEPG%FFQ/FVSBM/FUXPSLT
[ 18 ]
TensorFlow - An Introduction Chapter 1
How it works...
The preceding code is divided into three main parts. There is the import block that contains
all the libraries our code will use; in the present code, we use only TensorFlow. The JNQPSU
UFOTPSGMPXBTUG statement gives Python access to all TensorFlow's classes, methods,
and symbols. The second block contains the graph definition part; here, we build our
desired computational graph. In the present case, our graph consists of only one node, the
tensor constant message consisting of byte string, 8FMDPNFUPUIFFYDJUJOHXPSME
PG%FFQ/FVSBM/FUXPSLT. The third component of our code is running the
computational graph as Session; we created a Session using the XJUI keyword. Finally , in
the Session, we run the graph created above.
Let's now understand the output. The warning messages that are received tell you that
TensorFlow code could run at an even higher speed, which can be achieved by installing
TensorFlow from source (we will do this in a later recipe in this chapter). The information
messages received inform you about the devices used for computation. On their part, both
messages are quite harmless, but if you don't like seeing them, adding this two-line code
will do the trick:
JNQPSUPT
PTFOWJSPO< 5'@$11@.*/@-0(@-&7&- >
The code is to ignore all messages till level 2. Level 1 is for information, 2 for warnings, and
3 for error messages.
The program prints the result of running the graph created the graph is run using the
TFTTSVO
statement. The result of running the graph is fed to the QSJOU function, which
is further modified using the EFDPEF method. The TFTTSVO evaluates the tensor defined
in the message. The QSJOU function prints on TUEPVU the result of the evaluation:
C 8FMDPNFUPUIFFYDJUJOHXPSMEPG%FFQ/FVSBM/FUXPSLT
This says that the result is a CZUFTUSJOH. To remove string quotes and b (for byte), we
use the method EFDPEF
.
[ 19 ]
TensorFlow - An Introduction Chapter 1
To build a computation graph we define all the constants, variables, and operations that we
need to perform. Constants, variables, and placeholders will be dealt with in the next recipe.
Mathematical operations will be dealt in detail in the recipe for matrix manipulations. Here,
we describe the structure using a simple example of defining and executing a graph to add
two vectors.
Execution of the graph: The execution of the graph is performed using Session Object. The
Session Object encapsulates the environment in which tensor and Operation Objects are evaluated.
This is the place where actual calculations and transfer of information from one layer to
another takes place. The values of different tensor Objects are initialized, accessed, and
saved in Session Object only. Up to now the tensor Objects were just abstract definitions,
here they come to life.
How to do it...
We proceed with the recipe as follows:
1. We consider a simple example of adding two vectors, we have two inputs vectors
W@ and W@ they are to be fed as input to the "EE operation. The graph we want
to build is as follows:
[ 20 ]
TensorFlow - An Introduction Chapter 1
W@UGDPOTUBOU
<>
W@UGDPOTUBOU
<>
W@BEEUGBEE
W@W@:PVDBOBMTPXSJUFW@
W@JOTUFBE
XJUIUG4FTTJPO
BTTFTT
QSJO
TFTTSVO
W@BEE
The above two commands are equivalent to the following code. The advantage of
using XJUI block is that one need not close the session explicitly.
TFTTUG4FTTJPO
QSJOU
TFTSVO
UW@BEE
TFTTDMPTF
<>
How it works...
The building of a computational graph is very simple; you go on adding the variables and
operations and passing them through (flow the tensors) in the sequence you build your
neural network layer by layer. TensorFlow also allows you to use specific devices
(CPU/GPU) with different objects of the computation graph using XJUIUGEFWJDF
. In
our example, the computational graph consists of three nodes, W@ and W@ representing
the two vectors, and "EE is the operation to be performed on them.
[ 21 ]
TensorFlow - An Introduction Chapter 1
Now, to bring this graph to life, we first need to define a session object using
UG4FTTJPO
; we gave the name TFTT to our session object. Next, we run it using the SVO
method defined in Session class as follows:
SVO
GFUDIFTGFFE@EJDU/POFPQUJPOT/POFSVO@NFUBEBUB
This evaluates the tensor in GFUDIFT; our example has tensor W@BEE in fetches. The SVO
method will execute every tensor and every operation in the graph leading to W@BEE. If
instead of W@BEE, you have W@ in fetches, the result will be the value of vector W@:
<>
Fetches can be a single tensor/operation object or more, for example, if the fetches is <W@
W@W@BEE>, the output will be the following:
<BSSBZ
<>BSSBZ
<>BSSBZ
<>>
There's more...
You must be wondering why we have to write so many lines of code for a simple vector
addition or to print a small message. Well, you could have very conveniently done this
work in a one-liner:
QSJOU
UG4FTTJPO
SVO
UGBEE
UGDPOTUBOU
<>UGDPOTUBOU
<>
Writing this type of code not only affects the computational graph but can be memory
expensive when the same operation ( OP) is performed repeatedly in a for loop. Making a
habit of explicitly defining all tensor and operation objects not only makes the code more
readable but also helps you visualize the computational graph in a cleaner manner.
[ 22 ]
TensorFlow - An Introduction Chapter 1
If you are working on Jupyter Notebook or Python shell, it is more convenient to use
UG*OUFSBDUJWF4FTTJPO instead of UG4FTTJPO. *OUFSBDUJWF4FTTJPO makes itself the
default session so that you can directly call run the tensor Object using FWBM
without
explicitly calling the session, as described in the following example code:
TFTTUG*OUFSBDUJWF4FTTJPO
W@UGDPOTUBOU
<>
W@UGDPOTUBOU
<>
W@BEEUGBEE
W@W@
QSJOU
W@BEEFWBM
TFTTDMPTF
Constants
Variables
Placeholders
[ 23 ]
TensorFlow - An Introduction Chapter 1
Variables: We use variable tensors when the values require updating within a session. For
example, in the case of neural networks, the weights need to be updated during the training
session, which is achieved by declaring weights as variables. The variables need to be
explicitly initialized before use. Another important thing to note is that constants are stored
in the computation graph definition; they are loaded every time the graph is loaded. In
other words, they are memory expensive. Variables, on the other hand, are stored
separately; they can exist on the parameter server.
Placeholders: These are used to feed values into a TensorFlow graph. They are used along
with GFFE@EJDU to feed the data. They are normally used to feed new training examples
while training a neural network. We assign a value to a placeholder while running the
graph in the session. They allow us to create our operations and build the computation
graph without requiring the data. An important point to note is that placeholders do not
contain any data and thus there is no need to initialize them as well.
How to do it...
Let's start with constants:
U@UGDPOTUBOU
U@UGDPOTUBOU
<>
UG[FSPT
<./>UGEUZQF
4. We can also create tensor constants of the same shape as an existing Numpy
array or tensor constant as follows:
UG[FSPT@MJLF
U@
$SFBUFB[FSPNBUSJYPGTBNFTIBQFBTU@
UGPOFT@MJLF
U@
[ 24 ]
TensorFlow - An Introduction Chapter 1
$SFBUFTBPOFTNBUSJYPGTBNFTIBQFBTU@
5. We can create a tensor with all elements set to one; here, we create a ones matrix
of shape <./>:
UGPOFT
<./>UGEUZQF
1. We can generate a sequence of evenly spaced vectors, starting from start to stop,
within total num values:
UGMJOTQBDF
TUBSUTUPQOVN
SBOHF@UUGMJOTQBDF
8FHFU<>
UGSBOHF
TUBSUMJNJUEFMUB
Here is an example:
SBOHF@UUGSBOHF
3FTVMU<>
1. To create random values from a normal distribution of shape <./> with the
mean (default =0.0) and standard deviation (default=1.0) with seed, we can use
the following:
U@SBOEPNUGSBOEPN@OPSNBM
<>NFBOTUEEFWTFFE
3FTVMU<<><
>>
[ 25 ]
TensorFlow - An Introduction Chapter 1
U@SBOEPNUGUSVODBUFE@OPSNBM
<>TUEEFWTFFE
3FTVMU<<
>>
3. To create random values from a given gamma distribution of shape <./> in the
range <NJOWBM
EFGBVMUNBYWBM> with seed, perform as follows:
U@SBOEPNUGSBOEPN@VOJGPSN
<>NBYWBMTFFE
3FTVMU<<><
>>
UGSBOEPN@DSPQ
U@SBOEPN<>TFFE
Here, U@SBOEPN is an already defined tensor. This will result in a <> tensor
randomly cropped from tensor U@SBOEPN.
Many times we need to present the training sample in random order; we can use
UGSBOEPN@TIVGGMF
to randomly shuffle a tensor along its first dimension. If
U@SBOEPN is the tensor we want to shuffle, then we use the following:
UGSBOEPN@TIVGGMF
U@SBOEPN
5. Randomly generated tensors are affected by the value of the initial seed. To
obtain the same random numbers in multiple runs or sessions, the seed should be
set to a constant value. When there are large numbers of random tensors in use,
we can set the seed for all randomly generated tensors using
UGTFU@SBOEPN@TFFE
; the following command sets the seed for random
tensors for all sessions as :
UGTFU@SBOEPN@TFFE
[ 26 ]
TensorFlow - An Introduction Chapter 1
1. They are created using the variable class. The definition of variables also includes
the constant/random values from which they should be initialized. In the
following code, we create two different tensor variables, U@B and U@C. Both will
be initialized to random uniform distributions of shape <>, NJOWBM,
and NBYWBM:
SBOE@UUGSBOEPN@VOJGPSN
<>TFFE
U@BUG7BSJBCMF
SBOE@U
U@CUG7BSJBCMF
SBOE@U
2. In the following code, we define two variables weights and bias. The weights
variable is randomly initialized using normal distribution, with mean zeros and
standard deviation of two, the size of weights is 100e100. The bias consists of 100
elements each initialized to zero. Here we have also used the optional argument
name to give a name to the variable defined in the computational graph.
XFJHIUTUG7BSJBCMF
UGSBOEPN@OPSNBM
<>TUEEFW
CJBTUG7BSJBCMF
UG[FSPT<>OBNF CJBTFT
XFJHIUUG7BSJBCMF
XFJHIUTJOJUJBMJ[FE@WBMVF
OBNF X
JOUJBM@PQUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
CJBTUG7BSJBCMF
UG[FSPT
<>
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
CJBTJOJUJBMJ[FS
[ 27 ]
TensorFlow - An Introduction Chapter 1
6. Saving variables: We can save the variables using the 4BWFS class. To do this, we
define a TBWFS Operation Object:
TBWFSUGUSBJO4BWFS
UGQMBDFIPMEFS
EUZQFTIBQF/POFOBNF/POF
8. EUZQF specifies the data type of the placeholder and must be specified while
declaring the placeholder. Here, we define a placeholder for Y and calculate Z
Y using GFFE@EJDU for a random 4e5 matrix:
YUGQMBDFIPMEFS
GMPBU
ZY
EBUBUGSBOEPN@VOJGPSN
<>
XJUIUG4FTTJPO
BTTFTT
Y@EBUBTFTTSVO
EBUB
QSJOU
TFTTSVO
ZGFFE@EJDU\YY@EBUB^
How it works...
All constants, variables, and placeholders will be defined in the
computation graph section of the code. If we use the print statement in the
definition section, we will only get information about the type of tensor,
and not its value.
To find out the value, we need to create the session graph and explicitly use the SVO
command with the desired tensor values as GFUDIFT:
QSJOU
TFTTSVO
U@
8JMMQSJOUUIFWBMVFPGU@EFGJOFEJOTUFQ
[ 28 ]
TensorFlow - An Introduction Chapter 1
There's more...
Very often, we will need constant tensor objects with a large size; in this case, to optimize
memory, it is better to declare them as variables with a trainable flag set to 'BMTF:
U@MBSHFUG7BSJBCMF
MBSHF@BSSBZUSBJOBCMF'BMTF
TensorFlow was designed to work impeccably with Numpy, hence all the TensorFlow data
types are based on those of Numpy. Using UGDPOWFSU@UP@UFOTPS
, we can convert the
given value to tensor type and use it with TensorFlow functions and operators. This
function accepts Numpy arrays, Python Lists, and Python scalars and allows
interoperability with tensor Objects.
The following table lists some of the common TensorFlow supported data types (taken from
5FOTPS'MPXPSH):
Note that unlike Python/Numpy sequences, TensorFlow sequences are not iterable. Try the
following code:
GPSJJOUGSBOHF
[ 29 ]
TensorFlow - An Introduction Chapter 1
This recipe will show you how to get different types of matrices and how to perform
different matrix manipulation operations on them.
How to do it...
We proceed with the recipe as follows:
JNQPSUUFOTPSGMPXBTUG
4UBSUBO*OUFSBDUJWF4FTTJPO
TFTTUG*OUFSBDUJWF4FTTJPO
%FGJOFBY*EFOUJUZNBUSJY
*@NBUSJYUGFZF
QSJOU
*@NBUSJYFWBM
5IJTXJMMQSJOUBY*EFOUJUZNBUSJY
%FGJOFB7BSJBCMFJOJUJBMJ[FEUPBYJEFOUJUZNBUSJY
9UG7BSJBCMF
UGFZF
9JOJUJBMJ[FSSVO
*OJUJBMJ[FUIF7BSJBCMF
QSJOU
9FWBM
&WBMVBUFUIF7BSJBCMFBOEQSJOUUIFSFTVMU
$SFBUFBSBOEPNYNBUSJY
"UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
"JOJUJBMJ[FSSVO
.VMUJQMZUXPNBUSJDFT
QSPEVDUUGNBUNVM
"9
QSJOU
QSPEVDUFWBM
DSFBUFBSBOEPNNBUSJYPGTBOETTJ[FY
CUG7BSJBCMF
UGSBOEPN@VOJGPSN
<>EUZQFUGJOU
CJOJUJBMJ[FSSVO
[ 30 ]
TensorFlow - An Introduction Chapter 1
QSJOU
CFWBM
C@OFXUGDBTU
CEUZQFUGGMPBU
$BTUUPGMPBUEBUBUZQF
"EEUIFUXPNBUSJDFT
U@TVNUGBEE
QSPEVDUC@OFX
U@TVCQSPEVDUC@OFX
QSJOU
"9@C=OU@TVNFWBM
QSJOU
"9C=OU@TVCFWBM
JNQPSUUFOTPSGMPXBTUG
$SFBUFUXPSBOEPNNBUSJDFT
BUG7BSJBCMF
UGSBOEPN@OPSNBM
<>TUEEFW
CUG7BSJBCMF
UGSBOEPN@OPSNBM
<>TUEEFW
&MFNFOU8JTF.VMUJQMJDBUJPO
"BC
.VMUJQMJDBUJPOXJUIBTDBMBS
#UGTDBMBS@NVM
"
&MFNFOUXJTFEJWJTJPOJUTSFTVMUJT
$UGEJW
BC
&MFNFOU8JTFSFNBJOEFSPGEJWJTJPO
%UGNPE
BC
JOJU@PQUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU@PQ
XSJUFSUGTVNNBSZ'JMF8SJUFS
HSBQIT TFTTHSBQI
BC"@3#@3$@3%@3TFTTSVO
<BC"#$%>
QSJOU
B=OB=OC=OCBC=O"@3=OBC=O#@3
=OBC=O$@3=OBC=O%@3
XSJUFSDMPTF
[ 31 ]
TensorFlow - An Introduction Chapter 1
How it works...
All arithmetic operations of matrices like add, sub, div, multiply (elementwise
multiplication), mod, and cross require that the two tensor matrices should be of the same
data type. In case this is not so they will produce an error. We can use UGDBTU
to
convert Tensors from one data type to another.
There's more...
If we are doing division between integer tensors, it is better to use UGUSVFEJW
BC as it
first casts the integer tensors to floating points and then performs element-wise division.
How to do it...
1. The first step in using TensorBoard is to identify which OPs summaries you
would like to have. In the case of DNNs, it is customary to know how the loss
term (objective function) varies with time. In the case of Adaptive learning rate,
the learning rate itself varies with time. We can get the summary of the term we
require with the help of UGTVNNBSZTDBMBS OPs. Suppose, the variable loss
defines the error term and we want to know how it varies with time, then we can
do this as follows:
MPTTUG
UGTVNNBSZTDBMBS
MPTT MPTT
2. You can also visualize the distribution of gradients, weights, or even output of a
particular layer using UGTVNNBSZIJTUPHSBN:
PVUQVU@UFOTPSUGNBUNVM
JOQVU@UFOTPSXFJHIUT
CJBTFT
UGTVNNBSZIJTUPHSBN
PVUQVU PVUQVU@UFOTPS
[ 32 ]
TensorFlow - An Introduction Chapter 1
3. The summaries will be generated during the session. Instead of executing every
summary operation individually, you can define UGNFSHF@BMM@TVNNBSJFT
OPs in the computation graph to get all summaries in a single run.
4. The generated summary then needs to be written in an event file using
UGTVNNBSZ'JMFXSJUFS:
XSJUFSUGTVNNBSZ'JMFXSJUFS
TVNNBSZ@EJS TFTTHSBQI
5. This writes all the summaries and the graph in the TVNNBSZ@EJS directory.
6. Now, to visualize the summaries, you need to invoke TensorBoard from the
command line:
UFOTPSCPBSEMPHEJSTVNNBSZ@EJS
7. Next, open your browser and type the address IUUQMPDBMIPTU (or
the link you received after running the TensorBoard command).
8. You will see something like the following, with many tabs on the top. The
Graphs tab will display the graph:
[ 33 ]
TensorFlow - An Introduction Chapter 1
How to do it...
Here is how we proceed with the recipe:
QZUIPOUG@VQHSBEFQZJOGJMFPME@GJMFQZPVUGJMF
VQHSBEFE@GJMFQZ
3. For example, if you have a TensorFlow program file named UFTUQZ, you will
use the preceding command as follows:
QZUIPOUG@VQHSBEFQZJOGJMFUFTUQZPVUGJMFUFTU@QZ
QZUIPOUG@VQHSBEFQZJOUSFF*OQVU%*SPVUUSFF0VUQVU%JS
'PSFYBNQMFJGZPVIBWFBEJSFDUPSZMPDBUFEBU
IPNFVTFSNZ@EJSZPVDBONJHSBUFBMMUIFQZUIPOGJMFTJOUIF
EJSFDUPSZMPDBUFEBUIPNFVTFSNZEJS@QVTJOHUIFBCPWFDPNNBOE
BT
QZUIPOUG@VQHSBEFQZJOUSFFIPNFVTFSNZ@EJSPVUUSFF
IPNFVTFSNZ@EJS@Q
[ 34 ]
TensorFlow - An Introduction Chapter 1
6. In most cases, the directory also contains dataset files; you can ensure that non-
Python files are copied as well in the new directory (NZEJS@Q in the
preceding example) using the following:
QZUIPOUG@VQHSBEFQZJOUSFFIPNFVTFSNZ@EJSPVUUSFF
IPNFVTFSNZ@EJS@QDPQZPUIFSGJMFT5SVF
7. In all these cases, a SFQPSUUYU file is generated. This file contains the details of
conversion and any errors in the process.
8. Read the SFQPSUUYU file and manually upgrade the part of the code that the
script is unable to update.
There's more...
UG@VQHSBEFQZ has certain limitations:
XJUIUGWBSJBCMF@TDPQF
UGHFU@WBSJBCMF@TDPQF
SFTVTF5SVF
[ 35 ]
TensorFlow - An Introduction Chapter 1
Getting ready
At present, XLA is not included in the binary distributions of TensorFlow. One needs to
build it from source. To build TensorFlow from source, knowledge of LLVM and Bazel
along with TensorFlow is required. 5FOTPS'MPXPSH supports building from source in only
MacOS and Ubuntu. The steps needed to build TensorFlow from the source are as follows
(IUUQTXXXUFOTPSGMPXPSHJOTUBMMJOTUBMM@TPVSDFT):
HJUDMPOFIUUQTHJUIVCDPNUFOTPSGMPXUFOTPSGMPX
DEUFOTPSGMPX
HJUDIFDLPVU#SBODIXIFSF#SBODIJTUIFEFTJSFECSBODI
Bazel
TensorFlow Python dependencies
For the GPU version, NVIDIA packages to support TensorFlow
4. Configure the installation. In this step, you need to choose different options such
as XLA, Cuda support, Verbs, and so on:
DPOGJHVSF
CB[FMCVJMEDPOGJHPQU
UFOTPSGMPXUPPMTQJQ@QBDLBHFCVJME@QJQ@QBDLBHF
7. If you have a compatible GPU device and you want the GPU Support, then use:
CB[FMCVJMEDPOGJHPQUDPOGJHDVEB
UFOTPSGMPXUPPMTQJQ@QBDLBHFCVJME@QJQ@QBDLBHF
CB[FMCJOUFOTPSGMPXUPPMTQJQ@QBDLBHFCVJME@QJQ@QBDLBHF
UNQUFOTPSGMPX@QLH
[ 36 ]
TensorFlow - An Introduction Chapter 1
TVEPQJQJOTUBMMUNQUFOTPSGMPX@QLHUFOTPSGMPXQZOPOF
BOZXIM
How to do it...
TensorFlow generates TensorFlow graphs. With the help of XLA, it is possible to run the
TensorFlow graphs on any new kind of device.
$POGJHUPUVSOPO+*5DPNQJMBUJPO
DPOGJHUG$POGJH1SPUP
DPOGJHHSBQI@PQUJPOTPQUJNJ[FS@PQUJPOTHMPCBM@KJU@MFWFM
UG0QUJNJ[FS0QUJPOT0/@
TFTTUG4FTTJPO
DPOGJHDPOGJH
KJU@TDPQFUGDPOUSJCDPNQJMFSKJUFYQFSJNFOUBM@KJU@TDPQF
YUGQMBDFIPMEFS
OQGMPBU
XJUIKJU@TDPQF
ZUGBEE
YY5IFBEEXJMMCFDPNQJMFEXJUI9-"
3. We can also run computations via XLA by placing the operator on a specific XLA
device 9-"@$16 or 9-"@(16:
XJUIUGEFWJDF=
KPCMPDBMIPTUSFQMJDBUBTLEFWJDF9-"@(16
PVUQVUUGBEE
JOQVUJOQVU
AoT Compilation: Here, we use tfcompile as standalone to convert TensorFlow graphs into
executable code for different devices (mobile).
[ 37 ]
TensorFlow - An Introduction Chapter 1
tfcompile takes a subgraph, identified by the TensorFlow concepts of feeds and fetches, and
generates a function that implements that subgraph. The feeds are the input arguments for
the function, and the fetches are the output arguments for the function. All inputs must be
fully specified by the feeds; the resulting pruned subgraph cannot contain placeholder or
variable nodes. It is common to specify all placeholders and variables as feeds, which
ensures the resulting subgraph no longer contains these nodes. The generated function is
packaged as a cc_library, with a header file exporting the function signature, and an object
file containing the implementation. The user writes code to invoke the generated function
as appropriate.
As mentioned earlier, GPUs are much faster than CPUs because they have many small
cores. However, it is not always an advantage in terms of computational speed to use GPUs
for all types of computations. The overhead associated with GPUs can sometimes be more
computationally expensive than the advantage of parallel computation offered by GPUs. To
deal with this issue, TensorFlow has provisions to place computations on a particular
device. By default, if both CPU and GPU are present, TensorFlow gives priority to GPU.
How to do it...
TensorFlow represents devices as strings. Here, we will show you how one can manually
assign a device for matrix multiplication in TensorFlow. To verify that TensorFlow is
indeed using the device (CPU or GPU) specified, we create the session with the
MPH@EFWJDF@QMBDFNFOU flag set to 5SVF, namely,
DPOGJHUG$POGJH1SPUP
MPH@EFWJDF@QMBDFNFOU5SVF:
[ 38 ]
TensorFlow - An Introduction Chapter 1
1. If you are not sure about the device and want TensorFlow to choose the existing
and supported device, you can set the BMMPX@TPGU@QMBDFNFOU flag to 5SVF:
DPOGJHUG$POGJH1SPUP
BMMPX@TPGU@QMBDFNFOU5SVF
MPH@EFWJDF@QMBDFNFOU5SVF
XJUIUGEFWJDF
DQV
SBOE@UUGSBOEPN@VOJGPSN
<>EUZQFUGGMPBU
TFFE
BUG7BSJBCMF
SBOE@U
CUG7BSJBCMF
SBOE@U
DUGNBUNVM
BC
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFTTUG4FTTJPO
DPOGJH
TFTTSVO
JOJU
QSJOU
TFTTSVO
D
We can see that all the devices, in this case, are DQV .
[ 39 ]
TensorFlow - An Introduction Chapter 1
XJUIUGEFWJDF
HQV
SBOE@UUGSBOEPN@VOJGPSN
<>EUZQFUGGMPBU
TFFE
BUG7BSJBCMF
SBOE@U
CUG7BSJBCMF
SBOE@U
DUGNBUNVM
BC
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFTTUG4FTTJPO
DPOGJHUG$POGJH1SPUP
MPH@EFWJDF@QMBDFNFOU5SVF
TFTTSVO
JOJU
QSJOU
TFTTSVO
D
D<>
GPSEJO< HQV HQV >
XJUIUGEFWJDF
E
SBOE@UUGSBOEPN@VOJGPSN
<>
EUZQFUGGMPBUTFFE
BUG7BSJBCMF
SBOE@U
[ 40 ]
TensorFlow - An Introduction Chapter 1
CUG7BSJBCMF
SBOE@U
DBQQFOE
UGNBUNVM
BC
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFTT
UG4FTTJPO
DPOGJHUG$POGJH1SPUP
BMMPX@TPGU@QMBDFNFOU5SVFMPH@EFWJ
DF@QMBDFNFOU5SVF
TFTTSVO
JOJU
QSJOU
TFTTSVO
D
TFTTDMPTF
8. In this case, if the system has three GPU devices, then the first set of
multiplication will be carried on by HQV and the second set by HQV .
How it works...
The UGEFWJDF
argument selects the device (CPU or GPU). The XJUI block ensures the
operations for which the device is selected. All the variables, constants, and operations
defined within the XJUI block will use the device selected in UGEFWJDF
. Session
configuration is controlled using UG$POGJH1SPUP. By setting the
BMMPX@TPGU@QMBDFNFOU and MPH@EFWJDF@QMBDFNFOU flags, we tell TensorFlow to
automatically choose the available devices in case the specified device is not available, and
to give log messages as output describing the allocation of devices while the session is
executed.
The main reason for this is the availability of modern computing power such as GPUs and
tools like TensorFlow that make it easier to access GPUs and construct complex neural
networks in just a few lines of code.
[ 41 ]
TensorFlow - An Introduction Chapter 1
As a machine learning enthusiast, you must already be familiar with the concepts of neural
networks and deep learning, but for the sake of completeness, we will introduce the basics
here and explore what features of TensorFlow make it a popular choice for deep learning.
Neural networks are a biologically inspired model for computation and learning. Like a
biological neuron, they take weighted input from other cells (neurons or environment); this
weighted input undergoes a processing element and results in an output which can be
binary (fire or not fire) or continuous (probability, prediction). Artificial Neural Networks
(ANNs) are networks of these neurons, which can be randomly distributed or arranged in a
layered structure. These neurons learn through the set of weights and biases associated with
them.
The following figure gives a good idea about the similarity of the neural network in biology
and an artificial neural network:
[ 42 ]
TensorFlow - An Introduction Chapter 1
How to do it...
Any deep learning network consists of four important components: Dataset, defining the
model (network structure), Training/Learning, and Prediction/Evaluation. We can do all
these in TensorFlow; let's see how:
Dataset: DNNs depend on large amounts of data. The data can be collected or
generated or one can also use standard datasets available. TensorFlow supports
three main methods to read the data. There are different datasets available; some
of the datasets we will be using to train the models built in this book are as
follows:
MNIST: It is the largest database of handwritten digits (0 - 9). It consists of a
training set of 60,000 examples and a test set of 10,000 examples. The dataset is
maintained at Yann LeCun's home page (IUUQZBOOMFDVODPNFYECNOJTU).
The dataset is included in the TensorFlow library in
UFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTU.
[ 43 ]
TensorFlow - An Introduction Chapter 1
Reading the data: The data can be read in three ways in TensorFlow--feeding through
GFFE@EJDU, reading from files, and using preloaded data. We will be using the components
described in this recipe throughout the book for reading, and feeding the data. In the next
steps, you will learn each one of them.
1. Feeding: In this case, data is provided while running each step, using the
GFFE@EJDU argument in the SVO
or FWBM
function call. This is done with the
help of placeholders and this method allows us to pass Numpy arrays of data.
Consider the following part of code using TensorFlow:
ZUGQMBDFIPMEFS
UGGMPBU
YUGQMBDFIPMEFS
UGGMPBU
[ 44 ]
TensorFlow - An Introduction Chapter 1
XJUIUG4FTTJPOBTTFTT
9@"SSBZTPNF/VNQZ"SSBZ
:@"SSBZPUIFS/VNQZ"SSBZ
MPTT
TFTTSVO
MPTTGFFE@EJDU\Y9@"SSBZZ:@"SSBZ^
Here, Y and Z are the placeholders; using them, we pass on the array
containing 9 values and the array containing : values with the help of
GFFE@EJDU.
2. Reading from files: This method is used when the dataset is very large to ensure
that not all data occupies the memory at once (imagine the 60 GB YouTube-8m
dataset). The process of reading from files can be done in the following steps:
GJMFOBNF@RVFVFUGUSBJOTUSJOH@JOQVU@QSPEVDFS
GJMFT
XIFSFGJMFTJTUIFMJTUPGGJMFOBNFTDSFBUFEBCPWF
Reader is defined and used to read from files from the filename queue.
The reader is selected based on the input file format. The read method a
key identifying the file and record (useful while debugging) and a scalar
string value. For example, in the case of DTW file formats:
SFBEFSUG5FYU-JOF3FBEFS
LFZWBMVFSFBEFSSFBE
GJMFOBNF@RVFVF
[ 45 ]
TensorFlow - An Introduction Chapter 1
Decoder: One or more decoder and conversion ops are then used to
decode the WBMVF string into Tensors that make up the training example:
SFDPSE@EFGBVMUT<<><><>>
DPMDPMDPMUGEFDPEF@DTW
WBMVF
SFDPSE@EFGBVMUTSFDPSE@EFGBVMUT
3. Preloaded data: This is used when the dataset is small and can be loaded fully in
the memory. For this, we can store data either in a constant or variable. While
using a variable, we need to set the trainable flag to False so that the data does
not change while training. As TensorFlow constants:
1SFMPBEFEEBUBBTDPOTUBOU
USBJOJOH@EBUB
USBJOJOH@MBCFMT
XJUIUG4FTTJPOBTTFTT
Y@EBUBUG$POTUBOU
USBJOJOH@EBUB
Z@EBUBUG$POTUBOU
USBJOJOH@MBCFMT
1SFMPBEFEEBUBBT7BSJBCMFT
USBJOJOH@EBUB
USBJOJOH@MBCFMT
XJUIUG4FTTJPOBTTFTT
EBUB@YUGQMBDFIPMEFS
EUZQFUSBJOJOH@EBUBEUZQF
TIBQFUSBJOJOH@EBUBTIBQF
EBUB@ZUGQMBDFIPMEFS
EUZQFUSBJOJOH@MBCFMEUZQF
TIBQFUSBJOJOH@MBCFMTIBQF
Y@EBUBUG7BSJBCMF
EBUB@YUSBJOBCMF'BMTFDPMMFDUJPOT<>
Z@EBUBUG7BSJBCMF
EBUB@ZUSBJOBCMF'BMTFDPMMFDUJPOT<>
Conventionally, the data is divided into three parts--training data, validation data, and test
data.
[ 46 ]
TensorFlow - An Introduction Chapter 1
XJUIUG4FTTJPOBTTFTT
TFTTSVO
6. Evaluating the model: Once the network is trained, we evaluate the network
using QSFEJDU
on validation data and test data. The evaluations give us an
estimate of how well our model fits the dataset. We can thus avoid the common
mistakes of overfitting or underfitting. Once we are satisfied with our model, we
can deploy it in production.
There's more
In TensorFlow 1.3, a new feature called TensorFlow Estimators has been added. TensorFlow
Estimators make the task of creating the neural network models even easier, it is a higher
level API that encapsulates the process of training, evaluation, prediction and serving. It
provides the option of either using pre-made Estimators or one can write their own custom
Estimators. With pre-made Estimators, one no longer have to worry about building the
computational or creating a session, it handles it all.
At present TensorFlow Estimator has six pre-made Estimators. Another advantage of using
TensorFlow pre-made Estimators is that it also by itself creates summaries that can be
visualised on TensorBoard. More details about Estimators are available at IUUQTXXX
UFOTPSGMPXPSHQSPHSBNNFST@HVJEFFTUJNBUPST.
[ 47 ]
TensorFlow - An Introduction Chapter 1
How to do it...
Here are listed some of the common Python packages used:
1. Numpy: This is the fundamental package for scientific computing with Python. It
supports n-dimensional arrays and matrices. It also has a large collection of high-
level mathematical functions. It is a necessary package required by TensorFlow
and therefore, with QJQJOTUBMMUFOTPSGMPX, it is installed if not already
present.
2. Matplolib: This is the Python 2D plotting library. You can use it to create plots,
histograms, bar charts, error charts, scatterplots, and power spectra with just a
few lines of code. It can be installed using QJQ
QJQJOTUBMMNBUQMPUMJC
PSVTJOH"OBDPOEB
DPOEBJOTUBMMDDPOEBGPSHFNBUQMPUMJC
3. OS: This is included in the basic Python installation. It provides an easy and
portable way of using operating system-dependent functionality like reading,
writing, and changing files and directories.
4. Pandas: This provides various data structures and data analysis tools. Using
Pandas, you can read and write data between in-memory data structures and
different formats. We can read from DTW and text files. It can be installed using
either QJQJOTUBMM or DPOEBJOTUBMM.
5. Seaborn: This is a specialized statistical data visualization tool built on
Matplotlib.
6. H5fs: H5fs is a filesystem for Linux (also other operating systems with FUSE
implementation like macOS X) capable of operations on an HDFS (Hierarchical
Data Format Filesystem).
7. PythonMagick: It is the Python binding of the *NBHF.BHJDL library. It is a
library to display, convert, and edit raster image and vector image files. It
supports more than 200 image file formats. It can be installed using source build
available from *NBHF.BHJDL Certain XIM formats are also available for a
convenient QJQJOTUBMM (IUUQXXXMGEVDJFEV&HPIMLFQZUIPOMJCT
QZUIPONBHJDL).
[ 48 ]
TensorFlow - An Introduction Chapter 1
8. TFlearn: TFlearn is a modular and transparent deep learning library built on top
of TensorFlow. It provides a higher-level API to TensorFlow in order to facilitate
and speed up experimentation. It currently supports most of the recent deep
learning models, such as Convolutions, LSTM, BatchNorm, BiRNN, PReLU,
Residual networks, and Generative networks. It works only for TensorFlow 1.0 or
higher. To install, use QJQJOTUBMMUGMFBSO.
9. Keras: Keras too is a high-level API for neural networks, which uses TensorFlow
as its backend. It can run on top of Theano and CNTK as well. It is extremely
user-friendly, adding layers to it is just a one-line job. It can be installed using
QJQJOTUBMMLFSBT.
See also
Below you can find some weblinks for more information on installation of TensorFlow
IUUQTXXXUFOTPSGMPXPSHJOTUBMM
IUUQTXXXUFOTPSGMPXPSHJOTUBMMJOTUBMM@TPVSDFT
IUUQMMWNPSH
IUUQTCB[FMCVJME
[ 49 ]
Regression
2
This chapter shows how regression can be done using TensorFlow. In this chapter, we will
cover the following topics:
Introduction
Regression is one of the oldest and yet quite powerful tools for mathematical modelling,
classification, and prediction. Regression finds application in varied fields ranging from
engineering, physical science, biology, and the financial market to social sciences. It is the
basic tool in the hand of a data scientist.
Regression is normally the first algorithm that people in machine learning work with. It
allows us to make predictions from data by learning the relationship between the
dependent and independent variables. For example, in the case of house price estimation,
we determine the relationship between the area of the house (independent variable) and its
price (dependent variable); this relationship can be then used to predict the price of any
house given its area. We can have multiple independent variables impacting the dependent
variable. Thus, there are two important components of regression: the relationship between
independent and dependent variables, and the strength of impact of the different
independent variable on the dependent variable.
Regression Chapter 2
Linear regression: This is one of the most widely used modelling technique.
Existing for more than 200 years, it has been explored from almost all possible
angles. Linear regression assumes a linear relationship between the input
variables (X) and the single output variable (Y). It involves finding a linear
equation for predicted value Y of the form:
Here, X = (x1, x2, ..., xn) are the n input variables and W = (w1, w2, ...wn) are the
linear coefficients, with b as the bias term. The goal is to find the best
estimates for the coefficients W, such that the error in the predicted Y is
minimized. The linear coefficients Ws are estimated using the method of least
squares, that is, minimizing the sum of squared differences between
predicted values (Yhat) and observed values (Y). Thus, we try to minimize the
loss function:
Here, the sum is over all the training samples. Depending on the number and
type of input variable X, there are different types of linear regression: simple
linear regression (one input variable, one output variable), multiple linear
regression (many independent input variables, one output variable), or
multivariate linear regression (many independent input variables and
multiple output variables). For more on linear regression, you can refer to
IUUQTFOXJLJQFEJBPSHXJLJ-JOFBS@SFHSFTTJPO.
[ 51 ]
Regression Chapter 2
The goal now is to estimate weights W = ( w1, w2, ...wn) and bias term b. In
logistic regression, the coefficients are estimated using either maximum
likelihood estimator or stochastic gradient descent. The loss is conventionally
defined as a cross-entropy term given as follows:
Here, K is the total number of classes. More about logistic regression can be
read at IUUQTFOXJLJQFEJBPSHXJLJ-PHJTUJD@SFHSFTTJPO.
[ 52 ]
Regression Chapter 2
Getting ready
Declaring a MPTT function requires defining the coefficients as variables and the dataset as
placeholders. One can have a constant learning rate or changing learning rate and
regularization constant. In the following code, let N be the number of samples, O the number
of features, and 1 the number of classes. We should define these global parameters before
the code:
N
O
1
[ 53 ]
Regression Chapter 2
How to do it...
Let us now see how to proceed with the recipe:
1. In the case of standard linear regression, we have only one input variable and one
output variable:
1MBDFIPMEFSGPSUIF5SBJOJOH%BUB
9UGQMBDFIPMEFS
UGGMPBUOBNF 9
:UGQMBDFIPMEFS
UGGMPBUOBNF :
7BSJBCMFTGPSDPFGGJDJFOUTJOJUJBMJ[FEUP
XUG7BSJBCMF
XUG7BSJBCMF
5IF-JOFBS3FHSFTTJPO.PEFM
:@IBU9X
X
-PTTGVODUJPO
MPTTUGTRVBSF
::@IBUOBNFbMPTTb
2. In the case of multiple linear regression, the input variables are more than one,
while the output variable remains one. Now you can define 9 placeholder of
shape <NO>, where N is the number of samples and O is the number of features,
then the code is as follows:
1MBDFIPMEFSGPSUIF5SBJOJOH%BUB
9UGQMBDFIPMEFS
UGGMPBUOBNF'X'TIBQF<NO>
:UGQMBDFIPMEFS
UGGMPBUOBNF'Y'
7BSJBCMFTGPSDPFGGJDJFOUTJOJUJBMJ[FEUP
XUG7BSJBCMF
XUG7BSJBCMF
UGSBOEPN@OPSNBM
<O>
5IF-JOFBS3FHSFTTJPO.PEFM
:@IBUUGNBUNVM
9X
X
#.VMUJQMFMJOFBSSFHSFTTJPOMPTTGVODUJPO
MPTTUGSFEVDF@NFBO
UGTRVBSF
::@IBUOBNF'loss'
[ 54 ]
Regression Chapter 2
1MBDFIPMEFSGPSUIF5SBJOJOH%BUB
9UGQMBDFIPMEFS
UGGMPBUOBNF 9 TIBQF<NO>
:UGQMBDFIPMEFS
UGGMPBUOBNF : TIBQF<N1>
7BSJBCMFTGPSDPFGGJDJFOUTJOJUJBMJ[FEUP
XUG7BSJBCMF
UG[FSPT
<1>OBNFbCJBTb
XUG7BSJBCMF
UGSBOEPN@OPSNBM
<O>OBNFbXFJHIUTb
5IF-JOFBS3FHSFTTJPO.PEFM
:@IBUUGNBUNVM
9X
X
-PTTGVODUJPO
FOUSPQZUGOOTPGUNBY@DSPTT@FOUSPQZ@XJUI@MPHJUT
:@IBU:
MPTTUGSFEVDF@NFBO
FOUSPQZ
MBNEBUGDPOTUBOU
SFHVMBSJ[BUJPOQBSBNFUFS
SFHVMBSJ[BUJPO@QBSBNMBNEBUGSFEVDF@TVN
UGBCT
8
/FXMPTT
MPTT
SFHVMBSJ[BUJPO@QBSBN
MBNEBUGDPOTUBOU
SFHVMBSJ[BUJPOQBSBNFUFS
SFHVMBSJ[BUJPO@QBSBNMBNEBUGOOM@MPTT
8
/FXMPTT
MPTT
SFHVMBSJ[BUJPO@QBSBN
How it works...
You learned how to implement different types of MPTT functions. Depending on the
regression task at hand, you can choose the corresponding MPTT function or design your
own. It is also possible to combine L1 and L2 regularization in the loss term.
[ 55 ]
Regression Chapter 2
There's more...
The MPTT function should be convex in shape to ensure convergence. A smooth,
differentiable, convex MPTT function provides better convergence. As the learning proceeds,
the value of the MPTT function should decrease and eventually become stable.
Optimizers in TensorFlow
From your high school math, you must know that a function's first derivative is zero at its
maxima and minima. The gradient descent algorithm is based on the same principle--the
coefficients (weights and biases) are adjusted such that the gradient of the MPTT function
decreases. In regression, we use gradient descent to optimize the MPTT function and obtain
coefficients. In this recipe, you will learn how to use the gradient descent optimizer of
TensorFlow and some of its variants.
Getting ready
The update of coefficients (W and b) is done proportionally to the negative of the gradient of
the MPTT function. There are three variations of gradient descent depending on the size of
the training sample:
[ 56 ]
Regression Chapter 2
How to do it...
We proceed with the recipe as follows:
1. The first thing we decide is the optimizer that we want. TensorFlow provides you
with a wide variety of optimizers. We start with the most popular and simple
one, the gradient descent optimizer:
UGUSBJO(SBEJFOU%FTDFOU0QUJNJ[FS
MFBSOJOH@SBUF
NJOJNJ[F
MPTT
HMPCBM@TUFQ/POF
WBS@MJTU/POF
HBUF@HSBEJFOUT("5&@01
BHHSFHBUJPO@NFUIPE/POF
DPMPDBUF@HSBEJFOUT@XJUI@PQT'BMTF
OBNF/POF
HSBE@MPTT/POF
PQUJNJ[FSUGUSBJO(SBEJFOU%FTDFOU0QUJNJ[FS
MFBSOJOH@SBUF
USBJO@TUFQPQUJNJ[FSNJOJNJ[F
MPTT
&YFDVUJPO(SBQI
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
USBJO@TUFQGFFE@EJDU\99@EBUB::@EBUB^
5. The 9 and : data fed to GFFE@EJDU can be single 9 and : points (stochastic
gradient), the entire training set (vanilla), or batch.
[ 57 ]
Regression Chapter 2
6. Another variation in gradient descent is adding the momentum term (we will
find out more about this in $IBQUFS, Neural Networks - Perceptrons). For this, we
use the optimizer UGUSBJO.PNFOUVN0QUJNJ[FS
. It takes both
MFBSOJOH@SBUF and NPNFOUVN as JOJU arguments:
PQUJNJ[FSUGUSBJO.PNFOUVN0UJNJ[FS
MFBSOJOH@SBUF
NPNFOUVNNJOJNJ[F
MPTT
PQUJNJ[FSUGUSBJO"EBEFMUB0QUJNJ[FS
MFBSOJOH@SBUF
SIPNJOJNJ[F
MPTT
PQUJNJ[FSUGUSBJO3.4QSPQ0QUJNJ[FS
MFBSOJOH@SBUF
EFDBZNPNFOUVNNJOJNJ[F
MPTT
There are some fine differences between Adadelta and RMSprop. To find
out more about them, you can refer to IUUQXXXDTUPSPOUPFEV
_UJKNFODTDTMJEFTMFDUVSF@TMJEFT@MFDQEG and IUUQTBSYJW
PSHQEGQEG.
PQUJNJ[FSUGUSBJO"EBN0QUJNJ[FS
NJOJNJ[F
MPTT
UGUSBJO"EBHSBE0QUJNJ[FS"EBHSBE0QUJNJ[FS
UGUSBJO"EBHSBE%"0QUJNJ[FS"EBHSBE%VBM"WFSBHJOHPQUJNJ[FS
UGUSBJO'USM0QUJNJ[FS'PMMPXUIFSFHVMBSJ[FEMFBEFSPQUJNJ[FS
UGUSBJO1SPYJNBM(SBEJFOU%FTDFOU0QUJNJ[FS1SPYJNBM(%PQUJNJ[FS
UGUSBJO1SPYJNBM"EBHSBE0QUJNJ[FS1SPYJNBM"EBHSBEPQUJNJ[FS
[ 58 ]
Regression Chapter 2
There's more...
It is often recommended that you start with a high value of learning rate and reduce it as
the learning progresses. This helps in fine-tuning the training. We can use the TensorFlow
UGUSBJOFYQPOFOUJBM@EFDBZ method to achieve this. According to the TensorFlow
docs:
When training a model, it is often recommended to lower the learning rate as the training
progresses. This function applies an exponential decay function to a provided initial
learning rate. It requires a global_step value to compute the decayed learning rate. You can
just pass a TensorFlow variable that you increment at each training step.The function
returns the decayed learning rate.
Args:
learning_rate: A scalar float32 or float64 Tensor or a Python number. The initial learning
rate.
global_step: A scalar int32 or int64 Tensor or a Python number. Global step to use for the
decay computation. Must not be negative.
decay_steps: A scalar int32 or int64 Tensor or a Python number. Must be positive. See the
decay computation described earlier.
decay_rate: A scalar float32 or float64 Tensor or a Python number. The decay rate.
staircase: Boolean. If True decay the learning rate at discrete intervals
name: String. Optional name of the operation. Defaults to 'ExponentialDecay'.
Returns:
A scalar Tensor of the same type as learning_rate. The decayed learning rate.
See also
Below are some good links for different optimizers:
[ 59 ]
Regression Chapter 2
IUUQTXXXUFOTPSGMPXPSHBQJ@HVJEFTQZUIPOUSBJO0QUJNJ[FST: This is
the TensorFlow.org link, which details how to use different optimizers included
in TensorFlow.
IUUQTBSYJWPSHQEGQEG: The paper on Adam optimizer.
Getting ready
We will consider the Boston housing price dataset (IUUQMJCTUBUDNVFEVEBUBTFUT
CPTUPO) collected by Harrison and Rubinfield in 1978. The dataset contains 506 sample
cases. Each house is assigned 14 attributes:
[ 60 ]
Regression Chapter 2
How to do it`
We proceed with the recipe as follows:
import UFOTPSGMPXas UG
# Global parameters
%"5"@'*-&'boston_housing.csv'
#"5$)@4*;&
/6.@'&"563&4
2. Next, we define a function that will take as argument the filename and return
tensors in batches of size equal to #"5$)@4*;&:
EFG EBUB@HFOFSBUPS
GJMFOBNF
(FOFSBUFT5FOTPSTJOCBUDIFTPGTJ[F#BUDI@4*;&
"SHT4USJOH5FOTPS
'JMFOBNFGSPNXIJDIEBUBJTUPCFSFBE
3FUVSOT5FOTPST
GFBUVSF@CBUDIBOEMBCFM@CBUDI
G@RVFVFUGUSBJOTUSJOH@JOQVU@QSPEVDFS
GJMFOBNF
SFBEFSUG5FYU-JOF3FBEFS
TLJQ@IFBEFS@MJOFT# Skips the first
line
@WBMVFSFBEFSSFBE
G@RVFVF
4. We specify the data to use in case data is missing. Decode the DTW and select the
features we need. For the example, we choose RM, PTRATIO, and LSTAT:
SFDPSE@EFGBVMUT<<>GPS@JOSBOHF
/6.@'&"563&4>
EBUBUGEFDPEF@DTW
WBMVFSFDPSE@EFGBVMUTSFDPSE@EFGBVMUT
GFBUVSFTUGTUBDL
UGHBUIFS@OE
EBUB<<><><>>
MBCFMEBUB<>
[ 61 ]
Regression Chapter 2
NJOJNVNOVNCFSFMFNFOUTJOUIFRVFVFBGUFSB
EFRVFVFNJO@BGUFS@EFRVFVF#"5$)@4*;&
SFUVSOGFBUVSF@CBUDIMBCFM@CBUDI
def HFOFSBUF@EBUB
GFBUVSF@CBUDIMBCFM@CBUDI
with UG4FTTJPO
as TFTT
# intialize the queue threads
DPPSEUGUSBJO$PPSEJOBUPS
UISFBETUGUSBJOTUBSU@RVFVF@SVOOFST
DPPSEDPPSE
for @in SBOHF
# Generate 5 batches
GFBUVSFTMBCFMTTFTTSVO
<GFBUVSF@CBUDI
MBCFM@CBUDI>
QSJOU
GFBUVSFT"HI"
DPPSESFRVFTU@TUPQ
DPPSEKPJO
UISFBET
7. Now, we can use these two functions to get the data in batches. Here, we are just
printing the data; when learning, we will perform the optimization step at this
point:
if @@OBNF@@ @@NBJO@@
GFBUVSF@CBUDIMBCFM@CBUDIEBUB@HFOFSBUPS
<%"5"@'*-&>
HFOFSBUF@EBUB
GFBUVSF@CBUDIMBCFM@CBUDI
[ 62 ]
Regression Chapter 2
There's more...
We can use the TensorFlow control ops and tensor manipulations that we covered in the
first chapter to preprocess the data. For example, in the case of Boston house price, there are
about 16 data rows where the MEDV is . In most probability, these data points contain
missing or censored values and it would be advisable not to consider them for the training.
We can remove them from the training dataset with the following code:
DPOEJUJPOUGFRVBM
EBUB<>UGDPOTUBOU
EBUBUGXIFSF
DPOEJUJPOUG[FSPT
/6.@'&"563&4EBUB<>
Here, we first define a tensor boolean condition, which will be true if MEDV is equal to
. Then we use TensorFlow UGXIFSF
Op to assign all zeros if the condition is true.
Getting ready
Our goal is to predict the house price given in the last column (MEDV). In this recipe, we
load the dataset from the TensorFlow Contrib dataset directly. We optimize the coefficients
for an individual training sample using the stochastic gradient descent optimizer.
How to do it...
We proceed with the recipe as follows:
1. The first step is to import all the packages that we will need:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
[ 63 ]
Regression Chapter 2
2. In neural networks, all the inputs are added linearly to generate activity; for
effective training, the inputs should be normalized, so we define a function to
normalize the input data:
def OPSNBMJ[F
9
/PSNBMJ[FTUIFBSSBZ9
NFBOOQNFBO
9
TUEOQTUE
9
9
9NFBOTUE
SFUVSO9
3. Now we load the Boston house price dataset using TensorFlow DPOUSJC datasets
and separate it into 9@USBJO and :@USBJO. We can choose to normalize the data
here:
# Data
CPTUPOUGDPOUSJCMFBSOEBUBTFUTMPBE@EBUBTFU
'boston'
9@USBJO:@USBJOCPTUPOEBUB<>CPTUPOUBSHFU
9@USBJOOPSNBMJ[F
9@USBJO5IJTTUFQJTPQUJPOBMIFSF
O@TBNQMFTMFO
9@USBJO
5. We create TensorFlow variables for weight and bias with initial value zero:
7BSJBCMFTGPSDPFGGJDJFOUTJOJUJBMJ[FEUP
CUG7BSJBCMF
XUG7BSJBCMF
5IF-JOFBS3FHSFTTJPO.PEFM
:@IBU9X
C
# Loss function
MPTTUGTRVBSF
::@IBUOBNF'loss'
[ 64 ]
Regression Chapter 2
# Initializing Variables
JOJU@PQUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
UPUBM<>
10. Now, we start the computation graph. We run the training for 100 epochs:
# Computation Graph
with UG4FTTJPO
as TFTT
# Initialize variables
TFTTSVO
JOJU@PQ
XSJUFSUGTVNNBSZ'JMF8SJUFS
'graphs'TFTTHSBQI
# train the model for 100 epochs
for Jin SBOHF
UPUBM@MPTT
for YZin [JQ
9@USBJO:@USBJO
@MTFTTSVO
<PQUJNJ[FSMPTT>
GFFE@EJDU\9Y:Z^
UPUBM@MPTT
M
UPUBMBQQFOE
UPUBM@MPTTO@TBNQMFT
QSJOU
'Epoch {0}: Loss {1}'GPSNBU
J
UPUBM@MPTTO@TBNQMFT
XSJUFSDMPTF
C@WBMVFX@WBMVFTFTTSVO
<CX>
[ 65 ]
Regression Chapter 2
How it works...
From the plot, we can see that our simple linear regressor tries to fit a linear line to the
given dataset:
In the following graph, we can see that as our model learned the data, the MPTT function
decreased, as was expected:
[ 66 ]
Regression Chapter 2
The graph has two name scope nodes, Variable and Variable_1, they are the high-level
nodes representing bias and weights respectively. The node named gradient is also a high-
level node; expanding the node, we can see that it takes seven inputs and computes the
gradients that are then used by (SBEJFOU%FTDFOU0QUJNJ[FS to compute and apply
updates to weights and bias:
[ 67 ]
Regression Chapter 2
There's more...
Well, we performed simple linear regression, but how do we find out the performance of
our model? There are various ways to do this; statistically, we can compute R2 or divide our
data into train and cross-validation sets and check the accuracy (the loss term) for the
validation set.
[ 68 ]
Regression Chapter 2
How to do it...
Here is how we proceed with the recipe:
1. The first step is to import all the packages that we will need:
importUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
2. We need to normalize the feature data since the data ranges of all the features are
varied. We define a normalize function for it. Also, here we combine the bias
to weights by adding another input always fixed to one value; to do this we
define the function BQQFOE@CJBT@SFTIBQF
. This technique is sometimes used
to simplify programming:
EFGOPSNBMJ[F
9
/PSNBMJ[FTUIFBSSBZ9
NFBOOQNFBO
9
TUEOQTUE
9
9
9NFBOTUE
SFUVSO9
EFGBQQFOE@CJBT@SFTIBQF
GFBUVSFTMBCFMT
NGFBUVSFTTIBQF<>
OGFBUVSFTTIBQF<>
YOQSFTIBQF
OQD@<OQPOFT
NGFBUVSFT><NO
>
ZOQSFTIBQF
MBCFMT<N>
SFUVSOYZ
[ 69 ]
Regression Chapter 2
3. Now we load the Boston house price dataset using TensorFlow contrib datasets
and separate it into 9@USBJO and :@USBJO. Observe that this time, 9@USBJO
contains all features. We can choose to normalize the data here, we also use
append the bias and reshape the data for the network:
# Data
CPTUPOUGDPOUSJCMFBSOEBUBTFUTMPBE@EBUBTFU
'boston'
9@USBJO:@USBJOCPTUPOEBUBCPTUPOUBSHFU
9@USBJOOPSNBMJ[F
9@USBJO
9@USBJO:@USBJOBQQFOE@CJBT@SFTIBQF
9@USBJO:@USBJO
NMFO
9@USBJO#Number of training examples
O
# Number of features + bias
4. Declare the TensorFlow placeholders for the training data. Observe the change in
the shape of the 9 placeholder.
5. We create TensorFlow variables for weight and bias. This time, weights are
initialized with random numbers:
7BSJBCMFTGPSDPFGGJDJFOUT
XUG7BSJBCMF
UGSBOEPN@OPSNBM
<O>
6. Define the linear regression model to be used for prediction. Now we need matrix
multiplication to do the task:
# Loss function
MPTTUGSFEVDF@NFBO
UGTRVBSF
::@IBUOBNF'loss'
[ 70 ]
Regression Chapter 2
# Initializing Variables
JOJU@PQUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
UPUBM<>
withUG4FTTJPO
asTFTT
# Initialize variables
TFTTSVO
JOJU@PQ
XSJUFSUGTVNNBSZ'JMF8SJUFS
'graphs'TFTTHSBQI
# train the model for 100 epcohs
forJinSBOHF
@MTFTTSVO
<PQUJNJ[FSMPTT>GFFE@EJDU\99@USBJO:
:@USBJO^
UPUBMBQQFOE
M
QSJOU
'Epoch {0}: Loss {1}'GPSNBU
JM
XSJUFSDMPTF
X@WBMVFC@WBMVFTFTTSVO
<XC>
QMUQMPU
UPUBM
QMUTIPX
[ 71 ]
Regression Chapter 2
[ 72 ]
Regression Chapter 2
How it works...
In this recipe, we used all the 13 features to train the model. The important differences
between simple and multiple linear regression are in weights and the number of coefficients
is always equal to the number of input features. The following is the TensorBoard graph for
the multiple linear regression model that we built:
There's more...
We can predict the house price now using the coefficients learned from the model:
/
9@OFX9@USBJO</>
:@QSFE
OQNBUNVM
9@OFXX@WBMVF
C@WBMVFSPVOE
QSJOU
'Predicted value: ${0} Actual value: / ${1}'GPSNBU
:@QSFE<>
:@USBJO</>'\nDone'
[ 73 ]
Regression Chapter 2
For logistic regression, we use one-hot encoding for the output Y. Thus, we have 10 bits
representing the output; each bit can have a value either 0 or 1, and being one-hot means
that for each image in label Y, only one bit out of the 10 will have value 1, the rest will be
zeros. Here, you can see the image of the handwritten numeral 8, along with its hot encoded
value [0 0 0 0 0 0 0 0 1 0]:
How to do it...
Here is how we proceed with the recipe:
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMUNBUQMPUMJCJNBHFBTNQJNH
[ 74 ]
Regression Chapter 2
2. We take the input data of MNIST from the TensorFlow examples given in the
module JOQVU@EBUB. The POF@IPU flag is set to 5SVF to enable POF@IPU
encoding of labels. This results in generating two
tensors, NOJTUUSBJOJNBHFT of shape [55000,784] and NOJTUUSBJOMBCFMT
of shape [55000,10]. Each entry of NOJTUUSBJOJNBHFT is a pixel intensity with
the value ranging between 0 and 1:
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUBPOF@IPU5SVF
3. Create placeholders for the training dataset inputs Y and label Z to the
TensorFlow graph:
YUGQMBDFIPMEFS
UGGMPBU</POF>OBNF 9
ZUGQMBDFIPMEFS
UGGMPBU</POF>OBNF :
8UG7BSJBCMF
UG[FSPT
<>OBNF 8
CUG7BSJBCMF
UG[FSPT
<>OBNF C
XJUIUGOBNF@TDPQF
XY@CBTTDPQF
Z@IBUUGOOTPGUNBY
UGNBUNVM
Y8
C
6. Add summary OPs to collect data while training. We use the histogram summary
so that we can see how weights and bias change relative to each other's value
with time. We will be able to see this in the TensorBoard Histogram tab:
X@IUGTVNNBSZIJTUPHSBN
XFJHIUT8
C@IUGTVNNBSZIJTUPHSBN
CJBTFTC
[ 75 ]
Regression Chapter 2
7. Define the DSPTTFOUSPQZ and MPTT function, and also add name scope and
summary for better visualization. Here, we use the scalar summary to obtain the
variation in the MPTT function over time. The scalar summary is visible under the
Events tab:
XJUIUGOBNF@TDPQF
DSPTTFOUSPQZ BTTDPQF
MPTT
UGSFEVDF@NFBO
UGOOTPGUNBY@DSPTT@FOUSPQZ@XJUI@MPHJUT
MBCFMTZ
MPHJUTZ@IBU
UGTVNNBSZTDBMBS
DSPTTFOUSPQZ MPTT
XJUIUGOBNF@TDPQF
5SBJO BTTDPQF
PQUJNJ[FS
UGUSBJO(SBEJFOU%FTDFOU0QUJNJ[FS
NJOJNJ[F
MPTT
*OJUJBMJ[JOHUIFWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
NFSHFE@TVNNBSZ@PQUGTVNNBSZNFSHF@BMM
11. Now, we define the session and store the summaries in a defined folder:
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJUJOJUJBMJ[FBMMWBSJBCMFT
TVNNBSZ@XSJUFSUGTVNNBSZ'JMF8SJUFS
HSBQIT TFTTHSBQI
$SFBUFBOFWFOUGJMF
5SBJOJOH
GPSFQPDIJOSBOHF
NBY@FQPDIT
MPTT@BWH
OVN@PG@CBUDIJOU
NOJTUUSBJOOVN@FYBNQMFTCBUDI@TJ[F
GPSJJOSBOHF
OVN@PG@CBUDI
CBUDI@YTCBUDI@ZTNOJTUUSBJOOFYU@CBUDI
HFU
UIFOFYUCBUDIPGEBUB
@MTVNNBSZ@TUSTFTTSVO
<PQUJNJ[FSMPTT
NFSHFE@TVNNBSZ@PQ>GFFE@EJDU\YCBUDI@YTZCBUDI@ZT^3VO
UIFPQUJNJ[FS
MPTT@BWH
M
[ 76 ]
Regression Chapter 2
TVNNBSZ@XSJUFSBEE@TVNNBSZ
TVNNBSZ@TUS
FQPDIOVN@PG@CBUDI
J"EEBMMTVNNBSJFTQFSCBUDI
MPTT@BWHMPTT@BWHOVN@PG@CBUDI
QSJOU
&QPDI\^-PTT\^ GPSNBU
FQPDIMPTT@BWH
QSJOU
%POF
QSJOU
TFTTSVO
BDDVSBDZGFFE@EJDU\YNOJTUUFTUJNBHFTZ
NOJTUUFTUMBCFMT^
12. We get an accuracy of 86.5 percent after 30 epochs, 89.36 percent after 50 epochs,
and, after 100 epochs, the accuracy increases to 90.91 percent.
How it works...
We launch the TensorBoard using the tensor UFOTPSCPBSEMPHEJSHBSQIT. In the
browser, navigate to the web address MPDBMIPTU, to view the TensorBoard. The
graph of the preceding model is as follows:
[ 77 ]
Regression Chapter 2
Under the Histogram tab, we can see the histograms for weights and biases:
[ 78 ]
Regression Chapter 2
[ 79 ]
Regression Chapter 2
We can see that, with each time step, bias and weight both changed. The bias in our case
had a greater spread, which we can see from distributions in TensorBoard.
Under the Events tab, we can see the scalar summary, in this case, cross entropy. The
following graph shows that cross entropy loss decreases with time:
See also
These are some of the good resources if you are interested in finding out more:
[ 80 ]
Neural Networks - Perceptron
3
Since the last decade, neural networks have been at the forefront of machine learning
research and applications. Deep neural networks (DNNs), transfer learning, and
availability of computationally efficient GPUs have helped achieve significant progress in
the field of image recognition, speech recognition, and even text generation. In this chapter,
we will concentrate on the basic neural network perceptron, a fully connected layered
architecture of artificial neurons. The chapter will include the following recipes:
Activation functions
Single layer perceptron
Calculating Gradients of Backpropagation algorithm
MNIST classifier using MLP
Function approximation using MLP--predicting Boston house prices
Tuning hyperparameters
Higher level APIs-Keras
Introduction
Neural networks, also conventionally known as connectionist models, are inspired by the
human brain. Like the human brain, neural networks are a collection of a large number of
artificial neurons connected to each other via synaptic strengths called weights. Just as we
learn through examples provided to us by our elders, artificial neural networks learn by
examples presented to them as training datasets. With a sufficient number of training
datasets, artificial neural networks can generalize the information and can then be
employed for unseen data as well. Awesome, they sound like magic!
Neural Networks - Perceptron Chapter 3
Neural networks are not new; the first neural network model, McCulloch Pitts
(MCP) (IUUQWPSEFOLFSEFHHQIJMPTPQIZNDDVMMPDI@BMPHJDBMDBMDVMVTQEG) Model,
was proposed as early as 1943. (Yes, even before the first computer was built!) The model
could perform logical operations like AND/OR/NOT. The MCP model had fixed weights
and biases; there was no learning possible. This problem was resolved a few years later by
Frank Rosenblatt in 1958 (IUUQTCMPHTVNBTTFEVCSBJOXBSTGJMFT
SPTFOCMBUUQEG). He proposed the first learning neural network called Perceptron.
From that time, it was known that adding multiple layers of neurons and building a deep
and dense network will help neural networks solve complex tasks. Just as a mother is proud
of her child's achievements, scientists and engineers made tall claims about what they can
achieve using Neural Networks (NN) (IUUQTXXXZPVUVCFDPNXBUDIWK1)6M2JX%:).
The claims were not false, but it was simply not possible to accomplish them at that time
because of the hardware computation limitation and complex network structure. This
resulted in what is known as the AI Winters in the 1970s and 1980s; during these chills, the
progress in this field slowed down due to very little or almost no funding to AI-based
projects.
With the advent of DNNs and GPUs, the situation changed; today we have networks that
can perform better with less tuning parameters, techniques like dropout and transfer
learning the further reduce the training time, and lastly, the hardware companies that are
coming up with specialized hardware chips to perform fast NN-based computations.
The artificial neuron is at the heart of all neural networks. It consists of two major
components--an adder that (weighted) sums all the inputs to this neuron, and a processing
unit that takes the weighted sum and generates an output based on a predefined function,
called activation function. Each artificial neuron has its own set of weights and thresholds
(biases); it learns these weights and thresholds through different learning algorithms:
[ 82 ]
Neural Networks - Perceptron Chapter 3
Source: IUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMF3PTFOCMBUUQFSDFQUSPOQOH
When only one layer of such neurons is present, it is called Perceptron. The input layer is
called the zeroth layer as it simply buffers the input. The only layer of neurons present form
the output layer. Each neuron of the output layer has its own weights and thresholds. When
many such layers are present, the network is termed multi-layered perceptron (MLP). An
MLP has one or more hidden layers. These hidden layers have a different number of hidden
neurons. The neurons of each hidden layer have the same activation function:
The preceding figure shows an MLP with four inputs, five hidden layers, each with 4, 5, 6,
4, and 3 neurons respectively, and three neurons in the output layer. In MLP, all neurons of
the lower layer are connected to all the neurons of the layer just above it. Therefore, MLPs
are also called fully connected layers. The flow of information in MLPs is always from
input to output; there are no feedback or jumps present, therefore these networks are also
known as feedforward networks.
Perceptrons are trained using gradient descent algorithms. In $IBQUFS, Regression, you
learned about gradient descent; here we look at it a little deeper. Perceptrons learn through
supervised learning algorithms, that is, the network is provided by the desired output for
all the inputs present in the training dataset. At the output, we define an error function or
objective function J(W), such that when the network has completely learned all the training
data, the objective function will be minimum.
[ 83 ]
Neural Networks - Perceptron Chapter 3
The weights of the output layer and the hidden layers are updated such that the gradient of
the objective function decreases:
To understand it better, visualize a landscape full of hills, plateaus, and pits. The goal is to
come to the ground (global minima of the objective function). If you are standing on top
and you have to come down, it is an obvious choice that you will move down the hill, that
is, toward the negative slope (or negative gradient). In the same manner, the weights in the
Perceptron are changed proportionally to the negative of the gradient of the objective
function.
The higher the value of the gradient, the larger will be the change in weight values and vice
versa. Now, all this is quite fine, but we can land up with problems when we reach plateaus
as the gradient is zero and hence there is no change in weights. We can also land in
problems when we enter a small pit (local minima) because when we try to move to either
side, the gradient will increase, forcing the network to stay in the pit.
As discussed in $IBQUFS, Regression, there are various variants of gradient descent aimed
at increasing convergence and that avoid the problem of getting stuck at local minima or
plateaus (adding momentum, variable learning rate).
TensorFlow calculates these gradients automatically with the help of different optimizers.
The important thing to note, however, is that as TensorFlow will calculate gradients, which
will involve derivatives of the activation functions as well, it is important that the activation
function you choose is differentiable and preferably has a non-zero gradient throughout the
training scenario.
[ 84 ]
Neural Networks - Perceptron Chapter 3
One of the major ways gradient descent for perceptrons differ from $IBQUFS, Regression,
applications is that the objective function is defined for the output layer, but is used to find
the weight change of the neurons of the hidden layers as well. This is done using the
backpropagation (BPN) algorithm, where the error at the output is propagated backwards
to the hidden layers and used to determine the weight change. You will learn more about it
soon.
Activation functions
Every neuron must have an activation function. They are what gives the neuron the
nonlinear property necessary to model the complex nonlinear datasets. The function takes
the weighted sum of all the inputs and generates an output signal. You can think of it as a
transform between input and output. Using the proper activation function, we can bound
our output values in a defined range.
th th
If xj is the j input, Wj the weight connecting j input to our neuron, and b the bias of our
neuron, the output of the neuron (in biological terms, firing of the neuron) is decided by the
activation function, and mathematically it is expressed as follows:
Here, g represents the activation function. The argument to the activation function bWjxj+b
is called activity of the neuron.
Getting ready
Our response to a given input stimulus is governed by the activation function of our
neurons. Sometimes our response is a binary yes or no; for example, when presented with a
joke, we either laugh or do not laugh. At other times, the response seems linear, for
example, crying due to pain. At times, the response seems to be bounded within a range.
[ 85 ]
Neural Networks - Perceptron Chapter 3
Mimicking similar behavior, many different activation functions are used by artificial
neurons. In this recipe, you will learn how to define and use some of the common activation
functions in TensorFlow.
How to do it...
We proceed with activation functions as follows:
1. Threshold activation function: this is the simplest activation function. Here, the
neuron fires if the activity of the neuron is greater than zero; otherwise, it does
not fire. Here is the plot of the threshold activation function as the activity of the
neuron changes along with the code to implement the threshold activation
function in TensorFlow:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
5ISFTIPME"DUJWBUJPOGVODUJPO
EFGUISFTIPME
Y
DPOEUGMFTT
YUG[FSPT
UGTIBQF
YEUZQFYEUZQF
PVUUGXIFSF
DPOEUG[FSPT
UGTIBQF
Y
UGPOFT
UGTIBQF
Y
SFUVSOPVU
1MPUUJOH5ISFTIPME"DUJWBUJPO'VODUJPO
IOQMJOTQBDF
PVUUISFTIPME
I
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
ZTFTTSVO
PVU
QMUYMBCFM
"DUJWJUZPG/FVSPO
QMUZMBCFM
0VUQVUPG/FVSPO
QMUUJUMF
5ISFTIPME"DUJWBUJPO'VODUJPO
QMUQMPU
IZ
[ 86 ]
Neural Networks - Perceptron Chapter 3
2. Sigmoidal activation function: In this case, the output of the neuron is specified
by the function H
Y
FYQ
Y. In TensorFlow, there is a method,
UGTJHNPJE, which provides Sigmoid activation. The range of this function is
between 0 and 1. In shape, it looks like the alphabet S, hence the name Sigmoid:
1MPUUJOH4JHNPJEBM"DUJWBUJPOGVODUJPO
IOQMJOTQBDF
PVUUGTJHNPJE
I
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
ZTFTTSVO
PVU
QMUYMBCFM
"DUJWJUZPG/FVSPO
QMUZMBCFM
0VUQVUPG/FVSPO
QMUUJUMF
4JHNPJEBM"DUJWBUJPO'VODUJPO
QMUQMPU
IZ
[ 87 ]
Neural Networks - Perceptron Chapter 3
1MPUUJOH)ZQFSCPMJD5BOHFOU"DUJWBUJPOGVODUJPO
IOQMJOTQBDF
PVUUGUBOI
I
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
ZTFTTSVO
PVU
QMUYMBCFM
"DUJWJUZPG/FVSPO
QMUZMBCFM
0VUQVUPG/FVSPO
QMUUJUMF
)ZQFSCPMJD5BOHFOU"DUJWBUJPO'VODUJPO
QMUQMPU
IZ
[ 88 ]
Neural Networks - Perceptron Chapter 3
4. Linear activation function: In this case, the output of the neuron is the same as
the activity of the neuron. This function is not bounded on either side:
-JOFBS"DUJWBUJPO'VODUJPO
CUG7BSJBCMF
UGSBOEPN@OPSNBM
<>TUEEFW
XUG7BSJBCMF
UGSBOEPN@OPSNBM
<>TUEEFW
MJOFBS@PVUUGNBUNVM
9@JOX
C
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
PVUTFTTSVO
MJOFBS@PVU
QSJOU
PVU
5. Rectified linear units (ReLU) activation function is again built-in with the
TensorFlow library. The activation function is similar to the linear activation
function, but with one big change--for a negative value of the activity, the neuron
does not fire (zero output), and for a positive value of the activity, the output of
the neuron is the same as the given activity:
1MPUUJOH3F-6"DUJWBUJPOGVODUJPO
IOQMJOTQBDF
PVUUGOOSFMV
I
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
[ 89 ]
Neural Networks - Perceptron Chapter 3
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
ZTFTTSVO
PVU
QMUYMBCFM
"DUJWJUZPG/FVSPO
QMUZMBCFM
0VUQVUPG/FVSPO
QMUUJUMF
3F-6"DUJWBUJPO'VODUJPO
QMUQMPU
IZ
1MPUUJOH4PGUNBY"DUJWBUJPOGVODUJPO
IOQMJOTQBDF
PVUUGOOTPGUNBY
I
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
ZTFTTSVO
PVU
QMUYMBCFM
"DUJWJUZPG/FVSPO
QMUZMBCFM
0VUQVUPG/FVSPO
QMUUJUMF
4PGUNBY"DUJWBUJPO'VODUJPO
QMUQMPU
IZ
[ 90 ]
Neural Networks - Perceptron Chapter 3
How it works...
Following are the explanations of the functions:
Threshold activation function was used by McCulloch Pitts Neuron and initial
Perceptrons. It is not differentiable and is discontinuous at x=0. Therefore, it is not
possible to use this activation function to train using gradient descent or its
variants.
Sigmoid activation function was very popular at one time. If you look at the
curve, it looks like a continuous version of the threshold activation function. It
suffers from the vanishing gradient problem, that is, the gradient of the function
becomes zero near the two edges. This makes training and optimization difficult.
Hyperbolic Tangent activation function is again sigmoidal in shape and has
nonlinear properties. The function is centered at zero and has steeper derivatives
compared to sigmoid. Like sigmoid, this also suffers from the vanishing gradient
problem.
Linear activation function is, as the name suggests, linear in nature. The function
is unbounded from both sides [-inf, inf]. Its linearity is its major problem. The
sum of linear functions will be a linear function and the linear function of a linear
function too is a linear function. Thus, using this function, one cannot grasp the
non-linearities present in complex datasets.
[ 91 ]
Neural Networks - Perceptron Chapter 3
ReLU activation function is the rectified version of the linear activation function,
and this rectification allows it to capture non-linearities when used in multiple
layers. One of the major advantages of using ReLU is that it leads to sparse
activation; at any instant, all the neurons with negative activity are not firing.
This makes the network lighter in terms of computation. ReLU neurons suffer
from the dying ReLU problem, that is, neurons that do not fire will have their
gradients zero and, hence, will not be able to undergo any training and stay off
(dead). Despite this problem, ReLU today is one of the most used activation
functions for hidden layers.
Softmax activation function is popularly used as the activation function of the
output layer. The function is bounded in the range [0,1]. It is used to represent the
probability of a class in a multiclass classification problem. The sum of the output
of all the units will always be 1.
There's more...
Neural networks have been employed in a variety of tasks. These tasks can be broadly
classified into two categories: function approximation (regression) and classification.
Depending on the task at hand, one activation function may be better than the other.
Generally, it is better to use ReLU neuron for hidden layers. For classification tasks, softmax
is normally a better choice, and for regression problems, it is better to use sigmoid or
hyperbolic tangent.
See also
This link provides details of the activation functions defined in TensorFlow and
how to use them: IUUQTXXXUFOTPSGMPXPSHWFSTJPOTSBQJ@EPDT
QZUIPOOOBDUJWBUJPO@GVODUJPOT@
A nice summary on activation functions:
IUUQTFOXJLJQFEJBPSHXJLJ"DUJWBUJPO@GVODUJPO
[ 92 ]
Neural Networks - Perceptron Chapter 3
Getting ready
As perceptron uses the threshold activation function, we cannot use the TensorFlow
optimizers to update weights. We will have to use the weight update rule:
Here is the learning rate. For programming simplicity, bias can be added as one additional
weight with input fixed to +1. Then the preceding equation can be used to update both
weights and biases simultaneously.
How to do it...
Here is how we proceed with the single layer perceptron:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
)ZQFSQBSBNFUFST
FUBMFBSOJOHSBUFQBSBNFUFS
FQTJMPOFNJOJNVNBDDFQUFEFSSPS
NBY@FQPDIT.BYJNVN&QPDIT
5ISFTIPME"DUJWBUJPOGVODUJPO
EFGUISFTIPME
Y
DPOEUGMFTT
YUG[FSPT
UGTIBQF
YEUZQFYEUZQF
PVUUGXIFSF
DPOEUG[FSPT
UGTIBQF
Y
UGPOFT
UGTIBQF
Y
SFUVSOPVU
4. Specify the training data. In this example, we take a three input neuron (",#,$)
and train it to learn the logic "#
#$:
5SBJOJOH%BUB:"#
#$TVNPGUXPMJOFBSGVODUJPOT
5'
9@JO<
<5555>
<55'5>
[ 93 ]
Neural Networks - Perceptron Chapter 3
<5'55>
<5''5>
<'555>
<'5'5>
<''55>
<'''5>
>
:<
<5>
<5>
<'>
<'>
<5>
<'>
<'>
<'>
>
8UG7BSJBCMF
UGSBOEPN@OPSNBM
<>TUEEFWTFFE
IUGNBUNVM
9@JO8
:@IBUUISFTIPME
I
FSSPS::@IBU
NFBO@FSSPSUGSFEVDF@NFBO
UGTRVBSF
FSSPS
E8FUBUGNBUNVM
9@JOFSSPSUSBOTQPTF@B5SVF
USBJOUGBTTJHO
88
E8
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
FSS
FQPDI
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
XIJMFFSS FQTJMPOBOEFQPDINBY@FQPDIT
FQPDI
FSS@TFTTSVO
<NFBO@FSSPSUSBJO>
QSJOU
FQPDI\^NFBOFSSPS\^ GPSNBU
FQPDIFSS
QSJOU
5SBJOJOHDPNQMFUF
[ 94 ]
Neural Networks - Perceptron Chapter 3
There's more...
What do you think will happen if, instead of the threshold activation function, we use the
sigmoid activation function?
You guessed right; firstly, we can use the TensorFlow optimizer to update weights.
Secondly, the network will behave like the logistic regressor.
Forward pass: The inputs are fed to the network and the signal is propagated from the
input layer via the hidden layers, finally to the output layer. At the output layer, the error
and the MPTT function are computed.
Backward pass: In backward pass, the gradient of the MPTT function is computed first for
the output layer neurons and then for the hidden layer neurons. The gradients are then
used to update the weights.
[ 95 ]
Neural Networks - Perceptron Chapter 3
Getting ready
The network is first presented with M training pairs (X, Y), with X as the input and Y the
desired output. The input is propagated from input via the activation function g(h) through
the hidden layers, up to the output layers. The output Yhat is the output of the network,
giving the error = Y- Yhat.
Here, i varies over all the neurons of the output layer (1 to N). The change in weights Wij,
connecting ith output layer neuron to jth hidden layer neuron, can then be determined using
the gradient of J(W) and employing the chain rule for differentiation:
Here, Oj is the output of the hidden layer neuron, j and h represent activity. This was easy,
but now how do we find the update for weights Wjk connecting the neuron k from nth
hidden layer to the neuron j of n+1th hidden layer? The process is the same--we will use the
gradient of the MPTT function and chain rule for differentiation, but we will be calculating it
for Wjk this time:
Now that the equations are in place, let's see how to do it in TensorFlow. In this recipe, we
work with our same old the MNIST dataset (IUUQZBOOMFDVODPNFYECNOJTU).
[ 96 ]
Neural Networks - Perceptron Chapter 3
How to do it...
Let us now get started on the backpropagation algorithm:
JNQPSUUFOTPSGMPXBTUG
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUBPOF@IPU5SVF
%BUBTQFDJGJDDPOTUBOUT
O@JOQVU./*45EBUBJOQVU
JNHTIBQF
O@DMBTTFT./*45UPUBMDMBTTFT
EJHJUT
)ZQFSQBSBNFUFST
NBY@FQPDIT
MFBSOJOH@SBUF
CBUDI@TJ[F
TFFE
O@IJEEFO/VNCFSPGOFVSPOTJOUIFIJEEFOMBZFS
4. We will require the derivative of the TJHNPJE function for weight update, so we
define it:
EFGTJHNBQSJNF
Y
SFUVSOUGNVMUJQMZ
UGTJHNPJE
Y
UGTVCUSBDU
UGDPOTUBOU
UGTJHNPJE
Y
Y@JOUGQMBDFIPMEFS
UGGMPBU</POFO@JOQVU>
ZUGQMBDFIPMEFS
UGGMPBU</POFO@DMBTTFT>
[ 97 ]
Neural Networks - Perceptron Chapter 3
EFGNVMUJMBZFS@QFSDFQUSPO
YXFJHIUTCJBTFT
)JEEFOMBZFSXJUI3&-6BDUJWBUJPO
I@MBZFS@UGBEE
UGNBUNVM
YXFJHIUT< I >CJBTFT< I >
PVU@MBZFS@UGTJHNPJE
I@MBZFS@
0VUQVUMBZFSXJUIMJOFBSBDUJWBUJPO
I@PVUUGNBUNVM
PVU@MBZFS@XFJHIUT< PVU >
CJBTFT< PVU >
SFUVSOUGTJHNPJE
I@PVUI@PVUPVU@MBZFS@I@MBZFS@
XFJHIUT\
I UG7BSJBCMF
UGSBOEPN@OPSNBM
<O@JOQVUO@IJEEFO>TFFE
TFFE
PVU UG7BSJBCMF
UGSBOEPN@OPSNBM
<O@IJEEFOO@DMBTTFT>
TFFETFFE^
CJBTFT\
I UG7BSJBCMF
UGSBOEPN@OPSNBM
<O@IJEEFO>TFFE
TFFE
PVU UG7BSJBCMF
UGSBOEPN@OPSNBM
<O@DMBTTFT>TFFE
TFFE^
8. Create the computation graph for the forward pass, error, gradient, and update
calculations:
'PSXBSE1BTT
Z@IBUI@P@I@NVMUJMBZFS@QFSDFQUSPO
Y@JOXFJHIUT
CJBTFT
&SSPS
FSSZ@IBUZ
#BDLXBSE1BTT
EFMUB@UGNVMUJQMZ
FSSTJHNBQSJNF
I@
EFMUB@X@UGNBUNVM
UGUSBOTQPTF
P@EFMUB@
XUE@FSSPSUGNBUNVM
EFMUB@UGUSBOTQPTF
XFJHIUT< PVU >
EFMUB@UGNVMUJQMZ
XUE@FSSPSTJHNBQSJNF
I@
EFMUB@X@UGNBUNVM
UGUSBOTQPTF
Y@JOEFMUB@
FUBUGDPOTUBOU
MFBSOJOH@SBUF
6QEBUFXFJHIUT
TUFQ<
UGBTTJHO
XFJHIUT< I >UGTVCUSBDU
XFJHIUT< I >
[ 98 ]
Neural Networks - Perceptron Chapter 3
UGNVMUJQMZ
FUBEFMUB@X@
UGBTTJHO
CJBTFT< I >UGTVCUSBDU
CJBTFT< I >
UGNVMUJQMZ
FUBUGSFEVDF@NFBO
EFMUB@BYJT<>
UGBTTJHO
XFJHIUT< PVU >UGTVCUSBDU
XFJHIUT< PVU >
UGNVMUJQMZ
FUBEFMUB@X@
UGBTTJHO
CJBTFT< PVU >UGTVCUSBDU
CJBTFT< PVU >
UGNVMUJQMZ
FUBUGSFEVDF@NFBO
EFMUB@BYJT<>
>
BDDU@NBUUGFRVBM
UGBSHNBY
Z@IBUUGBSHNBY
Z
BDDVSBDZUGSFEVDF@TVN
UGDBTU
BDDU@NBUUGGMPBU
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
GPSFQPDIJOSBOHF
NBY@FQPDIT
CBUDI@YTCBUDI@ZTNOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
TFTTSVO
TUFQGFFE@EJDU\Y@JOCBUDI@YTZCBUDI@ZT^
JGFQPDI
BDD@UFTUTFTTSVO
BDDVSBDZGFFE@EJDU
\Y@JONOJTUUFTUJNBHFT
ZNOJTUUFTUMBCFMT^
BDD@USBJOTFTTSVO
BDDVSBDZGFFE@EJDU
\Y@JONOJTUUSBJOJNBHFT
ZNOJTUUSBJOMBCFMT^
QSJOU
&QPDI\^"DDVSBDZ5SBJO\^"DDVSBDZ
5FTU\^
GPSNBU
FQPDIBDD@USBJO
BDD@UFTU
[ 99 ]
Neural Networks - Perceptron Chapter 3
How it works...
Here, we are training the network for a batch size of 10. If we increase it, the network
performance decreases. Also, the accuracy of the trained network is checked on the test
data; the size of the test data on which it is tested is 1,000.
There's more...
Our one hidden layer multilayered perceptron gave an accuracy of 84.45 on training data
and 92.1 on test data. It is good but not good enough. The MNIST database is used as a
benchmark for classification problems in machine learning. Next, we see how using
TensorFlow's built-in optimizer affects network performance.
See also
The MNIST database: IUUQZBOOMFDVODPNFYECNOJTU
A simplified explanation of backpropagation algorithm: IUUQ
OFVSBMOFUXPSLTBOEEFFQMFBSOJOHDPNDIBQIUNM
Another intuitive explanation of backpropagation algorithm: IUUQDTO
HJUIVCJPPQUJNJ[BUJPO
One more on backpropagation algorithm, it gives detailed information, along
with derivation and how it can be applied to different neyworks: IUUQTQBHF
NJGVCFSMJOEFSPKBTOFVSBMDIBQUFS,QEG
Getting ready
In the backpropagation algorithm recipe, we defined layers, weights, loss, gradients, and
update through gradients manually. It is a good idea to do it manually with equations for
better understanding but this can be quite cumbersome as the number of layers in the
network increases.
[ 100 ]
Neural Networks - Perceptron Chapter 3
In this recipe, we will use powerful TensorFlow features such as Contrib (Layers) to define
neural network layers and TensorFlow's own optimizer to compute and apply gradients.
We saw in $IBQUFS, Regression, how to use different TensorFlow optimizers. The contrib
can be used to add various layers to the neural network model like adding building blocks.
The one method that we use here is UGDPOUSJCMBZFSTGVMMZ@DPOOFDUFE, defined in
TensorFlow documentation as follows:
GVMMZ@DPOOFDUFE
JOQVUT
OVN@PVUQVUT
BDUJWBUJPO@GOUGOOSFMV
OPSNBMJ[FS@GO/POF
OPSNBMJ[FS@QBSBNT/POF
XFJHIUT@JOJUJBMJ[FSJOJUJBMJ[FSTYBWJFS@JOJUJBMJ[FS
XFJHIUT@SFHVMBSJ[FS/POF
CJBTFT@JOJUJBMJ[FSUG[FSPT@JOJUJBMJ[FS
CJBTFT@SFHVMBSJ[FS/POF
SFVTF/POF
WBSJBCMFT@DPMMFDUJPOT/POF
PVUQVUT@DPMMFDUJPOT/POF
USBJOBCMF5SVF
TDPQF/POF
How to do it...
We proceed with the recipe as follows:
1. The first step is to change the MPTT function; though for classification, it is better
to use the cross-entropy MPTT function. We presently continue with the mean
square error (MSE):
MPTTUGSFEVDF@NFBO
UGTRVBSF
ZZ@IBUOBNF MPTT
[ 101 ]
Neural Networks - Perceptron Chapter 3
PQUJNJ[FSUGUSBJO(SBEJFOU%FTDFOU0QUJNJ[FS
MFBSOJOH@SBUF
MFBSOJOH@SBUF
USBJOPQUJNJ[FSNJOJNJ[F
MPTT
3. With just these two changes, we get an accuracy of only 61.3 percent for the test
dataset for the same set of hyperparameters. Increasing the NBY@FQPDI, we can
increase the accuracy, but that will not be an efficient use of TensorFlow abilities.
4. This is a classification problem, so it will be better to use cross-entropy loss, ReLU
activation function for hidden layers, and softmax for the output layer. Making
the required changes, the full code is as follows:
JNQPSUUFOTPSGMPXBTUG
JNQPSUUFOTPSGMPXDPOUSJCMBZFSTBTMBZFST
GSPNUFOTPSGMPXQZUIPOJNQPSUEFCVHBTUG@EFCVH
/FUXPSL1BSBNFUFST
O@IJEEFO
O@DMBTTFT
O@JOQVU
)ZQFSQBSBNFUFST
CBUDI@TJ[F
FUB
NBY@FQPDI
./*45JOQVUEBUB
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
UNQEBUBPOF@IPU5SVF
EFGNVMUJMBZFS@QFSDFQUSPO
Y
GDMBZFSTGVMMZ@DPOOFDUFE
YO@IJEEFO
BDUJWBUJPO@GOUGOOSFMVTDPQF GD
GDMBZFSTGVMMZ@DPOOFDUFE
GD
BDUJWBUJPO@GOUGOOSFMVTDPQF GD
PVUMBZFSTGVMMZ@DPOOFDUFE
GDO@DMBTTFT
BDUJWBUJPO@GO/POFTDPQF PVU
SFUVSOPVU
CVJMENPEFMMPTTBOEUSBJOPQ
YUGQMBDFIPMEFS
UGGMPBU</POFO@JOQVU>
OBNF QMBDFIPMEFS@Y
ZUGQMBDFIPMEFS
UGGMPBU</POFO@DMBTTFT>
OBNF QMBDFIPMEFS@Z
[ 102 ]
Neural Networks - Perceptron Chapter 3
Z@IBUNVMUJMBZFS@QFSDFQUSPO
Y
MPTT
UGSFEVDF@NFBO
UGOOTPGUNBY@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUTZ@IBU
MBCFMTZ
USBJOUGUSBJO"EBN0QUJNJ[FS
MFBSOJOH@SBUFFUBNJOJNJ[F
MPTT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
GPSFQPDIJOSBOHF
FQPDI@MPTT
CBUDI@TUFQTJOU
NOJTUUSBJOOVN@FYBNQMFTCBUDI@TJ[F
GPSJJOSBOHF
CBUDI@TUFQT
CBUDI@YCBUDI@ZNOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
@DTFTTSVO
<USBJOMPTT>
GFFE@EJDU\YCBUDI@YZCBUDI@Z^
FQPDI@MPTT
DCBUDI@TUFQT
QSJOU
&QPDIE-PTTG
FQPDIFQPDI@MPTT
5FTUNPEFM
DPSSFDU@QSFEJDUJPOUGFRVBM
UGBSHNBY
Z@IBU
UGBSHNBY
Z
BDDVSBDZUGSFEVDF@NFBO
UGDBTU
DPSSFDU@QSFEJDUJPO
UGGMPBU
QSJOU
"DDVSBDZBDDVSBDZFWBM
\YNOJTUUFTUJNBHFTZ
NOJTUUFTUMBCFMT^
How it works...
The modified MNIST MLP classifier gives us ~ 96 percent accuracy on the test dataset, with
only one hidden layer and within 10 epochs. Only in a few lines of code we got ~96 percent
accuracy, this is the power of TensorFlow:
[ 103 ]
Neural Networks - Perceptron Chapter 3
"multilayer feedforward networks with as few as one hidden layer are indeed capable of
universal approximation in a very precise and satisfactory sense."
In this recipe, we will show you how to use MLP for function approximation; specifically,
we will be predicting Boston house prices. We are already familiar with the dataset; in
$IBQUFS, Regression, we used regression techniques for the house price prediction, now
we will do the same using MLPs.
Getting ready
For function approximation, the MPTT function should be the MSE. The inputs should be
normalized and, while the hidden layer can be ReLU, the output layer should preferably be
sigmoid.
How to do it...
Here is how we start with function approximation using MLP:
JNQPSUUFOTPSGMPXBTUG
JNQPSUUFOTPSGMPXDPOUSJCMBZFSTBTMBZFST
GSPNTLMFBSOJNQPSUEBUBTFUT
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNTLMFBSONPEFM@TFMFDUJPOJNQPSUUSBJO@UFTU@TQMJU
GSPNTLMFBSOQSFQSPDFTTJOHJNQPSU.JO.BY4DBMFS
JNQPSUQBOEBTBTQE
JNQPSUTFBCPSOBTTOT
NBUQMPUMJCJOMJOF
[ 104 ]
Neural Networks - Perceptron Chapter 3
2. Load the dataset and create a Pandas dataframe to understand the data:
%BUB
CPTUPOEBUBTFUTMPBE@CPTUPO
EGQE%BUB'SBNF
CPTUPOEBUBDPMVNOTCPTUPOGFBUVSF@OBNFT
EG< UBSHFU >CPTUPOUBSHFU
6OEFSTUBOEJOH%BUB
EGEFTDSJCF
1MPUUJOHDPSSFMBUJPO
DPMPSNBQ@BYQMUTVCQMPUT
GJHTJ[F
DPSSEGDPSS
NFUIPE QFBSTPO
DNBQTOTEJWFSHJOH@QBMFUUF
BT@DNBQ5SVF
@TOTIFBUNBQ
DPSSDNBQDNBQTRVBSF5SVFDCBS@LXT\
TISJOL ^BYBYBOOPU5SVFBOOPU@LXT\ GPOUTJ[F
^
[ 105 ]
Neural Networks - Perceptron Chapter 3
5. From the preceding code, we can see that three parameters--3., 153"5*0, and
-45"5--have a correlation greater than 0.5 in magnitude. We choose them for the
training. Split the dataset into Train and Test datasets. We also use
.JO.BY4DBMFS to normalize our data set. One important change to note is that
since our neural network is using the Sigmoid activation function (output of
sigmoid can be between 0-1 only), we will have to normalize the target value : as
well:
$SFBUF5FTU5SBJO4QMJU
9@USBJO9@UFTUZ@USBJOZ@UFTUUSBJO@UFTU@TQMJU
EG<< 3.
[ 106 ]
Neural Networks - Perceptron Chapter 3
/FUXPSL1BSBNFUFST
NMFO
9@USBJO
O/VNCFSPGGFBUVSFT
O@IJEEFO/VNCFSPGIJEEFOOFVSPOT
)ZQFSQBSBNFUFST
CBUDI@TJ[F
FUB
NBY@FQPDI
EFGNVMUJMBZFS@QFSDFQUSPO
Y
GDMBZFSTGVMMZ@DPOOFDUFE
YO@IJEEFO
BDUJWBUJPO@GOUGOOSFMVTDPQF GD
PVUMBZFSTGVMMZ@DPOOFDUFE
GDBDUJWBUJPO@GOUGTJHNPJE
TDPQF PVU
SFUVSOPVU
8. Declare the placeholders for the training data and define the loss and optimizer:
CVJMENPEFMMPTTBOEUSBJOPQ
YUGQMBDFIPMEFS
UGGMPBUOBNF 9 TIBQF<NO>
ZUGQMBDFIPMEFS
UGGMPBUOBNF :
Z@IBUNVMUJMBZFS@QFSDFQUSPO
Y
DPSSFDU@QSFEJDUJPOUGTRVBSF
ZZ@IBU
NTFUGSFEVDF@NFBO
UGDBTU
DPSSFDU@QSFEJDUJPOGMPBU
USBJOUGUSBJO"EBN0QUJNJ[FS
MFBSOJOH@SBUFFUBNJOJNJ[F
NTF
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
$PNQVUBUJPO(SBQI
XJUIUG4FTTJPO
BTTFTT*OJUJBMJ[FWBSJBCMFT
TFTTSVO
JOJUXSJUFSUGTVNNBSZ'JMF8SJUFS
HSBQIT
TFTTHSBQI
USBJOUIFNPEFMGPSFQDPIT
GPSJJOSBOHF
NBY@FQPDI
[ 107 ]
Neural Networks - Perceptron Chapter 3
@MQTFTTSVO
<USBJOMPTTZ@IBU>GFFE@EJDU\Y
9@USBJOZZ@USBJO^
JGJ
QSJOU
&QPDI\^-PTT\^ GPSNBU
JM
QSJOU
5SBJOJOH%POF
QSJOU
0QUJNJ[BUJPO'JOJTIFE
5FTUNPEFMDPSSFDU@QSFEJDUJPOUGTRVBSF
ZZ@IBU
$BMDVMBUFBDDVSBDZ
BDDVSBDZUGSFEVDF@NFBO
UGDBTU
DPSSFDU@QSFEJDUJPOGMPBU
QSJOU
.FBO&SSPSBDDVSBDZFWBM
\Y9@USBJOZZ@USBJO^
QMUTDBUUFS
Z@USBJOQ
XSJUFSDMPTF
How it works...
With only one hidden layer, the model predicts the prices on the training dataset with a
mean error of 0.0071. The figure below shows the relationship between estimated and actual
price of the house:
[ 108 ]
Neural Networks - Perceptron Chapter 3
There's more...
Here, we used TensorFlow ops Layers (Contrib) to build neural network layers. It made our
work slightly easier as we were saved from declaring weights and biases for each layer
separately. The work can be further simplified if we use an API like Keras. Here is the code
for the same in Keras with TensorFlow as the backend:
/FUXPSL1BSBNFUFST
NMFO
9@USBJO
O/VNCFSPGGFBUVSFT
O@IJEEFO/VNCFSPGIJEEFOOFVSPOT
)ZQFSQBSBNFUFST
CBUDI
FUB
NBY@FQPDI
#VJME.PEFM
NPEFM4FRVFOUJBM
NPEFMBEE
%FOTF
O@IJEEFOJOQVU@EJNOBDUJWBUJPO SFMV
NPEFMBEE
%FOTF
BDUJWBUJPO TJHNPJE
NPEFMTVNNBSZ
4VNNBSJ[FUIFNPEFM
$PNQJMFNPEFM
NPEFMDPNQJMF
MPTT NFBO@TRVBSFE@FSSPS PQUJNJ[FS BEBN
'JUUIFNPEFM
NPEFMGJU
9@USBJOZ@USBJOWBMJEBUJPO@EBUB
9@UFTU
Z@UFTUFQPDITNBY@FQPDICBUDI@TJ[FCBUDIWFSCPTF
1SFEJDUUIFWBMVFTBOEDBMDVMBUF3.4&BOE3TDPSF
Z@UFTU@QSFENPEFMQSFEJDU
9@UFTU
Z@USBJO@QSFENPEFMQSFEJDU
9@USBJO
SS@TDPSF
Z@UFTUZ@UFTU@QSFE
SNTFNFBO@TRVBSFE@FSSPS
Z@UFTUZ@UFTU@QSFE
QSJOU
1FSGPSNBODF.FUSJDT3\G^3.4&\G^GPSNBU
SSNTF
[ 109 ]
Neural Networks - Perceptron Chapter 3
The preceding code gives the following result between predicted and actual values. We can
see that the result can be improved by removing the outliers (some houses with max price
irrespective of other parameters, the points at the right extreme):
Tuning hyperparameters
The neural network performance, as you must have observed by now, depends on a lot on
hyperparameters. Thus, it becomes important that one gains an understanding as to how
these parameters affect the network. Common examples of hyperparameters are learning
rate, regularizers, regularizing coefficient, dimensions of hidden layers, initial weight
values, and even the optimizer selected to optimize weights and biases.
How to do it...
Here is how we proceed with the recipe:
1. The first step in tuning hyperparameters is building the model. Build the model
in TensorFlow, exactly the way we have been.
2. Add a way to save the model in NPEFM@GJMF. In TensorFlow, this can be done
using a 4BWFS object. Then save it in the session:
TBWFSUGUSBJO4BWFS
XJUIUG4FTTJPO
BTTFTT%P
UIFUSBJOJOHTUFQTTBWF@QBUITBWFSTBWF
TFTT
UNQNPEFMDLQUQSJOU
.PEFMTBWFEJOGJMFTTBWF@QBUI
[ 110 ]
Neural Networks - Perceptron Chapter 3
3BOEPN$IPJDFHFOFSBUFSBOEPNWBMVFTPGMFBSOJOHSBUF
MZJOHCFUXFFOBOE
MFBSOJOH@SBUFOQSBOEPNSBOE
(SJE4FBSDIHFOFSBUFWBMVFTTUBSUJOHGSPNTFQBSBUFECZ
MFBSOJOH@SBUF<JGPSJJOOQBSBOHF
>
.BOVBM4FBSDIHJWFBOZWBMVFTZPVTFFNQMBVTJCMFNBOVBMMZ
MFBSOJOH@SBUF<>
5. We choose the parameters that give the best response to our chosen MPTT
function. So, we can define a maximum value of MPTT function at the start as
CFTU@MPTT (in the case of accuracy, you will choose the minimum accuracy you
desire from your model):
CFTU@MPTT
*UDBOCFBOZOVNCFSCVUJUXPVMECFCFUUFSJGZPVLFFQJUTBNF
BTUIFMPTTZPVBDIJFWFEGSPNZPVSCBTFNPEFMEFGJOFEJOTUFQT
BOE
6. Wrap your model in a for loop for the learning rate; any model that gives a better
estimate of loss is then saved:
-PBEBOEQSFQSPDFTTEBUB
)ZQFSQBSBNFUFST
5VOJOHFQPDIT<>
CBUDIFT<>
SNTF@NJO
GPSFQPDIJOFQPDIT
GPSCBUDIJOCBUDIFT
NPEFMHFU@NPEFM
NPEFMDPNQJMF
MPTT NFBO@TRVBSFE@FSSPS PQUJNJ[FS BEBN
NPEFMGJU
9@USBJOZ@USBJOWBMJEBUJPO@EBUB
9@UFTU
Z@UFTUFQPDITFQPDICBUDI@TJ[FCBUDIWFSCPTF
Z@UFTU@QSFENPEFMQSFEJDU
9@UFTU
SNTFNFBO@TRVBSFE@FSSPS
Z@UFTUZ@UFTU@QSFE
JGSNTFSNTF@NJO
SNTF@NJOSNTF
TFSJBMJ[FNPEFMUP+40/
NPEFM@KTPONPEFMUP@KTPO
[ 111 ]
Neural Networks - Perceptron Chapter 3
XJUIPQFO
NPEFMKTPOXBTKTPO@GJMF
KTPO@GJMFXSJUF
NPEFM@KTPO
TFSJBMJ[FXFJHIUTUP)%'
NPEFMTBWF@XFJHIUT
NPEFMIEG
QSJOU
4BWFENPEFMUPEJTL
There's more...
There is another approach called Bayesian optimization, which can also be used to tune
hyperparameters. In it, we define an acquisition function along with a Gaussian process.
The Gaussian process uses a set of previously evaluated parameters and resulting accuracy
to assume about unobserved parameters. The acquisition function using this information
suggests the next set of parameters. There is a wrapper available for even gradient-based
hyperparameter optimization IUUQTHJUIVCDPNMVDGSB3')0.
See also
Good introduction to different two excellent open-source packages for
hyperparameter optimization, Hyperopt and scikit-optimize: IUUQT
SPBNBOBMZUJDTDPNPQUJNJ[JOHUIFIZQFSQBSBNFUFSPGXIJDI
IZQFSQBSBNFUFSPQUJNJ[FSUPVTF
Another one on Hyperopt: IUUQGBTUNMDPNPQUJNJ[JOHIZQFSQBSBNTXJUI
IZQFSPQU
A detailed paper by Bengio and others on various algorithms for hyperparameter
optimization:IUUQTQBQFSTOJQTDDQBQFSBMHPSJUINTGPSIZQFS
QBSBNFUFSPQUJNJ[BUJPOQEG
Higher-level APIs-Keras
Keras is a higher-level API used with TensorFlow as the backend. Adding layers to it is as
easy as adding a single line of code. After the model architecture, using one line of code,
you can compile and fit the model. Later, it can be used for prediction. Declaration of
variables, placeholders, and even the session is managed by the API.
[ 112 ]
Neural Networks - Perceptron Chapter 3
How to do it...
We proceed with Keras as follows:
1. As the first step, we define the type of our model. Keras offers two types of
models: sequential and Model class API. Keras offers various types of neural
network layers:
*NQPSUUIFNPEFMBOEMBZFSTOFFEFE
GSPNLFSBTNPEFMJNQPSU4FRVFOUJBM
GSPNLFSBTMBZFSTJNQPSU%FOTF
NPEFM4FRVFOUJBM
3. We can use it to add as many layers as we want, with each hidden layer being fed
by the previous layer. We need to specify the input dimension only for the first
layer:
5IJTXJMMBEEBGVMMZDPOOFDUFEOFVSBMOFUXPSLMBZFSXJUI
OFVSPOTFBDIUBLJOHJOQVUTBOEXJUIBDUJWBUJPOGVODUJPO3F-6
NPEFBEE
%FOTF
JOQVU@EJNBDUJWBUJPO SFMV
NPEFMBEE
BDUJWBUJPO TJHNPJE
[ 113 ]
Neural Networks - Perceptron Chapter 3
4. Once the model is defined, we need to choose a MPTT function and optimizers.
Keras offers a variety
of MPTT@GVODUJPOT: NFBO@TRVBSFE@FSSPS, NFBO@BCTPMVUF@FSSPS, NFBO@BC
TPMVUF@QFSDFOUBHF@FSSPS, DBUFHPSJDBM@DSPTTFOUSPQZ; and
optimizers: sgd, RMSprop, Adagrad, Adadelta, Adam, and so on. With these two
decided, we can configure the learning process using DPNQJMF
TFMG
PQUJNJ[FSMPTTNFUSJDT/POFTBNQMF@XFJHIU@NPEF/POF:
NPEFMDPNQJMF
PQUJNJ[FS SNTQSPQ
MPTT DBUFHPSJDBM@DSPTTFOUSPQZ
NFUSJDT< BDDVSBDZ >
NPEFMGJU
EBUBMBCFMTFQPDITCBUDI@TJ[F
NPEFMQSFEJDU
UFTU@EBUBCBUDI@TJ[F
There's more...
Keras offers options to add convolutional layers, pooling layers, recurrent layers, and even
locally connected layers. The detailed description of each method is available in Keras
documentation at IUUQTLFSBTJPNPEFMTTFRVFOUJBM.
See also
McCulloch, Warren S., and Walter Pitts. A logical calculus of the ideas immanent in
nervous activity The bulletin of mathematical biophysics 5.4 (1943): 115-133.
IUUQWPSEFOLFSEFHHQIJMPTPQIZNDDVMMPDI@BMPHJDBMDBMDVMVTQEG
Rosenblatt, Frank (1957), The Perceptron--a perceiving and recognizing
automaton. Report 85-460-1, Cornell Aeronautical Laboratory. IUUQTCMPHT
VNBTTFEVCSBJOXBSTGJMFTSPTFOCMBUUQEG
The Thinking Machine, CBS Broadcast IUUQTXXXZPVUVCFDPNXBUDIW
K1)6M2JX%:
Hornik, Kurt, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward
networks are universal approximators. Neural networks 2.5 (1989): 359-366.
[ 114 ]
Convolutional Neural Networks
4
Convolutional Neural Networks (CNNs or sometimes referred to as ConvNets) are
fascinating. Over a short period of time, they became a disruptive technology, breaking all
the state-of-the-art results in multiple domains from text, to video, to speech--going well
beyond their original use for image processing. In this chapter, we will present a number of
recipes as follows:
Introduction
A CNN consists of many neural network layers. Two different types of layers,
convolutional and pooling, are typically alternated. The depth of each filter increases from
left to right in the network. The last stage is typically made of one or more fully connected
layers:
#PGZCORNGQH%QPXQNWVKQPCN0GWTCN0GVYQTMCUUGGPKUJVVRUEQOOQPUYKMKOGFKCQTIYKMK(KNG6[RKECNAEPPRPI
Convolutional Neural Networks Chapter 4
There are three key intuitions behind convnets: local receptive fields, shared weights, and
pooling. Let's review them together.
Let's define the feature map from one layer to another. Of course, we can have multiple
feature maps that learn independently from each hidden layer. For instance, we can start
with 28 x 28 input neurons to process MNIST images, and then recall k feature maps of size
23 x 23 neurons each (again with a stride of 5 x 5) in the next hidden layer.
A mathematical example
One simple way to understand convolution is to think about a sliding window function
applied to a matrix. In the following example, given the input matrix I and the kernel K, we
get the convolved output. The 3 x 3 kernel K (sometimes called filter or feature detector) is
multiplied element-wise with the input matrix to get one cell in the output convolved
matrix. All the other cells are obtained by sliding the window on I:
[ 116 ]
Convolutional Neural Networks Chapter 4
#PGZCORNGQHEQPXQNWVKQPCNQRGTCVKQPKPDQNFVJGEGNNUKPXQNXGFKPVJGEQORWVCVKQP
In this example, we decided to stop the sliding window as soon as we touch the borders of I
(so the output is 3 x 3). Alternatively, we could have chosen to pad the input with zeros (so
that the output would have been 5 x 5). This decision relates to the padding choice adopted.
Another choice is about the stride which is about the type of shift adopted by our sliding
windows. This can be one or more. A larger stride generates fewer applications of the
kernel and a smaller output size, while a smaller stride generates more output and retains
more information.
The size of the filter, the stride, and the type of padding are hyperparameters that can be
fine-tuned during the training of the network.
ConvNets in TensorFlow
In TensorFlow, if we want to add a convolutional layer, we will write the following:
UGOODPOWE
JOQVUGJMUFSTUSJEFTQBEEJOHVTF@DVEOO@PO@HQV/POF
EBUB@GPSNBU/POFOBNF/POF
JOQVU: A tensor, must be one of the following types: half, GMPBU, GMPBU.
GJMUFS: A tensor, must have the same type as input.
TUSJEFT: A list of ints. 1D of length 4. The stride of the sliding window for each
dimension of input. Must be in the same order as the dimension specified with
format.
QBEEJOH: A string from: 4".&, 7"-*%. The type of padding algorithm to use.
VTF@DVEOO@PO@HQV: An optional bool. Defaults to 5SVF.
[ 117 ]
Convolutional Neural Networks Chapter 4
#PGZCORNGQHEQPXQNWVKQPCNQRGTCVKQP
Pooling layers
Let's suppose that we want to summarize the output of a feature map. Again, we can use
the spatial contiguity of the output produced from a single feature map and aggregate the
values of a sub-matrix into one single output value synthetically describing the meaning
associated with that physical region.
Max pooling
One easy and common choice is the so-called max-pooling operator, which simply outputs
the maximum activation as observed in the region. In TensorFlow, if we want to define a
max pooling layer of size 2 x 2, we will write the following:
UGOONBY@QPPM
WBMVFLTJ[FTUSJEFTQBEEJOHEBUB@GPSNBU /)8$
OBNF/POF
[ 118 ]
Convolutional Neural Networks Chapter 4
WBMVF: A 4-D Tensor with shape [CBUDI, IFJHIU, XJEUI, DIBOOFMT]and type
UGGMPBU.
LTJ[F: A list of ints that has length >= 4. The size of the window for each
dimension of the input tensor.
TUSJEFT: A list of ints that has length >= 4. The stride of the sliding window for
each dimension of the input tensor.
QBEEJOH: A string, either 7"-*% or 4".&.
EBUB@GPSNBU: A string. /)8$ and /$)8 are supported.
OBNF: Optional name for the operation.
#PGZCORNGQHRQQNKPIQRGTCVKQP
Average pooling
Another choice is average-pooling, which simply aggregates a region into the average
values of the activation observed in that region.
[ 119 ]
Convolutional Neural Networks Chapter 4
ConvNets summary
CNNs are basically several layers of convolutions with nonlinear activation functions and
pooling layers applied to the results. Each layer applies different filters (hundreds or
thousands). The key observation to understand is that the filters are not pre-assigned, but
instead, they are learned during the training phase in such a way that a suitable loss
function is minimized. It has been observed that lower layers will learn to detect basic
features, while the higher layers detect progressively more sophisticated features such as
shapes or faces. Note that, thanks to pooling, individual neurons in later layers see more of
the original image, hence they are able to compose basic features learned in the earlier
layers.
So far we have described the basic concepts of ConvNets. CNNs apply convolution and
pooling operations in one dimension for audio and text data along the time dimension, in
two dimensions for images along the (height x width) dimensions, and in three dimensions
for videos along the (height x width x time) dimensions. For images, sliding the filter over
an input volume produces a map that provides the responses of the filter for each spatial
position.
In other words, a ConvNet has multiple filters stacked together that learn
to recognize specific visual features independently of the location in the
image. Those visual features are simple in the initial layers of the network
and then more and more sophisticated deeper in the network.
Getting ready
MNIST is a set of 60,000 images representing handwritten numbers. The goal of this recipe
is to recognize those numbers with high accuracy.
[ 120 ]
Convolutional Neural Networks Chapter 4
How to do it...
Let us start with the recipe:
1. Import UFOTPSGMPX, NBUQMPUMJC, SBOEPN, and OVNQZ. Then, import the NJOTU
data and perform one-hot encoding. Note the TensorFlow has some built-in
libraries to deal with ./*45 and we are going to use them:
GSPN@@GVUVSF@@JNQPSUEJWJTJPOQSJOU@GVODUJPO
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMU
JNQPSUOVNQZBTOQ
*NQPSU./*45EBUB
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUBPOF@IPU5SVF
2. Introspect some data to understand what ./*45 is. This gives us an idea of how
many images are in the training dataset and how many are in the test dataset. We
are also going to visualize a few numbers just to understand how they are
represented. The multi-flat output can give us a visual sense of how difficult it
can be to recognize a handwritten number, even for humans.
EFGUSBJO@TJ[F
OVN
QSJOU
5PUBM5SBJOJOH*NBHFTJO%BUBTFU
TUS
NOJTUUSBJOJNBHFTTIBQF
QSJOU
Y@USBJONOJTUUSBJOJNBHFT<OVN>
QSJOU
Y@USBJO&YBNQMFT-PBEFE
TUS
Y@USBJOTIBQF
Z@USBJONOJTUUSBJOMBCFMT<OVN>
QSJOU
Z@USBJO&YBNQMFT-PBEFE
TUS
Z@USBJOTIBQF
QSJOU
SFUVSOY@USBJOZ@USBJO
EFGUFTU@TJ[F
OVN
QSJOU
5PUBM5FTU&YBNQMFTJO%BUBTFU
TUS
NOJTUUFTUJNBHFTTIBQF
QSJOU
Y@UFTUNOJTUUFTUJNBHFT<OVN>
QSJOU
Y@UFTU&YBNQMFT-PBEFE
TUS
Y@UFTUTIBQF
Z@UFTUNOJTUUFTUMBCFMT<OVN>
QSJOU
Z@UFTU&YBNQMFT-PBEFE
TUS
Z@UFTUTIBQF
SFUVSOY@UFTUZ@UFTU
EFGEJTQMBZ@EJHJU
OVN
QSJOU
Z@USBJO<OVN>
MBCFMZ@USBJO<OVN>BSHNBY
BYJT
JNBHFY@USBJO<OVN>SFTIBQF
<>
QMUUJUMF
&YBNQMFE-BCFME
OVNMBCFM
[ 121 ]
Convolutional Neural Networks Chapter 4
QMUJNTIPX
JNBHFDNBQQMUHFU@DNBQ
HSBZ@S
QMUTIPX
EFGEJTQMBZ@NVMU@GMBU
TUBSUTUPQ
JNBHFTY@USBJO<TUBSU>SFTIBQF
<>
GPSJJOSBOHF
TUBSU
TUPQ
JNBHFTOQDPODBUFOBUF
JNBHFT
Y@USBJO<J>SFTIBQF
<>
QMUJNTIPX
JNBHFTDNBQQMUHFU@DNBQ
HSBZ@S
QMUTIPX
Y@USBJOZ@USBJOUSBJO@TJ[F
EJTQMBZ@EJHJU
OQSBOEPNSBOEJOU
Y@USBJOTIBQF<>
EJTQMBZ@NVMU@GMBU
#PGZCORNGQH/0+56JCPFYTKVVGPPWODGTU
1BSBNFUFST
MFBSOJOH@SBUF
USBJOJOH@JUFST
CBUDI@TJ[F
EJTQMBZ@TUFQ
/FUXPSL1BSBNFUFST
O@JOQVU
[ 122 ]
Convolutional Neural Networks Chapter 4
./*45EBUBJOQVU
JNHTIBQF
O@DMBTTFT
./*45UPUBMDMBTTFT
EJHJUT
ESPQPVU
%SPQPVUQSPCBCJMJUZUPLFFQVOJUT
4. Set up the TensorFlow computation graph input. Let's define two placeholders to
store predictions and true labels:
YUGQMBDFIPMEFS
UGGMPBU</POFO@JOQVU>
ZUGQMBDFIPMEFS
UGGMPBU</POFO@DMBTTFT>
LFFQ@QSPCUGQMBDFIPMEFS
UGGMPBU
5. Define a convolutional layer with input Y, weight 8, bias C, and a given stride.
The activation function is ReLU and padding is 4".&:
EFGDPOWE
Y8CTUSJEFT
YUGOODPOWE
Y8TUSJEFT<TUSJEFTTUSJEFT>
QBEEJOH 4".&
YUGOOCJBT@BEE
YC
SFUVSOUGOOSFMV
Y
6. Define a maxpool layer with input Y, with LTJ[F and padding 4".&:
EFGNBYQPPME
YL
SFUVSOUGOONBY@QPPM
YLTJ[F<LL>TUSJEFT<LL
>QBEEJOH 4".&
EFGDPOW@OFU
YXFJHIUTCJBTFTESPQPVU
SFTIBQFUIFJOQVUQJDUVSF
YUGSFTIBQF
YTIBQF<>
'JSTUDPOWPMVUJPOMBZFS
DPOWDPOWE
YXFJHIUT< XD >CJBTFT< CD >
.BY1PPMJOHVTFEGPSEPXOTBNQMJOH
DPOWNBYQPPME
DPOWL
4FDPOEDPOWPMVUJPOMBZFS
DPOWDPOWE
DPOWXFJHIUT< XD >CJBTFT< CD >
.BY1PPMJOHVTFEGPSEPXOTBNQMJOH
DPOWNBYQPPME
DPOWL
3FTIBQFDPOWPVUQVUUPNBUDIUIFJOQVUPGGVMMZDPOOFDUFE
MBZFS
GDUGSFTIBQF
DPOW<
XFJHIUT< XE >HFU@TIBQF
BT@MJTU
<>>
'VMMZDPOOFDUFEMBZFS
[ 123 ]
Convolutional Neural Networks Chapter 4
GDUGBEE
UGNBUNVM
GDXFJHIUT< XE >CJBTFT< CE >
GDUGOOSFMV
GD
%SPQPVU
GDUGOOESPQPVU
GDESPQPVU
0VUQVUUIFDMBTTQSFEJDUJPO
PVUUGBEE
UGNBUNVM
GDXFJHIUT< PVU >CJBTFT< PVU >
SFUVSOPVU
8. Define the layers weights and the bias. The first conv layer has a 5 x 5
convolution, 1 input, and 32 outputs. The second conv layer has 5 x 5
convolutions, 32 inputs, and 64 outputs. The fully connected layer has 7 x 7 x 64
inputs and 1,024 outputs, while the second layer has 1,024 inputs and 10 outputs
corresponding to the final digit classes. All the weights and bias are initialized
with SBOEPO@OPSNBM distribution:
XFJHIUT\
YDPOWJOQVUBOEPVUQVUT
XD UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
YDPOWJOQVUTBOEPVUQVUT
XD UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
GVMMZDPOOFDUFEJOQVUTBOEPVUQVUT
XE UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
JOQVUTPVUQVUTGPSDMBTTEJHJUT
PVU UG7BSJBCMF
UGSBOEPN@OPSNBM
<O@DMBTTFT>
^
CJBTFT\
CD UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
CD UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
CE UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
PVU UG7BSJBCMF
UGSBOEPN@OPSNBM
<O@DMBTTFT>
^
9. Build the convnet with the given weights and bias. Define the MPTT function
based on DSPTT@FOUSPQZ with MPHJUT and use the Adam optimizer for cost
minimization. After optimization, compute the accuracy:
QSFEDPOW@OFU
YXFJHIUTCJBTFTLFFQ@QSPC
DPTU
UGSFEVDF@NFBO
UGOOTPGUNBY@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUTQSFE
MBCFMTZ
PQUJNJ[FS
UGUSBJO"EBN0QUJNJ[FS
MFBSOJOH@SBUFMFBSOJOH@SBUFNJOJNJ[F
DPTU
DPSSFDU@QSFEJDUJPOUGFRVBM
UGBSHNBY
QSFEUGBSHNBY
Z
BDDVSBDZUGSFEVDF@NFBO
UGDBTU
DPSSFDU@QSFEJDUJPOUGGMPBU
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
[ 124 ]
Convolutional Neural Networks Chapter 4
10. Launch the graph and iterate for USBJOJOH@JUFSBUT times where each time a
CBUDI@TJ[F is taken in the input to run the optimizer. Note that we train by
using the NOJTUUSBJO data, which is separated from NJOTU. Each
EJTQMBZ@TUFQ, the current partial accuracy is computed. At the end, the
accuracy is computed on 2,048 test images with no dropout.
USBJO@MPTT<>
USBJO@BDD<>
UFTU@BDD<>
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
TUFQ
XIJMFTUFQUSBJOJOH@JUFST
CBUDI@YCBUDI@ZNOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
TFTTSVO
PQUJNJ[FSGFFE@EJDU\YCBUDI@YZCBUDI@Z
LFFQ@QSPCESPQPVU^
JGTUFQEJTQMBZ@TUFQ
MPTT@USBJOBDD@USBJOTFTTSVO
<DPTUBDDVSBDZ>
GFFE@EJDU\YCBUDI@Y
ZCBUDI@Z
LFFQ@QSPC
^
QSJOU*UFS
TUS
TUFQ
.JOJCBUDI-PTT
=
\G^GPSNBU
MPTT@USBJO
5SBJOJOH
"DDVSBDZ
=
\G^GPSNBU
BDD@USBJO
$BMDVMBUFBDDVSBDZGPSNOJTUUFTUJNBHFT
/PUFUIBUJOUIJTDBTFOPESPQPVU
BDD@UFTUTFTTSVO
BDDVSBDZ
GFFE@EJDU\YNOJTUUFTUJNBHFT
ZNOJTUUFTUMBCFMT
LFFQ@QSPC^
QSJOU5FTUJOH"DDVSBDZ
=
\G^GPSNBU
BDD@USBJO
USBJO@MPTTBQQFOE
MPTT@USBJO
USBJO@BDDBQQFOE
BDD@USBJO
UFTU@BDDBQQFOE
BDD@UFTU
TUFQ
11. Plot the Softmax loss per iteration, together with train and test accuracy:
FWBM@JOEJDFTSBOHF
USBJOJOH@JUFSTEJTQMBZ@TUFQ
1MPUMPTTPWFSUJNF
QMUQMPU
FWBM@JOEJDFTUSBJO@MPTT L
QMUUJUMF
4PGUNBY-PTTQFSJUFSBUJPO
QMUYMBCFM
*UFSBUJPO
[ 125 ]
Convolutional Neural Networks Chapter 4
QMUZMBCFM
4PGUNBY-PTT
QMUTIPX
1MPUUSBJOBOEUFTUBDDVSBDZ
QMUQMPU
FWBM@JOEJDFTUSBJO@BDD L MBCFM 5SBJO4FU"DDVSBDZ
QMUQMPU
FWBM@JOEJDFTUFTU@BDD S MBCFM 5FTU4FU"DDVSBDZ
QMUUJUMF
5SBJOBOE5FTU"DDVSBDZ
QMUYMBCFM
(FOFSBUJPO
QMUZMBCFM
"DDVSBDZ
QMUMFHFOE
MPD MPXFSSJHIU
QMUTIPX
Following is the output of the preceding code. We first look at Softmax per iteration:
#PGZCORNGQHNQUUFGETGCUG
#PGZCORNGQHVTCKPCPFVGUVCEEWTCE[KPETGCUG
[ 126 ]
Convolutional Neural Networks Chapter 4
How it works...
Using ConvNets, we increased our performance on the MNIST dataset reaching almost 95
percent accuracy. Our ConvNet consists of two layers combining convolutions, ReLU, and
maxpooling, followed by two fully connected layers with dropout. Training happens in
batches of size 128 with Adam used as an optimizer, a learning rate of 0.001, and a
maximum number of 500 iterations.
'ZCORNGUQH%+(#4KOCIGU
Getting ready
In this recipe, we use UGMFBSO--a higher-level framework--which abstracts some of the
TensorFlow internals and allows us to focus on the definition of deep networks. TFLearn is
available at IUUQUGMFBSOPSH and this code is part of the standard distribution.
(IUUQTHJUIVCDPNUGMFBSOUGMFBSOUSFFNBTUFSFYBNQMFT)
[ 127 ]
Convolutional Neural Networks Chapter 4
How to do it...
We proceed with the recipe as follows:
1. Import a few VUJMT and core layers for ConvNets, ESPQPVU, GVMMZ@DPOOFDUFE,
and NBY@QPPM. In addition, import a few modules useful for image processing
and image augmentation. Note that TFLearn provides some already defined
higher-level layers for ConvNets and this allows us to focus on the definition of
our code:
GSPN@@GVUVSF@@JNQPSUEJWJTJPOQSJOU@GVODUJPOBCTPMVUF@JNQPSU
JNQPSUUGMFBSO
GSPNUGMFBSOEBUB@VUJMTJNQPSUTIVGGMFUP@DBUFHPSJDBM
GSPNUGMFBSOMBZFSTDPSFJNQPSUJOQVU@EBUBESPQPVU
GVMMZ@DPOOFDUFE
GSPNUGMFBSOMBZFSTDPOWJNQPSUDPOW@ENBY@QPPM@E
GSPNUGMFBSOMBZFSTFTUJNBUPSJNQPSUSFHSFTTJPO
GSPNUGMFBSOEBUB@QSFQSPDFTTJOHJNQPSU*NBHF1SFQSPDFTTJOH
GSPNUGMFBSOEBUB@BVHNFOUBUJPOJNQPSU*NBHF"VHNFOUBUJPO
2. Load the CIFAR-10 data and separate it into X train data, Y train labels, 9@UFTU
for test, and :@UFTU for test labels. It might be useful to shuffle 9 and : to avoid
depending on a particular data configuration. The last step is to perform one-hot
encoding for both 9 and ::
%BUBMPBEJOHBOEQSFQSPDFTTJOH
GSPNUGMFBSOEBUBTFUTJNQPSUDJGBS
9:
9@UFTU:@UFTUDJGBSMPBE@EBUB
9:TIVGGMF
9:
:UP@DBUFHPSJDBM
:
:@UFTUUP@DBUFHPSJDBM
:@UFTU
3. Use *NBHF1SFQSPDFTTJOH
for Zero Center (with mean computed over the
whole dataset) and for STD Normalization (with std computed over the whole
dataset). The TFLearn data stream is designed to speed up training by
preprocessing data on CPU while GPU is performing model training.
3FBMUJNFEBUBQSFQSPDFTTJOH
JNH@QSFQ*NBHF1SFQSPDFTTJOH
JNH@QSFQBEE@GFBUVSFXJTF@[FSP@DFOUFS
JNH@QSFQBEE@GFBUVSFXJTF@TUEOPSN
[ 128 ]
Convolutional Neural Networks Chapter 4
4. Augment the dataset by performing random flip right and left and by random
rotation. This step is a simple trick used to increase the data available for the
training:
3FBMUJNFEBUBBVHNFOUBUJPO
JNH@BVH*NBHF"VHNFOUBUJPO
JNH@BVHBEE@SBOEPN@GMJQ@MFGUSJHIU
JNH@BVHBEE@SBOEPN@SPUBUJPO
NBY@BOHMF
5. Create the convolutional network with the images preparation and augmentation
defined earlier. The network consists of three convolutional layers. The first one
uses 32 convolutional filters, with size of filters 3 and activation function ReLU.
After that, there is a NBY@QPPM layer for the downsizing. Then there are two
convolutional filters in cascade with 64 convolutional filters, with size of filters 3
and activation function ReLU. After that, there is a NBY@QPPM for the downsizing
and a fully connected network with 512 neurons with activation function ReLU
followed by a dropout with probability 50 percent. The last layer is a fully
connected network with 10 neurons and activation function TPGUNBY to
determine the category of the handwritten digits. Note that this particular type of
ConvNet is known to be very effective with CIFAR-10. In this particular case, we
use the Adam optimizer with DBUFHPSJDBM@DSPTTFOUSPQZ and learning rate
:
$POWPMVUJPOBMOFUXPSLCVJMEJOH
OFUXPSLJOQVU@EBUB
TIBQF</POF>
EBUB@QSFQSPDFTTJOHJNH@QSFQ
EBUB@BVHNFOUBUJPOJNH@BVH
OFUXPSLDPOW@E
OFUXPSLBDUJWBUJPO SFMV
OFUXPSLNBY@QPPM@E
OFUXPSL
OFUXPSLDPOW@E
OFUXPSLBDUJWBUJPO SFMV
OFUXPSLDPOW@E
OFUXPSLBDUJWBUJPO SFMV
OFUXPSLNBY@QPPM@E
OFUXPSL
OFUXPSLGVMMZ@DPOOFDUFE
OFUXPSLBDUJWBUJPO SFMV
OFUXPSLESPQPVU
OFUXPSL
OFUXPSLGVMMZ@DPOOFDUFE
OFUXPSLBDUJWBUJPO TPGUNBY
OFUXPSLSFHSFTTJPO
OFUXPSLPQUJNJ[FS BEBN
MPTT DBUFHPSJDBM@DSPTTFOUSPQZ
MFBSOJOH@SBUF
[ 129 ]
Convolutional Neural Networks Chapter 4
6. Instantiate the ConvNet and run the train for 50 epochs with CBUDI@TJ[F:
5SBJOVTJOHDMBTTJGJFS
NPEFMUGMFBSO%//
OFUXPSLUFOTPSCPBSE@WFSCPTF
NPEFMGJU
9:O@FQPDITIVGGMF5SVFWBMJEBUJPO@TFU
9@UFTU
:@UFTU
TIPX@NFUSJD5SVFCBUDI@TJ[FSVO@JE DJGBS@DOO
How it works...
TFLearn hides many of the implementation details exposed by TensorFlow and, in many
cases, it allows us to focus on the definition of the ConvNet with a higher level of
abstraction. Our pipeline reaches an accuracy of 88 percent in 50 iterations. The following
image is a snapshot of the execution in a Jupyter notebook:
#PGZCORNGQH,WR[VGTGZGEWVKQPHQT%+(#4ENCUUKaECVKQP
[ 130 ]
Convolutional Neural Networks Chapter 4
There's more...
To install TFLearn, see the Installation Guide (IUUQUGMFBSOPSHJOTUBMMBUJPO) and if
you want to see more examples, there is a long list of already cooked solutions available
online (IUUQUGMFBSOPSHFYBNQMFT) .
Getting ready
We are going to implement the algorithm described in the paper, A Neural Algorithm of
Artistic Style (IUUQTBSYJWPSHBCT) by Leon A. Gatys, Alexander S. Ecker,
and Matthias Bethge. So, it would be good to read the paper first (IUUQTBSYJWPSHBCT
). This recipe will reuse a pretrained model VGG19 available online (IUUQ
XXXWMGFBUPSHNBUDPOWOFUNPEFMTCFUBJNBHFOFUWHHWFSZEFFQNBU), which
should be downloaded locally. Our style image will be a famous painting of Van Gogh
available online (IUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMF7BO(PHITUBSSZ@OJHIU
KQH), and our content image is a picture of Marilyn Monroe downloaded from Wikipedia
(IUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMF[email protected]@JO@KQH). The content
image will be repainted according to the Van Gogh style.
[ 131 ]
Convolutional Neural Networks Chapter 4
How to do it...
Let us start with the recipe:
JNQPSUPT
JNQPSUTZT
JNQPSUOVNQZBTOQ
JNQPSUTDJQZJP
JNQPSUTDJQZNJTD
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNNBUQMPUMJCQZQMPU
JNQPSUJNTIPX
GSPN1*-
JNQPSU*NBHFNBUQMPUMJCJOMJOFGSPN@@GVUVSF@@
JNQPSUEJWJTJPO
2. Then, set the input path for the image used to learn the style and for the content
image to be repainted according to the style:
065165@%*3 PVUQVU
4UZMFJNBHF
45:-&@*."(& EBUB4UBSSZ/JHIUKQH
$POUFOUJNBHFUPCFSFQBJOUFE
$0/5&/5@*."(& EBUB[email protected]@JO@KQH
3. Then we set up the noise ratio used during the image generation and the
emphasis we would like to put on content loss and on style loss when repainting
the content image. In addition to that, we store the path to the pretrained VGG
model and the mean computed during the VGG pretraining. This mean is already
known and it is subtracted from the input to the VGG model:
IPXNVDIOPJTFJTJOUIFJNBHF
/0*4&@3"5*0
)PXNVDIFNQIBTJTPODPOUFOUMPTT
#&5"
)PXNVDIFNQIBTJTPOTUZMFMPTT
"-1)"
UIF7((MBZFSQSFUSBJOFENPEFM
7((@.0%&- EBUBJNBHFOFUWHHWFSZEFFQNBU
[ 132 ]
Convolutional Neural Networks Chapter 4
5IFNFBOVTFEXIFOUIF7((XBTUSBJOFE
*UJTTVCUSBDUFEGSPNUIFJOQVUUPUIF7((NPEFM.&"/@7"-6&4
OQBSSBZ
<>SFTIBQF
DPOUFOU@JNBHFTDJQZNJTDJNSFBE
$0/5&/5@*."(&
JNTIPX
DPOUFOU@JNBHF
Here is the output of the preceding code (note that this image is in IUUQT
DPNNPOTXJLJNFEJBPSHXJLJ'JMF[email protected]@JO@KQH) :
5. Resize the style image and show it just to understand how it is. Note that the
content image and the style image have now the same size and the same number
of color channels:
TUZMF@JNBHFTDJQZNJTDJNSFBE
45:-&@*."(&
(FUTIBQFPGUBSHFUBOENBLFUIFTUZMFJNBHFUIFTBNF
UBSHFU@TIBQFDPOUFOU@JNBHFTIBQF
QSJOUUBSHFU@TIBQFUBSHFU@TIBQF
QSJOUTUZMF@TIBQFTUZMF@JNBHFTIBQF
SBUJPUBSHFU@TIBQF<>TUZMF@JNBHFTIBQF<>
QSJOUSFTJ[FSBUJPSBUJP
TUZMF@JNBHFTDJQZNJTDJNSFTJ[F
TUZMF@JNBHFUBSHFU@TIBQF
TDJQZNJTDJNTBWF
45:-&@*."(&TUZMF@JNBHF
JNTIPX
TUZMF@JNBHF
[ 133 ]
Convolutional Neural Networks Chapter 4
#PGZCORNGQH8KEGPV8CP)QIJRCKPVKPICUUGGPKPJVVRUEQOOQPUYKMKOGFKCQTIYKMK(KNG8CP)QIJUVCTT[APKIJVADCNNCPEGLRI
6. The next step is to define the VGG model as described in the original paper. Note
that the deep learning network is rather complex as it combines multiple
ConvNet layers with ReLU activation function and max pooling. An additional
note is that in the original paper for Transfer Styling (A Neural Algorithm of
Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge), many
experiments show that average pooling is actually outperforming max pooling.
So, we will use average pooling instead:
EFGMPBE@WHH@NPEFM
QBUIJNBHF@IFJHIUJNBHF@XJEUI
DPMPS@DIBOOFMT
3FUVSOTUIF7((NPEFMBTEFGJOFEJOUIFQBQFS
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTNBYQPPM
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTNBYQPPM
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
[ 134 ]
Convolutional Neural Networks Chapter 4
JTDPOW@
JTSFMV
JTNBYQPPM
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTNBYQPPM
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTDPOW@
JTSFMV
JTNBYQPPM
JTGVMMZDPOOFDUFE
JTSFMV
JTGVMMZDPOOFDUFE
JTSFMV
JTGVMMZDPOOFDUFE
JTTPGUNBY
WHHTDJQZJPMPBENBU
QBUI
WHH@MBZFSTWHH< MBZFST >
EFG@XFJHIUT
MBZFSFYQFDUFE@MBZFS@OBNF
3FUVSOUIFXFJHIUTBOECJBTGSPNUIF7((NPEFMGPS
BHJWFOMBZFS
8WHH@MBZFST<><MBZFS><><><><><>
CWHH@MBZFST<><MBZFS><><><><><>
MBZFS@OBNFWHH@MBZFST<><MBZFS><><><>
BTTFSUMBZFS@OBNFFYQFDUFE@MBZFS@OBNF
SFUVSO8C
EFG@SFMV
DPOWE@MBZFS
3FUVSOUIF3&-6GVODUJPOXSBQQFEPWFSB5FOTPS'MPXMBZFS
&YQFDUTB
$POWEMBZFSJOQVU
SFUVSOUGOOSFMV
DPOWE@MBZFS
EFG@DPOWE
QSFW@MBZFSMBZFSMBZFS@OBNF
[ 135 ]
Convolutional Neural Networks Chapter 4
3FUVSOUIF$POW%MBZFSVTJOHUIFXFJHIUTCJBTFTGSPNUIF
7((
NPEFMBU MBZFS
8C@XFJHIUT
MBZFSMBZFS@OBNF
8UGDPOTUBOU
8
CUGDPOTUBOU
OQSFTIBQF
C
CTJ[F
SFUVSOUGOODPOWE
QSFW@MBZFSGJMUFS8TUSJEFT<>
QBEEJOH 4".&
C
EFG@DPOWE@SFMV
QSFW@MBZFSMBZFSMBZFS@OBNF
3FUVSOUIF$POW%
3&-6MBZFSVTJOHUIFXFJHIUTCJBTFT
GSPNUIF7((
NPEFMBU MBZFS
SFUVSO@SFMV
@DPOWE
QSFW@MBZFSMBZFSMBZFS@OBNF
EFG@BWHQPPM
QSFW@MBZFS
3FUVSOUIF"WFSBHF1PPMJOHMBZFS
SFUVSOUGOOBWH@QPPM
QSFW@MBZFSLTJ[F<>
TUSJEFT<>QBEEJOH 4".&
$POTUSVDUTUIFHSBQINPEFM
HSBQI\^
HSBQI< JOQVU >UG7BSJBCMF
OQ[FSPT
JNBHF@IFJHIU
JNBHF@XJEUIDPMPS@DIBOOFMT
EUZQF GMPBU
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< JOQVU > DPOW@
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< DPOW@ > DPOW@
HSBQI< BWHQPPM >@BWHQPPM
HSBQI< DPOW@ >
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< BWHQPPM >
DPOW@
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< DPOW@ > DPOW@
HSBQI< BWHQPPM >@BWHQPPM
HSBQI< DPOW@ >
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< BWHQPPM >
DPOW@
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< DPOW@ >
DPOW@
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< DPOW@ >
DPOW@
HSBQI< DPOW@ >@DPOWE@SFMV
HSBQI< DPOW@ >
DPOW@
[ 136 ]
Convolutional Neural Networks Chapter 4
7. Define the content MPTT function as it has been described in the original paper:
EFGDPOUFOU@MPTT@GVOD
TFTTNPEFM
$POUFOUMPTTGVODUJPOBTEFGJOFEJOUIFQBQFS
EFG@DPOUFOU@MPTT
QY
/JTUIFOVNCFSPGGJMUFST
BUMBZFSM
/QTIBQF<>
.JTUIFIFJHIUUJNFTUIFXJEUIPGUIFGFBUVSFNBQ
BUMBZFSM
.QTIBQF<>QTIBQF<>SFUVSO
/.
UGSFEVDF@TVN
UGQPX
YQ
SFUVSO@DPOUFOU@MPTT
TFTTSVO
NPEFM< DPOW@ >NPEFM< DPOW@ >
8. Define which VGG layers we are going to reuse. If we would like to have softer
features, we will need to increase the weight of higher layers (DPOW@) and
decrease the weight of the lower layers (DPOW@). If we would like to have
harder features, we need to do the opposite:
45:-&@-":&34<
DPOW@
DPOW@
DPOW@
DPOW@
DPOW@
>
[ 137 ]
Convolutional Neural Networks Chapter 4
9. Define the style loss function as it has been described in the original paper:
EFGTUZMF@MPTT@GVOD
TFTTNPEFM
4UZMFMPTTGVODUJPOBTEFGJOFEJOUIFQBQFS
EFG@HSBN@NBUSJY
'/.
5IFHSBNNBUSJY(
'UUGSFTIBQF
'
./
SFUVSOUGNBUNVM
UGUSBOTQPTF
'U'U
EFG@TUZMF@MPTT
BY
5IFTUZMFMPTTDBMDVMBUJPO
/JTUIFOVNCFSPGGJMUFST
BUMBZFSM
/BTIBQF<>
.JTUIFIFJHIUUJNFTUIFXJEUIPGUIFGFBUVSFNBQ
BUMBZFSM
.BTIBQF<>BTIBQF<>
"JTUIFTUZMFSFQSFTFOUBUJPOPGUIFPSJHJOBMJNBHF
BUMBZFSM
"@HSBN@NBUSJY
B/.
(JTUIFTUZMFSFQSFTFOUBUJPOPGUIFHFOFSBUFEJNBHF
BUMBZFSM
(@HSBN@NBUSJY
Y/.
SFTVMU
/.UGSFEVDF@TVN
UGQPX
("
SFUVSOSFTVMU
&<@TUZMF@MPTT
TFTTSVO
NPEFM<MBZFS@OBNF>NPEFM<MBZFS@OBNF>
GPSMBZFS@OBNF@JO45:-&@-":&34>
8<XGPS@XJO45:-&@-":&34>
MPTTTVN
<8<M>&<M>GPSMJOSBOHF
MFO
45:-&@-":&34>
SFUVSOMPTT
10. Define a function to generate a noise image and intermix it with the content
image with a given ratio. Define two auxiliary methods to preprocess and save
images:
EFGHFOFSBUF@OPJTF@JNBHF
DPOUFOU@JNBHFOPJTF@SBUJP/0*4&@3"5*0
3FUVSOTBOPJTFJNBHFJOUFSNJYFEXJUIUIFDPOUFOUJNBHFBUBDFSUBJO
SBUJP
OPJTF@JNBHFOQSBOEPNVOJGPSN
DPOUFOU@JNBHF<>TIBQF<>
DPOUFOU@JNBHF<>TIBQF<>
[ 138 ]
Convolutional Neural Networks Chapter 4
DPOUFOU@JNBHF<>TIBQF<>BTUZQF
GMPBU
8IJUFOPJTFJNBHFGSPNUIFDPOUFOUSFQSFTFOUBUJPO5BLFBXFJHIUFE
BWFSBHF
PGUIFWBMVFT
JOQVU@JNBHFOPJTF@JNBHFOPJTF@SBUJP
DPOUFOU@JNBHF
OPJTF@SBUJP
SFUVSOJOQVU@JNBHF
EFGQSPDFTT@JNBHF
JNBHF
3FTJ[FUIFJNBHFGPSDPOWOFUJOQVUUIFSFJTOPDIBOHFCVUKVTU
BEEBOFYUSBEJNFOTJPO
JNBHFOQSFTIBQF
JNBHF
JNBHFTIBQF
*OQVUUPUIF7((NPEFMFYQFDUTUIFNFBOUPCFTVCUSBDUFE
JNBHFJNBHF.&"/@7"-6&4
SFUVSOJNBHF
EFGTBWF@JNBHF
QBUIJNBHF
0VUQVUTIPVMEBEECBDLUIFNFBO
JNBHFJNBHF
.&"/@7"-6&4
(FUSJEPGUIFGJSTUVTFMFTTEJNFOTJPOXIBUSFNBJOTJTUIFJNBHF
JNBHFJNBHF<>
JNBHFOQDMJQ
JNBHFBTUZQF
VJOU
TDJQZNJTDJNTBWF
QBUIJNBHF
TFTTUG*OUFSBDUJWF4FTTJPO
DPOUFOU@JNBHFMPBE@JNBHF
$0/5&/5@*."(&JNTIPX
DPOUFOU@JNBHF<>
We get the output of the preceding code as follows (note that we used an image from
https://commons.wikimedia.org/wiki/File:Marilyn_Monroe_in_1952.jpg) :
[ 139 ]
Convolutional Neural Networks Chapter 4
TUZMF@JNBHFMPBE@JNBHF
45:-&@*."(&JNTIPX
TUZMF@JNBHF<>
NPEFMMPBE@WHH@NPEFM
7((@.0%&-TUZMF@JNBHF<>TIBQF<>
TUZMF@JNBHF<>TIBQF<>TUZMF@JNBHF<>TIBQF<>QSJOU
NPEFM
15. Generate a random noise image that is used to bootstrap the repainting:
JOQVU@JNBHFHFOFSBUF@OPJTF@JNBHF
DPOUFOU@JNBHF
JNTIPX
JOQVU@JNBHF<>
TFTTSVO
UGJOJUJBMJ[F@BMM@WBSJBCMFT
$POTUSVDUDPOUFOU@MPTTVTJOHDPOUFOU@JNBHF
TFTTSVO
NPEFM< JOQVU >BTTJHO
DPOUFOU@JNBHF
DPOUFOU@MPTTDPOUFOU@MPTT@GVOD
TFTTNPEFM
$POTUSVDUTUZMF@MPTTVTJOHTUZMF@JNBHF
TFTTSVO
NPEFM< JOQVU >BTTJHO
TUZMF@JNBHF
TUZMF@MPTTTUZMF@MPTT@GVOD
TFTTNPEFM
[ 140 ]
Convolutional Neural Networks Chapter 4
$POTUSVDUUPUBM@MPTTBTXFJHIUFEDPNCJOBUJPOPGDPOUFOU@MPTTBOE
TZUMF@MPTT
UPUBM@MPTT#&5"DPOUFOU@MPTT
"-1)"TUZMF@MPTT
19. Build an optimizer to minimize the total loss. In this case, we adopt the Adam
optimizer:
5IFDPOUFOUJTCVJMUGSPNPOFMBZFSXIJMFUIFTUZMFJTGSPNGJWF
MBZFST5IFOXFNJOJNJ[FUIFUPUBM@MPTT
PQUJNJ[FSUGUSBJO"EBN0QUJNJ[FS
USBJO@TUFQPQUJNJ[FSNJOJNJ[F
UPUBM@MPTT
TFTTSVO
UGJOJUJBMJ[F@BMM@WBSJBCMFT
TFTTSVO
NPEFM< JOQVU >BTTJHO
JOQVU@JNBHF
21. Run the model for a fixed number of iterations and produce intermediate
repainted images:
TFTTSVO
UGJOJUJBMJ[F@BMM@WBSJBCMFT
TFTTSVO
NPEFM< JOQVU >BTTJHO
JOQVU@JNBHF
QSJOUTUBSUFEJUFSBUJPO
GPSJUJOSBOHF
*5&3"5*0/4
TFTTSVO
USBJO@TUFQ
QSJOUJU
JGJU
1SJOUFWFSZJUFSBUJPO
NJYFE@JNBHFTFTTSVO
NPEFM< JOQVU >
QSJOU
*UFSBUJPOE
JU
QSJOU
TVN
TFTTSVO
UGSFEVDF@TVN
NJYFE@JNBHF
QSJOU
DPTU TFTTSVO
UPUBM@MPTT
JGOPUPTQBUIFYJTUT
065165@%*3
PTNLEJS
065165@%*3
GJMFOBNF PVUQVUEQOH
JU
TBWF@JNBHF
GJMFOBNFNJYFE@JNBHF
[ 141 ]
Convolutional Neural Networks Chapter 4
22. In this image, we show how the content image has been repainted after 200, 400,
and 600 iterations:
#PGZCORNGQHUV[NGVTCPUHGT
How it works...
In this recipe, we have seen how to use style transfer to repaint a content image. A style
image has been provided as input to a neural network, which learned the key aspects
defining the style adopted by the painter. These aspects have been used to transfer the style
to the content image.
There's more...
Style transfer has been an area of active research since the original proposal in 2015. Many
new ideas have been proposed to accelerate the computation and extend the style transfer
to video analysis. Among many, two results are worth mentioning
[ 142 ]
Convolutional Neural Networks Chapter 4
The website, IUUQTEFFQBSUJP, allows you to play with your own images and repaint
your picture in the style of your favorite artist. An android app, an iPhone app, and a web
app are also available.
Following this line of thinking, computer vision researchers now commonly use pretrained
CNNs to generate representations for novel tasks, where the dataset may not be large
enough to train an entire CNN from scratch. Another common tactic is to take the
pretrained ImageNet network and then to fine-tune the entire network to the novel task.
The example proposed here has been inspired by Francois Chollet in a famous blog posting
for Keras. (IUUQTCMPHLFSBTJPCVJMEJOHQPXFSGVMJNBHFDMBTTJGJDBUJPONPEFMT
VTJOHWFSZMJUUMFEBUBIUNM)
Getting ready
The idea is to use the VGG16 network pretrained on a large dataset such as ImageNet. Note
that training could be rather computationally expensive and therefore makes sense to reuse
an already pretrained network:
#8))0GVYQTM
So, how to use a VGG16? Keras makes it easy as the library has a standard VGG16
application available as a library and precomputed weights are automatically downloaded.
Note that we explicitly omit the last layer and replace it with our custom layer, which will
be fine-tuned on the top of the prebuilt VGG16. In this example, you will learn how to
classify dogs and cats images available from Kaggle.
[ 143 ]
Convolutional Neural Networks Chapter 4
How to do it...
We proceed with the recipe as follows:
GSPNLFSBTJNQPSUBQQMJDBUJPOT
GSPNLFSBTQSFQSPDFTTJOHJNBHFJNQPSU*NBHF%BUB(FOFSBUPS
GSPNLFSBTJNQPSUPQUJNJ[FST
GSPNLFSBTNPEFMTJNQPSU4FRVFOUJBM.PEFM
GSPNLFSBTMBZFSTJNQPSU%SPQPVU'MBUUFO%FOTF
GSPNLFSBTJNQPSUPQUJNJ[FST
JNH@XJEUIJNH@IFJHIU
CBUDI@TJ[F
FQPDIT
USBJO@EBUB@EJS EBUBEPHT@BOE@DBUTUSBJO
WBMJEBUJPO@EBUB@EJS EBUBEPHT@BOE@DBUTWBMJEBUJPO
065$"5&(03*&4
065@$"5&(03*&4
OVNCFSPGUSBJOWBMJEBUJPOTBNQMFT
OC@USBJO@TBNQMFT
OC@WBMJEBUJPO@TBNQMFT
3. Load the pretrained on the ImageNet VGG16 network and omit the last layer
because we will add a custom classification network on the top of the prebuilt
VGG16 and replace the last classification layer:
MPBEUIF7((NPEFMQSFUSBJOFEPOJNBHFOFU
CBTF@NPEFMBQQMJDBUJPOT7((
XFJHIUTJNBHFOFU
JODMVEF@UPQ'BMTFJOQVU@TIBQF
JNH@XJEUIJNH@IFJHIU
CBTF@NPEFMTVNNBSZ
[ 144 ]
Convolutional Neural Networks Chapter 4
4. Freeze a certain number of lower layers for the pretrained VGG16 network. In
this case, we decide to freeze the initial 15 layers:
'SFF[FUIFMPXFSMBZFSTGPSMBZFSJOCBTF@NPEFMMBZFST<>
MBZFSUSBJOBCMF'BMTF
"EEDVTUPNUPMBZFSTCVJMEBDMBTTJGJFSNPEFMUPQVUPOUPQPG
UIFDPOWPMVUJPOBMNPEFMUPQ@NPEFM4FRVFOUJBM
UPQ@NPEFMBEE
'MBUUFO
JOQVU@TIBQFCBTF@NPEFMPVUQVU@TIBQF<>
[ 145 ]
Convolutional Neural Networks Chapter 4
UPQ@NPEFMBEE
%FOTF
BDUJWBUJPO SFMV
UPQ@NPEFMBEE
%SPQPVU
UPQ@NPEFMBEE
%FOTF
065@$"5&(03*&4
BDUJWBUJPO TJHNPJE
6. The custom network should be pretrained separately and here, we omit this part
for the sake of simplicity, leaving this task to the reader:
UPQ@NPEFMMPBE@XFJHIUT
UPQ@NPEFM@XFJHIUT@QBUI
7. Create a new network that is the juxtaposition of the pretrained VGG16 network
and our pretrained custom network:
DSFBUJOHUIFGJOBMNPEFMBDPNQPTJUJPOPG
QSFUSBJOFEBOE
NPEFM.PEFM
JOQVUTCBTF@NPEFMJOQVU
PVUQVUTUPQ@NPEFM
CBTF@NPEFMPVUQVU
DPNQJMFUIFNPEFM
NPEFMDPNQJMF
MPTTCJOBSZ@DSPTTFOUSPQZPQUJNJ[FS
PQUJNJ[FST4(%
MSNPNFOUVNNFUSJDT<BDDVSBDZ>
8. Retrain the juxtaposed new model still keeping the 15 lowest layers of VGG16
frozen. In this particular example, we also use an Image Augumentator to
increase our training set:
*OJUJBUFUIFUSBJOBOEUFTUHFOFSBUPSTXJUIEBUB"VHVNFOUBUJPO
USBJO@EBUBHFO*NBHF%BUB(FOFSBUPS
SFTDBMF
IPSJ[POUBM@GMJQ5SVF
UFTU@EBUBHFO*NBHF%BUB(FOFSBUPS
SFTDBMF
USBJO@HFOFSBUPSUSBJO@EBUBHFOGMPX@GSPN@EJSFDUPSZ
USBJO@EBUB@EJS
UBSHFU@TJ[F
JNH@IFJHIUJNH@XJEUI
CBUDI@TJ[FCBUDI@TJ[F
DMBTT@NPEF CJOBSZ
WBMJEBUJPO@HFOFSBUPSUFTU@EBUBHFOGMPX@GSPN@EJSFDUPSZ
WBMJEBUJPO@EBUB@EJS
UBSHFU@TJ[F
JNH@IFJHIUJNH@XJEUI
CBUDI@TJ[FCBUDI@TJ[F
DMBTT@NPEF CJOBSZ TIVGGMF'BMTF
NPEFMGJU@HFOFSBUPS
USBJO@HFOFSBUPS
TUFQT@QFS@FQPDIOC@USBJO@TBNQMFTCBUDI@TJ[F
FQPDITFQPDIT
WBMJEBUJPO@EBUBWBMJEBUJPO@HFOFSBUPS
WBMJEBUJPO@TUFQTOC@WBMJEBUJPO@TBNQMFTCBUDI@TJ[F
WFSCPTFXPSLFST
[ 146 ]
Convolutional Neural Networks Chapter 4
TDPSFNPEFMFWBMVBUF@HFOFSBUPS
WBMJEBUJPO@HFOFSBUPS
OC@WBMJEBUJPO@TBNQMFTCBUDI@TJ[F
TDPSFTNPEFMQSFEJDU@HFOFSBUPS
WBMJEBUJPO@HFOFSBUPS
OC@WBMJEBUJPO@TBNQMFTCBUDI@TJ[F
How it works...
A standard VGG16 network has been pretrained on the whole ImageNet with precomputed
weights downloaded from the Internet. This network has been then juxtaposed with a
custom network that has been also trained separately. Then, the juxtaposed network has
been retrained as a whole, keeping 15 lower layers of the VGG16 frozen.
This combination is very effective. It allows us to save significant computational power and
reuse the work already performed for VGG16 by performing a transfer learning of what the
network has learned on ImageNet, apply this learning to our new specific domain, and
therefore perform a fine-tuning classification task.
There's more...
There are a few rules of thumb to consider according to the specific classification task:
If the new dataset is small and similar to the ImageNet dataset, then we can
freeze all the VGG16 networks and retrain only the custom network. In this way,
we also minimize the risk of overfitting for the juxtaposed network:
If the new dataset is large and similar to the ImageNet dataset, then we can
retrain the whole juxtaposed network. We have still maintained the precomputed
weights as a starting point and perform a few iterations for the fine-tuning:
[ 147 ]
Convolutional Neural Networks Chapter 4
If the new dataset is very different from the ImageNet dataset, in practice it might
be still good to initialize with weights from a pretrained model. In this case, we
would have enough data and confidence to fine-tune through the entire network.
More information can be found online at IUUQDTOHJUIVCJPUSBOTGFS
MFBSOJOH.
#PGZCORNGQH)QQING&GGR&TGCOUCUUGGPKPIUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMF"VSFMJBBVSJUB
KQH
[ 148 ]
Convolutional Neural Networks Chapter 4
Getting ready
Download the pretrained Inception model from online (IUUQTHJUIVCDPNNBSUJOXJDLF
UFOTPSGMPXUVUPSJBMCMPCNBTUFSUFOTPSGMPX@JODFQUJPO@HSBQIQC).
How to do it...
We proceed with the recipes as follows:
JNQPSUOVNQZBTOQGSPNGVODUPPMT
JNQPSUQBSUJBMJNQPSU1*-*NBHF
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMU
2. Set up the path for the content image and the pretrained model. Start with a seed
image that is just random noise:
DPOUFOU@JNBHF EBUBHVMMJKQH
TUBSUXJUIBHSBZJNBHFXJUIBMJUUMFOPJTF
JNH@OPJTFOQSBOEPNVOJGPSN
TJ[F
NPEFM@GO EBUBUFOTPSGMPX@JODFQUJPO@HSBQIQC
3. Load the Inception network downloaded from the internet in a graph. Initialize a
TensorFlow session, load the graph with 'BTU('JMF
, and parse the graph
with 1BSTF'SPNTUSJOH
. After that, create an input as placeholder with the
QMBDFIPMEFS
method. The JNBHFOFU@NFBO is a precomputed constant that
will be removed from our content image to normalize the data. In fact, this is the
mean value observed during training and the normalization allows faster
convergence. The value will be subtracted from the input and stored in a
U@QSFQSPDFTTFE variable, which is then used to load the graph definition:
MPBEUIFHSBQI
HSBQIUG(SBQI
TFTTUG*OUFSBDUJWF4FTTJPO
HSBQIHSBQI
XJUIUGHGJMF'BTU('JMF
NPEFM@GO SC BTG
HSBQI@EFGUG(SBQI%FG
HSBQI@EFG1BSTF'SPN4USJOH
GSFBE
U@JOQVUUGQMBDFIPMEFS
OQGMPBUOBNF JOQVU EFGJOF
[ 149 ]
Convolutional Neural Networks Chapter 4
UIFJOQVUUFOTPS
JNBHFOFU@NFBO
U@QSFQSPDFTTFEUGFYQBOE@EJNT
U@JOQVUJNBHFOFU@NFBO
UGJNQPSU@HSBQI@EFG
HSBQI@EFG\ JOQVU U@QSFQSPDFTTFE^
4. Define some VUJM functions to visualize the image and transform the TF-graph
generating function into regular Python functions (see the following example to
resize):
IFMQFS
QZMJOUEJTBCMFVOVTFEWBSJBCMF
EFGTIPXBSSBZ
B
BOQVJOU
OQDMJQ
B
QMUJNTIPX
B
QMUTIPX
EFGWJTTUE
BT
/PSNBMJ[FUIFJNBHFSBOHFGPSWJTVBMJ[BUJPO
SFUVSO
BBNFBO
NBY
BTUE
FT
EFG5
MBZFS
)FMQFSGPSHFUUJOHMBZFSPVUQVUUFOTPS
SFUVSOHSBQIHFU@UFOTPS@CZ@OBNF
JNQPSUTMBZFS
EFGUGGVOD
BSHUZQFT
)FMQFSUIBUUSBOTGPSNT5'HSBQIHFOFSBUJOHGVODUJPOJOUPBSFHVMBS
POF
4FFSFTJ[FGVODUJPOCFMPX
QMBDFIPMEFSTMJTU
NBQ
UGQMBDFIPMEFSBSHUZQFT
EFGXSBQ
G
PVUG
QMBDFIPMEFST
EFGXSBQQFS
BSHTLX
SFUVSOPVUFWBM
EJDU
[JQ
QMBDFIPMEFSTBSHT
TFTTJPOLXHFU
TFTTJPO
SFUVSOXSBQQFS
SFUVSOXSBQ
EFGSFTJ[F
JNHTJ[F
JNHUGFYQBOE@EJNT
JNH
SFUVSOUGJNBHFSFTJ[F@CJMJOFBS
JNHTJ[F<>
SFTJ[FUGGVOD
OQGMPBUOQJOU
SFTJ[F
[ 150 ]
Convolutional Neural Networks Chapter 4
5. Compute the gradient ascent over the image. To increase the efficiency, apply a
tiled computation in which separate gradient ascents are computed on different
tiles. Random shifts are applied to the image to blur tile boundaries over multiple
iterations:
EFGDBMD@HSBE@UJMFE
JNHU@HSBEUJMF@TJ[F
$PNQVUFUIFWBMVFPGUFOTPSU@HSBEPWFSUIFJNBHFJOBUJMFEXBZ
3BOEPNTIJGUTBSFBQQMJFEUPUIFJNBHFUPCMVSUJMFCPVOEBSJFTPWFS
NVMUJQMFJUFSBUJPOT
T[UJMF@TJ[F
IXJNHTIBQF<>
TYTZOQSBOEPNSBOEJOU
T[TJ[F
JNH@TIJGUOQSPMM
OQSPMM
JNHTYTZ
HSBEOQ[FSPT@MJLF
JNH
GPSZJOSBOHF
NBY
IT[T[T[
GPSYJOSBOHF
NBY
XT[T[T[
TVCJNH@TIJGU<ZZ
T[YY
T[>
HTFTTSVO
U@HSBE\U@JOQVUTVC^
HSBE<ZZ
T[YY
T[>H
SFUVSOOQSPMM
OQSPMM
HSBETYTZ
6. Define the optimization object to reduce the mean of our input layer. The
HSBEJFOU function allows us to compute the symbolic gradient of our optimized
tensor by considering the input tensor. For efficiency, the image is split in a
number of octaves, which are then resized and added to an array of octaves.
Then, for each octave, we use the DBMD@HSBE@UJMFE function:
EFGSFOEFS@EFFQESFBN
U@PCKJNHJNH@OPJTF
JUFS@OTUFQPDUBWF@OPDUBWF@TDBMF
U@TDPSFUGSFEVDF@NFBO
U@PCKEFGJOJOHUIFPQUJNJ[BUJPOPCKFDUJWF
U@HSBEUGHSBEJFOUT
U@TDPSFU@JOQVU<>CFIPMEUIFQPXFSPGBVUPNBUJD
EJGGFSFOUJBUJPO
TQMJUUIFJNBHFJOUPBOVNCFSPGPDUBWFT
JNHJNH
PDUBWFT<>
GPS@JOSBOHF
PDUBWF@O
IXJNHTIBQF<>
MPSFTJ[F
JNH
OQJOU
OQGMPBU
IXPDUBWF@TDBMF
IJJNHSFTJ[F
MPIX
JNHMP
PDUBWFTBQQFOE
IJ
HFOFSBUFEFUBJMTPDUBWFCZPDUBWF
GPSPDUBWFJOSBOHF
PDUBWF@O
JGPDUBWF
[ 151 ]
Convolutional Neural Networks Chapter 4
IJPDUBWFT<PDUBWF>
JNHSFTJ[F
JNHIJTIBQF<>
IJ
GPS@JOSBOHF
JUFS@O
HDBMD@HSBE@UJMFE
JNHU@HSBE
JNH
H
TUFQ
OQBCT
HNFBO
F
UIJTXJMMVTVBMMZCFMJLFPSPDUBWFT
4UFQPVUQVUEFFQESFBNJNBHFWJBNBUQMPUMJC
TIPXBSSBZ
JNH
7. Load a specific content image and start dreaming. In this example, the face of the
author has been transformed into something resembling a wolf:
#PGZCORNGQH&GGR&TGCOVTCPUHQTOCVKQP1PGQHVJGCWVJQTUVTCPUHQTOGFKPVQCYQNH
How it works...
Neural networks store abstractions of the training images: lower layers memorize features
such as lines and edges, while higher layers memorize more sophisticated images features
such as eyes, faces, and noses. By applying a gradient ascent process, we maximize the
MPTT function and facilitate the discovery of a content image of patterns similar to the ones
memorized by higher layers. This results into a dreaming where the network sees trippy
images.
[ 152 ]
Convolutional Neural Networks Chapter 4
There's more...
Many websites allow you to play directly with DeepDreaming. Among others, I really like
%FFQ"SUJP (IUUQTEFFQBSUJP), which allows you to upload a content image and a
style image and perform the learning on the cloud.
See also
After the initial results presented in 2015, many new additional papers and blog postings
were presented about DeepDreaming:
[ 153 ]
Advanced Convolutional Neural
5
Networks
In this chapter, we will discuss how Convolutional Neural Networks (CNNs) are used for
deep learning in domains other than images. Our attention will be initially devoted to text
analysis and natural language processing (NLP). In this chapter, we will present a number
of recipes for:
Introduction
In the previous chapter, we saw how to apply ConvNets to images. During this chapter, we
will apply similar ideas to texts.
Advanced Convolutional Neural Networks Chapter 5
What do a text and an image have in common? At first glance, very little. However, if we
represent sentences or documents as a matrix then this matrix is not different from an image
matrix where each cell is a pixel. So, the next question is, how can we represent a text as a
matrix? Well, it is pretty simple: each row of a matrix is a vector which represents a basic
unit of the text. Of course, now we need to define what a basic unit is. A simple choice
could be to say that the basic unit is a character. Another choice would be to say that a basic
unit is a word, yet another choice is to aggregate similar words together and then denote
each aggregation (sometimes called cluster or embedding) with a representative symbol.
Note that regardless of the specific choice adopted for our basic units, we
need to have a 1:1 map from basic units into integer IDs so that a text can
be seen as a matrix. For instance, if we have a document with 10 lines of
text and each line is a 100-dimensional embedding, then we will represent
our text with a matrix 10 x 100. In this very particular image, a pixel is
turned on if that sentence x contains the embedding represented by
position y. You might also notice that a text is not really a matrix but more
a vector because two words located in adjacent rows of text have very little
in common. Indeed, there is a major difference with images where two
pixels located in adjacent columns most likely have some correlation.
Now you might wonder: I understand that you represent the text as a vector but, in doing
so, we lose the position of the words and this position should be important, shouldn't it?
Well, it turns out that in many real applications knowing whether a sentence contains a
particular basic unit (a char, a word, or an aggregate) or not is pretty accurate information,
even if we don't memorize where exactly in the sentence this basic unit is located.
Getting ready
TFLearn has libraries for automatically downloading the dataset from the network and for
facilitating the creation of the ConvNet, so let's go directly to the code.
[ 155 ]
Advanced Convolutional Neural Networks Chapter 5
How to do it...
We proceed with the recipe as follows:
1. Import TensorFlow, UGMFBSO, and the modules needed for building our network.
Then, Import IMDb libraries and perform one-hot encoding and padding:
JNQPSUUFOTPSGMPXBTUG
JNQPSUUGMFBSO
GSPNUGMFBSOMBZFSTDPSFJNQPSUJOQVU@EBUBESPQPVUGVMMZ@DPOOFDUFE
GSPNUGMFBSOMBZFSTDPOWJNQPSUDPOW@EHMPCBM@NBY@QPPM
GSPNUGMFBSOMBZFSTNFSHF@PQTJNQPSUNFSHF
GSPNUGMFBSOMBZFSTFTUJNBUPSJNQPSUSFHSFTTJPO
GSPNUGMFBSOEBUB@VUJMTJNQPSUUP@DBUFHPSJDBMQBE@TFRVFODFT
GSPNUGMFBSOEBUBTFUTJNQPSUJNEC
2. Load the dataset, pad the sentences to maxlength with 0s, and perform one-hot
encoding on the labels with two values corresponding to the true and one false
value. Note that the parameter O@XPSET is the number of words to keep in the
vocabulary. All extra words are set to unknown. Also, note that trainX and trainY
are sparse vectors because each review will most likely contain a subset of the
whole set of words:
*.%C%BUBTFUMPBEJOH
USBJOUFTU@JNECMPBE@EBUB
QBUI JNECQLM O@XPSET
WBMJE@QPSUJPO
USBJO9USBJO:USBJO
UFTU9UFTU:UFTU
QBEUIFTFRVFODF
USBJO9QBE@TFRVFODFT
USBJO9NBYMFOWBMVF
UFTU9QBE@TFRVFODFT
UFTU9NBYMFOWBMVF
POFIPUFODPEJOH
USBJO:UP@DBUFHPSJDBM
USBJO:OC@DMBTTFT
UFTU:UP@DBUFHPSJDBM
UFTU:OC@DMBTTFT
3. Print a few dimensions to inspect the just-processed data and understand what
the dimension of the problem is:
QSJOU
TJ[FUSBJO9USBJO9TJ[F
QSJOU
TJ[FUFTU9UFTU9TJ[F
QSJOU
TJ[FUFTU:UFTU:TJ[F
QSJOU
TJ[FUSBJO:USBJO:TJ[F
TJ[FUSBJO9
TJ[FUFTU9
TJ[FUFTU:
TJUFUSBJO:
[ 156 ]
Advanced Convolutional Neural Networks Chapter 5
4. Build an embedding for the text contained in the dataset. Just for now, consider
this step as a black box which takes the words and maps them into aggregates
(clusters) so that similar words are likely to appear in the same cluster. Note that
the vocabulary for the previous steps is discrete and sparse. With the embedding,
we will create a map which will embed each word into a continuous dense vector
space. Using this vector space representation will give us a continuous,
distributed representation of our vocabulary words. How to build embeddings is
something that will be discussed in detail when we talk about RNNs:
#VJMEBOFNCFEEJOH
OFUXPSLJOQVU@EBUB
TIBQF</POF>OBNF JOQVU
OFUXPSLUGMFBSOFNCFEEJOH
OFUXPSLJOQVU@EJNPVUQVU@EJN
#VJMEUIFDPOWOFU
CSBODIDPOW@E
OFUXPSLQBEEJOH WBMJE BDUJWBUJPO SFMV
SFHVMBSJ[FS-
CSBODIDPOW@E
OFUXPSLQBEEJOH WBMJE BDUJWBUJPO SFMV
SFHVMBSJ[FS-
CSBODIDPOW@E
OFUXPSLQBEEJOH WBMJE BDUJWBUJPO SFMV
SFHVMBSJ[FS-
OFUXPSLNFSHF
<CSBODICSBODICSBODI>NPEF DPODBU BYJT
OFUXPSLUGFYQBOE@EJNT
OFUXPSL
OFUXPSLHMPCBM@NBY@QPPM
OFUXPSL
OFUXPSLESPQPVU
OFUXPSL
OFUXPSLGVMMZ@DPOOFDUFE
OFUXPSLBDUJWBUJPO TPGUNBY
OFUXPSLSFHSFTTJPO
OFUXPSLPQUJNJ[FS BEBN MFBSOJOH@SBUF
MPTT DBUFHPSJDBM@DSPTTFOUSPQZ OBNF UBSHFU
7. Then we run the training with batch_size=32 and observe what is the accuracy
reached on the training and validation set. As you can see, we are able to get an
accuracy of 79% in predicting what is the sentiment expressed for movie reviews:
[ 157 ]
Advanced Convolutional Neural Networks Chapter 5
5SBJOJOH
NPEFMUGMFBSO%//
OFUXPSLUFOTPSCPBSE@WFSCPTF
NPEFMGJU
USBJO9USBJO:O@FQPDITIVGGMF5SVFWBMJEBUJPO@TFU
UFTU9
UFTU:TIPX@NFUSJD5SVFCBUDI@TJ[F
5SBJOJOH4UFQ]UPUBMMPTT0.09738]UJNFT
]"EBN]FQPDI]MPTTBDDJUFS
5SBJOJOH4UFQ]UPUBMMPTT0.09733]UJNFT
]"EBN]FQPDI]MPTTBDD]WBM@MPTT
WBM@BDDJUFS
How it works...
One-dimension ConvNets for sentiment analysis are discussed in detail in the paper
Convolutional Neural Networks for Sentence Classification, Yoon Kim, EMNLP 2014 (IUUQT
BSYJWPSHBCT) . Note that the model proposed by the paper retains some
pieces of information about the position, thanks to the filter windows operating over
consecutive words. The following image extracted from the paper graphically represents
the key intuitions beyond the network. At the beginning the text is represented as a vector
based on standard embeddings, providing us a compact representation in a one-
dimensional dense space. Then the matrix is processed with multiple standard one-
dimensional convolutional layers.
Note that the model uses multiple filters (with varying window sizes) to
obtain multiple features. After that, there is a max pool operation where
the idea is to capture the most important feature-the one with the highest
value for each feature map. For regularization, the article proposed to
adopt a dropout on the penultimate layer with a constraint on L2-norms of
the weight vectors. The final layer will output the sentiment as positive or
negative.
Filters are typically convolving on a continuous space. For images, this space is
the pixel matrix representation which is spatially continuous over height and
width. For texts, the continuous space is nothing more than the continuous
dimension naturally induced by continuous words. If we use only words
represented with one-hot encoding then space is sparse, if we use embedding
then the resulting space is dense because similar words are aggregated.
Images typically have three channels (RGB), while text naturally has only one
channel because we have no need to represent colors.
[ 158 ]
Advanced Convolutional Neural Networks Chapter 5
There is more...
The paper Convolutional Neural Networks for Sentence Classification, Yoon Kim, EMNLP 2014
(IUUQTBSYJWPSHBCT) performs an extensive set of experiments. Despite
little tuning of hyperparameters, a simple CNN with one layer of convolution performs
remarkably well for sentence classification. The paper shows that adopting a set of static
embedding - which will be discussed when we talk about RNNs - and building a very
simple ConvNet on the top of it, can actually improve the performance of sentiment
analysis significantly:
#PGZCORNGQHOQFGNCTEJKVGEVWTGCUUGGPKPIUUQTBSYJWPSHQEGQEG
The use of CNNs for text analysis is an active field of research. I suggest having a look at the
following article:
[ 159 ]
Advanced Convolutional Neural Networks Chapter 5
Getting ready
The first step is to select a specific category used for training the VGG16 on ImageNet. Let's
say that we take the category 20, which corresponds to the American Dipper bird shown in
the following picture:
#PGZCORNGQH#OGTKECP&KRRGTCUUGGPQPIUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMF"NFSJDBO@%JQQFSKQH
[ 160 ]
Advanced Convolutional Neural Networks Chapter 5
How to do it...
We proceed with the recipe as follows:
1. Import matplotlib and the modules used by keras-vis. In addition, also import the
pre-built VGG16 module. Keras makes it easy to deal with this pre-built network:
GSPNNBUQMPUMJCJNQPSUQZQMPUBTQMU
GSPNWJTVUJMTJNQPSUVUJMT
GSPNWJTVUJMTWHHOFUJNQPSU7((
GSPNWJTWJTVBMJ[BUJPOJNQPSUWJTVBMJ[F@DMBTT@BDUJWBUJPO
2. Access the VGG16 network by using the pre-built layers included in Keras and
trained with ImageNet weights:
#VJMEUIF7((OFUXPSLXJUI*NBHF/FUXFJHIUT
NPEFM7((
XFJHIUT JNBHFOFU JODMVEF@UPQ5SVF
NPEFMTVNNBSZ
QSJOU
.PEFMMPBEFE
3. This is how the VGG16 network looks internally. We have many ConvNets,
alternated with maxpool2D. Then, we have a Flatten layer followed by three
Dense layers. The last one is called predictions, and this layer should be able to
detect high-level features such as faces or, in our case, the shape of a bird. Note
that the top layer is explicitly included in our network because we want to
visualize what it learned:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
-BZFS
UZQF0VUQVU4IBQF1BSBN
JOQVU@
*OQVU-BZFS
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[ 161 ]
Advanced Convolutional Neural Networks Chapter 5
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GMBUUFO
'MBUUFO
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GD
%FOTF
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GD
%FOTF
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
QSFEJDUJPOT
%FOTF
/POF
5PUBMQBSBNT
5SBJOBCMFQBSBNT
/POUSBJOBCMFQBSBNT
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
.PEFMMPBEFE
#8))0GVYQTM
[ 162 ]
Advanced Convolutional Neural Networks Chapter 5
4. Now let's focus on inspecting how the last prediction layer looks internally by
focusing on American Dipper, which is the ID 20:
MBZFS@OBNF QSFEJDUJPOT
MBZFS@JEY<JEYGPSJEYMBZFSJOFOVNFSBUF
NPEFMMBZFSTJGMBZFSOBNF
MBZFS@OBNF><>
(FOFSBUFUISFFEJGGFSFOUJNBHFTPGUIFTBNFPVUQVUJOEFY
WJT@JNBHFT<>
GPSJEYJO<>
JNHWJTVBMJ[F@DMBTT@BDUJWBUJPO
NPEFMMBZFS@JEYGJMUFS@JOEJDFTJEY
NBY@JUFS
JNHVUJMTESBX@UFYU
JNHTUS
JEY
WJT@JNBHFTBQQFOE
JNH
5. Let's display the generated images for the specific layer given the features and
observe how the concept of the American Dipper bird is internally seen by the
network:
So, this is how a bird is internally represented by a neural network. It is a kind of trippy
image, but I swear that no particular kind of artificial drug has been given to the network
itself! This is just what this particular kind of artificial network has naturally learned.
6. Are you are still curious to understand a bit more? Well, let's pick an earlier layer
and represent how the network is internally seeing this same "NFSJDBO%JQQFS
training class:
MBZFS@OBNF CMPDL@DPOW
MBZFS@JEY<JEYGPSJEYMBZFSJOFOVNFSBUF
NPEFMMBZFSTJGMBZFSOBNF
MBZFS@OBNF><>
WJT@JNBHFT<>
GPSJEYJO<>
JNHWJTVBMJ[F@DMBTT@BDUJWBUJPO
NPEFMMBZFS@JEYGJMUFS@JOEJDFTJEY
NBY@JUFS
JNHVUJMTESBX@UFYU
JNHTUS
JEY
[ 163 ]
Advanced Convolutional Neural Networks Chapter 5
WJT@JNBHFTBQQFOE
JNH
TUJUDIFEVUJMTTUJUDI@JNBHFT
WJT@JNBHFT
QMUBYJT
PGG
QMUJNTIPX
TUJUDIFE
QMUUJUMF
MBZFS@OBNF
QMUTIPX
As expected, this particular layer is learning very basic features such as curves. However,
the true power of ConvNets is that the network infers more and more sophisticated features
the deeper we go into the models.
How it works...
The key idea of keras-vis visualization for Dense layers is to generate an input image that
maximizes the final Dense layer output corresponding to the bird class. So in reality what
the module does is reverse the problem. Given a specific trained Dense layer with its
weight, a new synthetic image is generated which best fits the layer itself.
A similar idea is used for each conv filter. In this case, note that the first ConvNet layer is
interpretable by simply visualizing its weights, as it is operating over raw pixels.
Subsequent conv filters operate over the outputs of previous conv filters, and therefore
visualizing them directly is not necessarily very interpretable. However, if we consider each
layer independently we can focus on generating only a synthetic input image which
maximizes the filter output.
[ 164 ]
Advanced Convolutional Neural Networks Chapter 5
There is more...
The keras-vis repository on GitHub (IUUQTHJUIVCDPNSBHIBLPULFSBTWJT) has a
good set of visualization examples of how to inspect a network internally, including the
very recent idea of attention maps, where the goal is to detect which part of an image
contributed the most to the training of a specific category (for example, tigers) when images
frequently contain other elements (for example, grass). The seed article is Deep Inside
Convolutional Networks: Visualising Image Classification Models and Saliency Maps, Karen
Simonyan, Andrea Vedaldi, Andrew Zisserman (IUUQTBSYJWPSHBCT), and
an example extracted from the Git repository is reported below where the network
understands by itself what the most salient part in an image to define a tiger is:
#PGZCORNGQHUCNKGPE[OCRUCUUGGPQPIUUQTHJUIVCDPNSBHIBLPULFSBTWJT
[ 165 ]
Advanced Convolutional Neural Networks Chapter 5
#PGZCORNGQHXGT[FGGRPGVYQTMEQPaIWTCVKQPUCUUGGPKPIUUQTBSYJWPSHQEGQEG
[ 166 ]
Advanced Convolutional Neural Networks Chapter 5
In 2015, having 16 or 19 layers was enough to consider the network deep, while today
(2017) we arrive at hundreds of layers. Note that VGG networks are very slow to train and
they require large weight space due to the depth and the number of fully-connected layers
at the end.
ResNet
ResNet has been introduced in Deep Residual Learning for Image Recognition, Kaiming He,
Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015, IUUQTBSYJWPSHBCT . This
network is very deep and can be trained using a standard stochastic descent gradient by
using a standard network component called the residual module, which is then used to
compose more complex networks (the composition is called network in network).
In comparison to VGG, ResNet is deeper, but the size of the model is smaller because a
global average pooling operation is used instead of full-dense layers.
[ 167 ]
Advanced Convolutional Neural Networks Chapter 5
Inception
Inception has been introduced in Rethinking the Inception Architecture for Computer Vision,
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna,
2015, IUUQTBSYJWPSHBCT and the key idea is to have multiple sizes of
convolutions acting as feature extraction and computing 1e1, 3e3, and 5e5 convolutions
within the same module. The outputs of these filters are then stacked along the channel
dimension and sent into the next layer in the network. This is described in the following
image:
Inception-v3 is described in Rethinking the Inception Architecture for Computer Vision, while
Inception-v4 is described in Inception-v4, Inception-ResNet and the Impact of Residual
Connections on Learning, Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alex Alemi,
2016, IUUQTBSYJWPSHBCT
[ 168 ]
Advanced Convolutional Neural Networks Chapter 5
Xception
Xception is an extension of Inception, introduced in Xception: Deep Learning with Depthwise
Separable Convolutions, Franfois Chollet, 2016, IUUQTBSYJWPSHBCT .
Xception uses a new concept called depthwise separable convolution operation which
allows it to outperform Inception-v3 on a large image classification dataset comprising 350
million images and 17,000 classes. Since the Xception architecture has the same number of
parameters as Inception-v3, the performance gains are not due to increased capacity but
rather to a more efficient use of model parameters.
Getting ready
This recipe uses Keras because the framework has a pre-cooked implementation of the
above modules. Keras automatically downloads the weights of each network the first time it
is used and stores these weights on your local disk. In other words, you don't need to
retrain the networks but can leverage training already available on the internet. This is true
under the assumption that you would like to classify your network on the 1,000 predefined
categories. In the next recipe, we will see how to start with these 1,000 categories and extend
them to a customized set with a process known as transfer learning.
How to do it...
We proceed with the recipe as follows:
1. Import the pre-built models and additional modules needed for processing and
showing images:
GSPNLFSBTBQQMJDBUJPOTJNQPSU3FT/FU
GSPNLFSBTBQQMJDBUJPOTJNQPSU*ODFQUJPO7
GSPNLFSBTBQQMJDBUJPOTJNQPSU9DFQUJPO5FOTPS'MPX0/-:
GSPNLFSBTBQQMJDBUJPOTJNQPSU7((
GSPNLFSBTBQQMJDBUJPOTJNQPSU7((
GSPNLFSBTBQQMJDBUJPOTJNQPSUJNBHFOFU@VUJMT
GSPNLFSBTBQQMJDBUJPOTJODFQUJPO@WJNQPSUQSFQSPDFTT@JOQVU
GSPNLFSBTQSFQSPDFTTJOHJNBHFJNQPSUJNH@UP@BSSBZ
GSPNLFSBTQSFQSPDFTTJOHJNBHFJNQPSUMPBE@JNH
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNNBUQMPUMJCQZQMPUJNQPSUJNTIPX
GSPN1*-JNQPSU*NBHF
NBUQMPUMJCJOMJOF
[ 169 ]
Advanced Convolutional Neural Networks Chapter 5
2. Define a map used for memorizing the size of images used for training the
networks. These are well-known constants for each model:
.0%&-4\
WHH
7((
WHH
7((
JODFQUJPO
*ODFQUJPO7
YDFQUJPO
9DFQUJPO
5FOTPS'MPX0/-:
SFTOFU
3FT/FU
^
3. Define an auxiliary function used for loading and converting each image. Note
that the pre-trained networks have been trained on a tensor with a shape that also
includes an additional dimension for CBUDI@TJ[F. Therefore, we need to add
this dimension to our image for compatibility:
EFGJNBHF@MPBE@BOE@DPOWFSU
JNBHF@QBUINPEFM
QJM@JN*NBHFPQFO
JNBHF@QBUI S
JNTIPX
OQBTBSSBZ
QJM@JN
JOJUJBMJ[FUIFJOQVUJNBHFTIBQF
BOEUIFQSFQSPDFTTJOHGVODUJPO
UIJTNJHIUOFFEUPCFDIBOHFE
JOQVU4IBQF.0%&-4<NPEFM><>
QSFQSPDFTTJNBHFOFU@VUJMTQSFQSPDFTT@JOQVU
JNBHFMPBE@JNH
JNBHF@QBUIUBSHFU@TJ[FJOQVU4IBQF
JNBHFJNH@UP@BSSBZ
JNBHF
UIFPSJHJOBMOFUXPSLTIBWFCFFOUSBJOFEPOBOBEEJUJPOBM
EJNFOTJPOUBLJOHJOUPBDDPVOUUIFCBUDITJ[F
XFOFFEUPBEEUIJTEJNFOTJPOGPSDPOTJTUFODZ
FWFOJGXFIBWFPOFJNBHFPOMZ
JNBHFOQFYQBOE@EJNT
JNBHFBYJT
JNBHFQSFQSPDFTT
JNBHF
SFUVSOJNBHF
4. Define an auxiliary function used for classifying the image and loop over the
predictions and display the rank-5 predictions along with the probabilities:
EFGDMBTTJGZ@JNBHF
JNBHF@QBUINPEFM
JNHJNBHF@MPBE@BOE@DPOWFSU
JNBHF@QBUINPEFM
/FUXPSL.0%&-4<NPEFM><>
NPEFM/FUXPSL
XFJHIUTJNBHFOFU
QSFETNPEFMQSFEJDU
JNH
1JNBHFOFU@VUJMTEFDPEF@QSFEJDUJPOT
QSFET
MPPQPWFSUIFQSFEJDUJPOTBOEEJTQMBZUIFSBOLQSFEJDUJPOT
BMPOHXJUIQSPCBCJMJUJFT
GPS
J
JNBHFOFU*%MBCFMQSPCJOFOVNFSBUF
1<>
QSJOU
\^\^\G^GPSNBU
J
MBCFMQSPC
[ 170 ]
Advanced Convolutional Neural Networks Chapter 5
#PGZCORNGQHOCECYCUUGGPKPIUUQTDPNNPOTXJLJNFEJBPSHXJLJ'JMF#MVFBOE:FMMPX.BDBX
KQH
DMBTTJGZ@JNBHF
JNBHFTQBSSPUKQHWHH
1. macaw: 99.77%
2. lorikeet: 0.07%
3. toucan: 0.06%
4. hornbill: 0.05%
5. jacamar: 0.01%
[ 171 ]
Advanced Convolutional Neural Networks Chapter 5
DMBTTJGZ@JNBHF
JNBHFTQBSSPUKQHSFTOFU
1. macaw: 97.93%
2. peacock: 0.86%
3. lorikeet: 0.23%
4. jacamar: 0.12%
5. jay: 0.12%
DMBTTJGZ@JNBHF
JNBHFTQBSSPU@DSPQQFEKQHSFTOFU
1. macaw: 99.98%
2. lorikeet: 0.00%
3. peacock: 0.00%
4. sulphur-crested_cockatoo: 0.00%
5. toucan: 0.00%
[ 172 ]
Advanced Convolutional Neural Networks Chapter 5
DMBTTJGZ@JNBHF
JNBHFTJODSFEJCMFIVMLKQHSFTOFU
1. comic_book: 99.76%
2. book_jacket: 0.19%
3. jigsaw_puzzle: 0.05%
4. menu: 0.00%
5. packet: 0.00%
#PGZCORNGQHEQOKEENCUUKaECVKQPCUUGGPKPIUUQTDPNJDWJOFHBNFTQPUDPNUIFJODSFEJCMFIVML
BOEUIFXJOEIPXMTXFOEJHP
DMBTTJGZ@JNBHF
JNBHFTDSPQQFE@QBOEBKQHSFTOFU
giant_panda: 99.04%
2. indri: 0.59%
3. lesser_panda: 0.17%
4. gibbon: 0.07%
5. titi: 0.05%
[ 173 ]
Advanced Convolutional Neural Networks Chapter 5
DMBTTJGZ@JNBHF
JNBHFTTQBDFTIVUUMFKQHSFTOFU
1. space_shuttle: 92.38%
2. triceratops: 7.15%
3. warplane: 0.11%
4. cowboy_hat: 0.10%
5. sombrero: 0.04%
DMBTTJGZ@JNBHF
JNBHFTTQBDFTIVUUMFKQHSFTOFU
1. space_shuttle: 99.96%
2. missile: 0.03%
3. projectile: 0.00%
4. steam_locomotive: 0.00%
5. warplane: 0.00%
[ 174 ]
Advanced Convolutional Neural Networks Chapter 5
DMBTTJGZ@JNBHF
JNBHFTTQBDFTIVUUMFKQHSFTOFU
1. space_shuttle: 93.21%
2. missile: 5.53%
3. projectile: 1.26%
4. mosque: 0.00%
5. beacon: 0.00%
DMBTTJGZ@JNBHF
JNBHFTTQBDFTIVUUMFKQHSFTOFU
1. space_shuttle: 49.61%
2. castle: 8.17%
3. crane: 6.46%
4. missile: 4.62%
5. aircraft_carrier: 4.24%
[ 175 ]
Advanced Convolutional Neural Networks Chapter 5
1. stopwatch: 100.00%
2. mink: 0.00%
3. hammer: 0.00%
4. black_grouse: 0.00%
5. web_site: 0.00%
DMBTTJGZ@JNBHF
JNBHFTQBSSPUKQHYDFQUJPO
1. backpack: 56.69%
2. military_uniform: 29.79%
3. bib: 8.02%
4. purse: 2.14%
5. ping-pong_ball: 1.52%
[ 176 ]
Advanced Convolutional Neural Networks Chapter 5
6. Define an auxiliary function used for showing the internal architecture of each
pre-built and pre-trained network:
EFGQSJOU@NPEFM
NPEFM
QSJOU
.PEFMNPEFM
/FUXPSL.0%&-4<NPEFM><>
NPEFM/FUXPSL
XFJHIUTJNBHFOFU
NPEFMTVNNBSZ
QSJOU@NPEFM
WHH
.PEFM WHH
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
-BZFS
UZQF0VUQVU4IBQF1BSBN
JOQVU@
*OQVU-BZFS
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[ 177 ]
Advanced Convolutional Neural Networks Chapter 5
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GMBUUFO
'MBUUFO
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GD
%FOTF
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GD
%FOTF
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
QSFEJDUJPOT
%FOTF
/POF
5PUBMQBSBNT
5SBJOBCMFQBSBNT
/POUSBJOBCMFQBSBNT
[ 178 ]
Advanced Convolutional Neural Networks Chapter 5
How it works...
We have used Keras Applications, pre-trained Keras learning models that are made
available alongside pre-trained weights. These models can be used for prediction, feature
extraction, and fine-tuning. In this case, we have used the model for prediction. We will see
in the next recipe how to use the model for fine-tuning and for building customized
classifiers on datasets that were originally not available when the models were originally
trained.
There is more...
Inception-v4 is not directly available in Keras as of July 2017, but it can be downloaded as a
separate module online (IUUQTHJUIVCDPNLFOUTPNNFSLFSBTJODFQUJPO7). Once
installed, the module will automatically download the weights the first time it is used.
AlexNet was one of the first stacked deep networks and it contained only eight layers, the
first five were convolutional layers followed by fully-connected layers. The network was
proposed in 2012 and significantly outperformed the second runner-up (top five error of
16% compared to runner-up with 26% error).
Recent research on deep neural networks has focused primarily on improving accuracy.
With equivalent accuracy, smaller DNN architectures offer at least three advantages:
[ 179 ]
Advanced Convolutional Neural Networks Chapter 5
Getting ready
One very simple idea is to use VGG16, and DCNN in general, for feature extraction. This
code implements the idea by extracting features from a specific layer.
How to do it...
We proceed with the recipe as follows:
1. Import the pre-built models and additional modules needed for processing and
showing images:
GSPNLFSBTBQQMJDBUJPOTWHHJNQPSU7((
GSPNLFSBTNPEFMTJNQPSU.PEFM
GSPNLFSBTQSFQSPDFTTJOHJNQPSUJNBHF
GSPNLFSBTBQQMJDBUJPOTWHHJNQPSUQSFQSPDFTT@JOQVU
JNQPSUOVNQZBTOQ
2. Select a specific layer from the network and get the features produced as output:
QSFCVJMUBOEQSFUSBJOFEEFFQMFBSOJOH7((NPEFM
CBTF@NPEFM7((
XFJHIUT JNBHFOFU JODMVEF@UPQ5SVF
GPSJMBZFSJOFOVNFSBUF
CBTF@NPEFMMBZFST
QSJOU
JMBZFSOBNFMBZFSPVUQVU@TIBQF
FYUSBDUGFBUVSFTGSPNCMPDL@QPPMCMPDL
NPEFM
.PEFM
JOQVUCBTF@NPEFMJOQVU
PVUQVUCBTF@NPEFMHFU@MBZFS
CMPDL@QPPM PVUQVU
[ 180 ]
Advanced Convolutional Neural Networks Chapter 5
3. Extract the features for a given image, as in the following code snippet:
JNH@QBUI DBUKQH
JNHJNBHFMPBE@JNH
JNH@QBUIUBSHFU@TJ[F
YJNBHFJNH@UP@BSSBZ
JNH
YOQFYQBOE@EJNT
YBYJT
YQSFQSPDFTT@JOQVU
Y
HFUUIFGFBUVSFTGSPNUIJTCMPDL
GFBUVSFTNPEFMQSFEJDU
Y
How it works...
Now, you might wonder why we want to extract the features from an intermediate layer in
a CNN. The key intuition is that: as the network learns to classify images into categories,
each layer learns to identify the features that are necessary to do the final classification.
Lower layers identify lower order features such as colors and edges, and higher layers
compose these lower order features into higher order features such as shapes or objects.
Hence the intermediate layer has the capability to extract important features from an image,
and these features are more likely to help with different kinds of classification.
This has multiple advantages. First, we can rely on publicly available large-scale training
and transfer this learning to novel domains. Second, we can save time for expensive large
training. Third, we can provide reasonable solutions even when we don't have a large
number of training examples for our domain. We also get a good starting network shape for
the task at hand, instead of guessing it.
Following this line of thinking, computer vision researchers now commonly use pre-trained
CNNs to generate representations for new tasks, where the dataset may not be large enough
to train an entire CNN from scratch. Another common tactic is to take the pre-trained
ImageNet network and then fine-tune the entire network to the novel task.
[ 181 ]
Advanced Convolutional Neural Networks Chapter 5
InceptionV3 Net is a very deep ConvNet developed by Google. Keras implements the full
network, shown in the following figure, and it comes pre-trained on ImageNet. The default
input size for this model is 299x299 on three channels:
#PGZCORNGQH+OCIG0GVX
Getting ready
This skeleton example is inspired by a scheme available online (IUUQTLFSBTJP
BQQMJDBUJPOT) on the Keras website. We suppose to have a training dataset D in a
domain, different from ImageNet. D has 1,024 features in input and 200 categories in
output.
How to do it...
We can proceed with the recipe as follows:
1. Import the pre-built models and additional modules needed for processing:
GSPNLFSBTBQQMJDBUJPOTJODFQUJPO@WJNQPSU*ODFQUJPO7
GSPNLFSBTQSFQSPDFTTJOHJNQPSUJNBHF
GSPNLFSBTNPEFMTJNQPSU.PEFM
GSPNLFSBTMBZFSTJNQPSU%FOTF(MPCBM"WFSBHF1PPMJOH%
GSPNLFSBTJNQPSUCBDLFOEBT,
DSFBUFUIFCBTFQSFUSBJOFENPEFM
CBTF@NPEFM*ODFQUJPO7
XFJHIUT JNBHFOFU JODMVEF@UPQ'BMTF
[ 182 ]
Advanced Convolutional Neural Networks Chapter 5
2. We use a trained Inception-v3 but we do not include the top model because we
want to fine-tune on D. The top level is a Dense layer with 1,024 inputs and the
last output level is a softmax Dense layer with 200 classes of output. Y
(MPCBM"WFSBHF1PPMJOH%
Y is used to convert the input to the correct
shape for the Dense layer to handle. In fact, CBTF@NPEFMPVUQVUUFOTPS has
the shape (samples, channels, rows, cols) for EJN@PSEFSJOHUI or (samples,
rows, cols, channels) for EJN@PSEFSJOHUG, but Dense needs them as
(samples, channels) (MPCBM"WFSBHF1PPMJOH% averages across (rows, cols). So
if you look at the last four layers (where JODMVEF@UPQ5SVF), you see these
shapes:
MBZFSOBNFMBZFSJOQVU@TIBQFMBZFSPVUQVU@TIBQF
NJYFE <
/POF
/POF
/POF
/POF>
/POF
BWH@QPPM
/POF
/POF
GMBUUFO
/POF
/POF
QSFEJDUJPOT
/POF
/POF
3. When you include @UPQ'BMTF, you are removing the last three layers and
exposing the NJYFE@ layer, so the (MPCBM"WFSBHF1PPMJOH% layer converts
the
/POF to
/POF, where each element in the
/POF
tensor is the average value for each corresponding (8,8) subtensor in the
/POF tensor:
BEEBHMPCBMTQBUJBMBWFSBHFQPPMJOHMBZFS
YCBTF@NPEFMPVUQVU
Y(MPCBM"WFSBHF1PPMJOH%
Y
MFU TBEEBGVMMZDPOOFDUFEMBZFSBTGJSTUMBZFS
Y%FOTF
BDUJWBUJPO SFMV
Y
BOEBMPHJTUJDMBZFSXJUIDMBTTFTBTMBTUMBZFS
QSFEJDUJPOT%FOTF
BDUJWBUJPO TPGUNBY
Y
NPEFMUPUSBJO
NPEFM.PEFM
JOQVUCBTF@NPEFMJOQVUPVUQVUQSFEJDUJPOT
4. All the convolutional levels are pre-trained so we freeze them during the training
of the full model.
JFGSFF[FBMMDPOWPMVUJPOBM*ODFQUJPOWMBZFST
GPSMBZFSJOCBTF@NPEFMMBZFST
MBZFSUSBJOBCMF'BMTF
[ 183 ]
Advanced Convolutional Neural Networks Chapter 5
5. The model is then compiled and trained for a few epochs so that the top layers
are trained:
DPNQJMFUIFNPEFM
TIPVMECFEPOFBGUFSTFUUJOHMBZFSTUPOPO
USBJOBCMF
NPEFMDPNQJMF
PQUJNJ[FS SNTQSPQ MPTT DBUFHPSJDBM@DSPTTFOUSPQZ
USBJOUIFNPEFMPOUIFOFXEBUBGPSBGFXFQPDIT
NPEFMGJU@HFOFSBUPS
6. Then we freeze the top layers in Inception and fine-tune the Inception layer. In
this example, we freeze the first 172 layers (a hyperparameter to tune):
XFDIPTFUPUSBJOUIFUPQJODFQUJPOCMPDLTJFXFXJMMGSFF[F
UIFGJSTUMBZFSTBOEVOGSFF[FUIFSFTU
GPSMBZFSJONPEFMMBZFST<>
MBZFSUSBJOBCMF'BMTF
GPSMBZFSJONPEFMMBZFST<>
MBZFSUSBJOBCMF5SVF
XFVTF4(%XJUIBMPXMFBSOJOHSBUF
GSPNLFSBTPQUJNJ[FSTJNQPSU4(%
NPEFMDPNQJMF
PQUJNJ[FS4(%
MSNPNFOUVN
MPTT DBUFHPSJDBM@DSPTTFOUSPQZ
XFUSBJOPVSNPEFMBHBJO
UIJTUJNFGJOFUVOJOHUIFUPQJODFQUJPO
CMPDLT
BMPOHTJEFUIFUPQ%FOTFMBZFST
NPEFMGJU@HFOFSBUPS
How it works...
Now we have a new deep network which re-uses the standard Inception-v3 network but it
is trained on a new domain D via transfer learning. Of course, there are many parameters to
fine-tune for achieving good accuracy. However, we are now re-using a very large pre-
trained network as a starting point via transfer learning. In doing so, we can save the need
for training on our machines by re-using what is already available in Keras.
[ 184 ]
Advanced Convolutional Neural Networks Chapter 5
There is more...
As of 2017, the problem of "computer vision" meant that the problem of finding patterns in
an image can be considered as solved, and this problem has an impact on our lives. For
instance:
The paper Dermatologist-level classification of skin cancer with deep neural networks,
Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter,
Helen M. Blau & Sebastian Thrun, 2017 IUUQTXXXOBUVSFDPNOBUVSF
KPVSOBMWOGVMMOBUVSFIUNM trains a CNN using a dataset of
129,450 clinical images consisting of 2,032 different diseases. They test the results
against 21 board-certified dermatologists on biopsy-proven clinical images with
two critical binary classification use cases: keratinocyte carcinomas versus benign
seborrheic keratoses; and malignant melanomas versus benign nevi. The CNN
achieves performance on par with all tested experts across both tasks,
demonstrating an artificial intelligence capable of classifying skin cancer with a
level of competence comparable to dermatologists.
The paper High-Resolution Breast Cancer Screening with Multi-View Deep
Convolutional Neural Networks, Krzysztof J. Geras, Stacey Wolfson, S. Gene Kim,
Linda Moy, Kyunghyun Cho, IUUQTBSYJWPSHBCT promises to
improve the breast cancer screening process through its innovative architecture,
which can handle the four standard views, or angles, without sacrificing a high
resolution. As opposed to the commonly used DCN architectures for natural
images, which work with images of 224 x 224 pixels, the MV-DCN is also capable
of using a resolution of 2600 x 2000 pixels.
[ 185 ]
Advanced Convolutional Neural Networks Chapter 5
So, you might wonder why learning to synthesize audio is so difficult. Well, each digital
sound we hear is based on 16,000 samples per second (sometimes 48,000 or more) and
building a predictive model where we learn to reproduce a sample based on all the
previous ones is a very difficult challenge. Nevertheless, there are experiments showing
that WaveNet has improved the current state-of-the-art Text-to-Speech (TTS) systems,
reducing the difference with human voices by 50 percent for both US English and Mandarin
Chinese.
What is even cooler is that DeepMind proved that WaveNet can be also used to teach to
computers how to generate the sound of musical instruments, such as piano music.
Now for some definitions. TTS systems are typically divided into two different classes:
Concatenative TTS, where single speech voice fragments are first memorized and
then recombined when the voice has to be reproduced. However, this approach
does not scale because it is possible to reproduce only the memorized voice
fragments and it is not possible to reproduce new speakers or different types of
audio without memorizing the fragments from the beginning.
Parametric TTS, where a model is created for storing all the characteristic features
of the audio to be synthesized. Before WaveNet, the audio generated with
parametric TTS was less natural than concatenative TTS. WaveNet improved the
state of the art by modeling directly the production of audio sounds, instead of
using intermediate signal processing algorithms used in the past.
In principle, WaveNet can be seen as a stack of 1D convolutional layers (we have seen 2D
convolution for images in Chapter 4) with a constant stride of one and with no pooling
layers. Note that the input and the output have by construction the same dimension, so the
ConvNets are well suited to model sequential data such as audio sounds. However, it has
been shown that in order to reach a large size for the receptive field in the output neuron it
is necessary to either use a massive number of large filters or prohibitively increase the
depth of the network. Remember that the receptive field of a neuron in a layer is the cross-
section of the previous layer from which neurons provide inputs. For this reason, pure
ConvNets are not so effective in learning how to synthesize audio.
The key intuition beyond WaveNet is the so-called dilated causal convolutions (sometimes
known as atrous convolutions), which simply means that some input values are skipped
when the filter of a convolutional layer is applied. Atrous is the bastardization of the French
expression c trous, meaning with holes. So an AtrousConvolution is a convolution with holes
As an example, in one dimension a filter w of size 3 with dilatation 1 would compute the
following sum.
[ 186 ]
Advanced Convolutional Neural Networks Chapter 5
So in short, in a D-dilated convolution, usually the stride is 1, but nothing prevents you
from using other strides. An example is given in the following figure with increased
dilatation (hole) sizes = 0, 1, 2:
#PGZCORNGQHFKNCVGFPGVYQTM
Thanks to this simple idea of introducing 'holes', it is possible to stack multiple dilated
convolutional layers with exponentially increasing filters and learn long range input
dependencies without having an excessively deep network.
A WaveNet is, therefore, a ConvNet where the convolutional layers have various dilation
factors allowing the receptive field to grow exponentially with depth, and therefore
efficiently cover thousands of audio timesteps.
When we train, the inputs are sounds recorded from human speakers. The waveforms are
quantized to a fixed integer range. A WaveNet defines an initial convolutional layer
accessing only the current and previous input. Then, there is a stack of dilated ConvNet
layers, still accessing only current and previous inputs. At the end, there is a series of Dense
layers which combines previous results, followed by a softmax activation function for
categorical outputs.
At each step, a value is predicted from the network and fed back into the input. At the same
time, a new prediction for the next step is computed. The loss function is the cross-entropy
between the output for the current step and the input at the next step.
[ 187 ]
Advanced Convolutional Neural Networks Chapter 5
#PGZCORNGQH05[PVJCTEJKVGEVWTGCUUGGPKPIUUQTNBHFOUBUFOTPSGMPXPSHOTZOUI
Getting ready
For this recipe, we are not going to write code, instead, I'll show you how to use some code
available online (IUUQTHJUIVCDPNUFOTPSGMPXNBHFOUBUSFFNBTUFSNBHFOUB
NPEFMTOTZOUI) and some cool demos you can find from Google Brain (IUUQT
BJFYQFSJNFOUTXJUIHPPHMFDPNTPVOENBLFS) . The interested reader can also have a look
the the paper Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders, Jesse Engel,
Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan,
Mohammad Norouzi, 5 Apr 2017. IUUQTBSYJWPSHBCT
How to do it...
We proceed with the recipe as follows:
DPOEBDSFBUFONBHFOUBQZUIPOKVQZUFS
TPVSDFBDUJWBUFNBHFOUB
[ 188 ]
Advanced Convolutional Neural Networks Chapter 5
2. Install the Magenta pip package and the librosa, which is used for reading audio
formats:
QJQJOTUBMMNBHFOUB
QJQJOTUBMMMJCSPTB
JNQPSUPT
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNNBHFOUBNPEFMTOTZOUIJNQPSUVUJMT
GSPNNBHFOUBNPEFMTOTZOUIXBWFOFUJNQPSUGBTUHFO
GSPN*1ZUIPOEJTQMBZJNQPSU"VEJP
NBUQMPUMJCJOMJOF
DPOGJH*OMJOF#BDLFOEGJHVSF@GPSNBU KQH
4. Then we load the demo sound downloaded from the internet and put it in the
same directory as the notebook. This will load 40,000 samples in about 2.5
seconds into the machine:
GSPNIUUQTXXXGSFFTPVOEPSHQFPQMF.VTUBSE1MVHTPVOET
GOBNF @@NVTUBSEQMVH@@CSFBLCFBUIJQIPQBCBSCQNXBW
TS
BVEJPVUJMTMPBE@BVEJP
GOBNFTBNQMF@MFOHUITSTS
TBNQMF@MFOHUIBVEJPTIBQF<>
QSJOU
\^TBNQMFT\^TFDPOET GPSNBU
TBNQMF@MFOHUITBNQMF@MFOHUI
GMPBU
TS
5. The next step is to encode the audio samples in a very compact representation
using a pre-trained NSynth model downloaded from the internet. This will give
us a 78 x 16 dimension encoding for every four seconds of audio, which we can
then decode or resynthesize. Our encoding is a tensor (GJMFTYY):
UJNFFODPEJOHGBTUHFOFODPEF
BVEJP NPEFMDLQU TBNQMF@MFOHUI
*/'0UFOTPSGMPX3FTUPSJOHQBSBNFUFSTGSPNNPEFMDLQU
$16UJNFTVTFSNJOTTZTTUPUBMNJOT
8BMMUJNFT
QSJOU
FODPEJOHTIBQF
[ 189 ]
Advanced Convolutional Neural Networks Chapter 5
6. Let's save the encoding that will be used later for re-synthesizing. In addition,
let's have a quick view of what the encoding shape is with a graphical
representation and compare it with the original audio signal. As you can see, the
encoding follows the beat presented in the original audio signal:
OQTBWF
GOBNF
OQZ FODPEJOH
GJHBYTQMUTVCQMPUT
GJHTJ[F
BYT<>QMPU
BVEJP
BYT<>TFU@UJUMF
"VEJP4JHOBM
BYT<>QMPU
FODPEJOH<>
BYT<>TFU@UJUMF
/4ZOUI&ODPEJOH
7. Now let's do a decoding of the encoding we just produced. In other words, we try
to reproduce the original audio from the compact representation with the intent
of understanding if the re-synthesized sound is similar to the original one.
Indeed, if you run the experiment and listen to the original audio and the re-
synthesized one, they sound very similar:
UJNFGBTUHFOTZOUIFTJ[F
FODPEJOHTBWF@QBUIT< HFO@
GOBNF>
TBNQMFT@QFS@TBWFTBNQMF@MFOHUI
[ 190 ]
Advanced Convolutional Neural Networks Chapter 5
How it works...
A WaveNet is a ConvNet where the convolutional layers have various dilation factors,
allowing the receptive field to grow exponentially with depth and therefore efficiently cover
thousands of audio timesteps. NSynth is an evolution of WaveNet where the original audio
is encoded using WaveNet-like processing for learning a compact representation. Then, this
compact representation is used to reproduce the original audio.
There is more...
As soon as we learn how to create a compact representation of audio via dilated
convolutions, we can play with these learnings and have fun. You will find very cool demos
on the internet:
1. For instance, you can see how the model learns the sound of different musical
instruments: (IUUQTNBHFOUBUFOTPSGMPXPSHOTZOUI)
[ 191 ]
Advanced Convolutional Neural Networks Chapter 5
2. Then, you can see how one model learned in one context can be re-mixed in
another context. For instance, by changing the speaker identity, we can use
WaveNet to say the same thing in different voices (IUUQTEFFQNJOEDPNCMPH
XBWFOFUHFOFSBUJWFNPEFMSBXBVEJP) .
3. Another very interesting experiment is to learn models for musical instruments
and then re-mix them in a such a way that we can create new musical
instruments that have never been heard before. This is really cool, and it opens
the path to a new range of possibilities that the ex-radio DJ sitting in me cannot
resist being super excited about. For instance, in this example, we combine a sitar
with an electric guitar, and this is a kind of cool new musical instrument. Not
excited enough? So what about combining a bowed bass with a dog's bark?
(IUUQTBJFYQFSJNFOUTXJUIHPPHMFDPNTPVOENBLFSWJFX) Have fun!:
[ 192 ]
Advanced Convolutional Neural Networks Chapter 5
For the sake of simplicity we will not re-train the combined network in 5 but, instead, will
use a pre-trained set of weights already available online (IUUQTBWJTJOHIHJUIVCJP
EFFQMFBSOJOHWJTVBMRB) . The interested reader can re-train the network on his own train
dataset made up of N images, N questions, and N answers. This is left as an optional
exercise. The network is inspired by the paper VQA: Visual Question Answering, Aishwarya
Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra,
Devi Parikh, 2015. (IUUQBSYJWPSHQEGWQEG) :
#PGZCORNGQH8KUWCN3 #CUUGGPKP7JTVBM2VFTUJPO"OTXFSJOHRCRGT
The only difference in our case is that we will concatenate the features produced by the
image layer with the features produced by the language layer.
[ 193 ]
Advanced Convolutional Neural Networks Chapter 5
How to do it...
We can proceed with the recipe as follows:
1. Load all the Keras modules needed by the recipe. These include spaCy for word
embedding, VGG16 for image features extraction, and LSTM for language
modeling. The remaining few additional modules are pretty standard:
NBUQMPUMJCJOMJOF
JNQPSUPTBSHQBSTF
JNQPSUOVNQZBTOQ
JNQPSUDWBTDW
JNQPSUTQBDZBTTQBDZ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNLFSBTNPEFMTJNQPSU.PEFM*OQVU
GSPNLFSBTMBZFSTDPSFJNQPSU%FOTF%SPQPVU3FTIBQF
GSPNLFSBTMBZFSTSFDVSSFOUJNQPSU-45.
GSPNLFSBTMBZFSTNFSHFJNQPSUDPODBUFOBUF
GSPNLFSBTBQQMJDBUJPOTWHHJNQPSU7((
GSPNLFSBTQSFQSPDFTTJOHJNQPSUJNBHF
GSPNLFSBTBQQMJDBUJPOTWHHJNQPSUQSFQSPDFTT@JOQVU
GSPNTLMFBSOFYUFSOBMTJNQPSUKPCMJC
JNQPSU1*-*NBHF
2. Define a few constants. Note that we assume that our corpus of questions has
NBY@MFOHUI@RVFTUJPOT and we know that we are going to use VGG16 to
extract 4,096 features describing our input image. In addition, we know that the
word embeddings are in a space of MFOHUI@GFBUVSF@TQBDF. Note that
we are going to use a set of pre-trained weights downloaded from the internet
(IUUQTHJUIVCDPNJBNBBEJUZB72"@%FNP):
NBQQJOHJE MBCFMTGPSDBUFHPSJFT
MBCFM@FODPEFS@GJMF@OBNF
6TFSTHVMMJ#PPLT5'DPEFHJUUFOTPSGMPX#PPL$IBQUFS'6--@MBCFMFODPEFS@U
SBJOWBMQLM
NBYMFOHUIBDSPTTDPSQVT
NBY@MFOHUI@RVFTUJPOT
7((PVUQVU
MFOHUI@WHH@GFBUVSFT
&NCFEEJOHPVUPVU
MFOHUI@GFBUVSF@TQBDF
QSFUSBJOFEXFJHIUT
72"@XFJHIUT@GJMF
6TFSTHVMMJ#PPLT5'DPEFHJUUFOTPSGMPX#PPL$IBQUFS72"@.0%&-@8&*()54I
EG
[ 194 ]
Advanced Convolutional Neural Networks Chapter 5
3. Use VGG16 for extracting features. Note that we explicitly extract them from the layer
fc2. This function returns 4,096 features given an input image:
JNBHFGFBUVSFT
EFGHFU@JNBHF@GFBUVSFT
JNH@QBUI7((NPEFM'VMM
HJWFOBOJNBHFSFUVSOTBUFOTPSXJUI
7((GFBUVSFT
4JODF7((XBTUSBJOFEBTBJNBHFPGYFWFSZOFXJNBHF
JTSFRVJSFEUPHPUISPVHIUIFTBNFUSBOTGPSNBUJPO
JNHJNBHFMPBE@JNH
JNH@QBUIUBSHFU@TJ[F
YJNBHFJNH@UP@BSSBZ
JNH
UIJTJTSFRVJSFECFDBVTFPGUIFPSJHJOBMUSBJOJOHPG7((XBTCBUDI
FWFOJGXFIBWFPOMZPOFJNBHFXFOFFEUPCFDPOTJTUFOU
YOQFYQBOE@EJNT
YBYJT
YQSFQSPDFTT@JOQVU
Y
GFBUVSFT7((NPEFM'VMMQSFEJDU
Y
NPEFM@FYUSBDUGFBUVSFT.PEFM
JOQVUT7((NPEFM'VMMJOQVU
PVUQVUT7((NPEFM'VMMHFU@MBZFS
GD PVUQVU
GD@GFBUVSFTNPEFM@FYUSBDUGFBUVSFTQSFEJDU
Y
GD@GFBUVSFTGD@GFBUVSFTSFTIBQF
MFOHUI@WHH@GFBUVSFT
SFUVSOGD@GFBUVSFT
[ 195 ]
Advanced Convolutional Neural Networks Chapter 5
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@DPOW
$POW%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
CMPDL@QPPM
.BY1PPMJOH%
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GMBUUFO
'MBUUFO
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GD
%FOTF
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
GD
%FOTF
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
QSFEJDUJPOT
%FOTF
/POF
5PUBMQBSBNT
5SBJOBCMFQBSBNT
/POUSBJOBCMFQBSBNT
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
4. Use spaCy to get a word embedding and map the input question into a space
(NBY@MFOHUI@RVFTUJPOT) where NBY@MFOHUI@RVFTUJPOT is the max
length of questions in our corpus, and 300 is the dimension of embeddings
produced by spaCy. Internally, spaCy uses an algorithm called gloVe (IUUQ
OMQTUBOGPSEFEVQSPKFDUTHMPWF). gloVe reduces a given token to a 300-
dimensional representation. Note that the question is padded to the
NBY@MFOHI@RVFTUJPOT with right 0 paddings:
FNCFEEJOH
EFGHFU@RVFTUJPO@GFBUVSFT
RVFTUJPO
HJWFOBRVFTUJPOBVOJDPEFTUSJOHSFUVSOTUIFUJNFTFSJFTWFDUPS
XJUIFBDIXPSE
UPLFOUSBOTGPSNFEJOUPBEJNFOTJPOSFQSFTFOUBUJPO
DBMDVMBUFEVTJOH(MPWF7FDUPS
XPSE@FNCFEEJOHTTQBDZMPBE
FO WFDUPST FO@HMPWF@DD@@N@WFDUPST
UPLFOTXPSE@FNCFEEJOHT
RVFTUJPO
OUPLFOTMFO
UPLFOT
JG
OUPLFOT NBY@MFOHUI@RVFTUJPOT
OUPLFOTNBY@MFOHUI@RVFTUJPOT
[ 196 ]
Advanced Convolutional Neural Networks Chapter 5
RVFTUJPO@UFOTPSOQ[FSPT
NBY@MFOHUI@RVFTUJPOT
GPSKJOYSBOHF
MFO
UPLFOT
RVFTUJPO@UFOTPS<K>UPLFOT<K>WFDUPS
SFUVSORVFTUJPO@UFOTPS
5. Load an image and get its salient features by using the previously defined image
feature extractor:
JNBHF@GJMF@OBNF HJSMKQH
JNH1*-*NBHFPQFO
JNBHF@GJMF@OBNF
JNHTIPX
HFUUIFTBMJFOUGFBUVSFT
NPEFM7((
XFJHIUT JNBHFOFU JODMVEF@UPQ5SVF
JNBHF@GFBUVSFTHFU@JNBHF@GFBUVSFT
JNBHF@GJMF@OBNFNPEFM
QSJOUJNBHF@GFBUVSFTTIBQF
6. Write a question and get its salient features by using the previously defined
sentence feature extractor:
RVFTUJPOV8IPJTJOUIJTQJDUVSF
MBOHVBHF@GFBUVSFTHFU@RVFTUJPO@GFBUVSFT
RVFTUJPO
QSJOUMBOHVBHF@GFBUVSFTTIBQF
7. Combine the two heterogeneous sets of features into one. In this network, we
have three LSTM layers which will take the creation of our language model into
account. Note that the LSTM will be discussed in detail in Chapter 6 and for now
we only use them as black boxes. The last LSTM returns 512 features which are
then used as inputs to a sequence of Dense and Dropout layers. The last layer is a
Dense one with a softmax activation function in a probability space of 1,000
potential answers:
DPNCJOF
EFGCVJME@DPNCJOFE@NPEFM
OVNCFS@PG@-45.
OVNCFS@PG@IJEEFO@VOJUT@-45.
OVNCFS@PG@EFOTF@MBZFST
OVNCFS@PG@IJEEFO@VOJUT
BDUJWBUJPO@GVODUJPO UBOI
ESPQPVU@QDU
JOQVUJNBHF
JOQVU@JNBHF*OQVU
TIBQF
MFOHUI@WHH@GFBUVSFT
OBNFJOQVU@JNBHF
NPEFM@JNBHF3FTIBQF
MFOHUI@WHH@GFBUVSFT
JOQVU@TIBQF
MFOHUI@WHH@GFBUVSFT
JOQVU@JNBHF
JOQVUMBOHVBHF
[ 197 ]
Advanced Convolutional Neural Networks Chapter 5
JOQVU@MBOHVBHF*OQVU
TIBQF
NBY@MFOHUI@RVFTUJPOTMFOHUI@GFBUVSF@TQBDF
OBNFJOQVU@MBOHVBHF
CVJMEBTFRVFODFPG-45.
NPEFM@MBOHVBHF-45.
OVNCFS@PG@IJEEFO@VOJUT@-45.
SFUVSO@TFRVFODFT5SVF
OBNFMTUN@
JOQVU@MBOHVBHF
NPEFM@MBOHVBHF-45.
OVNCFS@PG@IJEEFO@VOJUT@-45.
SFUVSO@TFRVFODFT5SVF
OBNFMTUN@
NPEFM@MBOHVBHF
NPEFM@MBOHVBHF-45.
OVNCFS@PG@IJEEFO@VOJUT@-45.
SFUVSO@TFRVFODFT'BMTF
OBNFMTUN@
NPEFM@MBOHVBHF
DPODBUFOBUF
NPEFMDPODBUFOBUF
<NPEFM@JNBHFNPEFM@MBOHVBHF>
%FOTF%SPQPVU
GPS@JOYSBOHF
OVNCFS@PG@EFOTF@MBZFST
NPEFM%FOTF
OVNCFS@PG@IJEEFO@VOJUT
LFSOFM@JOJUJBMJ[FS VOJGPSN
NPEFM
NPEFM%SPQPVU
ESPQPVU@QDU
NPEFM
NPEFM%FOTF
BDUJWBUJPO TPGUNBY
NPEFM
DSFBUFNPEFMGSPNUFOTPST
NPEFM.PEFM
JOQVUT<JOQVU@JNBHFJOQVU@MBOHVBHF>PVUQVUTNPEFM
SFUVSONPEFM
8. Build the combined network and show its summary just to understand how it
looks internally. Load the pre-trained weights and compile the model by using
the DBUFHPSJDBM@DSPTTFOUSPQZ loss function, with the rmsprop optimizer:
DPNCJOFE@NPEFMCVJME@DPNCJOFE@NPEFM
DPNCJOFE@NPEFMTVNNBSZ
DPNCJOFE@NPEFMMPBE@XFJHIUT
72"@XFJHIUT@GJMF
DPNCJOFE@NPEFMDPNQJMF
MPTT DBUFHPSJDBM@DSPTTFOUSPQZ
PQUJNJ[FS SNTQSPQ
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
-BZFS
UZQF0VUQVU4IBQF1BSBN$POOFDUFEUP
JOQVU@MBOHVBHF
*OQVU-BZFS
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
MTUN@
-45.
/POFJOQVU@MBOHVBHF<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
JOQVU@JNBHF
*OQVU-BZFS
/POF
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
[ 198 ]
Advanced Convolutional Neural Networks Chapter 5
MTUN@
-45.
/POFMTUN@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
SFTIBQF@
3FTIBQF
/POFJOQVU@JNBHF<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
MTUN@
-45.
/POFMTUN@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
DPODBUFOBUF@
$PODBUFOBUF
/POFSFTIBQF@<><>
MTUN@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
EFOTF@
%FOTF
/POFDPODBUFOBUF@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
ESPQPVU@
%SPQPVU
/POFEFOTF@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
EFOTF@
%FOTF
/POFESPQPVU@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
ESPQPVU@
%SPQPVU
/POFEFOTF@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
EFOTF@
%FOTF
/POFESPQPVU@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
ESPQPVU@
%SPQPVU
/POFEFOTF@<><>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@
EFOTF@
%FOTF
/POFESPQPVU@<><>
5PUBMQBSBNT
5SBJOBCMFQBSBNT
/POUSBJOBCMFQBSBNT
9. Use the pre-trained combined network for making the prediction. Note that in
this case we use weights already available online for this network, but the
interested reader can re-train the combined network on their own training set:
Z@PVUQVUDPNCJOFE@NPEFMQSFEJDU
<JNBHF@GFBUVSFTMBOHVBHF@GFBUVSFT>
5IJTUBTLIFSFJTSFQSFTFOUFEBTBDMBTTJGJDBUJPOJOUPBUPQBOTXFST
UIJTNFBOTTPNFPGUIFBOTXFSTXFSFOPUQBSUPGUSBJOJOHBOEUIVTXPVME
OPUTIPXVQJOUIFSFTVMU
5IFTFBOTXFSTBSFTUPSFEJOUIFTLMFBSO&ODPEFSDMBTT
[ 199 ]
Advanced Convolutional Neural Networks Chapter 5
MBCFMFODPEFSKPCMJCMPBE
MBCFM@FODPEFS@GJMF@OBNF
GPSMBCFMJOSFWFSTFE
OQBSHTPSU
Z@PVUQVU<>
QSJOUTUS
SPVOE
Z@PVUQVU<MBCFM>[GJMM
MBCFMFODPEFSJOWFSTF@USBOTGPSN
MBCFM
How it works...
The task of Visual Question Answering is tackled by using a combination of different deep
neural networks. A pre-trained VGG16 has been used to extract features from images, and a
sequence of LSTM has been used to extract features from questions previously mapped into
an embedding space. VGG16 is a CNN used for image feature extraction, while LSTM is an
RNN used for extracting temporal features representing the sequences. The combination of
these two is currently the state of the art for dealing with this type of network. A multi-
layered perceptron with dropout is then added on top of the combined models in order to
form our deep network.
There is more...
On the internet, you can find more experiments performed by Avi Singh (IUUQT
BWJTJOHIHJUIVCJPEFFQMFBSOJOHWJTVBMRB) where different models are
compared, including a simple 'bag-of-words' for language together with CNN for images,
an LSTM-only model, and an LSTM+CNN model - similar to the one discussed in this
recipe. The blog posting also discusses different training strategies for each model.
[ 200 ]
Advanced Convolutional Neural Networks Chapter 5
The first approach consists of classifying one video frame at a time by considering each of
them as a separate image processed with a 2D CNN. This approach simply reduces the
video classification problem to an image classification problem. Each video frame emits a
classification output, and the video is classified by taking into account the more frequently
chosen category for each frame.
The second approach consists of creating one single network where a 2D CNN is combined
with an RNN. The idea is that the CNN will take into account the image components and
the RNN will take into account the sequence information for each video. This type of
network can be very difficult to train because of the very high number of parameters to
optimize.
The fourth approach is based on smart intuition. Instead of using CNNs directly for
classification, they can be used for storing offline features for each frame in the video. The
idea is that feature extraction can be made very efficient with transfer learning, as shown in
a previous recipe. After all of the features are extracted, they can be passed as a set of inputs
into an RNN which will learn sequences across multiple frames and emit the final
classification.
The fifth approach is a simple variant of the fourth, where the final layer is an MLP instead
of an RNN. In certain situations, this approach can be simpler and less expensive in terms of
computational requirements.
The sixth approach is a variant of the fourth, where the phase of feature extraction is
realized with a 3D CNN which extracts spatial and visual features. These features are then
passed to either an RNN or an MLP.
Deciding on the best approach is strictly dependant on your specific application and there is
no definitive answer. The first three approaches are generally more computationally
expensive, while the last three approaches are less expensive and frequently achieve better
performance.
[ 201 ]
Advanced Convolutional Neural Networks Chapter 5
In this recipe, we show how to use the sixth approach by describing the results presented in
the paper Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks,
Montes, Alberto and Salvador, Amaia and Pascual, Santiago and Giro-i-Nieto, Xavier,
2016 (IUUQTBSYJWPSHBCT) . This work aims at solving the ActivityNet
Challenge IUUQBDUJWJUZOFUPSHDIBMMFOHFT. This challenge focuses on
recognizing high-level and goal-oriented activities from user-generated videos, similar to
those found in internet portals. The challenge is tailored to 200 activity categories in two
different tasks:
Untrimmed Classification Challenge: Given a long video, predict the labels of the
activities present in the video
Detection Challenge: Given a long video, predict the labels and temporal extents
of the activities present in the video
The architecture presented consists of two stages, as depicted in the following figure. The
first stage encodes the video information into a single vector representation for small video
clips. To achieve that, the C3D network is used. The C3D network uses 3D convolutions to
extract spatiotemporal features from the videos, which previously have been split into 16-
frame clips.
The second stage, once the video features are extracted, is that the activity on each clip is to
be classified. To perform this classification an RNN is used, more specifically an LSTM
network which tries to exploit long-term correlations and perform a prediction of the video
sequence. This stage is the one which has been trained:
#PGZCORNGQH%&400CUUGGPKPIUUQTJNBUHFVQDHJUIVCJPBDUJWJUZOFUDWQSX
[ 202 ]
Advanced Convolutional Neural Networks Chapter 5
How to do it...
For this recipe, we simply summarize the results presented online (IUUQTHJUIVCDPN
JNBUHFVQDBDUJWJUZOFUDWQSXCMPCNBTUFSNJTDTUFQ@CZ@TUFQ@HVJEFNE):
HJUDMPOFIUUQTHJUIVCDPNJNBUHFVQDBDUJWJUZOFUDWQSXHJU
2. Then we need to download the ActivityNet v1.3 dataset, which has a size of 600
GB:
DEEBUBTFU
5IJTXJMMEPXOMPBEUIFWJEFPTPOUIFEFGBVMUEJSFDUPSZ
TIEPXOMPBE@WJEFPTTIVTFSOBNFQBTTXPSE
5IJTXJMMEPXOMPBEUIFWJEFPTPOUIFEJSFDUPSZZPVTQFDJGZ
TIEPXOMPBE@WJEFPTTIVTFSOBNFQBTTXPSEQBUIZPVXBOU
3. The next step is to download the pre-trained weights for both the CNN3d and the
RNN:
DEEBUBNPEFMT
TIHFU@DE@TQPSUT@XFJHIUTTI
TIHFU@UFNQPSBM@MPDBUJPO@XFJHIUTTI
QZUIPOTDSJQUTSVO@BMM@QJQFMJOFQZJQBUIUPUFTUWJEFPNQ
How it works...
If you are interested in training the CNN3D and the RNN on your machines, then you can
find the specific commands used by this machine pipeline on the internet.
The intent is to present a high-level view of different approaches available for video
classification. Again, there is not only one single recipe, but instead multiple options, which
should be carefully chosen for your specific needs.
[ 203 ]
Advanced Convolutional Neural Networks Chapter 5
There is more...
CNN-LSTM architectures are a new RNN layer where the input of input transformations
and recurrent transformations are both convolutional. Despite the very similar name, CNN-
LSTM layers are therefore different from the combination of CNN and LSTM, as described
above. The model is described in the paper Convolutional LSTM Network: A Machine Learning
Approach for Precipitation Nowcasting, Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan
Yeung, Wai-kin Wong, Wang-chun Woo, 2015,(IUUQTBSYJWPSHBCT) and
in 2017 some people are starting to experiment using this module for video, but this is still
an area of active research.
[ 204 ]
Recurrent Neural Networks
6
*OUIJTDIBQUFSXFXJMMQSFTFOUBOVNCFSPGSFDJQFTDPWFSJOHUIFGPMMPXJOHUPQJDT
Introduction
In this chapter, we will discuss how Recurrent Neural Networks (RNNs) are used for deep
learning in domains where maintaining a sequential order is important. Our attention will
be mainly devoted to text analysis and natural language processing (NLP), but we will also
see examples of sequences used to predict the value of Bitcoins.
Recurrent Neural Networks Chapter 6
#PGZCORNGQHRTGFKEVKQPYKVJ6JGSWKEMDTQYPHQZUGPVGPEG
A simple variant is to store more than one prediction and therefore create a tree of possible
expansions as illustrated in the following figure:
#PGZCORNGQHVTGGRTGFKEVKQPYKVJ6JGSWKEMDTQYPHQZUGPVGPEG
[ 206 ]
Recurrent Neural Networks Chapter 6
However, a model based on sequences can be used in a very large number of additional
domains. In music, the next note in a composition will certainly depend on the previous
notes, and in a video, the next frame in a movie is certainly related to the previous frames.
Moreover, in certain situations, the current video frame, word, character, or musical note
will not only depend on the previous but also on the following ones.
A model based on a temporal sequence can be described with an RNN where for a given
input Xi at time i producing the output Yi, a memory of previous states at time [0, i-1] is fed
back to the network. This idea of feeding back previous states is depicted by a recurrent
loop, as shown in the following figures:
#PGZCORNGQHHGGFKPIDCEM
The recurrent relation can be conveniently expressed by unfolding the network, as illustrated
in the following figure:
#PGZCORNGQHWPHQNFKPICTGEWTTGPVEGNN
[ 207 ]
Recurrent Neural Networks Chapter 6
The simplest RNN cell consists of a simple tanh function, the hyperbolic tangent function, as
represented in the following figures:
#PGZCORNGQHUKORNGVCPJEGNN
'ZCORNGUQHITCFKGPV
[ 208 ]
Recurrent Neural Networks Chapter 6
Gradient clipping consists of imposing a maximum value to the the gradient so that it
cannot grow boundless. This simple solution, illustrated in the following figure, offers a
simple solution to the problem of exploding gradients:
#PGZCORNGQHITCFKGPVENKRRKPI
Solving the problem of a vanishing gradient requires a more complex model for memory
which allows it to selectively forget previous states and remember only the one that really
matters. Considering the following figure, the input is written in a memory M with a
probability p in [0,1], which is multiplied to weight the input.
In a similar way, the output is read with a probability p in [0,1], which is multiplied to
weight the output. One more probability is used to decide what to remember or forget:
#PGZCORNGQHOGOQT[EGNN
[ 209 ]
Recurrent Neural Networks Chapter 6
#PGZCORNGQHCP.56/EGNN
First, we need a logistic function g (see $IBQUFS, Regression) to compute a value between 0
and 1 and control which piece of information flows through the LSTM gates. Remember that
the logistic function is differentiable and therefore it allows backpropagation. Then, we
need an operator h that takes two matrices of the same dimensions and produces another
matrix where each element ij is the product of the elements ij of the original two matrices.
Similarly, we need an operator i that takes two matrices of the same dimensions and
produces another matrix where each element ij is the sum of the elements ij of the original
two matrices. With these basic blocks, we consider the input Xi at time i and simply
juxtapose it with output Yi-1 from the previous step.
[ 210 ]
Recurrent Neural Networks Chapter 6
Then we have two equations: si = d(Ws[Yi-1,xi]+bs) used to control via h how much of the
information ei = tanh(WC.[Yi-1,xi] + bc produced by the current cell should be added to the next
candidate value Ci via the operator i according to the scheme represented in the preceding
figure.
To implement what has discussed with the operators i and h, we need another equation
where the actual sums + and multiplications * take place: Ci=ft*Ci-1 + si*ei
Finally, we need to decide which part of the current cell should be sent to the Yi output. This
is simple: we take a logistic regression equation one more time and use this to control via an
h operation which part of the candidate value should go to the output. Here, there is a little
piece that deserves care and it is the use of the tanh function to squash
q the output into [-1,
1]. This latest step is described by the equation:
Now, I understand that this looks like a lot of math but there are two pieces of good news.
First, the math part is not so difficult after all if you understand what the goal is that we
want to achieve. Second, you can use LSTM cells as a blackbox drop-in replacement of
standard RNN cells and immediately get the benefit of solving the problem of a vanishing
gradient. For this reason, you really don't need to know all the math. You just take the
TensorFlow LSTM implementation from the library and use it.
[ 211 ]
Recurrent Neural Networks Chapter 6
Again, both GRUs and Peephole LSTM can be used as a blackbox drop-in for standard
RNN cells with no need to know the underlying math. Both cells can be used for solving the
problem of vanishing gradients and can be used for building deep neural networks:
'ZCORNGUQH5VCPFCTF.56/2GGR*QNG.56/CPF)47
[ 212 ]
Recurrent Neural Networks Chapter 6
#PGZCORNGQH400UGSWGPEGUCUUGGPKPJVVRMCTRCVJ[IKVJWDKQTPPGbGEVKXGPGUU
Machine translation is an example of non-synced sequences both in input and output: the
network reads the input text as a sequence and after reading the full text, it outputs the
target language.
Video classification is an example of synced sequences both in input and output: a video
input is a sequence of frames and for each frame, a classification label is provided in output.
[ 213 ]
Recurrent Neural Networks Chapter 6
#PGZCORNGQHVGZVIGPGTCVGFYKVJ400U
[ 214 ]
Recurrent Neural Networks Chapter 6
This section is inspired by the 2016 PhD thesis, Neural Machine Translation, by Minh-Thang
Luong (IUUQTHJUIVCDPNMNUIBOHUIFTJTCMPCNBTUFSUIFTJTQEG). The first key
concept is the presence of an encoder-decoder architecture, where an encoder transforms a
source sentence into a vector representing the meaning. This vector is then passed through a
decoder to produce a translation. Both the encoders and the decoders are RNNs that can
capture long-range dependencies in languages, for example, gender agreements and syntax
structures, without knowing them a priori and with no need to have a 1:1 mapping across
languages. This is a powerful capacity which enables very fluent translations:
#PGZCORNGQHGPEQFGTFGEQFGTCUUGGPKPJVVRUIKVJWDEQONOVJCPIVJGUKUDNQDOCUVGTVJGUKURFH
Let's see an example of an RNN translating the sentence She loves cute cats into Elle aime
les chats mignons.
There are two RNNs: one that acts as the encoder, and one that acts as the decoder. The
source sentence She loves cute cats is followed by a separator sign - and by the target
sentence Elle aime les chats mignons. These two concatenated sentences are given in input
to the encoder for training, and the decoder will produce the target Elle aime les chats
mignons. Of course, we need multiple examples like this one for achieving good training:
#PGZCORNGQHUGSWGPEGOQFGNUHQT0/6CUUGGPKPJVVRUIKVJWDEQONOVJCPIVJGUKUDNQDOCUVGTVJGUKURFH
[ 215 ]
Recurrent Neural Networks Chapter 6
Now there is a number of RNNs variants we can have. Let's look at some of them:
#PGZCORNGQH0GWTCNOCEJKPGVTCPUNCVKQPCUUGGPKPJVVRUIKVJWDEQONOVJCPIVJGUKUDNQDOCUVGTVJGUKURFH
[ 216 ]
Recurrent Neural Networks Chapter 6
In this recipe, we use NMT (Neural Machine Translation), a demo package for translation
available online on top of TensorFlow.
Getting ready
NMT is available at IUUQTHJUIVCDPNUFOTPSGMPXONU and the code is on GitHub.
How to do it...
We proceed with the recipe as follows:
HJUDMPOFIUUQTHJUIVCDPNUFOTPSGMPXONU
2. Download a training dataset. In this case, we will use the training set for
translating from Vietnamese to English. Other datasets are available at IUUQT
OMQTUBOGPSEFEVQSPKFDUTONU for additional languages, such as German and
Czech:
ONUTDSJQUTEPXOMPBE@JXTMUTIUNQONU@EBUB
&NCFEEJOH
FNCFEEJOH@FODPEFSWBSJBCMF@TDPQFHFU@WBSJBCMF
FNCFEEJOH@FODPEFS<TSD@WPDBC@TJ[FFNCFEEJOH@TJ[F>
-PPLVQFNCFEEJOH
FODPEFS@JOQVUT<NBY@UJNFCBUDI@TJ[F>
FODPEFS@FNC@JOQ<NBY@UJNFCBUDI@TJ[FFNCFEEJOH@TJ[F>
FODPEFS@FNC@JOQFNCFEEJOH@PQTFNCFEEJOH@MPPLVQ
FNCFEEJOH@FODPEFSFODPEFS@JOQVUT
[ 217 ]
Recurrent Neural Networks Chapter 6
#VJME3//DFMM
FODPEFS@DFMMUGOOSOO@DFMM#BTJD-45.$FMM
OVN@VOJUT
3VO%ZOBNJD3//
FODPEFS@PVUQVT<NBY@UJNFCBUDI@TJ[FOVN@VOJUT>
FODPEFS@TUBUF<CBUDI@TJ[FOVN@VOJUT>
FODPEFS@PVUQVUTFODPEFS@TUBUFUGOOEZOBNJD@SOO
FODPEFS@DFMMFODPEFS@FNC@JOQ
TFRVFODF@MFOHUITPVSDF@TFRVFODF@MFOHUIUJNF@NBKPS5SVF
5. After that, we need to define the decoder. So the first thing is to have a basic RNN
cell with UGOOSOO@DFMM#BTJD-45.$FMM, which is then used to create a basic
sampling decoder UGDPOUSJCTFRTFR#BTJD%FDPEFS, which is used to
perform dynamic decoding with the decoder
UGDPOUSJCTFRTFREZOBNJD@EFDPEF:
#VJME3//DFMM
EFDPEFS@DFMMUGOOSOO@DFMM#BTJD-45.$FMM
OVN@VOJUT
)FMQFS
IFMQFSUGDPOUSJCTFRTFR5SBJOJOH)FMQFS
EFDPEFS@FNC@JOQEFDPEFS@MFOHUITUJNF@NBKPS5SVF
%FDPEFS
EFDPEFSUGDPOUSJCTFRTFR#BTJD%FDPEFS
EFDPEFS@DFMMIFMQFSFODPEFS@TUBUF
PVUQVU@MBZFSQSPKFDUJPO@MBZFS
%ZOBNJDEFDPEJOH
PVUQVUT@UGDPOUSJCTFRTFREZOBNJD@EFDPEF
EFDPEFS
MPHJUTPVUQVUTSOO@PVUQVU
6. The last stage in the network is a softmax dense stage to transform the top hidden
states into a logit vector:
QSPKFDUJPO@MBZFSMBZFST@DPSF%FOTF
UHU@WPDBC@TJ[FVTF@CJBT'BMTF
[ 218 ]
Recurrent Neural Networks Chapter 6
7. Of course, we need to define the cross-entropy function and the loss used during
the training phase :
DSPTTFOUUGOOTQBSTF@TPGUNBY@DSPTT@FOUSPQZ@XJUI@MPHJUT
MBCFMTEFDPEFS@PVUQVUTMPHJUTMPHJUT
USBJO@MPTT
UGSFEVDF@TVN
DSPTTFOUUBSHFU@XFJHIUT
CBUDI@TJ[F
8. The next step is to define the steps needed for backpropagation, and use an
appropriate optimizer (in this case, Adam). Note that the gradient has been
clipped and that Adam uses a predefined learning rate:
$BMDVMBUFBOEDMJQHSBEJFOUT
QBSBNTUGUSBJOBCMF@WBSJBCMFT
HSBEJFOUTUGHSBEJFOUT
USBJO@MPTTQBSBNT
DMJQQFE@HSBEJFOUT@UGDMJQ@CZ@HMPCBM@OPSN
HSBEJFOUTNBY@HSBEJFOU@OPSN
0QUJNJ[BUJPO
PQUJNJ[FSUGUSBJO"EBN0QUJNJ[FS
MFBSOJOH@SBUF
VQEBUF@TUFQPQUJNJ[FSBQQMZ@HSBEJFOUT
[JQ
DMJQQFE@HSBEJFOUTQBSBNT
9. So now we can run the code and understand what the different executed steps
are. First, the training graph is created. Then the training iterations start. The
metric used for evaluations is bilingual evaluation understudy (BLEU). This
metric is the standard for evaluating the quality of text which has been machine-
translated from one natural language to another. Quality is considered to be the
correspondence between a machine and a human output. As you can see, this
value grows over time:
QZUIPONONUONUTSDWJUHUFOWPDBC@QSFGJYUNQONU@EBUBWPDBC
USBJO@QSFGJYUNQONU@EBUBUSBJOEFW@QSFGJYUNQONU@EBUBUTU
UFTU@QSFGJYUNQONU@EBUBUTUPVU@EJSUNQONU@NPEFM
OVN@USBJO@TUFQTTUFQT@QFS@TUBUTOVN@MBZFSTOVN@VOJUT
ESPQPVUNFUSJDTCMFV
+PCJE
<>
DSFBUJOHUSBJOHSBQI
OVN@MBZFSTOVN@SFTJEVBM@MBZFST
DFMM-45.GPSHFU@CJBT%SPQPVU8SBQQFSESPQPVU%FWJDF8SBQQFS
EFWJDFHQV
DFMM-45.GPSHFU@CJBT%SPQPVU8SBQQFSESPQPVU%FWJDF8SBQQFS
EFWJDFHQV
DFMM-45.GPSHFU@CJBT%SPQPVU8SBQQFSESPQPVU%FWJDF8SBQQFS
EFWJDFHQV
DFMM-45.GPSHFU@CJBT%SPQPVU8SBQQFSESPQPVU%FWJDF8SBQQFS
[ 219 ]
Recurrent Neural Networks Chapter 6
EFWJDFHQV
TUBSU@EFDBZ@TUFQMFBSOJOH@SBUFEFDBZ@TUFQTEFDBZ@GBDUPS
<>
4UBSUTUFQMS5IV4FQ
*OJUUSBJOJUFSBUPSTLJQQJOHFMFNFOUT
HMPCBMTUFQMSTUFQUJNFTXQT,QQMCMFV
HMPCBMTUFQMSTUFQUJNFTXQT,QQMCMFV
<>
HMPCBMTUFQMSTUFQUJNFTXQT,QQMCMFV
HMPCBMTUFQMSTUFQUJNFTXQT,QQMCMFV
HMPCBMTUFQMSTUFQUJNFTXQT,QQMCMFV
<>
&YUFSOBMFWBMVBUJPOHMPCBMTUFQ
EFDPEJOHUPPVUQVUUNQONU@NPEFMPVUQVU@EFW
EPOFOVNTFOUFODFTUJNFT5IV4FQ
CMFVEFW
TBWJOHIQBSBNTUPUNQONU@NPEFMIQBSBNT
&YUFSOBMFWBMVBUJPOHMPCBMTUFQ
EFDPEJOHUPPVUQVUUNQONU@NPEFMPVUQVU@UFTU
EPOFOVNTFOUFODFTUJNFT5IV4FQ
CMFVUFTU
TBWJOHIQBSBNTUPUNQONU@NPEFMIQBSBNT
<>
HMPCBMTUFQMSTUFQUJNFTXQT,QQMCMFV
How it works...
All the preceding code has been defined in IUUQTHJUIVCDPNUFOTPSGMPXONUCMPC
NBTUFSONUNPEFMQZ. The key idea is to have two RNNs packed together. The first one is the
encoder which works into an embedding space, mapping similar words very closely. The
encoder understands the meaning of training examples and it produces a tensor as output.
This tensor is then passed to the decoder simply by connecting the last hidden layer of the
encoder to the initial layer of the decoder. Note that the learning happens because of our
loss function based on cross-entropy with MBCFMTEFDPEFS@PVUQVUT.
[ 220 ]
Recurrent Neural Networks Chapter 6
The code learns how to translate and the progress is tracked through iterations by the BLEU
metric, as shown in the following figure:
#PGZCORNGQH$.'7OGVTKEKP6GPUQTDQCTF
[ 221 ]
Recurrent Neural Networks Chapter 6
#PGZCORNGQHUGSWGPEGOQFGNUHQT0/6YKVJRTQDCDKNKVKGUCUUGGPKPJVVRUIKVJWDEQONOVJCPIVJGUKUDNQDOCUVGTVJGUKURFH
[ 222 ]
Recurrent Neural Networks Chapter 6
How to do it...
We proceed with the recipe as follows:
1. Define the greedy strategy for sampling the decoder. This is easy because we can
use the library defined in UGDPOUSJCTFRTFR(SFFEZ&NCFEEJOH)FMQFS.
Since we don't know the exact length of the target sentence, we use a heuristic by
limiting it to be a maximum of twice the length of the source sentence:
)FMQFS
IFMQFSUGDPOUSJCTFRTFR(SFFEZ&NCFEEJOH)FMQFS
FNCFEEJOH@EFDPEFS
UGGJMM
<CBUDI@TJ[F>UHU@TPT@JEUHU@FPT@JE
%FDPEFS
EFDPEFSUGDPOUSJCTFRTFR#BTJD%FDPEFS
EFDPEFS@DFMMIFMQFSFODPEFS@TUBUF
PVUQVU@MBZFSQSPKFDUJPO@MBZFS
%ZOBNJDEFDPEJOH
PVUQVUT@UGDPOUSJCTFRTFREZOBNJD@EFDPEF
EFDPEFSNBYJNVN@JUFSBUJPOTNBYJNVN@JUFSBUJPOT
USBOTMBUJPOTPVUQVUTTBNQMF@JE
NBYJNVN@JUFSBUJPOTUGSPVOE
UGSFEVDF@NBY
TPVSDF@TFRVFODF@MFOHUI
2. We can now run the net, giving as input a sentence never seen before
(JOGFSFODF@JOQVU@GJMFUNQNZ@JOGFS@GJMF) and letting the network
translate the outcome
(JOGFSFODF@PVUQVU@GJMFUNQONU@NPEFMPVUQVU@JOGFS):
QZUIPONONUONU=
PVU@EJSUNQONU@NPEFM=
JOGFSFODF@JOQVU@GJMFUNQNZ@JOGFS@GJMFWJ=
JOGFSFODF@PVUQVU@GJMFUNQONU@NPEFMPVUQVU@JOGFS
How it works...
Two RNNs are packed together to form an encoder-decoder RNN network. The decoder
emits logits, which are greedily transformed into words of the target language. As an
example, an automatic translation from Vietnamese into English is shown here:
Input sentence in Vietnamese: Khi tji ckn nhl , Tji nghm rnng BocTripu Tiqn lr stt
nuvc twt nhtt trqn thx givi vr tji thuyng hzt bri " Ch{ng ta ch|ng c} g~ phi
ghen t.
[ 223 ]
Recurrent Neural Networks Chapter 6
Output sentence translated into English: When I'm a very good , I'm going to
see the most important thing about the most important and I'm not sure what I'm
going to say.
#PGZCORNGQHCVVGPVKQPOQFGNHQT0/6CUUGGPKPJVVRUIKVJWDEQONOVJCPIVJGUKUDNQDOCUVGTVJGUKURFH
[ 224 ]
Recurrent Neural Networks Chapter 6
First, the current target hidden state is used together with all previous source
states to derive attention weights which are used to pay more or less attention to
tokens previously seen in the sequence
Second, a context vector is created to summarize the results of the attention
weights
Third, the context vector is combined with the current target hidden state to
obtain the attention vector
How to do it...
We proceed with the recipe as follows:
BUUFOUJPO@TUBUFT<CBUDI@TJ[FNBY@UJNFOVN@VOJUT>
BUUFOUJPO@TUBUFTUGUSBOTQPTF
FODPEFS@PVUQVUT<>
$SFBUFBOBUUFOUJPONFDIBOJTN
BUUFOUJPO@NFDIBOJTNUGDPOUSJCTFRTFR-VPOH"UUFOUJPO
OVN@VOJUTBUUFOUJPO@TUBUFT
NFNPSZ@TFRVFODF@MFOHUITPVSDF@TFRVFODF@MFOHUI
2. Use the defined attention mechanism as a wrapper around the decoder cell by
means of an attention wrapper:
EFDPEFS@DFMMUGDPOUSJCTFRTFR"UUFOUJPO8SBQQFS
EFDPEFS@DFMMBUUFOUJPO@NFDIBOJTN
BUUFOUJPO@MBZFS@TJ[FOVN@VOJUT
3. Run the code to see the results. We immediately notice that the attention
mechanism produces a significant improvement in terms of the BLEU score:
QZUIPONONUONU=
BUUFOUJPOTDBMFE@MVPOH=
TSDWJUHUFO=
WPDBC@QSFGJYUNQONU@EBUBWPDBC=
USBJO@QSFGJYUNQONU@EBUBUSBJO=
EFW@QSFGJYUNQONU@EBUBUTU=
[ 225 ]
Recurrent Neural Networks Chapter 6
UFTU@QSFGJYUNQONU@EBUBUTU=
PVU@EJSUNQONU@BUUFOUJPO@NPEFM=
OVN@USBJO@TUFQT=
TUFQT@QFS@TUBUT=
OVN@MBZFST=
OVN@VOJUT=
ESPQPVU=
NFUSJDTCMFV
<>
4UBSUTUFQMS'SJ4FQ
*OJUUSBJOJUFSBUPSTLJQQJOHFMFNFOUT
HMPCBMTUFQMSTUFQUJNFTXQT,QQMCMFV
<>
'JOBMTUFQMSTUFQUJNFXQT,QQMEFWQQM
EFWCMFVUFTUQQMUFTUCMFV4BU4FQ
%POFUSBJOJOHUJNFT4BU4FQ
4UBSUFWBMVBUJOHTBWFECFTUNPEFMT
<>
MPBEFEJOGFSNPEFMQBSBNFUFSTGSPN
UNQONU@BUUFOUJPO@NPEFMCFTU@CMFVUSBOTMBUFDLQUUJNFT
TSDOIuOHCOCJxUsJpVH~LIjOH
SFG#VUZPVLOPXXIBU
ONU#VUXIBUEPZPVLOPX
<>
#FTUCMFVTUFQTUFQUJNFXQT,EFWQQMEFWCMFV
UFTUQQMUFTUCMFV4BU4FQ
How it works...
Attention is a mechanism for using the information acquired by the inner states of the
encoder RNN and combining this information with the final state of the decoder. The key
idea is that, in this way, it is possible to pay more or less attention to some tokens in the
source sequence. The gain obtained with attention is shown in the following figure
representing the BLEU score.
[ 226 ]
Recurrent Neural Networks Chapter 6
We notice a significant gain with respect to the same diagram given in our first recipe,
where no attention was used:
#PGZCORNGQH$.'7OGVTKEUYKVJCVVGPVKQPKP6GPUQTDQCTF
[ 227 ]
Recurrent Neural Networks Chapter 6
There's more...
It is worth remembering that seq2seq can be used for more than just Machine
Translation. Let's see some examples:
#PGZCORNGQH5[PVCZ0GVCUUGGPKPJVVRUTGUGCTEJIQQINGDNQIEQOCPPQWPEKPIU[PVCZPGVYQTNFUOQUVJVON
[ 228 ]
Recurrent Neural Networks Chapter 6
For the sake of simplicity, we will use the framework TFLearn (IUUQ
UGMFBSOPSH), which runs on top of TensorFlow. This example is part of
the standard distribution and it is available at IUUQTHJUIVCDPN
UGMFBSOUGMFBSOCMPCNBTUFSFYBNQMFTOMQMTUN@HFOFSBUPS@
TIBLFTQFBSFQZ . The model developed one is an RNN character-level
language model where the sequences considered are sequences of
characters and not words.
How to do it...
We proceed with the recipe as follows:
QJQJOTUBMM*UGMFBSO
JNQPSUPT
JNQPSUQJDLMF
GSPNTJYNPWFTJNQPSUVSMMJC
JNQPSUUGMFBSO
GSPNUGMFBSOEBUB@VUJMTJNQPSU
QBUITIBLFTQFBSF@JOQVUUYU
DIBS@JEY@GJMF DIBS@JEYQJDLMF
JGOPUPTQBUIJTGJMF
QBUI
VSMMJCSFRVFTUVSMSFUSJFWF
IUUQTSBXHJUIVCVTFSDPOUFOUDPNUGMFBSOUGMFB
SOHJUIVCJPNBTUFSSFTPVSDFTTIBLFTQFBSF@JOQVUUYUQBUI
3. Transform the input text into a vector and return the parsed sequences and
targets, along with the associated dictionary, by using
TUSJOH@UP@TFNJ@SFEVOEBOU@TFRVFODFT
, which returns a tuple (inputs,
targets, dictionary):
NBYMFO
DIBS@JEY/POF
JGPTQBUIJTGJMF
DIBS@JEY@GJMF
QSJOU
-PBEJOHQSFWJPVTDIBS@JEY
DIBS@JEYQJDLMFMPBE
PQFO
DIBS@JEY@GJMF SC
9:DIBS@JEY=
[ 229 ]
Recurrent Neural Networks Chapter 6
UFYUGJMF@UP@TFNJ@SFEVOEBOU@TFRVFODFT
QBUITFR@NBYMFONBYMFOSFEVO@TUFQ
QSF@EFGJOFE@DIBS@JEYDIBS@JEY
QJDLMFEVNQ
DIBS@JEYPQFO
DIBS@JEY@GJMF XC
4. Define an RNN made up of three LSTMs, each of which has 512 nodes and
returns the full sequence instead of the last sequence output only. Note that we
use drop-out modules with a probability of 50% for connecting the LSTM
modules. The last layer is a dense layer applying a softmax with length equal to
the dictionary size. The loss function is DBUFHPSJDBM@DSPTTFOUSPQZ and the
optimizer is Adam:
HUGMFBSOJOQVU@EBUB
</POFNBYMFOMFO
DIBS@JEY>
HUGMFBSOMTUN
HSFUVSO@TFR5SVF
HUGMFBSOESPQPVU
H
HUGMFBSOMTUN
HSFUVSO@TFR5SVF
HUGMFBSOESPQPVU
H
HUGMFBSOMTUN
H
HUGMFBSOESPQPVU
H
HUGMFBSOGVMMZ@DPOOFDUFE
HMFO
DIBS@JEYBDUJWBUJPO TPGUNBY
HUGMFBSOSFHSFTTJPO
HPQUJNJ[FS BEBN
MPTT DBUFHPSJDBM@DSPTTFOUSPQZ
MFBSOJOH@SBUF
5. Given the network defined in step 4, we can now generate the sequence with the
library GMFBSONPEFMTHFOFSBUPS4FRVFODF(FOFSBUPS (OFUXPSL,
EJDUJPOBSZDIBS@JEYTFR@NBYMFONBYMF, DMJQ@HSBEJFOUT
DIFDLQPJOU@QBUI NPEFM@TIBLFTQFBSF ):
NUGMFBSO4FRVFODF(FOFSBUPS
HEJDUJPOBSZDIBS@JEY
TFR@NBYMFONBYMFO
DMJQ@HSBEJFOUT
DIFDLQPJOU@QBUI NPEFM@TIBLFTQFBSF
6. For 50 iterations, we take a random sequence from the input text and we generate
a new text. Temperature is controlling the novelty of the created sequence; a
temperature close to 0 will look like samples used for training, while the higher
the temperature, the more the novelty:
GPSJJOSBOHF
TFFESBOEPN@TFRVFODF@GSPN@UFYUGJMF
QBUINBYMFO
NGJU
9:WBMJEBUJPO@TFUCBUDI@TJ[F
O@FQPDISVO@JE TIBLFTQFBSF
QSJOU
5&45*/(
QSJOU
5FTUXJUIUFNQFSBUVSFPG
QSJOU
NHFOFSBUF
UFNQFSBUVSFTFR@TFFETFFE
[ 230 ]
Recurrent Neural Networks Chapter 6
QSJOU
5FTUXJUIUFNQFSBUVSFPG
QSJOU
NHFOFSBUF
UFNQFSBUVSFTFR@TFFETFFE
How it works...
When a new unknown or forgotten work of art is about to be attributed to an author, there
are eminent academics who compare this work with other pieces of work attributed to the
author. What the academics do is to find common patterns in the text sequence of the well-
known works, hoping to find similar patterns in the unknown work.
This recipe works in a similar way: an RNN learns what the most peculiar patterns in
Shakespeare's writing are and then these patterns are used to generate new, never-seen-
before texts which well represent the style of the greatest English author.
First iteration
Here, the network is learning some basic structure, including the need to have dialog about
fictional characters (%*", 463:0/5, )3/5-(*13."3, and "3*-&/). However, the English is
still very poor and lots of words are not really in proper English:
5SBJOJOHTBNQMFT
7BMJEBUJPOTBNQMFT
5SBJOJOH4UFQ]UPUBMMPTT]UJNFT
]"EBN]FQPDI]MPTT]WBM@MPTTJUFS
5&45*/(
5FTUXJUIUFNQFSBUVSFPG
TUUIPVNBMJDF
*GUIPVDBTFHIPVHINFNFUPVENBNFNFBSE LF"GTXFLFXUFBL%ZOZXPME BT
UPPGNZUIPHUSPZBSEIBTTFWFIPSUIFOUIBUXPSEJUIHPMFIJFTVDD
[ 231 ]
Recurrent Neural Networks Chapter 6
DBJHIUGPN
%*"
"HSVPTDFFO*QFFZ
CZNZ
8JPVTFSBU4FCJOFXPVME
XBXUIJTBGFFBO
463:0/5
5FFWFOPVSUFSPOHBPVMUPODJNFCVDJDF JTGVSUVUVO
"NFNZTPSJWBTTBNVUNZQFBOU
"N
'FUIBUMFSDPNUIFSUIFOPNFNFQBBUVZDPSOTXSB[FONFBTHIPNO HFDPOTU
QIFBMF
"TZFSFENBUIUIZWBOT
*JNGPBUXPSFQPVHBOEUIJUNJKFXPNM
)3/5-(*13."3
* EEFSGPNRVFTGUIJZPGEPFEJMBTHIFMFIBODLPMNZDPSJSFIPVHBOHMF
,JHVXUSPMMZPVFFMFSEUIBNNZGPN*OPXMJUIB
5FTUXJUIUFNQFSBUVSFPG
TUUIPVNBMJDF
*GUIPVQSBMMTJU*IBSXJUIBOEUIFTPSUBGFUIFOPUIJOUPGUIFGPSFUIF
GJSXJUIXJUIUIFDFNFBUUIFJOEUIFDPVUIFSIJUZFUPGUIFTPOTFFJO
TPMMFTBOEUIBUOPUPGIFBSGPSFUIFIBUICVS
"3*-&/
.PSFZPVBUPUIFNBSFNFQFPETPSF
"OEGPSFTUSJOHUIFSFPVDLBOEBOEGFSUPUIFTPIBTUIFUIFBUFOEUIF
EPSFPGNBMMUIFTJTUIFUIFCPUDPVSEXJUFCFUIFUIPVMFUIFUPOFOHFBQF
BOEUIJTOPUUIFUIFCBMMCPPMNFUIFTPNFUIBUEFBST
5IFCFUPUIFUIFTUIFMFUUIFXJUIUIFUIFBSUPVMEGBNFCPPSTBOEOPUUP
OPUUIFEFBOFGFSFUIFXPNPVSIJUNVUITPUIBOEUIFFNFFOUUNZUPUIF
USFFSTBOEXPUIBOEXJ
5SBJOJOHTBNQMFT
7BMJEBUJPOTBNQMFT
5SBJOJOH4UFQ]UPUBMMPTT1.44823]UJNFT
]"EBN]FQPDI]MPTT]WBM@MPTTJUFS
[ 232 ]
Recurrent Neural Networks Chapter 6
5FTUXJUIUFNQFSBUVSFPG
JOUIJTLJOE
5)&4&64
5IFSFJTOPUUIBUCFTPUIFOUPUIFEFBUI
5PNBLFUIFCPEZBOEBMMUIFNJOE
#&/&%*$,
8FMMUIFSFTIBMMUIFUIJOHTUPOFFEUIFPGGFSUPPVSIFBSU
5POPUBSFIFXJUIIJN*IBWFTFFUIFIBOETBSFUPUSVFPGIJNUIBU*BN
OPU
5IFXIPNJOTPNFUIFGPSUVOFT
8IJDITIFXFSFCFUUFSOPUUPEPIJN
,*/()&/3:7*
*IBWFTPNFBTUBSUFSBOEBOETFFOUIFNPSFUPCFUIFCPZBOECFTVDIB
QMPDLBOEMPWFTPTBZBOE*XJMMCFIJTFOUJSF
"OEXIFONZNBTUFSTBSFBHPPEWJSUVFT
5IBUTFFUIFDSPXOPGPVSXPSTF
5IJTNBEFBDBMMFEHSBDFUPIFBSIJNBOEBOBTT
"OEUIFQSPWFTUBOETUBOE
There's more...
The blog post The Unreasonable Effectiveness of Recurrent Neural Networks (IUUQLBSQBUIZ
HJUIVCJPSOOFGGFDUJWFOFTT ) describes a fascinating set of examples of
RNN character-level language models, including the following:
[ 233 ]
Recurrent Neural Networks Chapter 6
We offer historical data from our Bitcoin Price Index through the following endpoint:
https://api.coindesk.com/v1/bpi/historical/close.json
By default, this will return the previous 31 days' worth of data. This endpoint accepts the
following optional parameters:
?index=[USD/CNY]The index to return data for. Defaults to USD.
?currency=<VALUE>The currency to return the data in, specified in ISO 4217 format.
Defaults to USD.
?start=<VALUE>&end=<VALUE> Allows data to be returned for a specific date range.
Must be listed as a pair of start and end parameters, with dates supplied in the YYYY-
MM-DD format, e.g. 2013-09-01 for September 1st, 2013.
?for=yesterday Specifying this will return a single value for the previous day. Overrides
the start/end parameter.
Sample Request:
IUUQTBQJDPJOEFTLDPNWCQJIJTUPSJDBMDMPTFKTPOTUBSU
FOE
Sample JSON Response:
{"bpi":{"2013-09-01":128.2597,"2013-09-02":127.3648,"2013-09-03":127.5915,"2013-0
9-04":120.5738,"2013-09-05":120.5333},"disclaimer":"This data was produced from the
CoinDesk Bitcoin Price Index. BPI value data returned as USD.","time":{"updated":"Sep
6, 2013 00:03:00 UTC","updatedISO":"2013-09-06T00:03:00+00:00"}}
How to do it...
Here is how we proceed with the recipe:
1. Clone the following GitHub repository. This is a project to encourage users to try
and experiment with the seq2seq neural network architecture:
HJUDMPOF
IUUQTHJUIVCDPNHVJMMBVNFDIFWBMJFSTFRTFRTJHOBMQSFEJDUJPOHJU
[ 234 ]
Recurrent Neural Networks Chapter 6
2. Given the preceding repository, consider the following functions which load and
normalize the Bitcoin historical data for the USD or EUR Bitcoin value. The
functions are defined in EBUBTFUQZ. Training and testing data are separated
according to the 80/20 rule. As a consequence, 20 percent of the testing data is the
most recent historical Bitcoin values. Every example contains 40 data points of
USD and then EUR data in the feature axis/dimension. Data is normalized
according to the mean and standard deviation. The function
HFOFSBUF@Y@Z@EBUB@W generates random samples of the training data
(respectively, the testing data) of size CBUDI@TJ[F:
EFGMPBE$VSSFODZ
DVSSXJOEPX@TJ[F
3FUVSOUIFIJTUPSJDBMEBUBGPSUIF64%PS&63CJUDPJOWBMVF*TEPOF
XJUIBOXFC"1*DBMM
DVSS64%]&63
'PSNPSFJOGPPOUIF63-DBMMJUJTJOTQJSFECZ
IUUQTHJUIVCDPN-FWJOPDPJOEFTLBQJOPEF
SSFRVFTUTHFU
IUUQBQJDPJOEFTLDPNWCQJIJTUPSJDBMDMPTFKTPOTUBSUFOE
DVSSFODZ\^GPSNBU
DVSS
EBUBSKTPO
UJNF@UP@WBMVFTTPSUFE
EBUB<CQJ>JUFNT
WBMVFT<WBMGPSLFZWBMJOUJNF@UP@WBMVFT>
LFQU@WBMVFTWBMVFT<>
9<>
:<>
GPSJJOSBOHF
MFO
LFQU@WBMVFTXJOEPX@TJ[F
9BQQFOE
LFQU@WBMVFT<JJ
XJOEPX@TJ[F>
:BQQFOE
LFQU@WBMVFT<J
XJOEPX@TJ[FJ
XJOEPX@TJ[F>
5PCFBCMFUPDPODBUPOJOOFSEJNFOTJPOMBUFSPO
9OQFYQBOE@EJNT
9BYJT
:OQFYQBOE@EJNT
:BYJT
SFUVSO9:
EFGOPSNBMJ[F
9:/POF
/PSNBMJTF9BOE:BDDPSEJOHUPUIFNFBOBOETUBOEBSE
EFWJBUJPOPGUIF9WBMVFTPOMZ
*UXPVMECFQPTTJCMFUPOPSNBMJ[FXJUIMBTUSBUIFSUIBONFBOTVDI
BT
MBTUTOQFYQBOE@EJNT
9<>BYJT
BTTFSU
MBTUT<>9<>BMM
\^\^\^
[ 235 ]
Recurrent Neural Networks Chapter 6
\^GPSNBU
MBTUT<>TIBQF9<>TIBQFMBTUT<>9<>
NFBOOQFYQBOE@EJNT
OQBWFSBHF
9BYJT
BYJT
TUEEFWOQFYQBOE@EJNT
OQTUE
9BYJT
BYJT
QSJOU
NFBOTIBQFTUEEFWTIBQF
QSJOU
9TIBQF:TIBQF
99NFBO
99
TUEEFW
JG:JTOPU/POF
BTTFSU:TIBQF9TIBQF
:TIBQF9TIBQF
::NFBO
::
TUEEFW
SFUVSO9:
SFUVSO9
EFGGFUDI@CBUDI@TJ[F@SBOEPN
9:CBUDI@TJ[F
3FUVSOTSBOEPNMZBOBMJHOFECBUDI@TJ[FPG9BOE:BNPOHBMMFYBNQMFT
5IFFYUFSOBMEJNFOTJPOPG9BOE:NVTUCFUIFCBUDITJ[F
FHDPMVNOFYBNQMF
9BOE:DBOCF/EJNFOTJPOBM
BTTFSU9TIBQF:TIBQF
9TIBQF:TIBQF
JEYFTOQSBOEPNSBOEJOU
9TIBQF<>TJ[FCBUDI@TJ[F
9@PVUOQBSSBZ
9<JEYFT>USBOTQPTF
:@PVUOQBSSBZ
:<JEYFT>USBOTQPTF
SFUVSO9@PVU:@PVU
9@USBJO<>
:@USBJO<>
9@UFTU<>
:@UFTU<>
EFGHFOFSBUF@Y@Z@EBUB@W
JT5SBJOCBUDI@TJ[F
3FUVSOGJOBODJBMEBUBGPSUIFCJUDPJO
'FBUVSFTBSF64%BOE&63JOUIFJOUFSOBMEJNFOTJPO
8FOPSNBMJ[F9BOE:EBUBBDDPSEJOHUPUIF9POMZUPOPU
TQPJMUIFQSFEJDUJPOTXFBTLGPS
'PSFWFSZXJOEPX
XJOEPXPSTFR@MFOHUI:JTUIFQSFEJDUJPOGPMMPXJOH
9
5SBJOBOEUFTUEBUBBSFTFQBSBUFEBDDPSEJOHUPUIF
SVMF
5IFSFGPSFUIFQFSDFOUPGUIFUFTUEBUBBSFUIFNPTU
SFDFOUIJTUPSJDBMCJUDPJOWBMVFT&WFSZFYBNQMFJO9DPOUBJOT
QPJOUTPG64%BOEUIFO&63EBUBJOUIFGFBUVSFBYJTEJNFOTJPO
*UJTUPCFOPUFEUIBUUIFSFUVSOFE9BOE:IBTUIFTBNFTIBQF
BOEBSFJOBUVQMF
QBTWBMVFTGPSFODPEFSBGUFSGPSEFDPEFS TQSFEJDUJPOT
[ 236 ]
Recurrent Neural Networks Chapter 6
TFR@MFOHUI
HMPCBM:@USBJO
HMPCBM9@USBJO
HMPCBM9@UFTU
HMPCBM:@UFTU
'JSTUMPBEXJUINFNPJ[BUJPO
JGMFO
:@UFTU
"1*DBMM
9@VTE:@VTEMPBE$VSSFODZ
64%
XJOEPX@TJ[FTFR@MFOHUI
9@FVS:@FVSMPBE$VSSFODZ
&63
XJOEPX@TJ[FTFR@MFOHUI
"MMEBUBBMJHOFE
9OQDPODBUFOBUF
9@VTE9@FVSBYJT
:OQDPODBUFOBUF
:@VTE:@FVSBYJT
9:OPSNBMJ[F
9:
4QMJU
9@USBJO9<JOU
MFO
9>
:@USBJO:<JOU
MFO
:>
9@UFTU9<JOU
MFO
9>
:@UFTU:<JOU
MFO
:>
JGJT5SBJO
SFUVSOGFUDI@CBUDI@TJ[F@SBOEPN
9@USBJO:@USBJOCBUDI@TJ[F
FMTF
SFUVSOGFUDI@CBUDI@TJ[F@SBOEPN
9@UFTU:@UFTUCBUDI@TJ[F
3. Generate the training, validation, and testing data, and define a number of
hyperparameters such as CBUDI@TJ[F, IJEEFO@EJN (the number of hidden
neurons in the RNN), and MBZFST@TUBDLFE@DPVOU (the number of stacked
recurrent cells). In addition, define a number of parameters for fine-tuning the
optimizer, such as the optimizer's learning rate, number of iterations, MS@EFDBZ
for the optimizer's simulated annealing, the optimizer's momentum, and L2
regularization for avoiding overfitting. Note that the GitHub repository has the
default CBUDI@TJ[F and OC@JUFST but I've obtained better results
with CBUDI@TJ[F and OC@JUFST:
GSPNEBUBTFUTJNQPSUHFOFSBUF@Y@Z@EBUB@W
HFOFSBUF@Y@Z@EBUBHFOFSBUF@Y@Z@EBUB@W
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
NBUQMPUMJCJOMJOF
TBNQMF@YTBNQMF@ZHFOFSBUF@Y@Z@EBUB
JT5SBJO5SVFCBUDI@TJ[F
QSJOU
%JNFOTJPOTPGUIFEBUBTFUGPS9BOE:USBJOJOH
FYBNQMFT
QSJOU
TBNQMF@YTIBQF
[ 237 ]
Recurrent Neural Networks Chapter 6
QSJOU
TBNQMF@ZTIBQF
QSJOU
TFR@MFOHUICBUDI@TJ[FPVUQVU@EJN
QSJOUTBNQMF@YTBNQMF@Z
*OUFSOBMOFVSBMOFUXPSLQBSBNFUFST
TFR@MFOHUITBNQMF@YTIBQF<>5JNFTFSJFTXJMMIBWFUIFTBNFQBTUBOE
GVUVSF
UPCFQSFEJDUFEMFOHIU
CBUDI@TJ[F-PXWBMVFVTFEGPSMJWFEFNPQVSQPTFTBOE
XPVMECFQPTTJCMFUPPDSBOLUIBUVQ
PVUQVU@EJNJOQVU@EJNTBNQMF@YTIBQF<>0VUQVUEJNFOTJPO
FH
NVMUJQMFTJHOBMTBUPODFUJFEJOUJNF
IJEEFO@EJN$PVOUPGIJEEFOOFVSPOTJOUIFSFDVSSFOUVOJUT
MBZFST@TUBDLFE@DPVOU/VNCFSPGTUBDLFESFDVSSFOUDFMMTPOUIF
OFVSBMEFQUIBYJT
0QUNJ[FS
MFBSOJOH@SBUF4NBMMMSIFMQTOPUUPEJWFSHFEVSJOHUSBJOJOH
OC@JUFST)PXNBOZUJNFTXFQFSGPSNBUSBJOJOHTUFQ
UIFSFGPSFIPX
NBOZUJNFTXFTIPXBCBUDI
MS@EFDBZEFGBVMU4JNVMBUFEBOOFBMJOH
NPNFOUVNEFGBVMU.PNFOUVNUFDIOJRVFJOXFJHIUTVQEBUF
MBNCEB@M@SFH-SFHVMBSJ[BUJPOPGXFJHIUTBWPJETPWFSGJUUJOH
UGOOTFRTFRUGDPOUSJCMFHBDZ@TFRTFR
UGOOSOO@DFMMUGDPOUSJCSOO
UGOOSOO@DFMM(36$FMMUGDPOUSJCSOO(36$FMM
UGSFTFU@EFGBVMU@HSBQI
TFTTDMPTF
TFTTUG*OUFSBDUJWF4FTTJPO
XJUIUGWBSJBCMF@TDPQF
4FRTFR
&ODPEFSJOQVUT
FOD@JOQ<
UGQMBDFIPMEFS
UGGMPBUTIBQF
/POFJOQVU@EJN
OBNFJOQ@\^GPSNBU
U
GPSUJOSBOHF
TFR@MFOHUI
>
%FDPEFSFYQFDUFEPVUQVUT
FYQFDUFE@TQBSTF@PVUQVU<
UGQMBDFIPMEFS
UGGMPBUTIBQF
/POFPVUQVU@EJN
OBNFFYQFDUFE@TQBSTF@PVUQVU@GPSNBU
U
GPSUJOSBOHF
TFR@MFOHUI
>
(JWFB(0UPLFOUPUIFEFDPEFS
:PVNJHIUXBOUUPSFWJTFXIBUJTUIFBQQFOEFEWBMVF
FOD@JOQ<>
[ 238 ]
Recurrent Neural Networks Chapter 6
EFD@JOQ<UG[FSPT@MJLF
FOD@JOQ<>EUZQFOQGMPBUOBNF(0>
FOD@JOQ<>
$SFBUFBAMBZFST@TUBDLFE@DPVOUAPGTUBDLFE3//T
(36DFMMTIFSF
DFMMT<>
GPSJJOSBOHF
MBZFST@TUBDLFE@DPVOU
XJUIUGWBSJBCMF@TDPQF
3//@\^ GPSNBU
J
DFMMTBQQFOE
UGOOSOO@DFMM(36$FMM
IJEEFO@EJN
DFMMTBQQFOE
UGOOSOO@DFMM#BTJD-45.$FMM
DFMMUGOOSOO@DFMM.VMUJ3//$FMM
DFMMT
'PSSFTIBQJOHUIFJOQVUBOEPVUQVUEJNFOTJPOTPGUIFTFRTFR3//
X@JOUG7BSJBCMF
UGSBOEPN@OPSNBM
<JOQVU@EJNIJEEFO@EJN>
C@JOUG7BSJBCMF
UGSBOEPN@OPSNBM
<IJEEFO@EJN>NFBO
X@PVUUG7BSJBCMF
UGSBOEPN@OPSNBM
<IJEEFO@EJNPVUQVU@EJN>
C@PVUUG7BSJBCMF
UGSBOEPN@OPSNBM
<PVUQVU@EJN>
SFTIBQFE@JOQVUT<UGOOSFMV
UGNBUNVM
JX@JO
C@JOGPSJJOFOD@JOQ>
)FSFUIFFODPEFSBOEUIFEFDPEFSVTFTUIFTBNFDFMM)08&7&3
UIFXFJHIUTBSFO UTIBSFEBNPOHUIFFODPEFSBOEEFDPEFSXFIBWFUXP
TFUTPGXFJHIUTDSFBUFEVOEFSUIFIPPEBDDPSEJOHUPUIBUGVODUJPO T
EFG
EFD@PVUQVUTEFD@NFNPSZUGOOTFRTFRCBTJD@SOO@TFRTFR
FOD@JOQ
EFD@JOQ
DFMM
PVUQVU@TDBMF@GBDUPSUG7BSJBCMF
OBNF0VUQVU@4DBMF'BDUPS
'JOBMPVUQVUTXJUIMJOFBSSFTDBMJOHTJNJMBSUPCBUDIOPSN
CVUXJUIPVUUIFOPSNQBSUPGCBUDIOPSNBMJ[BUJPOIFIF
SFTIBQFE@PVUQVUT<PVUQVU@TDBMF@GBDUPS
UGNBUNVM
JX@PVU
C@PVU
GPSJJOEFD@PVUQVUT>
.FSHFBMMUIFTVNNBSJFTBOEXSJUFUIFNPVUUPUNQCJUDPJO@MPHT
CZ
EFGBVMU
NFSHFEUGTVNNBSZNFSHF@BMM
USBJO@XSJUFSUGTVNNBSZ'JMF8SJUFS
UNQCJUDPJO@MPHT
TFTTHSBQI
5. Now let's run the TensorBoard and visualize the network consisting of an RNN
encoder and an RNN decoder:
UFOTPSCPBSEMPHEJSUNQCJUDPJO@MPHT
[ 239 ]
Recurrent Neural Networks Chapter 6
#PGZCORNGQHEQFGHQT$KVEQKPXCNWGRTGFKEVKQPCUUGGPKP6GPUQTDQCTF
[ 240 ]
Recurrent Neural Networks Chapter 6
6. Now let's define the loss function as an L2 loss with regularization to avoid
overfitting and to have a better generalization. The chosen optimizer is RMSprop
with the value of MFBSOJOH@SBUF, decay, and momentum as defined at step 3:
5SBJOJOHMPTTBOEPQUJNJ[FS
XJUIUGWBSJBCMF@TDPQF
-PTT
-MPTT
PVUQVU@MPTT
GPS@Z@:JO[JQ
SFTIBQFE@PVUQVUTFYQFDUFE@TQBSTF@PVUQVU
PVUQVU@MPTT
UGSFEVDF@NFBO
UGOOM@MPTT
@Z@:
-SFHVMBSJ[BUJPO
UPBWPJEPWFSGJUUJOHBOEUPIBWFBCFUUFS
HFOFSBMJ[BUJPODBQBDJUZ
SFH@MPTT
GPSUG@WBSJOUGUSBJOBCMF@WBSJBCMFT
JGOPU
#JBTJOUG@WBSOBNFPS0VUQVU@JOUG@WBSOBNF
SFH@MPTT
UGSFEVDF@NFBO
UGOOM@MPTT
UG@WBS
MPTTPVUQVU@MPTT
MBNCEB@M@SFHSFH@MPTT
XJUIUGWBSJBCMF@TDPQF
0QUJNJ[FS
PQUJNJ[FSUGUSBJO3.41SPQ0QUJNJ[FS
MFBSOJOH@SBUFEFDBZMS@EFDBZ
NPNFOUVNNPNFOUVN
USBJO@PQPQUJNJ[FSNJOJNJ[F
MPTT
7. Prepare for training in batch by generating the training data and by running the
optimizer on CBUDI@TJ[F examples from the dataset. Similarly, prepare for test
by generating test data on CBUDI@TJ[F examples from the dataset. Training runs
for OC@JUFST
iterations, and one iteration out of ten is used to test the results:
EFGUSBJO@CBUDI
CBUDI@TJ[F
5SBJOJOHTUFQUIBUPQUJNJ[FTUIFXFJHIUT
QSPWJEFETPNFCBUDI@TJ[F9BOE:FYBNQMFTGSPNUIFEBUBTFU
9:HFOFSBUF@Y@Z@EBUB
JT5SBJO5SVFCBUDI@TJ[FCBUDI@TJ[F
GFFE@EJDU\FOD@JOQ<U>9<U>GPSUJOSBOHF
MFO
FOD@JOQ^
GFFE@EJDUVQEBUF
\FYQFDUFE@TQBSTF@PVUQVU<U>:<U>GPSUJO
SBOHF
MFO
FYQFDUFE@TQBSTF@PVUQVU^
@MPTT@UTFTTSVO
<USBJO@PQMPTT>GFFE@EJDU
SFUVSOMPTT@U
EFGUFTU@CBUDI
CBUDI@TJ[F
5FTUTUFQEPFT/05PQUJNJ[FT8FJHIUTBSFGSP[FOCZOPU
EPJOHTFTTSVOPOUIFUSBJO@PQ
9:HFOFSBUF@Y@Z@EBUB
JT5SBJO'BMTFCBUDI@TJ[FCBUDI@TJ[F
[ 241 ]
Recurrent Neural Networks Chapter 6
GFFE@EJDU\FOD@JOQ<U>9<U>GPSUJOSBOHF
MFO
FOD@JOQ^
GFFE@EJDUVQEBUF
\FYQFDUFE@TQBSTF@PVUQVU<U>:<U>GPSUJO
SBOHF
MFO
FYQFDUFE@TQBSTF@PVUQVU^
MPTT@UTFTTSVO
<MPTT>GFFE@EJDU
SFUVSOMPTT@U<>
5SBJOJOH
USBJO@MPTTFT<>
UFTU@MPTTFT<>
TFTTSVO
UGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
GPSUJOSBOHF
OC@JUFST
USBJO@MPTTUSBJO@CBUDI
CBUDI@TJ[F
USBJO@MPTTFTBQQFOE
USBJO@MPTT
JGU
5FTUFS
UFTU@MPTTUFTU@CBUDI
CBUDI@TJ[F
UFTU@MPTTFTBQQFOE
UFTU@MPTT
QSJOU
4UFQ\^\^USBJOMPTT\^=U5&45MPTT\^GPSNBU
U
OC@JUFSTUSBJO@MPTTUFTU@MPTT
QSJOU
'JOUSBJOMPTT\^=U5&45MPTT\^GPSNBU
USBJO@MPTTUFTU@MPTT
5FTU
OC@QSFEJDUJPOT
QSJOU
-FU TWJTVBMJ[F\^QSFEJDUJPOTXJUIPVS
TJHOBMTGPSNBU
OC@QSFEJDUJPOT
9:HFOFSBUF@Y@Z@EBUB
JT5SBJO'BMTFCBUDI@TJ[FOC@QSFEJDUJPOT
GFFE@EJDU\FOD@JOQ<U>9<U>GPSUJOSBOHF
TFR@MFOHUI^
PVUQVUTOQBSSBZ
TFTTSVO
<SFTIBQFE@PVUQVUT>GFFE@EJDU<>
GPSKJOSBOHF
OC@QSFEJDUJPOT
QMUGJHVSF
GJHTJ[F
GPSLJOSBOHF
PVUQVU@EJN
QBTU9<KL>
FYQFDUFE:<KL>
QSFEPVUQVUT<KL>
MBCFM4FFO
QBTUWBMVFTJGLFMTF@OPMFHFOE@
MBCFM5SVFGVUVSFWBMVFTJGLFMTF@OPMFHFOE@
MBCFM1SFEJDUJPOTJGLFMTF@OPMFHFOE@
QMUQMPU
SBOHF
MFO
QBTUQBTUPCMBCFMMBCFM
QMUQMPU
SBOHF
MFO
QBTUMFO
FYQFDUFE
MFO
QBTUFYQFDUFEY
CMBCFMMBCFM
QMUQMPU
SBOHF
MFO
QBTUMFO
QSFE
MFO
QBTUQSFEPZ
[ 242 ]
Recurrent Neural Networks Chapter 6
MBCFMMBCFM
QMUMFHFOE
MPD CFTU
QMUUJUMF
1SFEJDUJPOTWTUSVFWBMVFT
QMUTIPX
[ 243 ]
Recurrent Neural Networks Chapter 6
#PGZCORNGQHDKVEQKPXCNWGRTGFKEVKQP
How it works...
An encoder-decoder layer stacked RNN with GRU basic cells is used to predict Bitcoin
value. RNNs are very good at learning sequences, and indeed the Bitcoin prediction is
reasonably accurate even with a simple model based on 2 layers and 12 GRU cells. Of
course, this prediction code is not an encouragement to invest in Bitcoin but only to discuss
a deep learning methodology. Moreover, more experiments are needed in order to verify
whether or not we have a situation of data overfitting.
[ 244 ]
Recurrent Neural Networks Chapter 6
There's more...
Predicting stock market values is a cool RNN application and there are a number of handy
packages available, for instance:
#PGZCORNGQH400UGSWGPEGUCUUGGPKPJVVRMCTRCVJ[IKVJWDKQTPPGbGEVKXGPGUU
[ 245 ]
Recurrent Neural Networks Chapter 6
How to do it...
We proceed with the recipe as follows:
1. If you want to create a one-to-one mapping, this is not an RNN but instead a
dense layer. Suppose to have a model already defined and you want to add a
Dense network. Then this is easily implemented in Keras:
NPEFM4FRVFOUJBM
NPEFMBEE
%FOTF
PVUQVU@TJ[FJOQVU@TIBQFJOQVU@TIBQF
NPEFM4FRVFOUJBM
NPEFMBEE
3FQFBU7FDUPS
OVNCFS@PG@UJNFTJOQVU@TIBQFJOQVU@TIBQF
NPEFMBEE
-45.
PVUQVU@TJ[FSFUVSO@TFRVFODFT5SVF
3. If you want to create a many-to-one option, this can be achieved with the
following LSTM code snippet:
NPEFM4FRVFOUJBM
NPEFMBEE
-45.
JOQVU@TIBQF
UJNFTUFQTEBUB@EJN
4. If you want to create a many-to-many option, this can be achieved with the
following LSTM code snippet when length of input and output matches the
number of recurrent steps:
NPEFM4FRVFOUJBM
NPEFMBEE
-45.
JOQVU@TIBQF
UJNFTUFQTEBUB@EJN
SFUVSO@TFRVFODFT5SVF
How it works...
Keras allows you to easily write various shapes of RNNs, including one-to-one, one-to-
many, many-to-one, and many-to-many mappings. The above examples explain how easy
is to implement them with Keras.
[ 246 ]
Unsupervised Learning
7
All the models that we have covered in this book up to now were based on the supervised
learning paradigm; the training dataset included the input and the desired label of that
input. This chapter, in contrast, focuses on the unsupervised learning paradigm. The
chapter will include the following topics:
Introduction
In machine learning, there are three different learning paradigms: supervised learning,
unsupervised learning, and reinforcement learning.
In supervised learning, also known as learning with a teacher, the network is provided
with both the inputs and the respective desired outputs. For example, in the MNIST dataset,
each image of the handwritten digit has a label signifying the digit value associated with it.
In reinforcement learning, also known as learning with a critic, the network is not provided
with the desired output; instead, the environment provides a feedback in terms of reward
or punishment. When its output is correct, the environment rewards the network, and when
the output is not correct, the environment punishes it.
Unsupervised Learning Chapter 7
Getting ready
PCA reduces the n-dimensional input data to r-dimensional input data, where r<n. In
simpler terms, PCA involves translating the origin and performing rotation of the axis such
that one of the axes (the principal axis) has the least variance with data points. A reduced
dimension dataset is obtained from the original dataset by performing this transformation
and then dropping (removing) the orthogonal axes with high variance. Here, we employ
the SVD method for PCA dimensionality reduction. Consider X, the n-dimensional data,
with p points Xp,n. Any real (pen) matrix can be decomposed as follows:
X = U b VT
Here, U and V are orthonormal matrices (that is, U.UT = VT.V= 1) of size p e n and n e n
respectively. b is a diagonal matrix of size n e n. Next, slice the b matrix to r columns
resulting in br; using U and V, we find the reduced dimension data points Yr:
Yr = Ubr
The code presented here has been adapted from this GitHub link:
IUUQTHJUIVCDPNFMJPSD.FEJVNCMPCNBTUFS1$"U4/&"&JQZOC
[ 248 ]
Unsupervised Learning Chapter 7
How to do it...
We proceed with the recipes as follows:
1. Import the modules that are required. We will be definitely using TensorFlow;
we will also need OVNQZ for some elementary matrix calculation, and
NBUQMPUMJC, NQM@UPPMLJU, and TFBCPSO for the plotting:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNNQM@UPPMLJUTNQMPUEJNQPSU"YFT%
JNQPSUTFBCPSOBTTOT
NBUQMPUMJCJOMJOF
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB
3. We define a class, 5'@1$", which will implement all our work. The class is
initialized as follows:
EFG@@JOJU@@
TFMGEBUBEUZQFUGGMPBU
TFMG@EBUBEBUB
TFMG@EUZQFEUZQF
TFMG@HSBQI/POF
TFMG@9/POF
TFMG@V/POF
TFMG@TJOHVMBS@WBMVFT/POF
TFMG@TJHNB/POF
4. The SVD of the given input data is calculated in the GJU method. The method
defines the computational graph and executes it to calculate the singular values
and the orthonormal matrix U. It takes TFMGEBUB to feed the placeholder
TFMG@9. UGTWE returns s (TJOHVMBS@WBMVFT) in shape [..., p] in descending
order of magnitude. We use UGEJBH to convert it to a diagonal matrix:
EFGGJU
TFMG
TFMG@HSBQIUG(SBQI
XJUITFMG@HSBQIBT@EFGBVMU
TFMG@9UGQMBDFIPMEFS
TFMG@EUZQF
TIBQFTFMG@EBUBTIBQF
[ 249 ]
Unsupervised Learning Chapter 7
1FSGPSN47%
TJOHVMBS@WBMVFTV@UGTWE
TFMG@9
$SFBUFTJHNBNBUSJY
TJHNBUGEJBH
TJOHVMBS@WBMVFT
XJUIUG4FTTJPO
HSBQITFMG@HSBQIBTTFTTJPO
TFMG@VTFMG@TJOHVMBS@WBMVFTTFMG@TJHNB
TFTTJPOSVO
<VTJOHVMBS@WBMVFTTJHNB>GFFE@EJDU\TFMG@9
TFMG@EBUB^
5. Now that we have the sigma matrix, the orthonormal U matrix, and the singular
values, we calculate the reduced dimension data by defining the SFEVDF method.
The method requires one of the two input arguments, O@EJNFOTJPOT or
LFFQ@JOGP. The O@EJNFOTJPOT argument represents the number of dimensions
we want to keep in the reduced dimension dataset. The LFFQ@JOGP argument, on
the other hand, decides the percentage of the information that we intend to keep
(a value of 0.8 means that we want to keep 80 percent of the original data). The
method creates a graph that slices the Sigma matrix and calculates the reduced
dimension dataset Yr:
EFGSFEVDF
TFMGO@EJNFOTJPOT/POFLFFQ@JOGP/POF
JGLFFQ@JOGP
/PSNBMJ[FTJOHVMBSWBMVFT
OPSNBMJ[FE@TJOHVMBS@WBMVFTTFMG@TJOHVMBS@WBMVFT
TVN
TFMG@TJOHVMBS@WBMVFT
JOGPSNBUJPOQFSEJNFOTJPO
JOGPOQDVNTVN
OPSNBMJ[FE@TJOHVMBS@WBMVFT
(FUUIFGJSTUJOEFYXIJDIJTBCPWFUIFHJWFOJOGPSNBUJPO
UISFTIPME
JUJUFS
JEYGPSJEYWBMVFJOFOVNFSBUF
JOGPJGWBMVF
LFFQ@JOGP
O@EJNFOTJPOTOFYU
JU
XJUITFMGHSBQIBT@EFGBVMU
$VUPVUUIFSFMFWBOUQBSUGSPNTJHNB
TJHNBUGTMJDF
TFMG@TJHNB<>
<TFMG@EBUBTIBQF<>O@EJNFOTJPOT>
1$"
QDBUGNBUNVM
TFMG@VTJHNB
XJUIUG4FTTJPO
HSBQITFMG@HSBQIBTTFTTJPO
SFUVSOTFTTJPOSVO
QDBGFFE@EJDU\TFMG@9
TFMG@EBUB^
[ 250 ]
Unsupervised Learning Chapter 7
6. Our 5'@1$" class is ready. Now, we will use it to reduce the MNIST data from
each input being of dimension 784 (28 x 28) to new data with each point of
dimension 3. Here, we retained only 10 percent of the information for better
viewing, but normally you would need to retain roughly 80 percent of the
information:
UG@QDBGJU
QDBUG@QDBSFEVDF
LFFQ@JOGP5IFSFEVDFEEJNFOTJPOT
EFQFOEFOUVQPOUIFPGJOGPSNBUJPO
QSJOU
PSJHJOBMEBUBTIBQF NOJTUUSBJOJNBHFTTIBQF
QSJOU
SFEVDFEEBUBTIBQF QDBTIBQF
7. Let's now plot the 55,000 data points in the three-dimensional space:
4FUTOTDPMPS@QBMFUUF
4FU
DPMPS@NBQQJOH\LFZWBMVFGPS
LFZWBMVFJOFOVNFSBUF
4FU^
DPMPSTMJTU
NBQ
MBNCEBYDPMPS@NBQQJOH<Y>NOJTUUSBJOMBCFMT
GJHQMUGJHVSF
BY"YFT%
GJH
BYTDBUUFS
QDB<>QDB<>QDB<>DDPMPST
[ 251 ]
Unsupervised Learning Chapter 7
How it works...
The preceding code performs dimensionality reduction on the MNIST images. Each original
image is of size 28 x 28; using the PCA method, we can reduce it to a smaller size. Normally,
for image data, dimensionality reduction is necessary. This is so because images are large in
size and contain a significant amount of redundant data.
There's more...
TensorFlow provides a technique called embeddings, which is a mapping of objects into
vectors. Embedding Projector of TensorBoard allows us to interactively visualize
embeddings from our model. Embedding Projector provides three ways for dimensionality
reduction: PCA, t-SNE, and Custom. We can use the Embedding Projector of TensorBoard
to achieve a similar result as the previous one. We will need to import the QSPKFDUPS class
from UFOTPSGMPXDPOUSJCUFOTPSCPBSEQMVHJOT to do the same from
UFOTPSGMPXDPOUSJCUFOTPSCPBSEQMVHJOT import QSPKFDUPS. We can do it in three
simple steps:
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
'MNIST_data'
JNBHFTUG7BSJBCMF
NOJTUUFTUJNBHFTOBNF'images'
with PQFO
NFUBEBUB'w'as NFUBEBUB@GJMF
for SPXin NOJTUUFTUMBCFMT
NFUBEBUB@GJMFXSJUF
'%d\n' SPX
with UG4FTTJPO
as TFTT
TBWFSUGUSBJO4BWFS
<JNBHFT>
TFTTSVO
JNBHFTJOJUJBMJ[FS
TBWFSTBWF
TFTTPTQBUIKPJO
-0(@%*3'images.ckpt'
DPOGJHQSPKFDUPS1SPKFDUPS$POGJH
# One can add multiple embeddings.
FNCFEEJOHDPOGJHFNCFEEJOHTBEE
FNCFEEJOHUFOTPS@OBNFJNBHFTOBNF
# Link this tensor to its metadata file (e.g. labels).
[ 252 ]
Unsupervised Learning Chapter 7
FNCFEEJOHNFUBEBUB@QBUINFUBEBUB
# Saves a config file that TensorBoard will read during
startup.
QSPKFDUPSWJTVBMJ[F@FNCFEEJOHT
UGTVNNBSZ'JMF8SJUFS
-0(@%*3
DPOGJH
The embeddings are ready and can now be seen using TensorBoard. Launch TensorBoard
through CLI UFOTPSCPBSEMPHEJSMPH, open TensorBoard in your web browser, and
go to the EMBEDDINGS tab. Here is the TensorBoard projection using PCA with the first
three principal components as the axis:
See also
IUUQTBSYJWPSHBCT
IUUQXXXDTPUBHPBDO[DPTDTUVEFOU@UVUPSJBMTQSJODJQBM@
DPNQPOFOUTQEG
IUUQNQMBCVDTEFEVUVUPSJBMTQDBQEG
IUUQQSPKFDUPSUFOTPSGMPXPSH
[ 253 ]
Unsupervised Learning Chapter 7
k-means clustering
The k-means clustering, as the name suggests, is a technique to cluster data, that is, partition
data into a specified number of data points. It is an unsupervised learning technique. It
works by identifying patterns in the given data. Remember the sorting hat from Harry
Potter fame? What it is doing in the book is clustering--dividing new (unlabeled) students
into four different clusters: Gryffindor, Ravenclaw, Hufflepuff, and Slytherin.
Humans are very good at grouping objects together; clustering algorithms try to give a
similar capability to computers. There are many clustering techniques available such as
Hierarchical, Bayesian, or Partional. The k-means clustering belongs to Partional clustering;
it partitions the data into k clusters. Each cluster has a center, called centroid. The number
of clusters k has to be specified by the user.
Getting ready
We will use TensorFlow ,NFBOT$MVTUFSJOH Estimator class to implement k-means. It is
defined in IUUQTHJUIVCDPNUFOTPSGMPXUFOTPSGMPXCMPCSUFOTPSGMPXDPOUSJC
MFBSOQZUIPOMFBSOFTUJNBUPSTLNFBOTQZ It creates a model to run k-means and
inference. According to the TensorFlow docs, once a ,NFBOT$MVTUFSJOH class object is
created, it is instantiated with the following @@JOJU@@ method:
@@JOJU@@
OVN@DMVTUFST
NPEFM@EJS/POF
JOJUJBM@DMVTUFST3"/%0.@*/*5
EJTUBODF@NFUSJD426"3&%@&6$-*%&"/@%*45"/$&
SBOEPN@TFFE
VTF@NJOJ@CBUDI5SVF
NJOJ@CBUDI@TUFQT@QFS@JUFSBUJPO
LNFBOT@QMVT@QMVT@OVN@SFUSJFT
SFMBUJWF@UPMFSBODF/POF
[ 254 ]
Unsupervised Learning Chapter 7
DPOGJH/POF
Args:
num_clusters: The number of clusters to train.
model_dir: The directory to save the model results and log files.
initial_clusters: Specifies how to initialize the clusters for training. See
clustering_ops.kmeans for the possible values.
distance_metric: The distance metric used for clustering. See
clustering_ops.kmeans for the possible values.
random_seed: Python integer. Seed for PRNG used to initialize centers.
use_mini_batch: If true, use the mini-batch k-means algorithm. Or else
assume full batch.
mini_batch_steps_per_iteration: The number of steps after which the
updated cluster centers are synced back to a master copy. See
clustering_ops.py for more details.
kmeans_plus_plus_num_retries: For each point that is sampled during
kmeans++ initialization, this parameter specifies the number of additional
points to draw from the current distribution before selecting the best. If a
negative value is specified, a heuristic is used to sample
O(log(num_to_sample)) additional points.
relative_tolerance: A relative tolerance of change in the loss between
iterations. Stops learning if the loss changes less than this amount. Note
that this may not work correctly if use_mini_batch=True.
config: See Estimator.
TensorFlow supports Euclidean distance and cosine distance as the measure of the centroid.
TensorFlow ,NFBOT$MVTUFSJOH provides various methods to interact with the
,NFBOT$MVTUFSJOH object. In the present recipe, we will use GJU
, DMVTUFST
, and
QSFEJDU@DMVTUFST@JEY
methods:
GJU
Y/POF
Z/POF
JOQVU@GO/POF
TUFQT/POF
CBUDI@TJ[F/POF
NPOJUPST/POF
NBY@TUFQT/POF
[ 255 ]
Unsupervised Learning Chapter 7
How to do it...
Here is how we proceed with the recipe:
JNQPSUOVNQZBTOQ
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNNBUQMPUMJCDPMPSTJNQPSU-JTUFE$PMPSNBQ
EBUBTFU*SJT
GSPNTLMFBSOJNQPSUEBUBTFUT
NBUQMPUMJCJOMJOF
JNQPSUTPNFEBUBUPQMBZXJUI
JSJTEBUBTFUTMPBE@JSJT
YJSJTEBUB<>XFPOMZUBLFUIFGJSTUUXPGFBUVSFT
ZJSJTUBSHFU
PSJHJOBMEBUBXJUIPVUDMVTUFSJOH
QMUTDBUUFS
IX@GSBNF<>IX@GSBNF<>
QMUYMBCFM
4FQJB-FOHUI
QMUZMBCFM
4FQJB8JEUI
[ 256 ]
Unsupervised Learning Chapter 7
4. We can see that there are no clusters obviously visible in the data. Now we define
JOQVU@GO, which will be used to feed the GJU method. Our input function
returns a TensorFlow constant, which is assigned the value and shape of x, and is
of the type GMPBU:
EFGJOQVU@GO
SFUVSOUGDPOTUBOU
OQBSSBZ
YUGGMPBUYTIBQF/POF
5. Now we use the ,NFBOT$MVTUFSJOH class; here, we already know the number of
classes as 3, so we set OVN@DMVTUFST. Normally, we are not aware of what the
number of clusters should be; in that case, a common methodology is the elbow
method:
LNFBOTUGDPOUSJCMFBSO,.FBOT$MVTUFSJOH
OVN@DMVTUFST
SFMBUJWF@UPMFSBODFSBOEPN@TFFE
LNFBOTGJU
JOQVU@GOJOQVU@GO
DMVTUFSTLNFBOTDMVTUFST
BTTJHONFOUTMJTU
LNFBOTQSFEJDU@DMVTUFS@JEFY
JOQVU@GOJOQVU@GO
[ 257 ]
Unsupervised Learning Chapter 7
EFG4DBUUFS1MPU
9:BTTJHONFOUT/POFDFOUFST/POF
JGBTTJHONFOUTJT/POF
BTTJHONFOUT<>MFO
9
GJHQMUGJHVSF
GJHTJ[F
DNBQ-JTUFE$PMPSNBQ
< SFE HSFFO CMVF >
QMUTDBUUFS
9:DBTTJHONFOUTDNBQDNBQ
JGDFOUFSTJTOPU/POF
QMUTDBUUFS
DFOUFST<>DFOUFST<>DSBOHF
MFO
DFOUFST
NBSLFS
TDNBQDNBQ
QMUYMBCFM
4FQJB-FOHUI
QMUZMBCFM
4FQJB8JEUI
[ 258 ]
Unsupervised Learning Chapter 7
How it works...
The preceding recipe used the k-means clustering estimator of TensorFlow to cluster the
given data into clusters. Here, as we know the number of clusters, we decided to keep
OVN@DMVTUFST, but in most cases with unlabeled data, one is never sure how many
clusters exist. One can determine the optimal number of clusters using the elbow method.
The method is based on the principle that we should choose the cluster number that reduces
that sum of squared error (SSE) distance. If k is the number of clusters, then as k increases,
the SSE decreases, with SSE = 0; when k is equal to the number of data points, each point is
its own cluster. We want a low value of k such that SSE is also low. In TensorFlow, we can
find the SSE using the TDPSF
method defined in the ,NFBOT$MVTUFSJOH class; the
method returns the total sum of distances to the nearest clusters:
TVN@EJTUBODFTLNFBOTTDPSF
JOQVU@GOJOQVU@GOTUFQT
For Iris data, if we plot SSE for different k values, we see that for k=3, the variance in SSE is
the highest; after that, it starts reducing, thus the elbow point is k=3:
There's more...
K-means clustering is very popular because it is fast, simple, and robust. It also has some
disadvantages: the biggest being that the user has to specify the number of clusters. Second,
the algorithm does not guarantee global optima. Third, it is very sensitive to outliers.
[ 259 ]
Unsupervised Learning Chapter 7
See also
Kanungo, Tapas, et al. An efficient k-means clustering algorithm: Analysis and
implementation. IEEE transactions on pattern analysis and machine intelligence
24.7 (2002): 881-892.
IUUQIPNFEFJCQPMJNJJUNBUUFVDD$MVTUFSJOHUVUPSJBM@IUNMLNFBOT
IUNM
Chen, Ke. On coresets for k-median and k-means clustering in metric and euclidean
spaces and their applications. SIAM Journal on Computing 39.3 (2009): 923-947.
IUUQTFOXJLJQFEJBPSHXJLJ%FUFSNJOJOH@UIF@OVNCFS@PG@DMVTUFST@JO@B@
EBUB@TFU
Self-organizing maps
Self-organized maps (SOM), sometimes known as Kohonen networks or Winner take all
units (WTU), are a very special kind of neural network, motivated by a distinctive feature
of the human brain. In our brain, different sensory inputs are represented in a topologically
ordered manner. Unlike other neural networks, neurons are not all connected to each other
via weights, instead, they influence each other's learning. The most important aspect of
SOM is that neurons represent the learned inputs in a topographic manner.
In SOM, neurons are usually placed at nodes of a (1D or 2D) lattice. Higher dimensions are
also possible but are rarely used in practice. Each neuron in the lattice is connected to all the
input units via weight matrix. Here, you can see a SOM with 3 x 4 (12 neurons) and seven
inputs. For clarity, only the weight vectors connecting all inputs to one neuron are shown.
In this case, each neuron will have seven elements, resulting in a combined weight matrix of
size (12 x 7):
[ 260 ]
Unsupervised Learning Chapter 7
Getting ready
In order to implement SOM, let's first understand how it works. As the first step, the
weights of the network are initialized to either some random value or by taking random
samples from the input. Each neuron occupying a space in the lattice will be assigned
specific locations. Now as an input is presented, the neuron with the least distance from the
input is declared Winner (WTU). This is done by measuring the distance between weight
vector (W) and input vectors (X) of all neurons:
Here, dj is the distance of weights of neuron j from input X. The neuron with minimum d
value is the winner.
[ 261 ]
Unsupervised Learning Chapter 7
Next, the weights of the winning neuron and its neighboring neurons are adjusted in a
manner to ensure that the same neuron is the winner if the same input is presented next
time. To decide which neighboring neurons need to be modified, the network uses a
neighborhood function f(r); normally, the Gaussian Mexican hat function is chosen as a
neighborhood function. The neighborhood function is mathematically represented as
follows:
Here, d is a time-dependent radius of influence of a neuron and d is its distance from the
winning neuron:
Another important property of the neighborhood function is that its radius reduces with
time. As a result, in the beginning, many neighboring neurons' weights are modified, but as
the network learns, eventually a few neurons' weights (at times, only one or none) are
modified in the learning process. The change in weight is given by the following equation:
E8= g(X-W)
We continue the process for all inputs and repeat it for a given number of iterations. As the
iterations progress, we reduce the learning rate and the radius by a factor dependent on the
iteration number.
[ 262 ]
Unsupervised Learning Chapter 7
How to do it...
We proceed with the recipe as follows:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
NBUQMPUMJCJOMJOF
2. Next, we declare a class, WTU, which will perform all the tasks. The class is
instantiated with NYO the size of the 2D SOM lattice, EJN the number of
dimensions in the input data, and the total number of iterations:
EFG@@JOJU@@
TFMGNOEJNOVN@JUFSBUJPOTFUBTJHNB
/POF
NYO5IFEJNFOTJPOPG%MBUUJDFJOXIJDIOFVSPOTBSF
BSSBOHFE
EJN%JNFOTJPOPGJOQVUUSBJOJOHEBUB
OVN@JUFSBUJPOT5PUBMOVNCFSPGUSBJOJOHJUFSBUJPOT
FUB-FBSOJOHSBUF
TJHNB5IFSBEJVTPGOFJHICPVSIPPEGVODUJPO
TFMG@NN
TFMG@OO
TFMG@OFJHICPVSIPPE<>
TFMG@UPQPHSBQIZ<>
TFMG@OVN@JUFSBUJPOTJOU
OVN@JUFSBUJPOT
TFMG@MFBSOFE'BMTF
3. In the @@JOJU@@ itself, we define the computation graph and the session.
4. If the network is not provided with any TJHNB value, it will take a default value,
which is normally half of the maximum dimension of the SOM lattice:
JGTJHNBJT/POF
TJHNBNBY
NO$POTUBOUSBEJVT
FMTF
TJHNBGMPBU
TJHNB
[ 263 ]
Unsupervised Learning Chapter 7
5. Next, in the graph, we declare variables for the weight matrix, placeholders for
the input, and computational steps to calculate the winner and update its and its
neighbors' weight. As SOM provides a topographic map, we also add ops to get
the topographic location of neurons:
TFMG@HSBQIUG(SBQI
#VJME$PNQVUBUJPO(SBQIPG40.
XJUITFMG@HSBQIBT@EFGBVMU
8FJHIU.BUSJYBOEUIFUPQPHSBQIZPGOFVSPOT
TFMG@8UG7BSJBCMF
UGSBOEPN@OPSNBM
<NOEJN>TFFE
TFMG@UPQPHSBQIZ
UGDPOTUBOU
OQBSSBZ
MJTU
TFMG@OFVSPO@MPDBUJPO
NO
1MBDFIPMEFSTGPSUSBJOJOHEBUB
TFMG@9UGQMBDFIPMEFS
GMPBU <EJN>
1MBDFIPMEFSUPLFFQUSBDLPGOVNCFSPGJUFSBUJPOT
TFMG@JUFSUGQMBDFIPMEFS
GMPBU
'JOEJOHUIF8JOOFSBOEJUTMPDBUJPO
EUGTRSU
UGSFEVDF@TVN
UGQPX
TFMG@8UGTUBDL
<TFMG@9
GPSJJOSBOHF
NO>
TFMG856@JEYUGBSHNJO
E
TMJDF@TUBSUUGQBE
UGSFTIBQF
TFMG856@JEY
<>OQBSSBZ
<<>>
TFMG856@MPDUGSFTIBQF
UGTMJDF
TFMG@UPQPHSBQIZ
TMJDF@TUBSU<><>
$IBOHFMFBSOJOHSBUFBOESBEJVTBTBGVODUJPOPGJUFSBUJPOT
MFBSOJOH@SBUFTFMG@JUFSTFMG@OVN@JUFSBUJPOT
@FUB@OFXFUBMFBSOJOH@SBUF
@TJHNB@OFXTJHNBMFBSOJOH@SBUF
$BMDVMBUJOH/FJHICPVSIPPEGVODUJPO
EJTUBODF@TRVBSFUGSFEVDF@TVN
UGQPX
UGTVCUSBDU
TFMG@UPQPHSBQIZUGTUBDL
<TFMG856@MPDGPSJJOSBOHF
N
O>
OFJHICPVSIPPE@GVODUGFYQ
UGOFHBUJWF
UGEJW
UGDBTU
EJTUBODF@TRVBSFGMPBUUGQPX
@TJHNB@OFX
NVMUJQMZMFBSOJOHSBUFXJUIOFJHICPVSIPPEGVOD
FUB@JOUP@(BNNBUGNVMUJQMZ
@FUB@OFXOFJHICPVSIPPE@GVOD
4IBQFJUTPUIBUJUDBOCFNVMUJQMJFEUPDBMDVMBUFE8
XFJHIU@NVMUJQMJFSUGTUBDL
<UGUJMF
UGTMJDF
FUB@JOUP@(BNNBOQBSSBZ
<J>OQBSSBZ
<><EJN>
GPSJJOSBOHF
NO>
EFMUB@8UGNVMUJQMZ
XFJHIU@NVMUJQMJFS
UGTVCUSBDU
UGTUBDL
<TFMG@9GPSJJOSBOHF
NO>TFMG@8
OFX@8TFMG@8
EFMUB@8
TFMG@USBJOJOHUGBTTJHO
TFMG@8OFX@8
[ 264 ]
Unsupervised Learning Chapter 7
*OJUJBMJ[F"MMWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFMG@TFTTUG4FTTJPO
TFMG@TFTTSVO
JOJU
6. We define a GJU method for the class, which executes the training op declared in
the default graph of the class. The method also calculates centroid grid:
EFGGJU
TFMG9
'VODUJPOUPDBSSZPVUUSBJOJOH
GPSJJOSBOHF
TFMG@OVN@JUFSBUJPOT
GPSYJO9
TFMG@TFTTSVO
TFMG@USBJOJOHGFFE@EJDU\TFMG@9Y
TFMG@JUFSJ^
4UPSFBDFOUSPJEHSJEGPSFBTZSFUSFJWBM
DFOUSPJE@HSJE<<>GPSJJOSBOHF
TFMG@N>
TFMG@8UTMJTU
TFMG@TFTTSVO
TFMG@8
TFMG@MPDBUJPOTMJTU
TFMG@TFTTSVO
TFMG@UPQPHSBQIZ
GPSJMPDJOFOVNFSBUF
TFMG@MPDBUJPOT
DFOUSPJE@HSJE<MPD<>>BQQFOE
TFMG@8UT<J>
TFMG@DFOUSPJE@HSJEDFOUSPJE@HSJE
TFMG@MFBSOFE5SVF
7. We define a function to determine the index and the location of the winning
neuron in the 2D lattice:
EFGXJOOFS
TFMGY
JEYTFMG@TFTTSVO
<TFMG856@JEYTFMG856@MPD>GFFE@EJDU
\TFMG@9Y^
SFUVSOJEY
EFG@OFVSPO@MPDBUJPO
TFMGNO
'VODUJPOUPHFOFSBUFUIF%MBUUJDFPGOFVSPOT
GPSJJOSBOHF
N
GPSKJOSBOHF
O
ZJFMEOQBSSBZ
<JK>
[ 265 ]
Unsupervised Learning Chapter 7
EFGHFU@DFOUSPJET
TFMG
'VODUJPOUPSFUVSOBMJTUPG N MJTUTXJUIFBDIJOOFS
MJTUDPOUBJOJOHUIF O DPSSFTQPOEJOHDFOUSPJEMPDBUJPOTBT%
/VN1ZBSSBZT
JGOPUTFMG@MFBSOFE
SBJTF7BMVF&SSPS
40.OPUUSBJOFEZFU
SFUVSOTFMG@DFOUSPJE@HSJE
EFGNBQ@WFDUT
TFMG9
'VODUJPOUPNBQFBDIJOQVUWFDUPSUPUIFSFMFWBOUOFVSPO
JOUIFMBUUJDF
JGOPUTFMG@MFBSOFE
SBJTF7BMVF&SSPS
40.OPUUSBJOFEZFU
UP@SFUVSO<>
GPSWFDUJO9
NJO@JOEFYNJO
<JGPSJJOSBOHF
MFO
TFMG@8UT>
LFZMBNCEBYOQMJOBMHOPSN
WFDU
TFMG@8UT<Y>
UP@SFUVSOBQQFOE
TFMG@MPDBUJPOT<NJO@JOEFY>
SFUVSOUP@SFUVSO
9. Now that our WTU class is ready, we read data from the DTW file and normalize
it:
EFGOPSNBMJ[F
EG
SFTVMUEGDPQZ
GPSGFBUVSF@OBNFJOEGDPMVNOT
NBY@WBMVFEG<GFBUVSF@OBNF>NBY
NJO@WBMVFEG<GFBUVSF@OBNF>NJO
SFTVMU<GFBUVSF@OBNF>
EG<GFBUVSF@OBNF>NJO@WBMVF
NBY@WBMVFNJO@WBMVF
SFUVSOSFTVMU
3FBEJOHJOQVUEBUBGSPNGJMF
JNQPSUQBOEBTBTQE
EGQESFBE@DTW
DPMPSTDTW 5IFMBTUDPMVNOPGEBUBGJMFJTB
MBCFM
EBUBOPSNBMJ[F
EG<< 3 ( # >>WBMVFT
OBNFEG< $PMPS/BNF >WBMVFT
O@EJNMFO
EGDPMVNOT
%BUBGPS5SBJOJOH
DPMPSTEBUB
[ 266 ]
Unsupervised Learning Chapter 7
DPMPS@OBNFTOBNF
10. Finally, we use our class to perform dimensionality reduction and arrange it in a
beautiful topographic map:
TPN856
O@EJNTJHNB
TPNGJU
DPMPST
(FUPVUQVUHSJE
JNBHF@HSJETPNHFU@DFOUSPJET
.BQDPMPVSTUPUIFJSDMPTFTUOFVSPOT
NBQQFETPNNBQ@WFDUT
DPMPST
1MPU
QMUJNTIPX
JNBHF@HSJE
QMUUJUMF
$PMPS(SJE40.
GPSJNJOFOVNFSBUF
NBQQFE
QMUUFYU
N<>N<>DPMPS@OBNFT<J>IB DFOUFS WB DFOUFS
CCPYEJDU
GBDFDPMPS XIJUF BMQIBMX
How it works...
SOMs are computationally expensive and thus are not really useful for a very large dataset.
Still, they are easy to understand and they can very nicely find the similarity between input
data. Thus, they have been employed for image segmentation and to determine word
similarity maps in NLP.
[ 267 ]
Unsupervised Learning Chapter 7
See also
This is a very good blog post, explaining SOM in simple language: IUUQXXX
BJKVOLJFDPNBOOTPNTPNIUNM
A brief introduction on SOM: IUUQTFOXJLJQFEJBPSHXJLJ4FMG
PSHBOJ[JOH@NBQ
The seminal paper by Kohonen on SOM: "The self-organizing map."
Neurocomputing 21.1 (1998): 1-6: IUUQTQEGTTFNBOUJDTDIPMBSPSHDB
BFBFGEFECDBGQEG
5QWTEG$[3YGTV[WU1YPYQTM%%$;5#JVVRUEQOOQPUYKMKOGFKCQTIYKPFGZRJREWTKF
[ 268 ]
Unsupervised Learning Chapter 7
RBM can be used for dimensionality reduction, feature extraction, and collaborative
filtering. The training in RBMs can be divided into three parts: forward pass, backward
pass, and then compare.
Getting ready
Let's see the expressions that we will need to make our RBM:
Forward pass: The information at visible units (V) is passed via weights (W) and biases (c)
to the hidden units (h0). The hidden unit may fire or not depending on the stochastic
probability (d is stochastic probability):
Backward pass: The hidden unit representation (h0) is then passed back to the visible units
through the same weights W, but different bias c, where they reconstruct the input. Again,
the input is sampled:
These two passes are repeated for k-steps or till the convergence is reached. According to
researchers, k=1 gives good results, so we will keep k = 1.
The joint configuration of the visible vector V and the hidden vector has an energy given as
follows:
Also associated with each visible vector V is free energy, the energy that a single
configuration would need to have in order to have the same probability as all of the
configurations that contain V:
[ 269 ]
Unsupervised Learning Chapter 7
Here, g is the learning rate. Similar expressions exist for the biases b and c.
How to do it...
We proceed with the recipe as follows:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
JNQPSUNBUQMPUMJCQZQMPUBTQMU
NBUQMPUMJCJOMJOF
2. Declare the class RBM, which will do the major task. The @@JOJU@@ will build the
complete graph, forward and backward pass, and the objective function; we will
use the TensorFlow built-in optimizer to update the weights and biases:
DMBTT3#.
PCKFDU
EFG@@JOJU@@
TFMGNO
N/VNCFSPGOFVSPOTJOWJTJCMFMBZFS
OOVNCFSPGOFVSPOTJOIJEEFOMBZFS
TFMG@NN
TFMG@OO
$SFBUFUIF$PNQVUBUJPOBMHSBQI
8FJHIUTBOECJBTFT
TFMG@8
UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
TFMG@NTFMG@O
TFMG@DUG7BSJBCMF
OQ[FSPT
TFMG@OBTUZQF
OQGMPBU
CJBTGPSIJEEFOMBZFS
TFMG@CUG7BSJBCMF
OQ[FSPT
TFMG@NBTUZQF
OQGMPBU
CJBTGPS7JTJCMFMBZFS
1MBDFIPMEFSGPSJOQVUT
TFMG@9UGQMBDFIPMEFS
GMPBU </POFTFMG@N>
'PSXBSE1BTT
@IUGOOTJHNPJE
UGNBUNVM
TFMG@9TFMG@8
TFMG@D
TFMGIUGOOSFMV
UGTJHO
@I
UGSBOEPN@VOJGPSN
UGTIBQF
@I
#BDLXBSEQBTT
@WUGOOTJHNPJE
UGNBUNVM
TFMGIUGUSBOTQPTF
TFMG@8
TFMG@C
TFMG7UGOOSFMV
UGTJHO
@W
UGSBOEPN@VOJGPSN
UGTIBQF
@W
[ 270 ]
Unsupervised Learning Chapter 7
0CKFDUJWF'VODUJPO
PCKFDUJWFUGSFEVDF@NFBO
TFMGGSFF@FOFSHZ
TFMG@9
UGSFEVDF@NFBO
TFMGGSFF@FOFSHZ
TFMG7
TFMG@USBJO@PQ
UGUSBJO(SBEJFOU%FTDFOU0QUJNJ[FS
FNJOJNJ[F
PCKFDUJWF
$SPTTFOUSPQZDPTU
SFDPOTUSVDUFE@JOQVUTFMGPOF@QBTT
TFMG@9
TFMGDPTU
UGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MBCFMTTFMG@9MPHJUTSFDPOTUSVDUFE@JOQVU
EFGGJU
TFMG9FQPDITCBUDI@TJ[F
/%9TIBQF
OVN@CBUDIFT/CBUDI@TJ[F
PCK<>
GPSJJOSBOHF
FQPDIT
9TIVGGMF
9
GPSKJOSBOHF
OVN@CBUDIFT
CBUDI9<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
@PCTFMGTFTTJPOSVO
<TFMG@USBJO@PQTFMGDPTU
>GFFE@EJDU\TFMG@9CBUDI^
JGK
QSJOU
USBJOJOHFQPDI\^DPTU
\^ GPSNBU
KPC
PCKBQQFOE
PC
SFUVSOPCK
4. There are other helper functions to calculate the logit error and return
reconstructed images from the network:
EFGTFU@TFTTJPO
TFMGTFTTJPO
TFMGTFTTJPOTFTTJPO
EFGGSFF@FOFSHZ
TFMG7
CUGSFTIBQF
TFMG@C
TFMG@N
UFSN@UGNBUNVM
7C
UFSN@UGSFTIBQF
UFSN@
UFSN@UGSFEVDF@TVN
UGOOTPGUQMVT
UGNBUNVM
7TFMG@8
TFMG@D
SFUVSOUFSN@
UFSN@
[ 271 ]
Unsupervised Learning Chapter 7
EFGPOF@QBTT
TFMG9
IUGOOTJHNPJE
UGNBUNVM
9TFMG@8
TFMG@D
SFUVSOUGNBUNVM
IUGUSBOTQPTF
TFMG@8
TFMG@C
EFGSFDPOTUSVDU
TFMG9
YUGOOTJHNPJE
TFMGPOF@QBTT
9
SFUVSOTFMGTFTTJPOSVO
YGFFE@EJDU\TFMG@99^
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUBPOF@IPU5SVF
US9US:UF9UF:NOJTUUSBJOJNBHFTNOJTUUSBJOMBCFMT
NOJTUUFTUJNBHFTNOJTUUFTUMBCFMT
9USBJOUS9BTUZQF
OQGMPBU
9UFTUUF9BTUZQF
OQGMPBU
@N9USBJOTIBQF
SCN3#.
N
*OJUJBMJ[FBMMWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
SCNTFU@TFTTJPO
TFTT
FSSSCNGJU
9USBJO
PVUSCNSFDPOTUSVDU
9FTU<>-FUVTSFDPOTUSVDU5FTU
%BUB
[ 272 ]
Unsupervised Learning Chapter 7
How it works...
Due to its ability to reconstruct images, RBM can be used to generate more data from the
existing data. We can see the original and reconstructed MNIST images by making a small
helper plotting code:
SPXDPM
JEYOQSBOEPNSBOEJOU
SPXDPM
GBYBSSQMUTVCQMPUT
SPXDPMTIBSFY5SVFTIBSFZ5SVFGJHTJ[F
GPSGJHSPXJO[JQ
<9UFTU@OPJTZPVU>BYBSS
GPSJBYJO[JQ
JEYSPX
BYJNTIPX
GJH<J>SFTIBQF
See also
Smolensky, Paul. Information processing in dynamical systems: Foundations of
harmony theory. No. CU-CS-321-86. COLORADO UNIV AT BOULDER DEPT OF
COMPUTER SCIENCE, 1986. (-JOL)
[ 273 ]
Unsupervised Learning Chapter 7
Getting ready
To make a recommender system using collaborative filtering, we need to modify our data.
As an illustration, we will use the movie dataset taken from IUUQTHSPVQMFOTPSH
EBUBTFUTNPWJFMFOT. The data consists of two EBU files: NPWJFTEBU and SBUJOHTEBU.
The NPWJFTEBU file contains three columns: MovieID, Title, and Genre for 3,883 movies.
The SBUJOHTEBU file contains four columns: UserID, MovieID, Rating, and Time. We need
to merge these two data files such that we are able to build an array, where, for each user,
we have a rating for all the 3,883 movies. The problem is that users normally do not rate all
movies, so we have non-zero (normalized) ratings only for some movies. The rest is made
zero and hence will not contribute to the hidden layer.
How to do it...
1. We will use the 3#. class created in the previous recipe. Let's define our RBM
network; the number of visible units will be the number of movies, in our case,
3,883 (NPWJFT@EG is the data frame containing data from the NPWJFTEBU file):
NMFO
NPWJFT@EG/VNCFSPGWJTJCMFVOJUT
O/VNCFSPG)JEEFOVOJUT
SFDPNNFOEFSSCN3#.
NO
[ 274 ]
Unsupervised Learning Chapter 7
2. We created a list, US9, of about 1,000 users with their normalized movie ratings
using Pandas merge and HSPVQCZ commands. The list is of size 1000 x 3883. We
use this to train our RBM:
9USBJOOQBSSBZ
US9
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
SFDPNNFOEFSTFU@TFTTJPO
TFTT
FSSSFDPNNFOEFSGJU
9USBJOFQPDIT
4. The network is now trained; we use it get the recommendation for a random user
with index 150 (it could be any existing user):
VTFS@JOEFY
YOQBSSBZ
<9USBJO<VTFS@JOEFY>>
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
SFDPNNFOEFSTFU@TFTTJPO
TFTT
PVUSFDPNNFOEFSSFDPOTUSVDU
YBTUZQF
OQGMPBU
[ 275 ]
Unsupervised Learning Chapter 7
5. The result is merged with the existing data frame and we can see the
recommendation score for this user:
There's more...
Prof Geoffrey Hinton led the University of Toronto team that won the Netflix competition
for the best collaborative filtering to predict user ratings for movies using RBM
The output from the hidden units of one RBM can be fed to visible units of another RBM,
the process can be repeated to form a stack of RBMs. This results in Stacked RBMs. Each
stacked RBM is trained independently assuming that others do not exist. A large number of
stacked RBMs form Deep Belief Networks (DBN). DBNs can be trained using both
supervised or unsupervised training. You will learn more about them in the next recipe.
[ 276 ]
Unsupervised Learning Chapter 7
Getting ready
We use the RBM class we have already created in the recipe Restricted Boltzmann Machine,
with just one change, we do not need to reconstruct the image after training now. Instead,
our stacked RBMs will be only forward passing the data up to the last MLP layer of DBN.
This is achieved by removing the SFDPOTUSVDU
function from the class, and replacing it
with the SCN@PVUQVU
function:
EFGSCN@PVUQVU
TFMG9
YUGOOTJHNPJE
UGNBUNVM
9TFMG@8
TFMG@D
SFUVSOTFMGTFTTJPOSVO
YGFFE@EJDU\TFMG@99^
For the data, we consider the Kaggle Face Emotion Recognition data, available at IUUQT
XXXLBHHMFDPNDDIBMMFOHFTJOSFQSFTFOUBUJPOMFBSOJOHGBDJBMFYQSFTTJPO
SFDPHOJUJPODIBMMFOHF. The data description given there is:
The data consists of 48 x 48 pixel grayscale images of faces. The faces have been
automatically registered so that the face is more or less centered and occupies about the
same amount of space in each image. The task is to categorize each face based on the
emotion shown in the facial expression into one of seven categories (0=Angry, 1=Disgust,
2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral).
train.csv contains two columns, "emotion" and "pixels". The "emotion" column contains
a numeric code ranging from 0 to 6, inclusive, for the emotion that is present in the image.
The "pixels" column contains a string surrounded in quotes for each image. The contents
of this string are space-separated pixel values in row major order. test.csv contains only the
"pixels" column and your task is to predict the emotion column.
The training set consists of 28,709 examples. The public test set used for the leaderboard
consists of 3,589 examples. The final test set, which was used to determine the winner of
the competition, consists of another 3,589 examples.
This dataset was prepared by Pierre-Luc Carrier and Aaron Courville, as part of an
ongoing research project. They have graciously provided the workshop organizers with a
preliminary version of their dataset to use for this contest.
The complete data is in one .DTW file called GFSDTW. We separate out training,
validation, and test data from this:
EBUBQESFBE@DTW
EBUBGFSDTW
US@EBUBEBUB<EBUB6TBHF5SBJOJOH>
UFTU@EBUBEBUB<EBUB6TBHF1VCMJD5FTU>
NBTLOQSBOEPNSBOE
MFO
US@EBUB
USBJO@EBUBUS@EBUB<NBTL>
WBM@EBUBUS@EBUB<_NBTL>
[ 277 ]
Unsupervised Learning Chapter 7
We will need to preprocess the data, that is, separate the pixels and emotion labels. For this
we make two function EFOTF@UP@POF@IPU
, it performs the one hot encoding for labels.
The second function is QSFQSPDFTT@EBUB
, which separates out individual pixels as an
array. With the help of these two functions, we generate the input feature and label of the
training, validation, and test dataset:
EFGEFOTF@UP@POF@IPU
MBCFMT@EFOTFOVN@DMBTTFT
OVN@MBCFMTMBCFMT@EFOTFTIBQF<>
JOEFY@PGGTFUOQBSBOHF
OVN@MBCFMTOVN@DMBTTFT
MBCFMT@POF@IPUOQ[FSPT
OVN@MBCFMTOVN@DMBTTFT
MBCFMT@POF@IPUGMBU<JOEFY@PGGTFU
MBCFMT@EFOTFSBWFM
>
SFUVSOMBCFMT@POF@IPU
EFGQSFQSPDFTT@EBUB
EBUBGSBNF
QJYFMT@WBMVFTEBUBGSBNFQJYFMTTUSTQMJU
UPMJTU
QJYFMT@WBMVFTQE%BUB'SBNF
QJYFMT@WBMVFTEUZQFJOU
JNBHFTQJYFMT@WBMVFTWBMVFT
JNBHFTJNBHFTBTUZQF
OQGMPBU
JNBHFTOQNVMUJQMZ
JNBHFT
MBCFMT@GMBUEBUBGSBNF<FNPUJPO>WBMVFTSBWFM
MBCFMT@DPVOUOQVOJRVF
MBCFMT@GMBUTIBQF<>
MBCFMTEFOTF@UP@POF@IPU
MBCFMT@GMBUMBCFMT@DPVOU
MBCFMTMBCFMTBTUZQF
OQVJOU
SFUVSOJNBHFTMBCFMT
Using the functions defined in the preceding code, we get the data in the format required
for training. We build the emotion detection DBN, based on the similar principle as
mentioned in this paper for MNIST: IUUQTXXXDTUPSPOUPFEV_IJOUPOBCTQT
GBTUODQEG.
How to do it...
We proceed with the recipe as follows:
1. We need to import the standard modules, TensorFlow, NumPy, and Pandas, for
reading the DTW file, and Matplolib:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUQBOEBTBTQE
JNQPSUNBUQMPUMJCQZQMPUBTQMU
[ 278 ]
Unsupervised Learning Chapter 7
2. The training, validation, and testing data is obtained using the helper functions:
9@USBJO:@USBJOQSFQSPDFTT@EBUB
USBJO@EBUB
9@WBM:@WBMQSFQSPDFTT@EBUB
WBM@EBUB
9@UFTU:@UFTUQSFQSPDFTT@EBUB
UFTU@EBUB
3. Let us explore our data a little. We plot the mean image and find the number of
images in each training, validation, and testing dataset:
&YQMPSF%BUB
NFBO@JNBHF9@USBJONFBO
BYJT
TUE@JNBHFOQTUE
9@USBJOBYJT
QSJOU
5SBJOJOH%BUBTFUIBT\^JNBHFTGPSNBU
MFO
9@USBJO
QSJOU
7BMJEBUJPO%BUBTFUIBT\^JNBHFTGPSNBU
MFO
9@WBM
QSJOU
5FTU%BUBTFUIBT\^JNBHFTGPSNBU
MFO
9@UFTU
QMUJNTIPX
NFBO@JNBHFSFTIBQF
DNBQ HSBZ
4. We also see the images from the training sample and their respective label:
DMBTTFT
< BOHSZ EJTHVTU GFBS IBQQZ TBE TVSQSJTF OFVUSBM >
OVN@DMBTTFTMFO
DMBTTFT
TBNQMFT@QFS@DMBTT
GPSZDMTJOFOVNFSBUF
DMBTTFT
JEYTOQGMBUOPO[FSP
OQBSHNBY
:@USBJOBYJTZ
JEYTOQSBOEPNDIPJDF
JEYTTBNQMFT@QFS@DMBTT
SFQMBDF'BMTF
[ 279 ]
Unsupervised Learning Chapter 7
GPSJJEYJOFOVNFSBUF
JEYT
QMU@JEYJOVN@DMBTTFT
Z
QMUTVCQMPU
TBNQMFT@QFS@DMBTTOVN@DMBTTFTQMU@JEY
QMUJNTIPX
9@USBJO<JEY>SFTIBQF
DNBQ HSBZ
QJYFMIFJHIUBOEXJEUI
QMUBYJT
PGG
JGJ
QMUUJUMF
DMT
QMUTIPX
5. Next, we define the RBM stack; each RBM takes the output of the previous RBM
as its input:
3#.@IJEEFO@TJ[FT<>DSFBUFMBZFSTPG3#.XJUI
TJ[FBOE
4FUJOQVUBTUSBJOJOHEBUB
JOQ99@USBJO
$SFBUFMJTUUPIPMEPVS3#.T
SCN@MJTU<>
4J[FPGJOQVUTJTUIFOVNCFSPGJOQVUTJOUIFUSBJOJOHTFU
JOQVU@TJ[FJOQ9TIBQF<>
'PSFBDI3#.XFXBOUUPHFOFSBUF
GPSJTJ[FJOFOVNFSBUF
3#.@IJEEFO@TJ[FT
QSJOU
3#. J JOQVU@TJ[F TJ[F
SCN@MJTUBQQFOE
3#.
JOQVU@TJ[FTJ[F
JOQVU@TJ[FTJ[F
[ 280 ]
Unsupervised Learning Chapter 7
This generates three RBMs: the first RBM with 2,304 (48 e 48) input and 1,500
hidden units, the second RBM with 1,500 input and 700 hidden units, and finally
the third RBM with 700 input and 400 hidden units.
6. We train each RBM one by one. The technique is also called greedy wise training.
In the original paper the number of epochs for training each RBM on MNIST was
30, so here, too, increasing the epochs should improve the performance of the
network:
(SFFEZXJTFUSBJOJOHPG3#.T
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
GPSSCNJOSCN@MJTU
QSJOU
/FX3#.
5SBJOBOFXPOF
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
SCNTFU@TFTTJPO
TFTT
FSSSCNGJU
JOQ9
JOQ9@OSCNSCN@PVUQVU
JOQ9
QSJOU
JOQ9@OTIBQF
JOQ9JOQ9@O
7. We define a %#/ class. In the class, we build the complete DBN with the three
layers of RBM and two additional MLP layers. The weights of RBM layers are
loaded from the pre-trained RBMs. We also declare methods to train and predict
the DBN; for fine-tuning, the network tries to minimize the mean square loss
function:
DMBTT%#/
PCKFDU
EFG@@JOJU@@
TFMGTJ[FT9:FUBNPNFOUVN
FQPDITCBUDI@TJ[F
*OJUJBMJ[FIZQFSQBSBNFUFST
TFMG@TJ[FTTJ[FT
QSJOU
TFMG@TJ[FT
TFMG@TJ[FTBQQFOE
TJ[FPGUIFGJSTU'$MBZFS
TFMG@99
TFMG@::
TFMG/MFO
9
TFMGX@MJTU<>
TFMGD@MJTU<>
TFMG@MFBSOJOH@SBUFFUB
TFMG@NPNFOUVNNPNFOUVN
TFMG@FQPDITFQPDIT
TFMG@CBUDITJ[FCBUDI@TJ[F
JOQVU@TJ[F9TIBQF<>
[ 281 ]
Unsupervised Learning Chapter 7
JOJUJBMJ[BUJPOMPPQ
GPSTJ[FJOTFMG@TJ[FT
<:TIBQF<>>
%FGJOFVQQFSMJNJUGPSUIFVOJGPSNEJTUSJCVUJPOSBOHF
NBY@SBOHFNBUITRSU
JOQVU@TJ[F
TJ[F
*OJUJBMJ[FXFJHIUTUISPVHIBSBOEPNVOJGPSN
EJTUSJCVUJPO
TFMGX@MJTUBQQFOE
OQSBOEPNVOJGPSN
NBY@SBOHFNBY@SBOHF<JOQVU@TJ[F
TJ[F>BTUZQF
OQGMPBU
*OJUJBMJ[FCJBTBT[FSPFT
TFMGD@MJTUBQQFOE
OQ[FSPT
<TJ[F>OQGMPBU
JOQVU@TJ[FTJ[F
#VJME%#/
$SFBUFQMBDFIPMEFSTGPSJOQVUXFJHIUTCJBTFTPVUQVU
TFMG@B</POF>
MFO
TFMG@TJ[FT
TFMG@X</POF>
MFO
TFMG@TJ[FT
TFMG@D</POF>
MFO
TFMG@TJ[FT
TFMG@B<>UGQMBDFIPMEFS
GMPBU</POF
TFMG@9TIBQF<>>
TFMGZUGQMBDFIPMEFS
GMPBU</POFTFMG@:TIBQF<>>
%FGJOFWBSJBCMFTBOEBDUJWBUJPOGVODUJPO
GPSJJOSBOHF
MFO
TFMG@TJ[FT
TFMG@X<J>UG7BSJBCMF
TFMGX@MJTU<J>
TFMG@D<J>UG7BSJBCMF
TFMGD@MJTU<J>
GPSJJOSBOHF
MFO
TFMG@TJ[FT
TFMG@B<J>UGOOTJHNPJE
UGNBUNVM
TFMG@B<J>
TFMG@X<J>
TFMG@D<J>
%FGJOFUIFDPTUGVODUJPO
DPTU
UGSFEVDF@NFBO
UGOOTPGUNBY@DSPTT@FOUSPQZ@XJUI@MPHJUT
MBCFMTTFMG
ZMPHJUTTFMG@B<>
DPTUUGSFEVDF@NFBO
UGTRVBSF
TFMG@B<>TFMGZ
%FGJOFUIFUSBJOJOHPQFSBUJPO
.PNFOUVN0QUJNJ[FS
NJOJNJ[JOHUIF$PTUGVODUJPO
TFMGUSBJO@PQ
UGUSBJO"EBN0QUJNJ[FS
MFBSOJOH@SBUFTFMG@MFBSOJOH@SBUFNJOJNJ[F
DPTU
1SFEJDUJPOPQFSBUJPO
TFMGQSFEJDU@PQUGBSHNBY
TFMG@B<>
[ 282 ]
Unsupervised Learning Chapter 7
MPBEEBUBGSPNSCN
EFGMPBE@GSPN@SCNT
TFMGECO@TJ[FTSCN@MJTU
$IFDLJGFYQFDUFETJ[FTBSFDPSSFDU
BTTFSUMFO
ECO@TJ[FTMFO
TFMG@TJ[FT
GPSJJOSBOHF
MFO
TFMG@TJ[FT
$IFDLJGGPSFBDI3#/UIFFYQFDUFETJ[FTBSFDPSSFDU
BTTFSUECO@TJ[FT<J>TFMG@TJ[FT<J>
*GFWFSZUIJOHJTDPSSFDUCSJOHPWFSUIFXFJHIUTBOE
CJBTFT
GPSJJOSBOHF
MFO
TFMG@TJ[FT
TFMGX@MJTU<J>SCN@MJTU<J>@8
TFMGD@MJTU<J>SCN@MJTU<J>@D
EFGTFU@TFTTJPO
TFMGTFTTJPO
TFMGTFTTJPOTFTTJPO
5SBJOJOHNFUIPE
EFGUSBJO
TFMGWBM@YWBM@Z
'PSFBDIFQPDI
OVN@CBUDIFTTFMG/TFMG@CBUDITJ[F
CBUDI@TJ[FTFMG@CBUDITJ[F
GPSJJOSBOHF
TFMG@FQPDIT
'PSFBDITUFQ
GPSKJOSBOHF
OVN@CBUDIFT
CBUDITFMG@9<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
CBUDI@MBCFMTFMG@:<KCBUDI@TJ[F
K
CBUDI@TJ[F
CBUDI@TJ[F>
TFMGTFTTJPOSVO
TFMGUSBJO@PQ
GFFE@EJDU\TFMG@B<>CBUDITFMGZCBUDI@MBCFM^
GPSKJOSBOHF
MFO
TFMG@TJ[FT
3FUSJFWFXFJHIUTBOECJBTFT
TFMGX@MJTU<K>TFTTSVO
TFMG@X<K>
TFMGD@MJTU<K>TFTTSVO
TFMG@D<K>
USBJO@BDDOQNFBO
OQBSHNBY
TFMG@:BYJT
TFMGTFTTJPOSVO
TFMGQSFEJDU@PQGFFE@EJDU\TFMG@B<>TFMG@9
TFMGZTFMG@:^
WBM@BDDOQNFBO
OQBSHNBY
WBM@ZBYJT
TFMGTFTTJPOSVO
TFMGQSFEJDU@PQGFFE@EJDU\TFMG@B<>WBM@Y
TFMGZWBM@Z^
[ 283 ]
Unsupervised Learning Chapter 7
QSJOU
FQPDI
TUS
J
TUS
TFMG@FQPDIT
5SBJOJOH"DDVSBDZ
TUS
USBJO@BDD
7BMJEBUJPO"DDVSBDZ
TUS
WBM@BDD
EFGQSFEJDU
TFMG9
SFUVSOTFMGTFTTJPOSVO
TFMGQSFEJDU@PQ
GFFE@EJDU\TFMG@B<>9^
8. Now, we train instantiate a %#/ object and train it. And predict the labels for the
test data:
O/FU%#/
3#.@IJEEFO@TJ[FT9@USBJO:@USBJOFQPDIT
XJUIUG4FTTJPO
BTTFTT
*OJUJBMJ[F7BSJBCMFT
TFTTSVO
UGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
O/FUTFU@TFTTJPO
TFTT
O/FUMPBE@GSPN@SCNT
3#.@IJEEFO@TJ[FTSCN@MJTU
O/FUUSBJO
9@WBM:@WBM
Z@QSFEO/FUQSFEJDU
9@UFTU
How it works...
The RBMs learn the hidden representations/features of the model using unsupervised
learning and then the fully connected layers added along with pre-trained RBMs are
finetuned.
The accuracy here varies largely on image representation. In the preceding recipe, we have
used no image processing, only grayscale images scaled between 0 to 1. But if we add the
image processing as done in the following paper, it increases the accuracy further-IUUQ
EFFQMFBSOJOHOFUXQDPOUFOUVQMPBETEMTWNQEG. Thus we multiply each
image by 100.0/255.0 in the QSFQSPDFTT@EBUB function, and add these few lines of the code
to main code:
TUE@JNBHFOQTUE
9@USBJOBYJT
9@USBJOOQEJWJEF
OQTVCUSBDU
9@USBJONFBO@JNBHFTUE@JNBHF
9@WBMOQEJWJEF
OQTVCUSBDU
9@WBMNFBO@JNBHFTUE@JNBHF
9@UFTUOQEJWJEF
OQTVCUSBDU
9@UFTUNFBO@JNBHFTUE@JNBHF
[ 284 ]
Unsupervised Learning Chapter 7
There's more...
In the preceding example, without preprocessing, the accuracy of the three data sets is
roughly ~40 percent. But when we add preprocessing, the accuracy for the training data
increases to 90 percent, but for validation and testing, we still get an accuracy of ~45
percent.
There are many changes which can be introduced to improve the result. First, the dataset
that we used in the recipe is the Kaggle dataset with only 22,000 images. If you observe
these images you will find the addition of the step of filtering only faces will improve the
result. Another strategy, as mentioned in the following paper, is increasing the size of
hidden layers instead of reducing them-IUUQTXXXDTTXBSUINPSFFEV_NFFEFODT
TQBQFST,FWJO7JODFOUQEG.
Another change that has been found really successful in identifying the emotions is using
facial-keypoints instead of training for the whole face, IUUQDTTUBOGPSEFEV
QSPK.D-BVHIMJO-F#BZBOCBU3FDPHOJ[JOH&NPUJPOT8JUI%FFQ#FMJFG/FUTQEG.
With the preceding recipe, you can play around with these changes and explore how
performance improves. May the GPU force be with you!
[ 285 ]
Autoencoders
8
Autoencoders are feed-forward, non-recurrent neural networks, which learn by
unsupervised learning. They have an inherent capability to learn a compact representation
of data. They are at the centre of deep belief networks and find applications in image
reconstruction, clustering, machine translation, and much more. In this chapter, you will
learn and implement different variants of autoencoders and eventually learn how to stack
autoencoders. The chapter includes the following topics:
Vanilla autoencoder
Sparse autoencoder
Denoising autoencoder
Convolutional autoencoders
Stacked autoencoders
Introduction
Autoencoders, also known as Diabolo networks or autoassociators, was initially proposed
in the 1980s by Hinton and the PDP group [1]. They are feedforward networks, without any
feedback, and they learn via unsupervised learning. Like multiplayer perceptrons of
$IBQUFS, Neural Networks-Perceptrons, they use the backpropagation algorithm to learn,
but with a major difference--the target is the same as the input.
yh
x
Autoencoders Chapter 8
The second network uses the encoded signal y as its input and performs another
transformation f to get a reconstructed signal r:
rf
yf
h
x
We define error e as the difference between the original input x and the reconstructed signal
r, e = x - r. The network then learns by reducing the mean squared error (MSE), and the
error is propagated back to the hidden layers as in the case of MLPs. The following figure
shows an autoencoder with encoder and decoder highlighted separately. Autoencoders can
have weight sharing, that is, weights of decoder and encoder are simply a transpose of each
other, which can help the network learn faster as the number of training parameters is less,
but at the same time reduces the degrees of freedom of the network. They are very similar
to RBMs of $IBQUFS, Unsupervised Learning, but with one big difference--the state of the
neurons in autoencoders is deterministic, while in RBMs, the state of the neurons is
probabilistic:
[ 287 ]
Autoencoders Chapter 8
Depending on the size of the hidden layer, autoencoders are classified as Undercomplete
(hidden layer has fewer neurons than the input layer) or Overcomplete (hidden layer has
more neurons than the input layer). Depending on the restriction/constraints imposed on
the loss, we have various types of autoencoders: Sparse Autoencoders, Denoising
Autoencoders, and Convolution Autoencoders. In this chapter, you will learn about these
variations in autoencoders and implement them using TensorFlow.
See Also
Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. Learning
internal representations by error propagation. No. ICS-8506. California Univ San
Diego La Jolla Inst for Cognitive Science, 1985. (IUUQXXXDTUPSPOUPFEV
_GSJU[BCTQTQEQQEG)
[ 288 ]
Autoencoders Chapter 8
Vanilla autoencoders
The vanilla autoencoder, as proposed by Hinton, consists of only one hidden layer. The
number of neurons in the hidden layer is less than the number of neurons in the input (or
output) layer. This results in producing a bottleneck effect on the flow of information in the
network, and therefore we can think of the hidden layer as a bottleneck layer, restricting the
information that would be stored. Learning in the autoencoder consists of developing a
compact representation of the input signal at the hidden layer so that the output layer can
faithfully reproduce the original input:
#WVQGPEQFGTYKVJCUKPINGJKFFGPNC[GT
[ 289 ]
Autoencoders Chapter 8
Getting ready
This recipe will use autoencoders for image reconstruction; we will train the autoencoder on
MNIST database and use it to reconstruct the test images.
How to do it...
We proceed with the recipe as follows:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
JNQPSUNBUQMPUMJCQZQMPUBTQMU
NBUQMPUMJCJOMJOF
2. Next, we take the MNIST data from the TensorFlow examples--the important
thing to note here is that the labels are not one-hot encoded, simply because we
are not using labels to train the network. Autoencoders learn via unsupervised
learning:
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB
US9US:UF9UF:NOJTUUSBJOJNBHFTNOJTUUSBJOMBCFMT
NOJTUUFTUJNBHFTNOJTUUFTUMBCFMT
3. Next, we declare a class "VUP&ODPEFS; the class has the JOJU method to
initialize weights, biases, and placeholders for the autoencoder. We can also build
the complete graph in the JOJU method. The class also has methods for FODPEFS,
EFDPEFS, setting session (TFU@TFTTJPO), and GJU. The autoencoder we build
here uses the simple MSE as the MPTT function, and we try to optimize it using
"EBN0QUJNJ[FS:
DMBTT"VUP&ODPEFS
PCKFDU
EFG@@JOJU@@
TFMGNOFUB
N/VNCFSPGOFVSPOTJOJOQVUPVUQVUMBZFS
OOVNCFSPGOFVSPOTJOIJEEFOMBZFS
TFMG@NN
TFMG@OO
TFMGMFBSOJOH@SBUFFUB
[ 290 ]
Autoencoders Chapter 8
$SFBUFUIF$PNQVUBUJPOBMHSBQI
8FJHIUTBOECJBTFT
TFMG@8UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
TFMG@NTFMG@O
TFMG@8UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
TFMG@OTFMG@N
TFMG@CUG7BSJBCMF
OQ[FSPT
TFMG@OBTUZQF
OQGMPBUCJBT
GPSIJEEFOMBZFS
TFMG@CUG7BSJBCMF
OQ[FSPT
TFMG@NBTUZQF
OQGMPBUCJBT
GPSPVUQVUMBZFS
1MBDFIPMEFSGPSJOQVUT
TFMG@9UGQMBDFIPMEFS
GMPBU </POFTFMG@N>
TFMGZTFMGFODPEFS
TFMG@9
TFMGSTFMGEFDPEFS
TFMGZ
FSSPSTFMG@9TFMGS
TFMG@MPTTUGSFEVDF@NFBO
UGQPX
FSSPS
TFMG@PQU
UGUSBJO"EBN0QUJNJ[FS
TFMGMFBSOJOH@SBUFNJOJNJ[F
TFMG@MPTT
EFGFODPEFS
TFMGY
IUGNBUNVM
YTFMG@8
TFMG@C
SFUVSOUGOOTJHNPJE
I
EFGEFDPEFS
TFMGY
IUGNBUNVM
YTFMG@8
TFMG@C
SFUVSOUGOOTJHNPJE
I
EFGTFU@TFTTJPO
TFMGTFTTJPO
TFMGTFTTJPOTFTTJPO
EFGSFEVDFE@EJNFOTJPO
TFMGY
ITFMGFODPEFS
Y
SFUVSOTFMGTFTTJPOSVO
IGFFE@EJDU\TFMG@9Y^
EFGSFDPOTUSVDU
TFMGY
ITFMGFODPEFS
Y
STFMGEFDPEFS
I
SFUVSOTFMGTFTTJPOSVO
SGFFE@EJDU\TFMG@9Y^
EFGGJU
TFMG9FQPDITCBUDI@TJ[F
/%9TIBQF
[ 291 ]
Autoencoders Chapter 8
OVN@CBUDIFT/CBUDI@TJ[F
PCK<>
GPSJJOSBOHF
FQPDIT
9TIVGGMF
9
GPSKJOSBOHF
OVN@CBUDIFT
CBUDI9<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
@PCTFMGTFTTJPOSVO
<TFMG@PQUTFMG@MPTT>
GFFE@EJDU\TFMG@9CBUDI^
JGKBOEJ
QSJOU
USBJOJOHFQPDI\^CBUDI\^DPTU\^ GPSNBU
JPC
K
PCKBQQFOE
PC
SFUVSOPCK
To be able to use the autoencoder after training, we also define two utility
functions--SFEVDFE@EJNFOTJPO, which gives the output of the encoder network,
and SFDPOTUSVDU, which reconstructs the final image.
4. We convert the input data to a GMPBU for the training, initialize all the variables,
and start the computation session. In computation, we are, at present, only
testing the reconstruction power of the autoencoder:
9USBJOUS9BTUZQF
OQGMPBU
9UFTUUF9BTUZQF
OQGMPBU
@N9USBJOTIBQF
BVUP&ODPEFS"VUP&ODPEFS
N
*OJUJBMJ[FBMMWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
BVUP&ODPEFSTFU@TFTTJPO
TFTT
FSSBVUP&ODPEFSGJU
9USBJOFQPDIT
PVUBVUP&ODPEFSSFDPOTUSVDU
9UFTU<>
5. We can verify if our network indeed optimized the MSE while training by
plotting error versus epoch. For a good training, the error should reduce with
FQPDIT:
QMUQMPU
FSS
QMUYMBCFM
FQPDIT
QMUZMBCFM
DPTU
[ 292 ]
Autoencoders Chapter 8
We can see that loss/cost is decreasing as the network learns and by the time we
reach 5,000 epochs, it is almost oscillating about a line. This means further
increasing epochs will not be useful. If we want to now improve our training, we
should change the hyperparameters like learning rate, batch size, and optimizer
used.
6. Let's now see the reconstructed images. Here, you can see the original and
reconstructed images together as generated by our autoencoder:
1MPUUJOHPSJHJOBMBOESFDPOTUSVDUFEJNBHFT
SPXDPM
JEYOQSBOEPNSBOEJOU
SPXDPM
GBYBSSQMUTVCQMPUT
SPXDPMTIBSFY5SVFTIBSFZ5SVF
GJHTJ[F
GPSGJHSPXJO[JQ
<9UFTUPVU>BYBSS
GPSJBYJO[JQ
JEYSPX
BYJNTIPX
GJH<J>SFTIBQF
[ 293 ]
Autoencoders Chapter 8
How it works...
It is interesting to note that in the preceding code, we reduced the dimensions of the input
from 784 to 256 and our network could still reconstruct the original image. Let's compare
our performance with RBM (Chapter 7, Unsupervised Learning ) with the same dimension of
the hidden layer:
We can see that the images reconstructed by the autoencoder are much crisper than the
ones reconstructed by RBM. The reason is that, in the autoencoder, there are additional
weights (weights from hidden to decoder output layer) to be trained and hence retain
learning. As the autoencoder learns more, it performs better than RBM, even when both
compact the information to the same dimensions.
[ 294 ]
Autoencoders Chapter 8
There's more...
Autoencoders like PCA can be used for dimensionality reduction, but while PCA can only
represent linear transformations, we can use nonlinear activation functions in autoencoders,
thus introducing non-linearities in our encodings. Here is the result reproduced from the
Hinton paper, Reducing the dimensionality of data with Neural Networks. The result compares
the result of a PCA (A) with that of stacked RBMs as autoencoders with architecture
consisting of 784-1000-500-250-2:
As we will see later when an autoencoder is made using stacked autoencoders, each
autoencoder is initially pretrained individually, and then the entire network is fine-tuned
for better performance.
[ 295 ]
Autoencoders Chapter 8
Sparse autoencoder
The autoencoder that we saw in the previous recipe worked more like an identity network--
they simply reconstruct the input. The emphasis is to reconstruct the image at the pixel
level, and the only constraint is the number of units in the bottleneck layer; while it is
interesting, pixel-level reconstruction does not ensure that the network will learn abstract
features from the dataset. We can ensure that the network learns abstract features from the
dataset by adding further constraints.
In sparse autoencoders, a sparse penalty term is added to the reconstruction error, which
tries to ensure that fewer units in the bottleneck layer will fire at any given time. If m is the
total number of input patterns, then we can define a quantity @IBU (you can check the
mathematical details in Andrew Ng's Lecture at IUUQTXFCTUBOGPSEFEVDMBTT
DTBTQBSTF"VUPFODPEFS@OFXQEG), which measures the net activity (how many
times on an average it fires) for each hidden layer unit. The basic idea is to put a
constraint @IBU, such that it is equal to the sparsity parameter . This results in adding a
regularization term for sparsity in the loss function so that now the MPTT function is as
follows:
MPTT.FBOTRVBSFEFSSPS
3FHVMBSJ[BUJPOGPSTQBSTJUZQBSBNFUFS
This regularization term will penalize the network if @IBU deviates from ; one standard
way to do this is to use Kullback-Leiber (KL) divergence between and @IBU.
Getting Ready...
Before starting with the recipe, let's explore the KL divergence, DKL a little more. It is a non-
symmetric measure of the difference between two distributions, in our case, and @IBU.
When and @IBU are equal, then it is zero, otherwise, it increases monotonically as @IBU
diverges from h. Mathematically, it is expressed as follows:
[ 296 ]
Autoencoders Chapter 8
Here is the plot of DKL for a fixed h=0.3, where we can see that when h_hat=0.3, DKL=0;
otherwise, it is increasing monotonically on both sides:
How to do it...
We proceed with the recipe as follows:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
JNQPSUNBUQMPUMJCQZQMPUBTQMU
NBUQMPUMJCJOMJOF
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB
US9US:UF9UF:NOJTUUSBJOJNBHFTNOJTUUSBJOMBCFMT
NOJTUUFTUJNBHFTNOJTUUFTUMBCFMT
[ 297 ]
Autoencoders Chapter 8
EFGLM@EJW
TFMGSIPSIP@IBU
UFSN@OVNUGDPOTUBOU
SIP
UFSN@EFOUGDPOTUBOU
SIP@IBU
LMTFMGMPHGVOD
SIPSIP@IBU
TFMGMPHGVOD
UFSN@OVN
UFSN@EFO
SFUVSOLM
EFGMPHGVOD
TFMGYY
SFUVSOUGNVMUJQMZ
YUGMPH
UGEJW
YY
Here, BMQIB is the weightage given to the sparsity constraint. The complete code
for the class is as follows:
DMBTT4QBSTF"VUP&ODPEFS
PCKFDU
EFG@@JOJU@@
TFMGNOFUB
N/VNCFSPGOFVSPOTJOJOQVUPVUQVUMBZFS
OOVNCFSPGOFVSPOTJOIJEEFOMBZFS
TFMG@NN
TFMG@OO
TFMGMFBSOJOH@SBUFFUB
$SFBUFUIF$PNQVUBUJPOBMHSBQI
8FJHIUTBOECJBTFT
TFMG@8UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
TFMG@NTFMG@O
TFMG@8UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
TFMG@OTFMG@N
TFMG@CUG7BSJBCMF
OQ[FSPT
TFMG@OBTUZQF
OQGMPBUCJBT
GPSIJEEFOMBZFS
TFMG@CUG7BSJBCMF
OQ[FSPT
TFMG@NBTUZQF
OQGMPBUCJBT
GPSPVUQVUMBZFS
1MBDFIPMEFSGPSJOQVUT
TFMG@9UGQMBDFIPMEFS
GMPBU </POFTFMG@N>
TFMGZTFMGFODPEFS
TFMG@9
[ 298 ]
Autoencoders Chapter 8
TFMGSTFMGEFDPEFS
TFMGZ
FSSPSTFMG@9TFMGS
TFMG@MPTTUGSFEVDF@NFBO
UGQPX
FSSPS
BMQIBF
LM@EJW@MPTTUGSFEVDF@TVN
TFMGLM@EJW
UGSFEVDF@NFBO
TFMGZ
MPTTTFMG@MPTT
BMQIBLM@EJW@MPTT
TFMG@PQU
UGUSBJO"EBN0QUJNJ[FS
TFMGMFBSOJOH@SBUFNJOJNJ[F
MPTT
EFGFODPEFS
TFMGY
IUGNBUNVM
YTFMG@8
TFMG@C
SFUVSOUGOOTJHNPJE
I
EFGEFDPEFS
TFMGY
IUGNBUNVM
YTFMG@8
TFMG@C
SFUVSOUGOOTJHNPJE
I
EFGTFU@TFTTJPO
TFMGTFTTJPO
TFMGTFTTJPOTFTTJPO
EFGSFEVDFE@EJNFOTJPO
TFMGY
ITFMGFODPEFS
Y
SFUVSOTFMGTFTTJPOSVO
IGFFE@EJDU\TFMG@9Y^
EFGSFDPOTUSVDU
TFMGY
ITFMGFODPEFS
Y
STFMGEFDPEFS
I
SFUVSOTFMGTFTTJPOSVO
SGFFE@EJDU\TFMG@9Y^
EFGLM@EJW
TFMGSIPSIP@IBU
UFSN@OVNUGDPOTUBOU
SIP
UFSN@EFOUGDPOTUBOU
SIP@IBU
LMTFMGMPHGVOD
SIPSIP@IBU
TFMGMPHGVOD
UFSN@OVN
UFSN@EFO
SFUVSOLM
EFGMPHGVOD
TFMGYY
SFUVSOUGNVMUJQMZ
YUGMPH
UGEJW
YY
EFGGJU
TFMG9FQPDITCBUDI@TJ[F
/%9TIBQF
[ 299 ]
Autoencoders Chapter 8
OVN@CBUDIFT/CBUDI@TJ[F
PCK<>
GPSJJOSBOHF
FQPDIT
9TIVGGMF
9
GPSKJOSBOHF
OVN@CBUDIFT
CBUDI9<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
@PCTFMGTFTTJPOSVO
<TFMG@PQUTFMG@MPTT>
GFFE@EJDU\TFMG@9CBUDI^
JGK
QSJOU
USBJOJOHFQPDI\^CBUDI\^DPTU
\^ GPSNBU
JPCK
PCKBQQFOE
PC
SFUVSOPCK
9USBJOUS9BTUZQF
OQGMPBU
9UFTUUF9BTUZQF
OQGMPBU
@N9USBJOTIBQF
TBF4QBSTF"VUP&ODPEFS
N
*OJUJBMJ[FBMMWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
TBFTFU@TFTTJPO
TFTT
FSSTBFGJU
9USBJOFQPDIT
PVUTBFSFDPOTUSVDU
9UFTU<>
5. Let's see the variation of mean squared reconstruction loss as the network learns:
QMUQMPU
FSS
QMUYMBCFM
FQPDIT
QMUZMBCFM
3FDPOTUSVDUJPO-PTT
.4&
[ 300 ]
Autoencoders Chapter 8
1MPUUJOHPSJHJOBMBOESFDPOTUSVDUFEJNBHFT
SPXDPM
JEYOQSBOEPNSBOEJOU
SPXDPM
GBYBSSQMUTVCQMPUT
SPXDPMTIBSFY5SVFTIBSFZ5SVF
GJHTJ[F
GPSGJHSPXJO[JQ
<9UFTUPVU>BYBSS
GPSJBYJO[JQ
JEYSPX
BYJNTIPX
GJH<J>SFTIBQF
[ 301 ]
Autoencoders Chapter 8
How it works...
The main code of the sparse autoencoder, as you must have noticed, is exactly the same as
that of the vanilla autoencoder, which is so because the sparse autoencoder has just one
major change--the addition of the KL divergence loss to ensure sparsity of the hidden
(bottleneck) layer. However, if you compare the two reconstructions, you can see that the
sparse autoencoder is much better than the standard encoder, even with the same number
of units in the hidden layer:
The reconstruction loss after training the vanilla autoencoder for the MNIST dataset is 0.022,
while for the sparse autoencoder, it is 0.006. Thus, adding constraint forces the network to
learn hidden representations of the data.
[ 302 ]
Autoencoders Chapter 8
There's More...
The compact representation of the inputs is stored in weights; let's visualize the weights
learned by the network. Here are the weights of the encoder layer for the standard and
sparse autoencoder respectively. We can see that, in the standard autoencoder, many
hidden units have their very large weights, suggesting they are overworked:
See Also
IUUQXFCFOHSJMMJOPJTFEV_IBOKDTCL,-EJWFSHFODFQEG
IUUQTFOXJLJQFEJBPSHXJLJ,VMMCBDL&-FJCMFS@EJWFSHFODF
[ 303 ]
Autoencoders Chapter 8
Denoising autoencoder
The two autoencoders that we have explored in the previous two recipes are examples
of Undercomplete Autoencoders because the hidden layer in them has a lower dimension
as compared to the input (output) layer. Denoising autoencoder belongs to the class of
Overcomplete Autoencoders because it works better when the dimensions of the hidden
layer are more than the input layer.
A denoising autoencoder learns from a corrupted (noisy) input; it feeds its encoder network
the noisy input and then the reconstructed image from the decoder is compared with the
original input. The idea is that this will help the network learn how to denoise an input. It
will no longer just make a pixel-wise comparison, but in order to denoise, it will learn the
information of neighbouring pixels as well.
Getting Ready
A denoising autoencoder will also have the KL divergence penalty term; it will differ from
the sparse autoencoder of the previous recipe in two main aspects. First, O@IJEEFO, the
number of hidden units in the bottleneck layer is more than the number of units in the input
layer, m, that is O@IJEEFO N. Second, the input to the encoder is corrupted input; to do
this in TensorFlow, we add the function corrupt, which adds noise to the input:
EFGDPSSVQUJPO
YOPJTF@GBDUPSDPSSVQUJPOPGUIFJOQVU
OPJTZ@JNHTY
OPJTF@GBDUPSOQSBOEPNSBOEO
YTIBQF
OPJTZ@JNHTOQDMJQ
OPJTZ@JNHT
SFUVSOOPJTZ@JNHT
How to do it...
1. Like always, the first step is to import the necessary modules--TensorFlow,
numpy to manipulate input data, matplotlib to plot, and so on:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
JNQPSUNBUQMPUMJCQZQMPUBTQMU
JNQPSUNBUI
NBUQMPUMJCJOMJOF
[ 304 ]
Autoencoders Chapter 8
2. Load the data from TensorFlow examples. We have used the standard MNIST
database for illustration in all the recipes of this chapter to provide you with a
benchmark between different autoencoders.
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB
US9US:UF9UF:NOJTUUSBJOJNBHFTNOJTUUSBJOMBCFMT
NOJTUUFTUJNBHFTNOJTUUFTUMBCFMT
DMBTT%FOPJTJOH"VUP&ODPEFS
PCKFDU
EFG@@JOJU@@
TFMGNOFUB
N/VNCFSPGOFVSPOTJOJOQVUPVUQVUMBZFS
OOVNCFSPGOFVSPOTJOIJEEFOMBZFS
TFMG@NN
TFMG@OO
TFMGMFBSOJOH@SBUFFUB
$SFBUFUIF$PNQVUBUJPOBMHSBQI
8FJHIUTBOECJBTFT
TFMG@8UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
TFMG@NTFMG@O
TFMG@8UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
TFMG@OTFMG@N
TFMG@CUG7BSJBCMF
OQ[FSPT
TFMG@OBTUZQF
OQGMPBUCJBTGPS
IJEEFOMBZFS
TFMG@CUG7BSJBCMF
OQ[FSPT
TFMG@NBTUZQF
OQGMPBUCJBTGPS
PVUQVUMBZFS
1MBDFIPMEFSGPSJOQVUT
TFMG@9UGQMBDFIPMEFS
GMPBU </POFTFMG@N>
TFMG@9@OPJTZUGQMBDFIPMEFS
GMPBU </POFTFMG@N>
TFMGZTFMGFODPEFS
TFMG@9@OPJTZ
TFMGSTFMGEFDPEFS
TFMGZ
FSSPSTFMG@9TFMGS
TFMG@MPTTUGSFEVDF@NFBO
UGQPX
FSSPS
[ 305 ]
Autoencoders Chapter 8
TFMG@MPTTUGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MBCFMT
TFMG@9MPHJUTTFMGS
BMQIB
LM@EJW@MPTTUGSFEVDF@TVN
TFMGLM@EJW
UGSFEVDF@NFBO
TFMGZ
MPTTTFMG@MPTT
BMQIBLM@EJW@MPTT
TFMG@PQUUGUSBJO"EBN0QUJNJ[FS
TFMGMFBSOJOH@SBUFNJOJNJ[F
MPTT
EFGFODPEFS
TFMGY
IUGNBUNVM
YTFMG@8
TFMG@C
SFUVSOUGOOTJHNPJE
I
EFGEFDPEFS
TFMGY
IUGNBUNVM
YTFMG@8
TFMG@C
SFUVSOUGOOTJHNPJE
I
EFGTFU@TFTTJPO
TFMGTFTTJPO
TFMGTFTTJPOTFTTJPO
EFGSFDPOTUSVDU
TFMGY
ITFMGFODPEFS
Y
STFMGEFDPEFS
I
SFUVSOTFMGTFTTJPOSVO
SGFFE@EJDU\TFMG@9Y^
EFGLM@EJW
TFMGSIPSIP@IBU
UFSN@OVNUGDPOTUBOU
SIP
UFSN@EFOUGDPOTUBOU
SIP@IBU
LMTFMGMPHGVOD
SIPSIP@IBU
TFMGMPHGVOD
UFSN@OVNUFSN@EFO
SFUVSOLM
EFGMPHGVOD
TFMGYY
SFUVSOUGNVMUJQMZ
YUGMPH
UGEJW
YY
EFGDPSSVQU
TFMGY
SFUVSOYUGDBTU
UGSBOEPN@VOJGPSN
TIBQFUGTIBQF
Y
NJOWBMNBYWBMUGGMPBU
EFGHFU8FJHIUT
TFMG
SFUVSOTFMGTFTTJPOSVO
<TFMG@8TFMG@8TFMG@CTFMG@C>
EFGGJU
TFMG99PSHFQPDITCBUDI@TJ[F
/%9TIBQF
OVN@CBUDIFT/CBUDI@TJ[F
PCK<>
GPSJJOSBOHF
FQPDIT
9TIVGGMF
9
[ 306 ]
Autoencoders Chapter 8
GPSKJOSBOHF
OVN@CBUDIFT
CBUDI9<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
CBUDI09PSH<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
@PCTFMGTFTTJPOSVO
<TFMG@PQUTFMG@MPTT>GFFE@EJDU\TFMG@9
CBUDI0TFMG@9@OPJTZCBUDI^
JGK
QSJOU
USBJOJOHFQPDI\^CBUDI\^DPTU\^ GPSNBU
JPCK
PCKBQQFOE
PC
SFUVSOPCK
It is also possible to add noise to the autoencoder object. In that case, you will use the
corrupt method defined in the class, TFMG@9@OPJTZTFMGDPSSVQU
TFMG@9
TFMG@9
, and the fit method will also change to the following:
EFGGJU
TFMG9FQPDITCBUDI@TJ[F
/%9TIBQF
OVN@CBUDIFT/CBUDI@TJ[F
PCK<>
GPSJJOSBOHF
FQPDIT
9TIVGGMF
9
GPSKJOSBOHF
OVN@CBUDIFT
CBUDI9<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
@PCTFMGTFTTJPOSVO
<TFMG@PQUTFMG@MPTT>
GFFE@EJDU\TFMG@9CBUDI^
JGK
QSJOU
USBJOJOHFQPDI\^CBUDI\^DPTU
\^ GPSNBU
JPCK
PCKBQQFOE
PC
SFUVSOPCK
4. Now, we use the corruption function defined earlier to generate a noisy image
and feed it to the session:
O@IJEEFO
9USBJOUS9BTUZQF
OQGMPBU
9USBJO@OPJTZDPSSVQUJPO
9USBJOBTUZQF
OQGMPBU
9UFTUUF9BTUZQF
OQGMPBU
OPJTF9UFTUOQSBOEPNSBOEJOU
9UFTUTIBQFBTUZQF
OQGMPBU
9UFTU@OPJTZDPSSVQUJPO
9UFTUBTUZQF
OQGMPBU9UFTU
OPJTF
@N9USBJOTIBQF
EBF%FOPJTJOH"VUP&ODPEFS
NO@IJEEFO
*OJUJBMJ[FBMMWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
[ 307 ]
Autoencoders Chapter 8
TFTTSVO
JOJU
EBFTFU@TFTTJPO
TFTT
FSSEBFGJU
9USBJO@OPJTZ9USBJOFQPDIT
PVUEBFSFDPOTUSVDU
9UFTU@OPJTZ<>
88CCEBFHFU8FJHIUT
SFEEBFSFEVDFE@EJNFOTJPO
9USBJO
QMUQMPU
FSS
QMUYMBCFM
FQPDIT
QMUZMBCFM
3FDPOTUSVDUJPO-PTT
.4&
6. The reconstructed images, when noisy images from the test dataset are presented
to the trained network, are as follows:
1MPUUJOHPSJHJOBMBOESFDPOTUSVDUFEJNBHFT
SPXDPM
JEYOQSBOEPNSBOEJOU
SPXDPM
GBYBSSQMUTVCQMPUT
SPXDPMTIBSFY5SVFTIBSFZ5SVFGJHTJ[F
GPSGJHSPXJO[JQ
<9UFTU@OPJTZPVU>BYBSS
GPSJBYJO[JQ
JEYSPX
BYJNTIPX
GJH<J>SFTIBQF
[ 308 ]
Autoencoders Chapter 8
See Also
IUUQTDTTUBOGPSEFEVQFPQMFLBSQBUIZDPOWOFUKTEFNPBVUPFODPEFS
IUNM
IUUQCMBDLFDIPHJUIVCJPCMPHNBDIJOFMFBSOJOHEFOPJTJOH
BVUPFODPEFSUFOTPSGMPXIUNM
Convolutional autoencoders
Researchers have found that Convolutional Neural Networks (CNN) work best with
images because they can extract the spatial information hidden in the image. It is thus
natural to assume that if encoder and decoder network consists of CNN, it will work better
than the rest of the autoencoders, and so we have Convolutional Autoencoders (CAE). In
Chapter 4, Convolutional Neural Networks, the process of convolution and max-pooling
was explained, which we will use as a base to understand how convolutional autoencoders
work.
A CAE is one where both the encoder and decoder are CNN networks. The convolutional
network of the encoder learns to encode the input as a set of signals and then the decoder
CNN tries to reconstruct the input from them. They work as general purpose feature
extractors and learn the optimal filters needed to capture the features from the input.
[ 309 ]
Autoencoders Chapter 8
Getting Ready...
From Chapter 4, Convolutional Neural Network, you had learned that, as convolutional layers
are added, the information passed to the next layer reduces in spatial extent, but in an
autoencoder, the reconstructed image should be of the same size and depth as the input
image. This implies that the decoder should somehow do both image resizing and
convolution to reconstruct the original image. One way of increasing the spatial extent
along with convolution is with the help of transposed convolution layers. These are easily
implemented in TensorFlow by UGOODPOWE@USBOTQPTF, but it was found that
transposed convolution layers result in artefacts in the final image. Augustus Odena et al.
[1], in their work, showed that these artefacts can be avoided by resizing the layers using
either the nearest neighbour or bilinear interpolation (upsampling) followed by a
convolutional layer. They achieved best results with nearest neighbour interpolation,
implemented via UGJNBHFSFTJ[F@JNBHFT; we will follow the same approach here.
How to do it...
1. The first step, as always, is including the necessary modules:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
JNQPSUNBUQMPUMJCQZQMPUBTQMU
JNQPSUNBUI
NBUQMPUMJCJOMJOF
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB
US9US:UF9UF:NOJTUUSBJOJNBHFTNOJTUUSBJOMBCFMT
NOJTUUFTUJNBHFTNOJTUUFTUMBCFMT
3. Define the network parameters. Here, we also calculate the spatial dimensions of
the output of each max-pool layer; we need this information to upsample the
image in the decoder network:
/FUXPSL1BSBNFUFST
I@JOX@JO*NBHFTJ[FIFJHIUBOEXJEUI
L,FSOFMTJ[F
QQPPM
T4USJEFTJONBYQPPM
GJMUFST\^
[ 310 ]
Autoencoders Chapter 8
BDUJWBUJPO@GOUGOOSFMV
$IBOHFJOEJNFOTJPOTPGJNBHFBGUFSFBDI.BY1PPM
I@MX@MJOU
OQDFJM
GMPBU
I@JOGMPBU
T
JOU
OQDFJM
GMPBU
X@JOGMPBU
T)FJHIUBOEXJEUITFDPOE
FODPEFSEFDPEFSMBZFS
I@MX@MJOU
OQDFJM
GMPBU
I@MGMPBU
T
JOU
OQDFJM
GMPBU
X@MGMPBU
T)FJHIUBOEXJEUIUIJSE
FODPEFSEFDPEFSMBZFS
4. Create placeholders for input (noisy image) and target (corresponding clear
image):
9@OPJTZUGQMBDFIPMEFS
UGGMPBU
/POFI@JOX@JOOBNF JOQVUT
9UGQMBDFIPMEFS
UGGMPBU
/POFI@JOX@JOOBNF UBSHFUT
&ODPEFS
DPOWUGMBZFSTDPOWE
9@OPJTZGJMUFST<>
LLQBEEJOH TBNF
BDUJWBUJPOBDUJWBUJPO@GO
0VUQVUTJ[FI@JOYX@JOYGJMUFST<>
NBYQPPMUGMBZFSTNBY@QPPMJOHE
DPOW
QQ
TTQBEEJOH TBNF
0VUQVUTJ[FI@MYX@MYGJMUFST<>
DPOWUGMBZFSTDPOWE
NBYQPPMGJMUFST<>
LLQBEEJOH TBNF
BDUJWBUJPOBDUJWBUJPO@GO
0VUQVUTJ[FI@MYX@MYGJMUFST<>
NBYQPPMUGMBZFSTNBY@QPPMJOHE
DPOW
QQ
TTQBEEJOH TBNF
0VUQVUTJ[FI@MYX@MYGJMUFST<>
DPOWUGMBZFSTDPOWE
NBYQPPMGJMUFST<>
LLQBEEJOH TBNF
BDUJWBUJPOBDUJWBUJPO@GO
0VUQVUTJ[FI@MYX@MYGJMUFST<>
FODPEFEUGMBZFSTNBY@QPPMJOHE
DPOW
QQ
TTQBEEJOH TBNF
0VUQVUTJ[FI@MTYX@MTYGJMUFST<>/PXYY
%FDPEFS
VQTBNQMFUGJNBHFSFTJ[F@OFBSFTU@OFJHICPS
FODPEFE
I@MX@M
0VUQVUTJ[FI@MYX@MYGJMUFST<>
DPOWUGMBZFSTDPOWE
VQTBNQMFGJMUFST<>
LLQBEEJOH TBNF
BDUJWBUJPOBDUJWBUJPO@GO
0VUQVUTJ[FI@MYX@MYGJMUFST<>
VQTBNQMFUGJNBHFSFTJ[F@OFBSFTU@OFJHICPS
DPOW
I@MX@M
0VUQVUTJ[FI@MYX@MYGJMUFST<>
DPOWUGMBZFSTDPOWE
VQTBNQMFGJMUFST<>
LLQBEEJOH TBNF
BDUJWBUJPOBDUJWBUJPO@GO
0VUQVUTJ[FI@MYX@MYGJMUFST<>
VQTBNQMFUGJNBHFSFTJ[F@OFBSFTU@OFJHICPS
DPOW
I@JOX@JO
0VUQVUTJ[FI@JOYX@JOYGJMUFST<>
[ 311 ]
Autoencoders Chapter 8
DPOWUGMBZFSTDPOWE
VQTBNQMFGJMUFST<>
LLQBEEJOH TBNF
BDUJWBUJPOBDUJWBUJPO@GO
0VUQVUTJ[FI@JOYX@JOYGJMUFST<>
MPHJUTUGMBZFSTDPOWE
DPOW
LLQBEEJOH TBNF
BDUJWBUJPO/POF
0VUQVUTJ[FI@JOYX@JOY
EFDPEFEUGOOTJHNPJE
MPHJUTOBNF EFDPEFE
MPTTUGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MBCFMT9MPHJUTMPHJUT
DPTUUGSFEVDF@NFBO
MPTT
PQUUGUSBJO"EBN0QUJNJ[FS
NJOJNJ[F
DPTU
TFTTUG4FTTJPO
FQPDIT
CBUDI@TJ[F
4FU TIPXNVDIOPJTFXF SFBEEJOHUPUIF./*45JNBHFT
OPJTF@GBDUPS
TFTTSVO
UGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
FSS<>
GPSJJOSBOHF
FQPDIT
GPSJJJOSBOHF
NOJTUUSBJOOVN@FYBNQMFTCBUDI@TJ[F
CBUDINOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
(FUJNBHFTGSPNUIFCBUDI
JNHTCBUDI<>SFTIBQF
I@JOX@JO
"EESBOEPNOPJTFUPUIFJOQVUJNBHFT
OPJTZ@JNHTJNHT
OPJTF@GBDUPSOQSBOEPNSBOEO
JNHTTIBQF
$MJQUIFJNBHFTUPCFCFUXFFOBOE
OPJTZ@JNHTOQDMJQ
OPJTZ@JNHT
/PJTZJNBHFTBTJOQVUTPSJHJOBMJNBHFTBTUBSHFUT
CBUDI@DPTU@TFTTSVO
<DPTUPQU>GFFE@EJDU\9@OPJTZOPJTZ@JNHT9
JNHT^
FSSBQQFOE
CBUDI@DPTU
JGJJ
QSJOU
&QPDI\^\^5SBJOJOHMPTT\^GPSNBU
JFQPDITCBUDI@DPTU
[ 312 ]
Autoencoders Chapter 8
QMUQMPU
FSS
QMUYMBCFM
FQPDIT
QMUZMBCFM
$SPTT&OUSPQZ-PTT
GJHBYFTQMUTVCQMPUT
SPXTDPMTTIBSFY5SVFTIBSFZ5SVF
GJHTJ[F
JO@JNHTNOJTUUFTUJNBHFT<>
OPJTZ@JNHTJO@JNHT
OPJTF@GBDUPSOQSBOEPNSBOEO
JO@JNHTTIBQF
OPJTZ@JNHTOQDMJQ
OPJTZ@JNHT
SFDPOTUSVDUFETFTTSVO
EFDPEFEGFFE@EJDU\9@OPJTZ
OPJTZ@JNHTSFTIBQF
^
GPSJNBHFTSPXJO[JQ
<OPJTZ@JNHTSFDPOTUSVDUFE>BYFT
GPSJNHBYJO[JQ
JNBHFTSPX
BYJNTIPX
JNHSFTIBQF
[ 313 ]
Autoencoders Chapter 8
TFTTDMPTF
How it Works...
The preceding CAE is a denoising CAE, and we can see that it is better in denoising the
image as compared to the simple denoising autoencoder consisting of only one bottleneck
layer.
There's More...
CAE has been used by researchers for semantic segmentation. An interesting read is a 2015
paper by Badrinayanan et al, Segnet: a Deep Convolutional Encoder-Decoder Architecture for
Image Segmentation (IUUQTBSYJWPSHQEGQEG). The network uses the
convolutional layers of VGG16 as its Encoder Network and contains a hierarchy of
decoders, one corresponding to each encoder as its Decoder Network. The decoders use the
max-pooling indices received from the corresponding Encoder and perform nonlinear
upsampling of the input feature map. The link of the paper is given in the See Also section
of this recipe along with the GitHub link.
See Also
1. IUUQTEJTUJMMQVCEFDPOWDIFDLFSCPBSE
2. IUUQTQHBMFPOFFVOFVSBMOFUXPSLTDPOWPMVUJPOBM
BVUPFODPEFST
3. IUUQTBSYJWPSHQEGQEG
4. IUUQTHJUIVCDPNBSBIVTLZ5FOTPSGMPX4FHNFOUBUJPO
[ 314 ]
Autoencoders Chapter 8
Stacked autoencoder
Autoencoders covered so far (except for CAEs) consisted only of a single-layer encoder and
a single-layer decoder. However, it is possible for us to have multiple layers in encoder and
decoder networks; using deeper encoder and decoder networks can allow the autoencoder
to represent complex features. The structure so obtained is called a Stacked Autoencoder
(Deep Autoencoders); the features extracted by one encoder are passed on to the next
encoder as input. The stacked autoencoder can be trained as a whole network with an aim
to minimize the reconstruction error, or each individual encoder/decoder network can be
first pretrained using the unsupervised method you learned earlier, and then the complete
network is fine-tuned. It has been pointed out that, by pretraining, also called Greedy layer-
wise training, the results are better.
Getting Ready
In the recipe we will use the Greedy layer-wise approach to train the stacked autoencoders;
to make the task easier, we will use shared weights, thus the corresponding
encoder/decoder weights would be the transpose of each other.
How to do it...
We proceed with the recipe as follows:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
JNQPSUNBUQMPUMJCQZQMPUBTQMU
NBUQMPUMJCJOMJOF
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB
US9US:UF9UF:NOJTUUSBJOJNBHFTNOJTUUSBJOMBCFMT
NOJTUUFTUJNBHFTNOJTUUFTUMBCFMT
[ 315 ]
Autoencoders Chapter 8
DMBTT4UBDLFE"VUP&ODPEFS
PCKFDU
EFG@@JOJU@@
TFMGMJTUFUB
MJTU<JOQVU@EJNFOTJPOIJEEFO@MBZFS@IJEEFO@MBZFS@O>
/MFO
MJTU
TFMG@NMJTU<>
TFMGMFBSOJOH@SBUFFUB
$SFBUFUIF$PNQVUBUJPOBMHSBQI
TFMG@8\^
TFMG@C\^
TFMG@9\^
TFMG@9< >UGQMBDFIPMEFS
GMPBU </POFMJTU<>>
GPSJJOSBOHF
/
MBZFS \^ GPSNBU
J
QSJOU
"VUP&ODPEFS-BZFS\^\^ \^ GPSNBU
MBZFSMJTU<J>
MJTU<J
>
TFMG@8< &
MBZFS>UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
MJTU<J>
MJTU<J
>OBNF 8UT&ODPEFS
MBZFS
TFMG@C< &
MBZFS>
UG7BSJBCMF
OQ[FSPT
MJTU<J
>BTUZQF
OQGMPBUOBNF #JBT&ODPEFS
MBZF
S
TFMG@9<MBZFS>UGQMBDFIPMEFS
GMPBU </POFMJTU<J
>>
TFMG@8< %
MBZFS>UGUSBOTQPTF
TFMG@8< &
MBZFS>4IBSFEXFJHIUT
TFMG@C< %
MBZFS>
UG7BSJBCMF
OQ[FSPT
MJTU<J>BTUZQF
OQGMPBUOBNF #JBT%FDPEFS
MBZFS
1MBDFIPMEFSGPSJOQVUT
TFMG@9@OPJTZUGQMBDFIPMEFS
GMPBU </POFTFMG@N>
[ 316 ]
Autoencoders Chapter 8
TFMGUSBJO@PQT\^
TFMGPVU\^
GPSJJOSBOHF
/
MBZFS \^ GPSNBU
J
QSFW@MBZFS \^ GPSNBU
J
PQUTFMGQSFUSBJO
TFMG@9<QSFW@MBZFS>MBZFS
TFMGUSBJO@PQT<MBZFS>PQU
TFMGPVU<MBZFS>TFMGPOF@QBTT
TFMG@9<QSFW@MBZFS>TFMG@8< &
MBZFS>
TFMG@C< &
MBZFS>TFMG@C< %
MBZFS>
5. We build the computation graph for the fine-tuning of the complete stacked
autoencoder. To do this, we make use of the class methods, FODPEFS and
EFDPEFS:
TFMGZTFMGFODPEFS
TFMG@9@OPJTZ/&ODPEFSPVUQVU
TFMGSTFMGEFDPEFS
TFMGZ/%FDPEFSPVQVU
PQUJNJ[FSUGUSBJO"EBN0QUJNJ[FS
TFMGMFBSOJOH@SBUF
FSSPSTFMG@9< >TFMGS3FDPOTUSVDUJPO&SSPS
TFMG@MPTTUGSFEVDF@NFBO
UGQPX
FSSPS
TFMG@PQUPQUJNJ[FSNJOJNJ[F
TFMG@MPTT
6. Finally, we define the class method, GJU, to perform the batch-wise pretraining of
each autoencoder, followed by fine-tuning. While pretraining, we use the non-
corrupted input, and for fine-tuning, we use the corrupted input. This allows us
to use the stacked autoencoder to reconstruct even from the noisy input:
EFGGJU
TFMG9USBJO9US@OPJTZMBZFSTFQPDITCBUDI@TJ[F
/%9USBJOTIBQF
OVN@CBUDIFT/CBUDI@TJ[F
9@OPJTZ\^
9\^
9@OPJTZ< >9US@OPJTZ
9< >9USBJO
[ 317 ]
Autoencoders Chapter 8
GPSJJOSBOHF
MBZFST
9JO9<TUS
J>
QSJOU
1SFUSBJOJOH-BZFS J
GPSFJOSBOHF
GPSKJOSBOHF
OVN@CBUDIFT
CBUDI9JO<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
TFMGTFTTJPOSVO
TFMGUSBJO@PQT<TUS
J
>GFFE@EJDU\TFMG@9<TUS
J>
CBUDI^
QSJOU
1SFUSBJOJOH'JOJTIFE
9<TUS
J
>TFMGTFTTJPOSVO
TFMGPVU<TUS
J
>GFFE@EJDU
\TFMG@9<TUS
J>9JO^
PCK<>
GPSJJOSBOHF
FQPDIT
GPSKJOSBOHF
OVN@CBUDIFT
CBUDI9USBJO<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
CBUDI@OPJTZ9US@OPJTZ<KCBUDI@TJ[F
KCBUDI@TJ[F
CBUDI@TJ[F>
@PCTFMGTFTTJPOSVO
<TFMG@PQUTFMG@MPTT>GFFE@EJDU\TFMG@9< >
CBUDITFMG@9@OPJTZCBUDI@OPJTZ^
JGK
QSJOU
USBJOJOHFQPDI\^CBUDI\^DPTU\^ GPSNBU
JPCK
PCKBQQFOE
PC
SFUVSOPCK
EFGFODPEFS
TFMG9/
Y9
GPSJJOSBOHF
/
MBZFS \^ GPSNBU
J
IJEEFO&UGOOTJHNPJE
UGNBUNVM
YTFMG@8< &
MBZFS>
TFMG@C< &
MBZFS>
YIJEEFO&
SFUVSOY
EFGEFDPEFS
TFMG9/
Y9
GPSJJOSBOHF
/
MBZFS \^ GPSNBU
J
IJEEFO%UGOOTJHNPJE
UGNBUNVM
YTFMG@8< %
MBZFS>
TFMG@C< %
MBZFS>
YIJEEFO%
SFUVSOY
EFGTFU@TFTTJPO
TFMGTFTTJPO
[ 318 ]
Autoencoders Chapter 8
TFMGTFTTJPOTFTTJPO
EFGSFDPOTUSVDU
TFMGYO@MBZFST
ITFMGFODPEFS
YO@MBZFST
STFMGEFDPEFS
IO@MBZFST
SFUVSOTFMGTFTTJPOSVO
SGFFE@EJDU\TFMG@9< >Y^
EFGQSFUSBJO
TFMG9MBZFS
ZUGOOTJHNPJE
UGNBUNVM
9TFMG@8< &
MBZFS>
TFMG@C< &
MBZFS>
SUGOOTJHNPJE
UGNBUNVM
ZTFMG@8< %
MBZFS>
TFMG@C< %
MBZFS>
0CKFDUJWF'VODUJPO
FSSPS9S3FDPOTUSVDUJPO&SSPS
MPTTUGSFEVDF@NFBO
UGQPX
FSSPS
PQUUGUSBJO"EBN0QUJNJ[FS
NJOJNJ[F
MPTTWBS@MJTU
<TFMG@8< &
MBZFS>TFMG@C< &
MBZFS>TFMG@C< %
MBZFS>>
SFUVSOPQU
EFGPOF@QBTT
TFMG98CD
IUGOOTJHNPJE
UGNBUNVM
98
C
SFUVSOI
9USBJOUS9BTUZQF
OQGMPBU
9USBJO@OPJTZDPSSVQUJPO
9USBJOBTUZQF
OQGMPBU
9UFTUUF9BTUZQF
OQGMPBU
9UFTU@OPJTZDPSSVQUJPO
9UFTUBTUZQF
OQGMPBU
@N9USBJOTIBQF
MJTU<N>-JTUXJUIOVNCFSPGOFVSPOTJO&BDIIJEEFOMBZFS
TUBSUJOHGSPNJOQVUMBZFS
O@MBZFSTMFO
MJTU
BVUP&ODPEFS4UBDLFE"VUP&ODPEFS
MJTU
*OJUJBMJ[FBMMWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
JOJU
BVUP&ODPEFSTFU@TFTTJPO
TFTT
FSSBVUP&ODPEFSGJU
9USBJO9USBJO@OPJTZO@MBZFSTFQPDIT
PVUBVUP&ODPEFSSFDPOTUSVDU
9UFTU@OPJTZ<>O@MBZFST
[ 319 ]
Autoencoders Chapter 8
QMUQMPU
FSS
QMUYMBCFM
FQPDIT
QMUZMBCFM
'JOF5VOJOH3FDPOTUSVDUJPO&SSPS
10. Let's now check the performance of our network. Here is the denoised
handwritten images, when the network is presented with the noisy test images:
How it works...
The experiments on the stacked autoencoder suggest that pretraining should be done with
low values of learning rate. This ensures better convergence and performance during the
fine tuning.
[ 320 ]
Autoencoders Chapter 8
There's More...
This whole chapter was about autoencoders, and, while right now they are used only for
dimensionality reduction and information retrieval, they generate a lot of interest. Firstly,
because they are unsupervised, and secondly because they can be employed along with
FCNs. They can help us deal with the curse of dimensionality. Researchers have
demonstrated that they can also be used for classification and anomaly detection.
See Also
A nice tutorial on stacked autoencoders: IUUQVGMEMTUBOGPSEFEVXJLJ
JOEFYQIQ4UBDLFE@"VUPFODPEFST.
Schwenk, Holger. "The diabolo classifier." Neural Computation 10.8 (1998):
2175-2200.
Sakurada, Mayu, and Takehisa Yairi. "Anomaly detection using autoencoders
with nonlinear dimensionality reduction." Proceedings of the MLSDA 2014 2nd
Workshop on Machine Learning for Sensory Data Analysis. ACM, 2014.
Cool TensorBoard visualization and implementation of stacked autoencoders:
IUUQTHJUIVCDPNDNHSFFO5FOTPS'MPX%FFQ"VUPFODPEFS
[ 321 ]
Reinforcement Learning
9
This chapter introduces reinforcement learning (RL)the least explored and yet most
promising learning paradigm. The chapter includes the following topics:
Introduction
In March 2016, AlphaGo--the program made by Google's DeepMind--defeated the world's
best Go player, 18-time world champion Lee Sedol, by 4 to 1. The match was historic
because Go is a notoriously difficult game for computers to play, with:
possible legal board positions. Playing and winning Go cannot be done by simple brute
force. It requires skill, creativity, and, as professional Go players say, intuition.
This remarkable feat was accomplished by AlphaGo with the help of RL algorithm-based
deep neural networks combined with a state-of-the-art tree search algorithm. This chapter
introduces RL and some algorithms that we employ to perform RL.
Reinforcement Learning Chapter 9
So, the first question that arises is what is RL and how is it different from supervised and
unsupervised learning, which we explored in earlier chapters?
Anyone who owns a pet knows that the best strategy to train a pet is rewarding it for
desirable behavior and punishing it for bad behavior. The RL, also called learning with a
critic, is a learning paradigm where the agent learns in the same manner. The Agent here
corresponds to our network (program); it can perform a set of Actions (a), which brings
about a change in the State (s) of the environment and, in turn, the Agent perceives whether
it gets a reward or punishment.
For example, in the case of a dog, the dog is our Agent, the voluntary muscle movements
that the dog makes are the actions, and the ground is the environment; the dog perceives
our reaction to its action in terms of giving it a bone as a reward:
#FCRVGFHTQO4GKPHQTEGOGPV.GCTPKPICP+PVTQFWEVKQPD[5WVVQPCPF$CTVQ
Even our brain has a group of subcortical nuclei situated at the base of the
forebrain called basal ganglia, which, according to neuroscience, are
responsible for action selection, that is, help us decide which of several
possible actions to execute at any given time.
[ 323 ]
Reinforcement Learning Chapter 9
The aim of the agent is to maximize the rewards and reduce the punishments. There are
various challenges involved in making this decision, the most important one being how to
maximize future rewards, also known as the temporal credit assignment problem. The
agent decides its action based on some policy (); the agent learns this policy () based on
its interaction with the environment. There are various policy learning algorithms; we will
explore some of them in this chapter. The agent infers the optimal policy (*) by a process of
trial and error, and to learn the optimal policy, the agent requires an environment to
interact with; we will be using OpenAI Gym, which provides different environments.
We have given here only a review of the basic concepts involved in RL; we assume that you
are familiar with the Markov concepts decision process, discount factor, and value function
(state value and action value).
In this chapter, and the recipes that follow, we define an episode as one run of the game, for
example, solving one sudoku. Typically, an agent will play many episodes to learn an
optimal policy, the one which maximizes the rewards.
It is really amazing to see how the RL agent, without any implicit knowledge of the game,
learns to play and, not only play, even beat humans in these games.
Getting ready
The first thing to do is install OpenAI Gym; a minimal installation can be done using QJQ
JOTUBMMHZN. The OpenAI gym provides a variety of environments, such as Atari, board
games, and 2D or 3D physics engines. The minimal installation works on Windows and
supports only basic environments--algorithmic, UPZ@UFYU, and DMBTTJD@DPOUSPM--but if
you want to explore other environments, they will need more dependencies. The full make
is supported in OS X and Ubuntu. Detailed instructions can be read at OpenAI Gym's
GitHub link (IUUQTHJUIVCDPNPQFOBJHZNJOTUBMMJOHEFQFOEFODJFTGPSTQFDJGJD
FOWJSPONFOUT).
[ 324 ]
Reinforcement Learning Chapter 9
How to do it...
Let's start with the recipe:
JNQPSUHZN
FOW@OBNF #SFBLPVUW
FOWHZNNBLF
FOW@OBNF
PCTFOWSFTFU
QSJOU
PCTTIBQF
6. The number of actions possible can be checked using the command BDUJPOT
FOWBDUJPO@TQBDF. We can see from the result of this that, for Breakout-v4, we
have four possible actions: NoOp, Fire, Left, and Right. The total number of
actions can be obtained by calling the FOWBDUJPO@TQBDFO command.
7. Let's define an agent with a random policy. The agent chooses any of the four
possible actions randomly:
EFGSBOEPN@QPMJDZ
O
BDUJPOOQSBOEPNSBOEJOU
O
SFUVSOBDUJPO
8. We next allow our random agent to play for 1,000 steps, using PCTSFXBSE
EPOFJOGPFOWTUFQ
BDUJPO:
GPSTUFQJOSBOHF
TUFQTNBY
BDUJPOSBOEPN@QPMJDZ
FOWBDUJPO@TQBDFO
PCTSFXBSEEPOFJOGPFOWTUFQ
BDUJPO
[ 325 ]
Reinforcement Learning Chapter 9
FOWSFOEFS
JGEPOF
JNHFOWSFOEFS
NPEF SHC@BSSBZ
QMUJNTIPX
JNH
QMUTIPX
QSJOU
5IFHBNFJTPWFSJO\^TUFQTGPSNBU
TUFQ
CSFBL
The PCT tells the agent what the environment looks like; for our environment, it
corresponds to an RGB image of size 210 x 160 x 3. The agent is rewarded either 0
or 1 at each step, the SFXBSE as per the OpenAI Gym wiki is <JOGJOG>. Once
the game is over, the environment returns EPOF as 5SVF. The JOGP can be useful
for debugging but is not used by the agent. The FOWSFOEFS
command pops
up a window that shows the present state of the environment; when you include
this command, you can see through the popped-up window how the agent is
trying to play and learn. It is better to comment it when the agent is under
training to save time.
FOWDMPTF
How it works...
The preceding code implements a random agent; the agent randomly selects one of the four
actions:
[ 326 ]
Reinforcement Learning Chapter 9
Another important thing to observe is that in this environment, the action space is Discrete,
while the observation space is of type Box. The terms Discrete and Box in reference to space
(action/observation) in OpenAI refer to the allowed values. The Discrete space allows a
fixed range of non-negative numbers, in our case, (0,1,2,3). The Box space, on the other
hand, represents an n-dimensional box, thus any valid observation would be an array of 210
e 160 e 3 numbers for Pac-Man.
There's more...
OpenAI Gym consists of many different environments and many are added to it by its
active contributing community. To get a list of all the existing environments, you can run
this simple code (taken from IUUQTHJUIVCDPNPQFOBJHZN):
GSPNHZNJNQPSUFOWT
FOW@JET<TQFDJEGPSTQFDJOFOWTSFHJTUSZBMM
>
QSJOU
5PUBM/VNCFSPGFOWJSPONFOUTBSFMFO
FOW@JET
GPSFOW@JEJOTPSUFE
FOW@JET
QSJOU
FOW@JE
At present, OpenAI Gym has 777 environments in total. Here is the image of Pac-Man using
the same random agent as earlier:
[ 327 ]
Reinforcement Learning Chapter 9
See also
The details about different environments can be obtained from IUUQTHZN
PQFOBJDPNFOWT
Wiki pages are maintained for some environments at IUUQTHJUIVCDPN
PQFOBJHZNXJLJ
Details regarding installation instructions and dependencies can be obtained
from IUUQTHJUIVCDPNPQFOBJHZN
Getting ready
The agents in this recipe are not learning any policy; they make their decision based on their
initial set of weights (fixed policy). The agent picks the actions based on the probability
given by the neural network. The decision that each agent makes is based only on the
present observation of the environment.
We implement this by a fully connected neural network. The inputs to the NN are
decided by the observation space of the environment; the number of output neurons is
decided by the number of possible discrete actions. The Pac-Man game has nine discrete
possible actions--NoOp, Turn Right, Turn Left, Turn Up, Turn Down, Move Left, Move
Right, Move Up, and Move Down--and so our NN has nine output neurons.
[ 328 ]
Reinforcement Learning Chapter 9
How to do it...
Let's start with the recipe:
1. The first step, as always, is importing modules. Besides the usual modules, in this
case, we will import HZN so that we can use the different environments provided
by it:
JNQPSUHZN
JNQPSUOVNQZBTOQ
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMU
2. Next, we create a 3M"HFOU class. The class is made with three methods--the
@@JOJU@@ method initializes the NN size and creates the computational graph.
Here, we employed the TensorFlow function, UGNVMUJOPNJBM, to make the
decision of the possible action to take. The function returns the action based on
the sigmoidal values of the nine output neurons of our network. This ensures that
the network chooses the final action probabilistically. The QSFEJDU method
returns the action predicted by the NN. The HFU@XFJHIUT method helps us get
the weights and biases of the winner agent:
DMBTT3M"HFOU
PCKFDU
EFG@@JOJU@@
TFMGNOJOJ'BMTF8/POFC/POF
TFMG@HSBQIUG(SBQI
XJUITFMG@HSBQIBT@EFGBVMU
TFMG@9UGQMBDFIPMEFS
UGGMPBUTIBQF
N
JGJOJ'BMTF
TFMG8UG7BSJBCMF
UGSBOEPN@OPSNBM
<NO>
USBJOBCMF'BMTF
TFMGCJBT
UG7BSJBCMF
UGSBOEPN@OPSNBM
<O>USBJOBCMF'BMTF
FMTF
TFMG88
TFMGCJBTC
PVUUGOOTJHNPJE
UGNBUNVM
TFMG@9TFMG8
TFMGCJBT
TFMG@SFTVMUUGNVMUJOPNJBM
PVU
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFMG@TFTTUG4FTTJPO
TFMG@TFTTSVO
JOJU
EFGQSFEJDU
TFMG9
BDUJPOTFMG@TFTTSVO
TFMG@SFTVMUGFFE@EJDU\TFMG@9
9^
[ 329 ]
Reinforcement Learning Chapter 9
SFUVSOBDUJPO
EFGHFU@XFJHIUT
TFMG
8CTFMG@TFTTSVO
<TFMG8TFMGCJBT>
SFUVSO8C
EFGQMBZ@POF@FQJTPEF
FOWBHFOU
PCTFOWSFTFU
JNH@QSFQSFQSPDFTT@JNBHF
PCT
EPOF'BMTF
U
XIJMFOPUEPOFBOEU
FOWSFOEFS
5IJTDBOCFDPNNFOUFEUPTQFFEVQ
U
BDUJPOBHFOUQSFEJDU
JNH@QSF
QSJOU
UBDUJPO
PCTSFXBSEEPOFJOGPFOWTUFQ
BDUJPO
JNH@QSFQSFQSPDFTT@JNBHF
PCT
JGEPOF
CSFBL
SFUVSOU
EFGQMBZ@NVMUJQMF@FQJTPEFT
FOW5JOJ'BMTF8/POFC/POF
FQJTPEF@MFOHUITOQFNQUZ
5
PCTFOWSFTFU
JNH@QSFQSFQSPDFTT@JNBHF
PCT
JGJOJ'BMTF
BHFOU3M"HFOU
JNH@QSFTIBQF<>FOWBDUJPO@TQBDFO
FMTF
BHFOU3M"HFOU
JNH@QSFTIBQF<>FOWBDUJPO@TQBDFOJOJ8
C
GPSJJOSBOHF
5
FQJTPEF@MFOHUIT<J>QMBZ@POF@FQJTPEF
FOWBHFOU
BWH@MFOHUIFQJTPEF@MFOHUITNFBO
QSJOU
BWHMFOHUIBWH@MFOHUI
JGJOJ'BMTF
8CBHFOUHFU@XFJHIUT
SFUVSOBWH@MFOHUI8C
[ 330 ]
Reinforcement Learning Chapter 9
EFGSBOEPN@TFBSDI
FOW
FQJTPEF@MFOHUIT<>
CFTU
GPSUJOSBOHF
QSJOU
"HFOU\^SFQPSUJOHGPSNBU
U
BWH@MFOHUIXUTCJBTQMBZ@NVMUJQMF@FQJTPEFT
FOW
FQJTPEF@MFOHUITBQQFOE
BWH@MFOHUI
JGBWH@MFOHUI CFTU
CFTU@XUXUT
CFTU@CJBTCJBT
CFTUBWH@MFOHUI
SFUVSOFQJTPEF@MFOHUITCFTU@XUCFTU@CJBT
6. The environment returns an observation field each time it progresses a step. This
observation is with three color channels; to feed it to the NN, the observation
needs to be preprocessed and, at present, the only preprocessing we do is convert
it to grayscale, increase the contrast, and reshape it into a row vector:
EFGQSFQSPDFTT@JNBHF
JNH
JNHJNHNFBO
BYJTUPHSBZTDBMF
JNH<JNH>#SJOHBCPVUBCFUUFSDPOUSBTU
JNH
JNH/PSNBMJ[FJNBHFGSPNUP
NOJNHTIBQF
SFUVSOJNHSFTIBQF
NO
7. The NN agents are instantiated one by one and the best is selected. For
computational efficiency, we have at present searched only 10 agents, each
playing 10 games. The one that can play the longest is considered the best:
JG@@OBNF@@ @@NBJO@@
FOW@OBNF #SFBLPVUW
FOW@OBNF .T1BDNBOW
FOWHZNNBLF
FOW@OBNF
FQJTPEF@MFOHUIT8CSBOEPN@TFBSDI
FOW
QMUQMPU
FQJTPEF@MFOHUIT
QMUTIPX
QSJOU
'JOBM3VOXJUICFTU"HFOU
QMBZ@NVMUJQMF@FQJTPEFT
FOWJOJ5SVF88CC
[ 331 ]
Reinforcement Learning Chapter 9
We can see that our random agent too can play the game for an average length of 615.5. Not
bad!
Q: S i AjR
[ 332 ]
Reinforcement Learning Chapter 9
The agent starts with some arbitrary initial value of Q, and, as the agent selects an action
a and receives a reward r, it updates the state s' (which depends on the past
state s and action a) and the Q value:
Here, k is the learning rate and l is the discount factor. The first term preserves the old
value of Q, and the second term provides an improved estimate of the Q value (it includes
the present reward and discounted rewards for future actions). This will reduce the Q value
when the resultant state is undesirable, thus ensuring that the agent will not choose the
same action the next time this state is encountered. Similarly, when the resultant state is
desirable, the corresponding Q value will increase.
Getting ready
We will train a linear neural network to solve the $BSU1PMFW environment (IUUQT
HJUIVCDPNPQFOBJHZNXJLJ$BSU1PMFW). The goal here is to balance the pole on the
cart; the observation state consists of four continuous-valued parameters: Cart Position
[-2.4, 2.4], Cart Velocity [-, ], Pole Angle [~-41.8, ~41.8], and Pole Velocity at Tip [-,
]. The balancing can be achieved by pushing the cart either to left or right, so the action
space consists of two possible actions. You can see the $BSU1PMFW environment space:
[ 333 ]
Reinforcement Learning Chapter 9
Now, for Q learning, we need to find a way to quantize the continuous-valued observation
states. This is achieved using the class 'FBUVSF5SBOTGPSN; the class first generates 20,000
random samples of observation space examples. The randomly generated observation space
examples are standardized using the scikit 4UBOEBSE4DBMFS class. Then scikit's
3#'4BNQMFS is employed with different variances to cover different parts of the
observation space. The 'FBUVSF5SBOTGPSNFS class is instantiated with the random
observation space examples, which are used to train the 3#'4BNQMFS using
the GJU@USBOTGPSN function method.
Later, the USBOTGPSN method is employed to transform the continuous observation space
to this featurized representation:
DMBTT'FBUVSF5SBOTGPSNFS
EFG@@JOJU@@
TFMGFOW
PCT@FYBNQMFTOQSBOEPNSBOEPN
QSJOU
PCT@FYBNQMFTTIBQF
TDBMFS4UBOEBSE4DBMFS
TDBMFSGJU
PCT@FYBNQMFT
6TFEUPDPOWFSUFBTUBUFUPBGFBUVSJ[FTSFQSFTFOBUJPO
8FVTF3#'LFSOFMTXJUIEJGGFSFOUWBSJBODFTUPDPWFSEJGGFSFOUQBSUT
PGUIFTQBDF
GFBUVSJ[FS'FBUVSF6OJPO
<
DBSU@QPTJUJPO3#'4BNQMFS
HBNNBO@DPNQPOFOUT
DBSU@WFMPDJUZ3#'4BNQMFS
HBNNBO@DPNQPOFOUT
QPMF@BOHMF3#'4BNQMFS
HBNNBO@DPNQPOFOUT
[ 334 ]
Reinforcement Learning Chapter 9
QPMF@WFMPDJUZ3#'4BNQMFS
HBNNBO@DPNQPOFOUT
>
GFBUVSF@FYBNQMFT
GFBUVSJ[FSGJU@USBOTGPSN
TDBMFSUSBOTGPSN
PCT@FYBNQMFT
QSJOU
GFBUVSF@FYBNQMFTTIBQF
TFMGEJNFOTJPOTGFBUVSF@FYBNQMFTTIBQF<>
TFMGTDBMFSTDBMFS
TFMGGFBUVSJ[FSGFBUVSJ[FS
EFGUSBOTGPSN
TFMGPCTFSWBUJPOT
TDBMFETFMGTDBMFSUSBOTGPSN
PCTFSWBUJPOT
SFUVSOTFMGGFBUVSJ[FSUSBOTGPSN
TDBMFE
How to do it...
We proceed with the recipe as follows:
1. The first step is importing the necessary modules. This time, besides our usual
TensorFlow, Numpy, and Matplotlib, we will be importing Gym and some
classes from scikit:
JNQPSUOVNQZBTOQ
JNQPSUUFOTPSGMPXBTUG
JNQPSUHZN
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNTLMFBSOQJQFMJOFJNQPSU'FBUVSF6OJPO
GSPNTLMFBSOQSFQSPDFTTJOHJNQPSU4UBOEBSE4DBMFS
GSPNTLMFBSOLFSOFM@BQQSPYJNBUJPOJNQPSU3#'4BNQMFS
DMBTT/FVSBM/FUXPSL
EFG@@JOJU@@
TFMG%
FUB
TFMG8UG7BSJBCMF
UGSBOEPN@OPSNBM
TIBQF
%OBNF X
TFMG9UGQMBDFIPMEFS
UGGMPBUTIBQF
/POF%OBNF 9
TFMG:UGQMBDFIPMEFS
UGGMPBUTIBQF
/POFOBNF :
[ 335 ]
Reinforcement Learning Chapter 9
NBLFQSFEJDUJPOBOEDPTU
:@IBUUGSFTIBQF
UGNBUNVM
TFMG9TFMG8<>
FSSTFMG::@IBU
DPTUUGSFEVDF@TVN
UGQPX
FSS
PQTXFXBOUUPDBMMMBUFS
TFMGUSBJO@PQ
UGUSBJO(SBEJFOU%FTDFOU0QUJNJ[FS
FUBNJOJNJ[F
DPTU
TFMGQSFEJDU@PQ:@IBU
TUBSUUIFTFTTJPOBOEJOJUJBMJ[FQBSBNT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFMGTFTTJPOUG4FTTJPO
TFMGTFTTJPOSVO
JOJU
EFGUSBJO
TFMG9:
TFMGTFTTJPOSVO
TFMGUSBJO@PQGFFE@EJDU\TFMG99TFMG:
:^
EFGQSFEJDU
TFMG9
SFUVSOTFMGTFTTJPOSVO
TFMGQSFEJDU@PQGFFE@EJDU\TFMG99^
3. The next important class is the "HFOU class that uses the /FVSBM/FUXPSL class to
create a learning agent. The class on instantiation creates an agent with two linear
NNs, each with 2,000 input neurons and 1 output neuron. (Essentially, this means
that the agent has 2 neurons each with 2,000 inputs as the input layer of the NN
does not do any processing). The "HFOU class has methods defined to predict the
output of the two NNs and update the weights of the two NNs. The agent here
uses Epsilon Greedy Policy for exploration during the training phase. At each
step, the agent either chooses an action with the highest Q value or a random
action, depending on the value of epsilon (FQT); epsilon is annealed during the
training process so that, initially, the agent takes lots of random actions
(exploration) but as training progresses, the actions with maximum Q value are
taken (exploitation). This is called the Exploration-Exploitation trade-off: we
allow the agent to explore random actions over the exploited course of actions,
which allows the agent to try new random actions and learn from them:
DMBTT"HFOU
EFG@@JOJU@@
TFMGFOWGFBUVSF@USBOTGPSNFS
TFMGFOWFOW
TFMGBHFOU<>
TFMGGFBUVSF@USBOTGPSNFSGFBUVSF@USBOTGPSNFS
GPSJJOSBOHF
FOWBDUJPO@TQBDFO
NPEFM/FVSBM/FUXPSL
GFBUVSF@USBOTGPSNFSEJNFOTJPOT
TFMGBHFOUBQQFOE
NPEFM
[ 336 ]
Reinforcement Learning Chapter 9
EFGQSFEJDU
TFMGT
9TFMGGFBUVSF@USBOTGPSNFSUSBOTGPSN
<T>
SFUVSOOQBSSBZ
<NQSFEJDU
9<>GPSNJOTFMGBHFOU>
EFGVQEBUF
TFMGTB(
9TFMGGFBUVSF@USBOTGPSNFSUSBOTGPSN
<T>
TFMGBHFOU<B>USBJO
9<(>
EFGTBNQMF@BDUJPO
TFMGTFQT
JGOQSBOEPNSBOEPN
FQT
SFUVSOTFMGFOWBDUJPO@TQBDFTBNQMF
FMTF
SFUVSOOQBSHNBY
TFMGQSFEJDU
T
EFGQMBZ@POF
FOWNPEFMFQTHBNNB
PCTFOWSFTFU
EPOF'BMTF
UPUBMSFXBSE
JUFST
XIJMFOPUEPOFBOEJUFST
BDUJPONPEFMTBNQMF@BDUJPO
PCTFQT
QSFW@PCTPCT
PCTSFXBSEEPOFJOGPFOWTUFQ
BDUJPO
FOWSFOEFS
$BODPNNFOUJUUPTQFFEVQ
JGEPOF
SFXBSE
VQEBUFUIFNPEFM
OFYUNPEFMQSFEJDU
PCT
BTTFSU
MFO
OFYUTIBQF
(SFXBSE
HBNNBOQNBY
OFYU
NPEFMVQEBUF
QSFW@PCTBDUJPO(
[ 337 ]
Reinforcement Learning Chapter 9
JGSFXBSE
UPUBMSFXBSE
SFXBSE
JUFST
5. Now that all the functions and classes are in place, we define our agent and
environment (in this case, $BSU1PMFW ). The agent plays in total 1,000
episodes and learns by interacting with the environment with the help of the
value function:
JG@@OBNF@@ @@NBJO@@
FOW@OBNF $BSU1PMFW
FOWHZNNBLF
FOW@OBNF
GU'FBUVSF5SBOTGPSNFS
FOW
BHFOU"HFOU
FOWGU
HBNNB
/
UPUBMSFXBSETOQFNQUZ
/
SVOOJOH@BWHOQFNQUZ
/
GPSOJOSBOHF
/
FQTOQTRSU
O
UPUBMSFXBSEQMBZ@POF
FOWBHFOUFQTHBNNB
UPUBMSFXBSET<O>UPUBMSFXBSE
SVOOJOH@BWH<O>UPUBMSFXBSET<NBY
O
O
>NFBO
JGO
QSJOU
FQJTPEF\^UPUBMSFXBSE\^FQT\^BWH
SFXBSE
MBTU\^GPSNBU
OUPUBMSFXBSEFQT
SVOOJOH@BWH<O>
QSJOU
BWHSFXBSEGPSMBTUFQJTPEFT
UPUBMSFXBSET<>NFBO
QSJOU
UPUBMTUFQTUPUBMSFXBSETTVN
QMUQMPU
UPUBMSFXBSET
QMUYMBCFM
FQJTPEFT
QMUZMBCFM
5PUBM3FXBSET
QMUTIPX
QMUQMPU
SVOOJOH@BWH
QMUYMBCFM
FQJTPEFT
QMUZMBCFM
3VOOJOH"WFSBHF
QMUTIPX
FOWDMPTF
[ 338 ]
Reinforcement Learning Chapter 9
6. The following is the plot for total rewards and running average reward as the
agent learned through the game. According to the Cart-Pole wiki, a reward of 200
means that the agent won the episode after being trained for 1,000 episodes; our
agent managed to reach an average reward of 195.7 while playing 100 episodes,
which is a remarkable feat:
[ 339 ]
Reinforcement Learning Chapter 9
There's more...
The same logic can be used to make an agent for other environments of OpenAI; however,
for Atari games such as Breakout or Pac-Man, the observation space is not an array of just
four numbers. Instead, it is very large (210 e 160 = 33,600 pixels with 3 RGB values); without
some form of quantization, the possible states are infinite and do not yield good results
with this simplistic NN. We will resolve this issue using CNN in our deep Q learning
recipe.
[ 340 ]
Reinforcement Learning Chapter 9
See also
Although there are a large number of web links on Q learning, some of the useful ones are
as follows:
https://en.wikipedia.org/wiki/Q learning
IUUQNOFNTUVEJPPSHQBUIGJOEJOHRMFBSOJOHUVUPSJBMIUN
IUUQBSUJOUJOGPIUNM"SU*OU@IUNM
IUUQTNFEJVNDPNFNFSHFOUGVUVSFTJNQMFSFJOGPSDFNFOUMFBSOJOHXJUI
UFOTPSGMPXQBSURMFBSOJOHXJUIUBCMFTBOEOFVSBMOFUXPSLT
EE
This recipe is based on the original DQN paper, Playing Atari with Deep
Reinforcement Learning by DeepMind. In the paper, they used a concept
called experience replay, which involved randomly sampling the previous
game moves (state, action reward, next state).
Getting ready
As mentioned in the previous recipe, Q learning to balance CartPole, for Atari games such as
Pac-Man or Breakout, we need to preprocess the observation state space, which consists of
33,600 pixels with 3 RGB values. Each of these pixels can take any value between 0 and 255.
Our QSFQSPDFTT function should be able to quantize the pixel possible values and, at the
same time, reduce the observation state space.
We make use of Scipy's JNSFTJ[F function to downsample the image. The following
functions QSFQSPDFTT the image before it is fed to the DQN:
EFGQSFQSPDFTT
JNH
JNH@UFNQJNH<>$IPPTFUIFJNQPSUBOUBSFBPGUIFJNBHF
JNH@UFNQJNH@UFNQNFBO
BYJT$POWFSUUP(SBZTDBMF
[ 341 ]
Reinforcement Learning Chapter 9
%PXOTBNQMFJNBHFVTJOHOFBSFTUOFJHICPVSJOUFSQPMBUJPO
JNH@UFNQJNSFTJ[F
JNH@UFNQTJ[F
*.@4*;&*.@4*;&
JOUFSQ OFBSFTU
SFUVSOJNH@UFNQ
*.@4*;& is a global parameter--in the code, we take its value to be 80. The function has
comments describing each process. Here, you can see the observation space before and after
the preprocessing:
Another important thing to note is that the present observation space does not give a
complete picture of the game; for example, seeing the preceding figure, you cannot
determine whether the paddle was moving toward left or right. Thus, to completely
understand the present state of the game, we need to consider the sequences of actions and
observations. In the recipe, we consider four sequences of actions and observations to
determine the current situation and train the agent. This is accomplished with the help of
the TUBUF@VQEBUF function, which appends the present observation state to
the previous state, thus generating a sequence of states:
EFGVQEBUF@TUBUF
TUBUFPCT
PCT@TNBMMQSFQSPDFTT
PCT
SFUVSOOQBQQFOE
TUBUF<>OQFYQBOE@EJNT
PCT@TNBMMBYJT
Lastly, to deal with the stability issues while training, we use the concept of
UBSHFU@OFUXPSL, which is a copy of the DQN but is not updated as often. We use the
target network to generate the target value function for the DQN network, while the DQN
is updated at each step/episode and the UBSHFU@OFUXPSL is updated (made the same as
DQN) after a regular interval. As all updates take place within the TensorFlow session, we
use the name scope to differentiate the UBSHFU@OFUXPSL and DQN network.
[ 342 ]
Reinforcement Learning Chapter 9
How to do it...
1. We import the necessary modules. We are using the TZT module's
TUEPVUGMVTI
to help us force Python to flush the data in the standard output
(computer monitor in our case). The SBOEPN module is used to derive random
samples from the experience replay buffer (the buffer where we store the past
experience). The EBUFUJNF module is used to keep track of the time spent in
training:
JNQPSUHZN
JNQPSUTZT
JNQPSUSBOEPN
JNQPSUOVNQZBTOQ
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMU
GSPNEBUFUJNFJNQPSUEBUFUJNF
GSPNTDJQZNJTDJNQPSUJNSFTJ[F
2. We define the hyperparameters for the training; you can experiment by changing
them. These parameters define the minimum and maximum size of the
experience replay buffer and the number of episodes after which the target
network is updated:
."9@&91&3*&/$&4
.*/@&91&3*&/$&4
5"3(&5@61%"5&@1&3*0%
*.@4*;&
,
3. The class %2/ is defined; its constructor builds the CNN network using
the UGDPOUSJCMBZFSTDPOWE function and defines the cost and training ops:
DMBTT%2/
EFG@@JOJU@@
TFMG,TDPQFTBWF@QBUI NPEFMTBUBSJDLQU
TFMG,,
TFMGTDPQFTDPQF
TFMGTBWF@QBUITBWF@QBUI
XJUIUGWBSJBCMF@TDPQF
TDPQF
JOQVUTBOEUBSHFUT
TFMG9UGQMBDFIPMEFS
UGGMPBUTIBQF
/POF
*.@4*;&*.@4*;&OBNF 9
[ 343 ]
Reinforcement Learning Chapter 9
UFOTPSGMPXDPOWPMVUJPOOFFETUIFPSEFSUPCF
OVN@TBNQMFTIFJHIUXJEUIDPMPS
TPXFOFFEUPUSBOQPTFMBUFS
TFMG(UGQMBDFIPMEFS
UGGMPBUTIBQF
/POF
OBNF (
TFMGBDUJPOTUGQMBDFIPMEFS
UGJOUTIBQF
/POF
OBNF BDUJPOT
DBMDVMBUFPVUQVUBOEDPTU
DPOWPMVUJPOBMMBZFST
;TFMG9
;UGUSBOTQPTF
;<>
DOOUGDPOUSJCMBZFSTDPOWE
;
BDUJWBUJPO@GOUGOOSFMV
DOOUGDPOUSJCMBZFSTDPOWE
DOO
BDUJWBUJPO@GOUGOOSFMV
DOOUGDPOUSJCMBZFSTDPOWE
DOO
BDUJWBUJPO@GOUGOOSFMV
GVMMZDPOOFDUFEMBZFST
GDUGDPOUSJCMBZFSTGMBUUFO
DOO
GDUGDPOUSJCMBZFSTGVMMZ@DPOOFDUFE
GD
GJOBMPVUQVUMBZFS
TFMGQSFEJDU@PQ
UGDPOUSJCMBZFSTGVMMZ@DPOOFDUFE
GD,
TFMFDUFE@BDUJPO@WBMVFTUGSFEVDF@TVN
TFMGQSFEJDU@PQ
UGPOF@IPU
TFMGBDUJPOT,
SFEVDUJPO@JOEJDFT<>
TFMGDPTUUGSFEVDF@NFBO
UGTRVBSF
TFMG(
TFMFDUFE@BDUJPO@WBMVFT
TFMGUSBJO@PQUGUSBJO3.41SPQ0QUJNJ[FS
FNJOJNJ[F
TFMGDPTU
EFGTFU@TFTTJPO
TFMGTFTTJPO
TFMGTFTTJPOTFTTJPO
EFGQSFEJDU
TFMGTUBUFT
SFUVSOTFMGTFTTJPOSVO
TFMGQSFEJDU@PQGFFE@EJDU\TFMG9
TUBUFT^
[ 344 ]
Reinforcement Learning Chapter 9
EFGVQEBUF
TFMGTUBUFTBDUJPOTUBSHFUT
D@TFMGTFTTJPOSVO
<TFMGDPTUTFMGUSBJO@PQ>
GFFE@EJDU\
TFMG9TUBUFT
TFMG(UBSHFUT
TFMGBDUJPOTBDUJPOT
^
SFUVSOD
EFGTBNQMF@BDUJPO
TFMGYFQT
*NQMFNFOUTFQTJMPOHSFFEZBMHPSJUIN
JGOQSBOEPNSBOEPN
FQT
SFUVSOOQSBOEPNDIPJDF
TFMG,
FMTF
SFUVSOOQBSHNBY
TFMGQSFEJDU
<Y><>
5. We also define methods to load and save the network as the training can take
time:
EFGMPBE
TFMG
TFMGTBWFSUGUSBJO4BWFS
UGHMPCBM@WBSJBCMFT
MPBE@XBT@TVDDFTT5SVF
USZ
TBWF@EJS KPJO
TFMGTBWF@QBUITQMJU
<>
DLQUUGUSBJOHFU@DIFDLQPJOU@TUBUF
TBWF@EJS
MPBE@QBUIDLQUNPEFM@DIFDLQPJOU@QBUI
TFMGTBWFSSFTUPSF
TFMGTFTTJPOMPBE@QBUI
FYDFQU
QSJOU
OPTBWFENPEFMUPMPBETUBSUJOHOFXTFTTJPO
MPBE@XBT@TVDDFTT'BMTF
FMTF
QSJOU
MPBEFENPEFM\^GPSNBU
MPBE@QBUI
TBWFSUGUSBJO4BWFS
UGHMPCBM@WBSJBCMFT
FQJTPEF@OVNCFSJOU
MPBE@QBUITQMJU
<>
EFGTBWF
TFMGO
TFMGTBWFSTBWF
TFMGTFTTJPOTFMGTBWF@QBUIHMPCBM@TUFQO
QSJOU
4"7&%.0%&-\^GPSNBU
O
[ 345 ]
Reinforcement Learning Chapter 9
6. The method to copy the parameters of the main DQN network to the target
network is as follows:
EFGDPQZ@GSPN
TFMGPUIFS
NJOF<UGPSUJOUGUSBJOBCMF@WBSJBCMFT
JG
UOBNFTUBSUTXJUI
TFMGTDPQF>
NJOFTPSUFE
NJOFLFZMBNCEBWWOBNF
PUIFST<UGPSUJOUGUSBJOBCMF@WBSJBCMFT
JG
UOBNFTUBSUTXJUI
PUIFSTDPQF>
PUIFSTTPSUFE
PUIFSTLFZMBNCEBWWOBNF
PQT<>
GPSQRJO[JQ
NJOFPUIFST
BDUVBMTFMGTFTTJPOSVO
R
PQQBTTJHO
BDUVBM
PQTBQQFOE
PQ
TFMGTFTTJPOSVO
PQT
EFGMFBSO
NPEFMUBSHFU@NPEFMFYQFSJFODF@SFQMBZ@CVGGFSHBNNB
CBUDI@TJ[F
4BNQMFFYQFSJFODFT
TBNQMFTSBOEPNTBNQMF
FYQFSJFODF@SFQMBZ@CVGGFSCBUDI@TJ[F
TUBUFTBDUJPOTSFXBSETOFYU@TUBUFTEPOFTNBQ
OQBSSBZ
[JQ
TBNQMFT
$BMDVMBUFUBSHFUT
OFYU@2TUBSHFU@NPEFMQSFEJDU
OFYU@TUBUFT
OFYU@2OQBNBY
OFYU@2TBYJT
UBSHFUTSFXBSET
OQJOWFSU
EPOFTBTUZQF
OQGMPBUHBNNB
OFYU@2
6QEBUFNPEFM
MPTTNPEFMVQEBUF
TUBUFTBDUJPOTUBSHFUT
SFUVSOMPTT
[ 346 ]
Reinforcement Learning Chapter 9
8. Now that we have all the ingredients defined in the main code, we use them to
build and train a DQN network to play the game of Atari. The code is well
commented and is an extension of the previous Q learning code with an addition
of the Experience Replay buffer, so you should not have trouble understanding it:
JG@@OBNF@@ @@NBJO@@
IZQFSQBSBNFUFST
HBNNB
CBUDI@T[
OVN@FQJTPEFT
UPUBM@U
FYQFSJFODF@SFQMBZ@CVGGFS<>
FQJTPEF@SFXBSETOQ[FSPT
OVN@FQJTPEFT
MBTU@@BWHT<>
FQTJMPOGPS&QTJMPO(SFFEZ"MHPSJUIN
FQTJMPO
FQTJMPO@NJO
FQTJMPO@DIBOHF
FQTJMPOFQTJMPO@NJO
$SFBUF"UBSJ&OWJSPONFOU
FOWHZNFOWTNBLF
#SFBLPVUW
$SFBUFPSJHJOBMBOEUBSHFU/FUXPSLT
NPEFM%2/
,,HBNNBHBNNBTDPQFNPEFM
UBSHFU@NPEFM%2/
,,HBNNBHBNNBTDPQFUBSHFU@NPEFM
XJUIUG4FTTJPO
BTTFTT
NPEFMTFU@TFTTJPO
TFTT
UBSHFU@NPEFMTFU@TFTTJPO
TFTT
TFTTSVO
UGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
NPEFMMPBE
QSJOU
'JMMJOHFYQFSJFODFSFQMBZCVGGFS
PCTFOWSFTFU
PCT@TNBMMQSFQSPDFTT
PCT
TUBUFOQTUBDL
<PCT@TNBMM>BYJT
'JMMFYQFSJFODFSFQMBZCVGGFS
GPSJJOSBOHF
.*/@&91&3*&/$&4
BDUJPOOQSBOEPNSBOEJOU
,
PCTSFXBSEEPOF@FOWTUFQ
BDUJPO
OFYU@TUBUFVQEBUF@TUBUF
TUBUFPCT
FYQFSJFODF@SFQMBZ@CVGGFSBQQFOE
TUBUFBDUJPOSFXBSE
[ 347 ]
Reinforcement Learning Chapter 9
OFYU@TUBUFEPOF
JGEPOF
PCTFOWSFTFU
PCT@TNBMMQSFQSPDFTT
PCT
TUBUFOQTUBDL
<PCT@TNBMM>BYJT
FMTF
TUBUFOFYU@TUBUF
1MBZBOVNCFSPGFQJTPEFTBOEMFBSO
GPSJJOSBOHF
OVN@FQJTPEFT
UEBUFUJNFOPX
3FTFUUIFFOWJSPONFOU
PCTFOWSFTFU
PCT@TNBMMQSFQSPDFTT
PCT
TUBUFOQTUBDL
<PCT@TNBMM>BYJT
BTTFSU
TUBUFTIBQF
MPTT/POF
UPUBM@UJNF@USBJOJOH
OVN@TUFQT@JO@FQJTPEF
FQJTPEF@SFXBSE
EPOF'BMTF
XIJMFOPUEPOF
6QEBUFUBSHFUOFUXPSL
JGUPUBM@U5"3(&5@61%"5&@1&3*0%
UBSHFU@NPEFMDPQZ@GSPN
NPEFM
QSJOU
$PQJFENPEFMQBSBNFUFSTUPUBSHFU
OFUXPSLUPUBM@UTQFSJPET
UPUBM@U5"3(&5@61%"5&@1&3*0%
5BLFBDUJPO
BDUJPONPEFMTBNQMF@BDUJPO
TUBUFFQTJMPO
PCTSFXBSEEPOF@FOWTUFQ
BDUJPO
PCT@TNBMMQSFQSPDFTT
PCT
OFYU@TUBUFOQBQQFOE
TUBUF<>
OQFYQBOE@EJNT
PCT@TNBMMBYJT
FQJTPEF@SFXBSE SFXBSE
3FNPWFPMEFTUFYQFSJFODFJGSFQMBZCVGGFSJTGVMM
JGMFO
FYQFSJFODF@SFQMBZ@CVGGFS
."9@&91&3*&/$&4
FYQFSJFODF@SFQMBZ@CVGGFSQPQ
[ 348 ]
Reinforcement Learning Chapter 9
4BWFUIFSFDFOUFYQFSJFODF
FYQFSJFODF@SFQMBZ@CVGGFSBQQFOE
TUBUFBDUJPO
SFXBSEOFYU@TUBUFEPOF
5SBJOUIFNPEFMBOELFFQNFBTVSFPGUJNF
U@EBUFUJNFOPX
MPTTMFBSO
NPEFMUBSHFU@NPEFM
FYQFSJFODF@SFQMBZ@CVGGFSHBNNBCBUDI@T[
EUEBUFUJNFOPX
U@
UPUBM@UJNF@USBJOJOH
EUUPUBM@TFDPOET
OVN@TUFQT@JO@FQJTPEF
TUBUFOFYU@TUBUF
UPUBM@U
FQTJMPONBY
FQTJMPOFQTJMPO@DIBOHF
FQTJMPO@NJO
EVSBUJPOEBUFUJNFOPX
U
FQJTPEF@SFXBSET<J>FQJTPEF@SFXBSE
UJNF@QFS@TUFQUPUBM@UJNF@USBJOJOH
OVN@TUFQT@JO@FQJTPEF
MBTU@@BWHFQJTPEF@SFXBSET<NBY
JJ
>NFBO
MBTU@@BWHTBQQFOE
MBTU@@BWH
QSJOU
&QJTPEFJ%VSBUJPOEVSBUJPO/VN
TUFQTOVN@TUFQT@JO@FQJTPEF
3FXBSEFQJTPEF@SFXBSE5SBJOJOHUJNFQFS
TUFQGUJNF@QFS@TUFQ
"WH3FXBSE
-BTUG
MBTU@@BWH&QTJMPOGFQTJMPO
JGJ
NPEFMTBWF
J
TZTTUEPVUGMVTI
1MPUT
QMUQMPU
MBTU@@BWHT
QMUYMBCFM
FQJTPEFT
QMUZMBCFM
"WFSBHF3FXBSET
QMUTIPX
FOWDMPTF
[ 349 ]
Reinforcement Learning Chapter 9
We can see from the preceding figure that the agent is reaching higher rewards with
training, and the situation is clearer with the plot of average reward per 100 episodes:
[ 350 ]
Reinforcement Learning Chapter 9
This is only after the first 500 episodes of training; for better results, you will need to train it
much longer, ~10,000 episodes.
There's more...
Training the agent takes lots of episodes, which is both time-and memory-consuming.
OpenAI Gym provides a wrapper to save the game as a video and so, instead of using
render, you can use the wrapper to save videos and later monitor how the agent learned. AI
engineers and enthusiasts can upload these videos to show their results. To do so, we need
to first import wrappers, then create the environment, and finally use Monitor. By default, it
will store the video of 1, 8, 27, 64, ... and then every 1,000th episode (episode numbers with
perfect cubes); each training, by default, is saved in one folder. The code to be added for this
is as follows:
JNQPSUHZN
GSPNHZNJNQPSUXSBQQFST
FOWHZNNBLF
#SFBLPVUW
FOWXSBQQFST.POJUPS
FOW TBWFQBUI
In case you want to use the same folder in the next training, you can add GPSDF5SVF to
Monitor.
[ 351 ]
Reinforcement Learning Chapter 9
See also
Mnih, Volodymyr, and others, Playing Atari with deep reinforcement learning,
arXiv preprint arXiv:1312.5602 (2013) (IUUQTBSYJWPSHQEGQEG)
Mnih, Volodymyr, et al. Human-level control through deep reinforcement
learning, Nature 518.7540 (2015): 529-533
A cool implementation of DQN to play Atari: IUUQTHJUIVCDPNEFWTJTUFST
%2/UFOTPSGMPX
Getting ready
The game of Pong is a two-player game, where the goal is to bounce the ball past the other
player. The agent can move the paddle up or down (and, yes, the standard NoOp). One of
the players in the OpenAI environment is a decent AI player who knows how to play the
game well. Our goal is to train the second agent using policy gradients. Our agent gets
proficient with each game it plays. While the code has been built to run only for 500
episodes, we should add a provision to save the agent state at specified checkpoints and, in
the next run, start by loading the previous checkpoint. We achieve this by first declaring a
saver, then using the TensorFlow TBWFSTBWF method to save the present network state
(checkpoint), and lastly load the network from the last saved checkpoint. The following
methods of the class 1PMJDZ/FUXPSL, defined in the How to do it section of this
recipe, perform this work:
EFGMPBE
TFMG
TFMGTBWFSUGUSBJO4BWFS
UGHMPCBM@WBSJBCMFT
MPBE@XBT@TVDDFTT5SVFZFT* NCFJOHPQUJNJTUJD
USZ
[ 352 ]
Reinforcement Learning Chapter 9
TBWF@EJS KPJO
TFMGTBWF@QBUITQMJU
<>
DLQUUGUSBJOHFU@DIFDLQPJOU@TUBUF
TBWF@EJS
MPBE@QBUIDLQUNPEFM@DIFDLQPJOU@QBUI
TFMGTBWFSSFTUPSF
TFMGTFTTJPOMPBE@QBUI
FYDFQU
QSJOU
OPTBWFENPEFMUPMPBETUBSUJOHOFXTFTTJPO
MPBE@XBT@TVDDFTT'BMTF
FMTF
QSJOU
MPBEFENPEFM\^GPSNBU
MPBE@QBUI
TBWFSUGUSBJO4BWFS
UGHMPCBM@WBSJBCMFT
FQJTPEF@OVNCFSJOU
MPBE@QBUITQMJU
<>
How to do it...
1. The code for this recipe is based on the Andrej Karpathy blog (IUUQLBSQBUIZ
HJUIVCJPSM) and a part of it has been adapted from code by Sam
Greydanus (IUUQTHJTUHJUIVCDPNLBSQBUIZ
BDGFGDCDFFBD).
2. We have the usual imports:
JNQPSUOVNQZBTOQ
JNQPSUHZN
JNQPSUNBUQMPUMJCQZQMPUBTQMU
JNQPSUUFOTPSGMPXBTUG
3. We define our 1PMJDZ/FUXPSL class. During the class construction, the model
hyperparameters are also initialized. The @@JOJU@@ method defines the
placeholders for input state, TFMGUG@Y; predicted action, TFMGUGZ;
corresponding reward, TFMGUG@FQS; network weights; and ops to predict action
value, training, and updating. You can see that the class construction also initiates
an interactive TensorFlow session:
DMBTT1PMJDZ/FUXPSL
PCKFDU
EFG@@JOJU@@
TFMG/@4*;&IHBNNBFUBF
EFDBZTBWF@QBUI NPEFMTQPOHDLQU
TFMGHBNNBHBNNB
[ 353 ]
Reinforcement Learning Chapter 9
TFMGTBWF@QBUITBWF@QBUI
1MBDFIPMEFSTGPSQBTTJOHTUBUF
TFMGUG@YUGQMBDFIPMEFS
EUZQFUGGMPBUTIBQF</POF
/@4*;&/@4*;&>OBNFUG@Y
TFMGUG@ZUGQMBDFIPMEFS
EUZQFUGGMPBUTIBQF</POF
O@BDUJPOT>OBNFUG@Z
TFMGUG@FQSUGQMBDFIPMEFS
EUZQFUGGMPBUTIBQF</POF
>OBNFUG@FQS
8FJHIUT
YBWJFS@MUGUSVODBUFE@OPSNBM@JOJUJBMJ[FS
NFBO
TUEEFW/@4*;&EUZQFUGGMPBU
TFMG8UGHFU@WBSJBCMF
8</@4*;&/@4*;&I>
JOJUJBMJ[FSYBWJFS@M
YBWJFS@MUGUSVODBUFE@OPSNBM@JOJUJBMJ[FS
NFBO
TUEEFWOQTRSU
IEUZQFUGGMPBU
TFMG8UGHFU@WBSJBCMF
8<IO@BDUJPOT>
JOJUJBMJ[FSYBWJFS@M
#VJME$PNQVUBUJPO
UGSFXBSEQSPDFTTJOH
OFFEUG@EJTDPVOUFE@FQSGPSQPMJDZ
HSBEJFOUXJ[BSESZ
UG@EJTDPVOUFE@FQSTFMGUG@EJTDPVOU@SFXBSET
TFMGUG@FQS
UG@NFBOUG@WBSJBODFUGOONPNFOUT
UG@EJTDPVOUFE@FQS
<>TIJGU/POFOBNFSFXBSE@NPNFOUT
UG@EJTDPVOUFE@FQSUG@NFBO
UG@EJTDPVOUFE@FQSUGTRSU
UG@WBSJBODF
F
%FGJOF0QUJNJ[FSDPNQVUFBOEBQQMZHSBEJFOUT
TFMGUG@BQSPCTFMGUG@QPMJDZ@GPSXBSE
TFMGUG@Y
MPTTUGOOM@MPTT
TFMGUG@ZTFMGUG@BQSPC
PQUJNJ[FSUGUSBJO3.41SPQ0QUJNJ[FS
FUBEFDBZEFDBZ
UG@HSBETPQUJNJ[FSDPNQVUF@HSBEJFOUT
MPTT
WBS@MJTUUGUSBJOBCMF@WBSJBCMFT
HSBE@MPTTUG@EJTDPVOUFE@FQS
TFMGUSBJO@PQPQUJNJ[FSBQQMZ@HSBEJFOUT
UG@HSBET
*OJUJBMJ[F7BSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFMGTFTTJPOUG*OUFSBDUJWF4FTTJPO
TFMGTFTTJPOSVO
JOJU
TFMGMPBE
[ 354 ]
Reinforcement Learning Chapter 9
4. We define a method to calculate the discounted rewards. This ensures that the
agent takes into account not only the present reward, but also future rewards.
The discounted reward at any time t is given by Rt = blkrt+k, where the summation
is over k o [0,p], and l is the discount factor with value lying between 0 and 1. In
our code, we have used gamma = 0.99:
EFGUG@EJTDPVOU@SFXBSET
TFMGUG@SUG@S_<HBNF@TUFQT>
EJTDPVOU@GMBNCEBBWBTFMGHBNNB
W
UG@S@SFWFSTFUGTDBO
EJTDPVOU@GUGSFWFSTF
UG@S<>
UG@EJTDPVOUFE@SUGSFWFSTF
UG@S@SFWFSTF<>
SFUVSOUG@EJTDPVOUFE@S
EFGUG@QPMJDZ@GPSXBSE
TFMGYY_<%>
IUGNBUNVM
YTFMG8
IUGOOSFMV
I
MPHQUGNBUNVM
ITFMG8
QUGOOTPGUNBY
MPHQ
SFUVSOQ
EFGQSFEJDU@61
TFMGY
GFFE\TFMGUG@YOQSFTIBQF
Y
^
BQSPCTFMGTFTTJPOSVO
TFMGUG@BQSPCGFFE
SFUVSOBQSPC
6. The 1PMJDZ/FUXPSL agent updates the weights using the VQEBUF method:
EFGVQEBUF
TFMGGFFE
SFUVSOTFMGTFTTJPOSVO
TFMGUSBJO@PQGFFE
EPXOTBNQMJOH
EFGQSFQSPDFTT
*
QSFQSPYYVJOUGSBNFJOUP
Y%GMPBU
WFDUPS
**<>DSPQ
**<>EPXOTBNQMFCZGBDUPSPG
*<*>FSBTFCBDLHSPVOE
CBDLHSPVOEUZQF
[ 355 ]
Reinforcement Learning Chapter 9
*<*>FSBTFCBDLHSPVOE
CBDLHSPVOEUZQF
*<*>FWFSZUIJOHFMTF
QBEEMFTCBMMKVTUTFUUP
SFUVSO*BTUZQF
OQGMPBUSBWFM
8. The rest is simple--we create a game environment, define arrays to hold (state,
action, reward, state), and make the agent learn for a very large number of
episodes (with a break or continuously, depending solely on your computational
resources). An important thing to note here is that the agent is not learning per
action step. Instead, the agent uses the complete set of (state, action, reward,
state) of one episode to correct its policy. This can be memory-expensive:
JG@@OBNF@@ @@NBJO@@
$SFBUF(BNF&OWJSPONFOU
FOW@OBNF1POHW
FOWHZNNBLF
FOW@OBNF
FOWXSBQQFST.POJUPS
FOW UNQQPOH GPSDF5SVF
O@BDUJPOTFOWBDUJPO@TQBDFO/VNCFSPGQPTTJCMFBDUJPOT
*OJUJBMJ[JOH(BNFBOE4UBUF
UBDUJPOSFXBSETUBUF
U
YTSTZT<><><>
PCTFOWSFTFU
QSFW@Y/POF
SVOOJOH@SFXBSE/POF
SVOOJOH@SFXBSET<>
SFXBSE@TVN
O
EPOF'BMTF
O@TJ[F
OVN@FQJTPEFT
$SFBUF"HFOU
BHFOU1PMJDZ/FUXPSL
O@TJ[F
USBJOJOHMPPQ
XIJMFOPUEPOFBOEOOVN@FQJTPEFT
1SFQSPDFTTUIFPCTFSWBUJPO
DVS@YQSFQSPDFTT
PCT
YDVS@YQSFW@YJGQSFW@YJTOPU/POFFMTF
OQ[FSPT
O@TJ[FO@TJ[F
QSFW@YDVS@Y
1SFEJDUUIFBDUJPO
BQSPCBHFOUQSFEJDU@61
YBQSPCBQSPC<>
[ 356 ]
Reinforcement Learning Chapter 9
BDUJPOOQSBOEPNDIPJDF
O@BDUJPOTQBQSPC
QSJOU
BDUJPO
MBCFMOQ[FSPT@MJLF
BQSPCMBCFM<BDUJPO>
4UFQUIFFOWJSPONFOUBOEHFUOFXNFBTVSFNFOUT
PCTSFXBSEEPOFJOGPFOWTUFQ
BDUJPO
FOWSFOEFS
SFXBSE@TVN
SFXBSE
SFDPSEHBNFIJTUPSZ
YTBQQFOE
YZTBQQFOE
MBCFMSTBQQFOE
SFXBSE
JGEPOF
VQEBUFSVOOJOHSFXBSE
SVOOJOH@SFXBSESFXBSE@TVNJGSVOOJOH@SFXBSEJT/POF
FMTFSVOOJOH@SFXBSE
SFXBSE@TVN
SVOOJOH@SFXBSETBQQFOE
SVOOJOH@SFXBSE
GFFE\BHFOUUG@YOQWTUBDL
YTBHFOUUG@FQS
OQWTUBDL
STBHFOUUG@ZOQWTUBDL
ZT^
BHFOUVQEBUF
GFFE
QSJOUQSPHSFTTDPOTPMF
JGO
QSJOU
FQ\^SFXBSE\^NFBOSFXBSE
\G^ GPSNBU
OSFXBSE@TVNSVOOJOH@SFXBSE
FMTF
QSJOU
=UFQ\^SFXBSE\^ GPSNBU
OSFXBSE@TVN
4UBSUOFYUFQJTPEFBOETBWFNPEFM
YTSTZT<><><>
PCTFOWSFTFU
O
UIF/FYU&QJTPEF
SFXBSE@TVN
JGO
BHFOUTBWF
EPOF'BMTF
QMUQMPU
SVOOJOH@SFXBSET
QMUYMBCFM
FQJTPEFT
QMUZMBCFM
3VOOJOH"WFSHF
QMUTIPX
FOWDMPTF
[ 357 ]
Reinforcement Learning Chapter 9
The following figure shows the average running reward as the agent learns for the first 500
episodes:
How it works...
Weights were initialized using Xavier initialization, which ensures that our weights are
neither too large nor too small; both cases impede the learning of the network. In Xavier
initialization, the weights are assigned a value from a distribution with zero mean and a
specific variance 2/(nin+nout), where nin and nout are respectively the numbers of inputs and
output of the layer. To know more about Xavier initialization, you can refer to Glorot and
Bengio's 2009 paper; details are given in the See also section.
[ 358 ]
Reinforcement Learning Chapter 9
There's more...
Anyone who sees the agent learn to play for the first time will be amazed by it--it seems so
human-like. The initial moves are always clumsy; slowly, the agent learns which way to go,
though it is slow and often misses the ball. However, as the learning continues, the agent
becomes an expert.
Yet it is very unlike us; once we learn to play a game, we can easily use that knowledge in
any other similar situation. The RL agent will not be able to do so--even a simple thing such
as changing the size of environment space will bring it back to zero. Transfer learning, a
technique being investigated by researchers, can help the agent use the knowledge it
learned in one environment space in another environment space, and perhaps pave the way
for true artificial general intelligence one day.
AlphaGo Zero
Recently, DeepMind published an article about AlphaGo Zero, the latest evolution of
AlphaGo. According to the results they have published, AlphaGo Zero is even more
powerful and the strongest Go player in History. AlphaGo starts tabula rasa, that is, it starts
from a blank state, and it uses only the board states and the games it plays against itself to
tune the neural network and predict the right moves.
AlphaGo Zero uses a deep neural network, that takes as an input the raw board
representations (present and history) and outputs both move probabilities and a value.
Thus this neural network combines the role of both policy network and value network. The
network is trained from games of self-play, unlike previous AlphaGo versions (they were
trained using supervised learning). At each position, a Monte Carlo Tree Search (MCTS) is
performed, guided by the neural network. The neural network is trained by a self-play
reinforcement learning algorithm that uses MCTS to play each move.
Initially, the neural network has its weights randomly initialized. At each iteration step,
many games of self-play are generated. At each time step, an MCTS search is performed for
the possible policies using the previous iteration of the neural network., then a move is
played by sampling the search probabilities. This is repeated till this particular game
terminates. The game state, the policy taken and rewards for each time step of the game are
stored. In parallel, the neural network is trained from the data sampled uniformly among
all the time steps of the previous iteration(s) of self-play. The weights of the neural network
are adjusted so as to minimize the error between the predicted value and the self-play
winner, and to maximize the similarity of the neural network move probabilities to the
search probabilities.
[ 359 ]
Reinforcement Learning Chapter 9
With only 3 days of training on a single machine with four TPUs, AlphaGo Zero beat
AlphaGo by 100-0. AlphaGo Zero is based solely on RL. The detail of its implementation
can be read in the paper Mastering the game of Go without human knowledge published in
Nature, October 2017.
See also
IUUQTBSYJWPSHQEGQEG
IUUQVGBMNGGDVOJD[_TUSBLBDPVSTFTOQGMTVUUPO
CPPLESBGUTFQQEG
IUUQLBSQBUIZHJUIVCJPSM
Xavier Glorot and Yoshua Bengio, Understanding the difficulty of training deep
feedforward neural networks, Proceedings of the Thirteenth International
Conference on Artificial Intelligence and Statistics, 2010, IUUQQSPDFFEJOHT
NMSQSFTTWHMPSPUBHMPSPUBQEG
[ 360 ]
Mobile Computation
10
In this chapter, we will discuss using deep learning on mobile devices and we will present a
number of recipes for the following:
Introduction
In this section, we will present a number of use cases for mobile deep learning. This is a
very different situation from the desktop or cloud deep learning where GPUs and electricity
are commonly available. In fact, on a mobile device, it is very important to preserve the
battery and GPUs are frequently not available. However, deep learning can be very useful
in a number of situations. Let's review them:
Image recognition: Modern phones have powerful cameras and users are keen to
try effects on images and pictures. Frequently, it is also important to understand
what is in the pictures, and there are multiple pre-trained models that can be
adapted for this, as discussed in the chapters dedicated to CNNs. A good
example of a model used for image recognition is given
at IUUQTHJUIVCDPN5FOTPS'MPXNPEFMTUSFFNBTUFSPGGJDJBMSFTOFU
Mobile Computation Chapter 10
[ 362 ]
Mobile Computation Chapter 10
Getting ready
We will use Android studio, the official integrated development environment (IDE) for
Google's Android operating system.
How to do it...
We proceed with installing TensorFlow mobile for macOS and Android as follows:
[ 363 ]
Mobile Computation Chapter 10
[ 364 ]
Mobile Computation Chapter 10
[ 365 ]
Mobile Computation Chapter 10
[ 366 ]
Mobile Computation Chapter 10
#PGZCORNGQHETGCVKPI6GPUQT(NQYOQDKNGCRRNKECVKQPKP#PFTQKF5VWFKQHQWTVJUVGR
BEEFEGPSBVUPNBUJDBMMZDPOOFDUUP5FOTPS'MPXWJBNBWFO
SFQPTJUPSJFT\
KDFOUFS
NBWFO\
VSM IUUQTHPPHMFCJOUSBZDPN5FOTPS'MPX
^
^
EFQFOEFODJFT\
DPNQJMFGJMF5SFF
EJS MJCT JODMVEF< KBS >
BOESPJE5FTU$PNQJMF
DPNBOESPJETVQQPSUUFTUFTQSFTTPFTQSFTTP
DPSF \
FYDMVEFHSPVQ DPNBOESPJETVQQPSU NPEVMF TVQQPSUBOOPUBUJPOT
^
[ 367 ]
Mobile Computation Chapter 10
DPNQJMF DPNBOESPJETVQQPSUBQQDPNQBUW
DPNQJMF DPNBOESPJETVQQPSUDPOTUSBJOUDPOTUSBJOUMBZPVU
BEEFEGPSBVUPNBUJDBMMZDPNQJMF5FOTPS'MPX
DPNQJMF PSH5FOTPS'MPX5FOTPS'MPXBOESPJE
UFTU$PNQJMF KVOJUKVOJU
^
[ 368 ]
Mobile Computation Chapter 10
[ 369 ]
Mobile Computation Chapter 10
How it works...
Setting up Android TensorFlow with Android Studio is very simple. You just need to add a
few configuration lines to your app's CVJMEHSBEMF file and Android Studio will do
everything on your behalf.
[ 370 ]
Mobile Computation Chapter 10
There's more...
If you want to build directly from a TensorFlow source then you need to install Bazel and
TensorFlow. Bazel is a fast, scalable, multi-language, and extensible build system. Google
uses the build tool Blaze internally and released the open source part of the Blaze tool as
Bazel; the name is an anagram of Blaze.
VTSCJOSVCZF
DVSMGT4-=
IUUQTSBXHJUIVCVTFSDPOUFOUDPN)PNFCSFXJOTUBMMNBTUFSJOTUBMM
CSFXJOTUBMMCB[FM
CB[FMWFSTJPO
CSFXVQHSBEFCB[FM
HJUDMPOFIUUQTHJUIVCDPN5FOTPS'MPX5FOTPS'MPXHJU
Getting ready
TensorFlow mobile Android apps are available on GitHub at the following address:
IUUQTHJUIVCDPN5FOTPS'MPX5FOTPS'MPXUSFFNBTUFS5FOTPS'MPXFYBNQMFT
BOESPJE . In October 2017, the page contained the following examples:
TF Classify: Uses the Google Inception model to classify camera frames in real
time, displaying the top results in an overlay on the camera image.
[ 371 ]
Mobile Computation Chapter 10
How to do it...
We proceed with the recipe as follows:
1. The best way to install the packages is to use the pre-built APK which is created
every night. Point your browser to IUUQTDJ5FOTPS'MPXPSHWJFX/JHIUMZ
KPCOJHIUMZBOESPJE and download 5FOTPS'MPX@EFNPBQL as shown in the
following screenshot:
[ 372 ]
Mobile Computation Chapter 10
2. Install the applications on your device. In my example, I will use the Pixel XL
emulated device available in Android Studio. This is a terminal device emulated
directly from inside Android Studio. The command BECEFWJDFT lists all the
attached devices. In this case, I have an emulator for the Pixel XL and i can install
the 5FOTPS'MPX@EFNPBQL.
BECEFWJDFT
-JTUPGEFWJDFTBUUBDIFE
FNVMBUPSEFWJDF
BECJOTUBMMS5FOTPS'MPX@EFNPBQL
After the installation, the emulator will have a new set of TensorFlow applications available
for use as shown in the following figure.
[ 373 ]
Mobile Computation Chapter 10
3. Run your favorite application. For instance, the following image is an example of
TF Stylize, used for restyling the camera preview image to that of a number of
different art styles via Transfer Learning:
[ 374 ]
Mobile Computation Chapter 10
How it works...
Installing TensorFlow examples for Android is pretty easy if you use the nightly build
demo and the BEC tool for installing the APK on the device.
[ 375 ]
Mobile Computation Chapter 10
Getting ready
We will use the Xcode development environment and CocoaPods for pre-installing
TensorFlow. I will assume that Xcode is already installed in your environment. If not,
please download it from IUUQTEFWFMPQFSBQQMFDPNYDPEF.
How to do it...
We will proceed with the recipe as follows:
TVEPHFNJOTUBMMDPDPBQPET
QPETFUVQ
4FUUJOHVQ$PDPB1PETNBTUFSSFQP
VTSMPDBMHJUDVSSFOUCJOHJUDMPOF
IUUQTHJUIVCDPN$PDPB1PET4QFDTHJUNBTUFSQSPHSFTT
$MPOJOHJOUP NBTUFS
SFNPUF$PVOUJOHPCKFDUTEPOF
SFNPUF$PNQSFTTJOHPCKFDUT
EPOF
SFNPUF5PUBM
EFMUBSFVTFE
EFMUBQBDL
SFVTFE
3FDFJWJOHPCKFDUT
.J#].J#T
EPOF
3FTPMWJOHEFMUBT
EPOF
$IFDLJOHPVUGJMFT
EPOF
DE5FOTPS'MPX5FOTPS'MPXFYBNQMFTJPTCFODINBSL
QPEJOTUBMM
"OBMZ[JOHEFQFOEFODJFT
%PXOMPBEJOHEFQFOEFODJFT
*OTUBMMJOH5FOTPS'MPXFYQFSJNFOUBM
(FOFSBUJOH1PETQSPKFDU
*OUFHSBUJOHDMJFOUQSPKFDU
<>1MFBTFDMPTFBOZDVSSFOU9DPEFTFTTJPOTBOEVTF
AUG@CFODINBSL@FYBNQMFYDXPSLTQBDFAGPSUIJTQSPKFDUGSPNOPXPO
4FOEJOHTUBUT
1PEJOTUBMMBUJPODPNQMFUF5IFSFJTEFQFOEFODZGSPNUIF1PEGJMF
BOEUPUBMQPEJOTUBMMFE
[ 376 ]
Mobile Computation Chapter 10
3. Download some sample data from Inception v1. Extract the label and graph files
into the data folders inside both the TJNQMF and DBNFSB folders:
NLEJSQ_HSBQIT
DVSMP_HSBQITJODFQUJPOI[JQ=
IUUQTTUPSBHFHPPHMFBQJTDPNEPXOMPBE5FOTPS'MPXPSHNPEFMTJODFQ
UJPOI[JQ=
VO[JQ_HSBQITJODFQUJPOI[JQE_HSBQITJODFQUJPOI
DQ_HSBQITJODFQUJPOI5FOTPS'MPXFYBNQMFTJPTCFODINBSLEBUB
DQ_HSBQITJODFQUJPOI5FOTPS'MPXFYBNQMFTJPTDBNFSBEBUB
DQ_HSBQITJODFQUJPOI5FOTPS'MPXFYBNQMFTJPTTJNQMFEBUB
4. Download an image used as a test from and copy it into the benchmark directory:
IUUQTVQMPBEXJLJNFEJBPSHXJLJQFEJBDPNNPOT(SBDF@)PQQFSKQH
DQHSBDF@IPQQFSKQHCFODINBSLEBUB
#PKOCIGQH)TCEG*QRRGT
[ 377 ]
Mobile Computation Chapter 10
5. Open the sample project used previously. The following command will open
Xcode with TensorFlow already available: After that, run compile, as shown in
the following code and image:
PQFOUG@CFODINBSL@FYBNQMFYDXPSLTQBDF
[ 378 ]
Mobile Computation Chapter 10
6. Look at the results in the iPhone simulator. The image used in step 4 is
recognized as being an image of a military uniform, according to the Inception v1
categories:
#PGZCORNGQH+RJQPGCRRNKECVKQPHQT6GPUQTcQYEQORWVCVKQP
[ 379 ]
Mobile Computation Chapter 10
How it works...
Xcode and CocoaPods are used for compiling a TensorFlow application used for classifying
images in different Inception categories. The result is visualized using an iPhone simulator.
There's more...
You can use TensorFlow directly from your applications. More information is available
here: IUUQTHJUIVCDPN5FOTPS'MPX5FOTPS'MPXCMPCNBTUFS5FOTPS'MPXFYBNQMFT
JPT3&"%.&NE.
Getting ready
We are going to use Bazel for building the different components of TensorFlow. Therefore,
the first step is to make sure that both Bazel and TensorFlow are installed.
How to do it...
We proceed with the optimization as follows:
[ 380 ]
Mobile Computation Chapter 10
VTSCJOSVCZF
DVSMGT4-=
IUUQTSBXHJUIVCVTFSDPOUFOUDPN)PNFCSFXJOTUBMMNBTUFSJOTUBMM
CSFXJOTUBMMCB[FM
CB[FMWFSTJPO
CSFXVQHSBEFCB[FM
HJUDMPOFIUUQTHJUIVCDPN5FOTPS'MPX5FOTPS'MPXHJU
DE_5FOTPS'MPX
CB[FMCVJME5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQI
<>$PNQJMJOH
5FOTPS'MPXDPSFLFSOFMTDXJTF@PQ@HSFBUFSDD
*/'0'SPN-JOLJOH
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQI
DMBOHXBSOJOHBSHVNFOUVOVTFEEVSJOHDPNQJMBUJPO QUISFBE <
8VOVTFEDPNNBOEMJOFBSHVNFOU>
5BSHFU5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQIVQUP
EBUF
CB[FMCJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQI
*/'0&MBQTFEUJNFT$SJUJDBM1BUIT
NLEJSQ_HSBQIT
DVSMP_HSBQITJODFQUJPOI[JQ=
IUUQTTUPSBHFHPPHMFBQJTDPNEPXOMPBE5FOTPS'MPXPSHNPEFMTJODFQ
UJPOI[JQ=
VO[JQ_HSBQITJODFQUJPOI[JQE_HSBQITJODFQUJPOI
8. Summarize the Inception graph and notice the number of const parameters:
13.46Million. Each of them is stored with 32-bit floats, and this is quite expensive:
CB[FMCJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQI
JO@HSBQI6TFSTHVMMJHSBQIT5FOTPS'MPX@JODFQUJPO@HSBQIQC
'PVOEQPTTJCMFJOQVUT
OBNFJOQVUUZQFGMPBU
TIBQF<>
/PWBSJBCMFTTQPUUFE
'PVOEQPTTJCMFPVUQVUT
OBNFPVUQVUPQ*EFOUJUZ
OBNFPVUQVU
PQ*EFOUJUZ
OBNFPVUQVUPQ*EFOUJUZ
'PVOE
.DPOTUQBSBNFUFST
WBSJBCMF
[ 381 ]
Mobile Computation Chapter 10
QBSBNFUFSTBOEDPOUSPM@FEHFT
OPEFTBTTJHOFEUPEFWJDF DQV 0QUZQFTVTFE$POTU
#JBT"EE3FMV$POW%.BY1PPM$PODBU3FTIBQF
.BU.VM4PGUNBY*EFOUJUZ"WH1PPM-3/1MBDFIPMEFS
5PVTFXJUI5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFMUSZUIFTF
BSHVNFOUT
CB[FMSVO5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFM
HSBQI6TFSTHVMMJHSBQIT5FOTPS'MPX@JODFQUJPO@HSBQIQC
TIPX@GMPQTJOQVU@MBZFSJOQVUJOQVU@MBZFS@UZQFGMPBU
JOQVU@MBZFS@TIBQFPVUQVU@MBZFSPVUQVUPVUQVUPVUQVU
9. Compile the tool for quantizing the const operations down to 8 bits:
CB[FMCVJME5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQI
*/'0'SPN-JOLJOH
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQI
DMBOHXBSOJOHBSHVNFOUVOVTFEEVSJOHDPNQJMBUJPO QUISFBE <
8VOVTFEDPNNBOEMJOFBSHVNFOU>
5BSHFU5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIVQUP
EBUF
CB[FMCJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQI
*/'0&MBQTFEUJNFT$SJUJDBM1BUIT
CB[FMCJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQI
JO@HSBQI6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQI
QCPVU@HSBQIUNQ5FOTPS'MPX@JODFQUJPO@RVBOUJ[FEQC
JOQVUT .VM PVUQVUT TPGUNBY
USBOTGPSNT RVBOUJ[F@XFJHIUT
*
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIDD>"QQMZJOH
RVBOUJ[F@XFJHIUT
MTMBI
6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQIQC
SXSHVMMJ./PW
6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQIQC
MTMBIUNQ5FOTPS'MPX@JODFQUJPO@RVBOUJ[FEQC
SXSSHVMMJXIFFM.0DU
UNQ5FOTPS'MPX@JODFQUJPO@RVBOUJ[FEQC
[ 382 ]
Mobile Computation Chapter 10
How it works...
Quantization helps to reduce the size of a model by downsizing the const operations from
32 bit down to 8 bits. In general, the model will not suffer significant degradation of
performance. However, this has to be verified on a case by case basis.
Getting ready
We are going to use Bazel for building the different components of TensorFlow. Therefore,
the first step is to make sure that both Bazel and TensorFlow are installed.
How to do it...
We proceed with profiling as follows:
VTSCJOSVCZF
DVSMGT4-=
IUUQTSBXHJUIVCVTFSDPOUFOUDPN)PNFCSFXJOTUBMMNBTUFSJOTUBMM
CSFXJOTUBMMCB[FM
CB[FMWFSTJPO
CSFXVQHSBEFCB[FM
HJUDMPOFIUUQTHJUIVCDPN5FOTPS'MPX5FOTPS'MPXHJU
[ 383 ]
Mobile Computation Chapter 10
DE_5FOTPS'MPX
CB[FMCVJMEDPQU5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFM
*/'0'PVOEUBSHFU
5BSHFU5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFMVQUPEBUF
CB[FMCJO5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFM
*/'0&MBQTFEUJNFT$SJUJDBM1BUIT
CB[FMCJO5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFM
HSBQI6TFSTHVMMJHSBQIT5FOTPS'MPX@JODFQUJPO@HSBQIQC
TIPX@SVO@PSEFSGBMTFTIPX@UJNFGBMTFTIPX@NFNPSZGBMTF
TIPX@TVNNBSZUSVFTIPX@GMPQTUSVF
(SBQI<6TFSTHVMMJHSBQIT5FOTPS'MPX@JODFQUJPO@HSBQIQC>
*OQVUMBZFST<JOQVU>
*OQVUTIBQFT<>
*OQVUUZQFT<GMPBU>
0VUQVUMBZFST<PVUQVU>
/VNSVOT<>
*OUFSJOGFSFODFEFMBZ
TFDPOET<>
*OUFSCFODINBSLEFMBZ
TFDPOET<>
/VNUISFBET<>
#FODINBSLOBNF<>
0VUQVUQSFGJY<>
4IPXTJ[FT<>
8BSNVQSVOT<>
-PBEJOH5FOTPS'MPX
(PUDPOGJHEFWJDFT
3VOOJOHCFODINBSLGPSNBYJUFSBUJPOTNBYTFDPOETXJUIPVU
EFUBJMFETUBUMPHHJOHXJUITTMFFQCFUXFFOJOGFSFODFT
DPVOUGJSTUDVSSNJONBYBWH
TUE
3VOOJOHCFODINBSLGPSNBYJUFSBUJPOTNBYTFDPOETXJUIPVU
EFUBJMFETUBUMPHHJOHXJUITTMFFQCFUXFFOJOGFSFODFT
DPVOUGJSTUDVSSNJONBYBWH
TUE
3VOOJOHCFODINBSLGPSNBYJUFSBUJPOTNBYTFDPOETXJUI
EFUBJMFETUBUMPHHJOHXJUITTMFFQCFUXFFOJOGFSFODFT
DPVOUGJSTUDVSSNJONBYBWH
TUE
"WFSBHFJOGFSFODFUJNJOHTJOVT8BSNVQOPTUBUT
XJUITUBUT
[ 384 ]
Mobile Computation Chapter 10
CB[FMCVJMEDPQUDPOGJHBOESPJE@BSN=
5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFM
BECQVTICB[FMCJO5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFM=
EBUBMPDBMUNQ
BECQVTIUNQ5FOTPS'MPX@JODFQUJPO@HSBQIQCEBUBMPDBMUNQ
BECQVTI_HVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQIQC
EBUBMPDBMUNQ
6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQIQC
GJMFQVTIFE.#T
CZUFTJOT
BECTIFMM
HFOFSJD@Y
EBUBMPDBMUNQCFODINBSL@NPEFM
HSBQIEBUBMPDBMUNQ5FOTPS'MPX@JODFQUJPO@HSBQIQC
TIPX@SVO@PSEFSGBMTFTIPX@UJNFGBMTFTIPX@NFNPSZGBMTF
TIPX@TVNNBSZUSVF
How it works...
This model spends a lot of time on Conv2D operations, as expected. Overall this takes about
77.5 percent of the average time on my desktop machine. If you run this on a mobile device,
it is critical to take the time to execute each layer in the neural network and make sure they
are under control. The other aspect to consider is the memory occupation. In this case, it is
~10 Mb for the desktop execution.
[ 385 ]
Mobile Computation Chapter 10
Getting ready
We are going to use Bazel for building the different components of TensorFlow. Therefore,
the first step is to make sure that both Bazel and TensorFlow are installed.
How to do it...
Here is how we proceed with transforming a TensorFlow:
VTSCJOSVCZF
DVSMGT4-=
IUUQTSBXHJUIVCVTFSDPOUFOUDPN)PNFCSFXJOTUBMMNBTUFSJOTUBMM
CSFXJOTUBMMCB[FM
CB[FMWFSTJPO
CSFXVQHSBEFCB[FM
HJUDMPOFIUUQTHJUIVCDPN5FOTPS'MPX5FOTPS'MPXHJU
[ 386 ]
Mobile Computation Chapter 10
CB[FMSVO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQI
JO@HSBQI6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQI
QC
8"3/*/(6TFSTHVMMJ5FOTPS'MPX5FOTPS'MPXDPSF#6*-%JO
JODMVEFTBUUSJCVUFPGDD@MJCSBSZSVMF
5FOTPS'MPXDPSFGSBNFXPSL@IFBEFST@MJC
FYUFSOBMOTZODQVCMJD SFTPMWFTUP FYUFSOBMOTZODQVCMJD
OPUCFMPXUIFSFMBUJWFQBUIPGJUTQBDLBHF 5FOTPS'MPXDPSF 5IJT
XJMMCFBOFSSPSJOUIFGVUVSF4JODFUIJTSVMFXBTDSFBUFECZUIF
NBDSP DD@IFBEFS@POMZ@MJCSBSZ UIFFSSPSNJHIUIBWFCFFODBVTFECZ
UIFNBDSPJNQMFNFOUBUJPOJO
6TFSTHVMMJ5FOTPS'MPX5FOTPS'MPX5FOTPS'MPXC[M
*/'0'PVOEUBSHFU
5BSHFU5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQIVQUP
EBUF
CB[FMCJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQI
*/'0&MBQTFEUJNFT$SJUJDBM1BUIT
*/'03VOOJOHDPNNBOEMJOFCB[FM
CJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTTVNNBSJ[F@HSBQI
JO@HSBQI6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQI
QC
'PVOEQPTTJCMFJOQVUT
OBNFJOQVUUZQFGMPBU
TIBQF<>
/PWBSJBCMFTTQPUUFE
'PVOEQPTTJCMFPVUQVUT
OBNFPVUQVUPQ*EFOUJUZ
OBNFPVUQVU
PQ*EFOUJUZ
OBNFPVUQVUPQ*EFOUJUZ
'PVOE
.DPOTUQBSBNFUFST
WBSJBCMF
QBSBNFUFSTBOEDPOUSPM@FEHFT
OPEFTBTTJHOFEUPEFWJDF DQV 0QUZQFTVTFE$POTU
#JBT"EE3FMV$POW%.BY1PPM$PODBU3FTIBQF
.BU.VM4PGUNBY*EFOUJUZ"WH1PPM-3/1MBDFIPMEFS
5PVTFXJUI5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFMUSZUIFTF
BSHVNFOUT
CB[FMSVO5FOTPS'MPXUPPMTCFODINBSLCFODINBSL@NPEFM
HSBQI6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQIQC
TIPX@GMPQTJOQVU@MBZFSJOQVUJOQVU@MBZFS@UZQFGMPBU
JOQVU@MBZFS@TIBQFPVUQVU@MBZFSPVUQVUPVUQVUPVUQVU
5. Strip all the nodes used for training, which are not needed when the graph is
used for inference on mobile devices:
CB[FMSVO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQI
JO@HSBQI6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQI
QCPVU@HSBQIUNQPQUJNJ[FE@JODFQUJPO@HSBQIQC
USBOTGPSNTTUSJQ@VOVTFE@OPEFTGPME@DPOTUBOUT
JHOPSF@FSSPSTUSVF
GPME@CBUDI@OPSNTGPME@PME@CBUDI@OPSNT
[ 387 ]
Mobile Computation Chapter 10
8"3/*/(6TFSTHVMMJ5FOTPS'MPX5FOTPS'MPXDPSF#6*-%JO
JODMVEFTBUUSJCVUFPGDD@MJCSBSZSVMF
5FOTPS'MPXDPSFGSBNFXPSL@IFBEFST@MJC
FYUFSOBMOTZODQVCMJD SFTPMWFTUP FYUFSOBMOTZODQVCMJD
OPUCFMPXUIFSFMBUJWFQBUIPGJUTQBDLBHF 5FOTPS'MPXDPSF 5IJT
XJMMCFBOFSSPSJOUIFGVUVSF4JODFUIJTSVMFXBTDSFBUFECZUIF
NBDSP DD@IFBEFS@POMZ@MJCSBSZ UIFFSSPSNJHIUIBWFCFFODBVTFECZ
UIFNBDSPJNQMFNFOUBUJPOJO
6TFSTHVMMJ5FOTPS'MPX5FOTPS'MPX5FOTPS'MPXC[M
*/'0'PVOEUBSHFU
5BSHFU5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIVQUP
EBUF
CB[FMCJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQI
*/'0&MBQTFEUJNFT$SJUJDBM1BUIT
*/'03VOOJOHDPNNBOEMJOFCB[FM
CJO5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQI
JO@HSBQI6TFSTHVMMJHSBQITJODFQUJPOI5FOTPS'MPX@JODFQUJPO@HSBQI
QC PVU@HSBQIUNQPQUJNJ[FE@JODFQUJPO@HSBQIQC
USBOTGPSNTTUSJQ@VOVTFE@OPEFTGPME@DPOTUBOUT
JHOPSF@FSSPSTUSVF
GPME@CBUDI@OPSNTGPME@PME@CBUDI@OPSNT
*
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIDD>"QQMZJOH
TUSJQ@VOVTFE@OPEFT
*
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIDD>"QQMZJOH
GPME@DPOTUBOUT
*
5FOTPS'MPXDPSFQMBUGPSNDQV@GFBUVSF@HVBSEDD>:PVS$16
TVQQPSUTJOTUSVDUJPOTUIBUUIJT5FOTPS'MPXCJOBSZXBTOPUDPNQJMFE
UPVTF44&"79"79'."
&
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIDD>
GPME@DPOTUBOUT*HOPSJOHFSSPS.VTUTQFDJGZBUMFBTUPOFUBSHFUUP
GFUDIPSFYFDVUF
*
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIDD>"QQMZJOH
GPME@CBUDI@OPSNT
*
5FOTPS'MPXUPPMTHSBQI@USBOTGPSNTUSBOTGPSN@HSBQIDD>"QQMZJOH
GPME@PME@CBUDI@OPSNT
[ 388 ]
Mobile Computation Chapter 10
How it works...
In order to create a lighter model that can be loaded on the device, we have deleted all the
unneeded nodes with the TUSJQ@VOVTFE@OPEFT rule which was applied by the graph'
transforms tool. This operation removes all operations used for learning and leaves the ones
used for inference intact.
[ 389 ]
Generative Models and
11
CapsNet
In this chapter, we will present a number of recipes for:
Introduction
In this chapter, we will discuss how Generative Adversarial Networks (GANs) are used
for deep learning in domains where the key method is to train an image generator by
simultaneously training a discriminator to challenge the latter for improvement. The same
method can be applied to domains different from Images. In addition, we will discuss the
Variational Autoencoder.
GANs have been defined as the most interesting idea in the last 10 years in ML (IUUQTXXX
RVPSBDPN8IBUBSFTPNFSFDFOUBOEQPUFOUJBMMZVQDPNJOHCSFBLUISPVHITJOEFFQ
MFBSOJOH) by Yann LeCun, one of the fathers of Deep Learning. GANs are able to learn
how to reproduce synthetic data which looks real. For instance, computers can learn how to
paint and create realistic images. The idea was originally proposed by Ian Goodfellow who
worked with the University of Montreal, Google Brain, and recently with OpenAI (IUUQT
PQFOBJDPN).
Generative Models and CapsNet Chapter 11
So what is a GAN?
The key process of GAN can be easily understood by considering it analogous to art forgery,
which is the process of creating works of art which are falsely credited to other, usually
more famous, artists. GANs train two neural networks simultaneously.
The Generator G(Z) is the one that makes the forgery happen, and the Discriminator D(Y)
is the one that can judge how realistic the reproduction is based on its observations of
authentic pieces of art and copies. D(Y) takes an input Y (for instance an image) and
expresses a vote to judge how real the input is. In general, a value close to zero denotes real,
while a value close to one denotes forgery. G(Z) takes an input from a random noise Z and
trains itself to fool D into thinking that whatever G(Z) produces is real. So the goal of
training the discriminator D(Y) is to maximize D(Y) for every image from the true data
distribution and to minimize D(Y) for every image not from the true data distribution. So G
and D play an opposite game: hence the name adversarial training. Note that we train G
and D in an alternating manner, where each of their objectives is expressed as a loss
function optimized via a gradient descent. The generative model learns how to forge better
and better, and the discriminative model learns how to recognize forgery better and better.
[ 391 ]
Generative Models and CapsNet Chapter 11
Of course, GANs find the equilibrium in a game with two players. For effective learning, it
is necessary that if a player successfully moves downhill in a round of updates, the same
update must move the other player downhill too. Think about it! If the forger learns how to
fool the judge on every occasion, then the forger himself has nothing more to learn.
Sometimes the two players eventually reach an equilibrium, but this is not always
guaranteed and the two players can continue playing for a long time. An example of this
from both sides is provided in the following figure:
#PGZCORNGQHEQPXGTIGPEGHQT)GPGTCVQTCPF&KUETKOKPCVQT
[ 392 ]
Generative Models and CapsNet Chapter 11
Now let us see how a GAN can learn to forge the MNIST dataset. In this case, it is a
combination of GAN and ConvNets used for the generator and the discriminator networks.
In the beginning, the generator creates nothing understandable, but after a few iterations,
synthetic forged numbers are progressively clearer and clearer. In the following figure, the
panels are ordered by increasing training epochs and you can see the quality improvement
among the panels:
[ 393 ]
Generative Models and CapsNet Chapter 11
5IFJNQSPWFEJNBHFJTBTGPMMPXT
8FDBOTFFGVSUIFSJNQSPWFNFOUTJOUIFGPMMPXJOHJNBHF
[ 394 ]
Generative Models and CapsNet Chapter 11
One of the coolest uses of GAN is doing arithmetics on faces in the generator's vector Z. In
other words, if we stay in the space of synthetic forged images, it is possible to see things
like this: [smiling woman] - [neutral woman] + [neutral man] = [smiling man], or like
this: [man with glasses] - [man without glasses] + [woman without glasses] = [woman
with glasses]. The following figure was taken from: Unsupervised Representation Learning
with Deep Convolutional Generative Adversarial Networks, Alec Radford, Luke Metz, Soumith
Chintala, 2016, IUUQTBSYJWPSHBCT
[ 395 ]
Generative Models and CapsNet Chapter 11
[ 396 ]
Generative Models and CapsNet Chapter 11
#PGZCORNGQHIGPGTCVGFDGFTQQOU
Album covers: these images are not real but generated by the GAN. Album covers looks
like real:
[ 397 ]
Generative Models and CapsNet Chapter 11
#PGZCORNGQHIGPGTCVGFCNDWOEQXGTU
[ 398 ]
Generative Models and CapsNet Chapter 11
Discriminator architecture.
Getting ready
This recipe is based on the code available at IUUQTHJUIVCDPN5FOHEB)BO("/
5FOTPS'MPX
How to do it...
We proceed with the recipe as follows:
HJUDMPOFIUUQTHJUIVCDPN5FOHEB)BO("/5FOTPS'MPX
EFGYBWJFS@JOJU
TJ[F
JO@EJNTJ[F<>
YBWJFS@TUEEFWUGTRSU
JO@EJN
SFUVSOYBWJFS@TUEEFW
3. Define the generator for the input X. First we define a matrix W1 with the
dimension [100, K=128], initialized according to a normal distribution. Note that
100 is an arbitrary value for Z, the initial noise used by our generator. Then, we
define the bias B1 with the dimension [K=256]. Similarly, we define a matrix W2
with the dimension [,, -] and a bias B2 with the dimension [L=784].
The two matrices W1 and W2 are initialized by using the YBWJFS@JOJU defined
in step 1, while B1 and B2 are initialized by using
UGDPOTUBOU@JOJUJBMJ[FS
. After that, we compute the multiplication
between matrices X * W1, sum the bias of B1, and pass it through a RELU
activation function for obtaining fc1. This dense layer is then connected with the
next dense layer which was created by multiplying the matrices fc1 with W2 and
summing the bias of B2. The result is then passed via a sigmoid function. These
steps are used to define our two layers' neural network used for the generator:
EFGHFOFSBUPS
9
XJUIUGWBSJBCMF@TDPQF
HFOFSBUPS
[ 399 ]
Generative Models and CapsNet Chapter 11
,
-
8UGHFU@WBSJBCMF
(@8 <,>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFWYBWJFS@JOJU
<,>
#UGHFU@WBSJBCMF
(@# <,>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
8UGHFU@WBSJBCMF
(@8 <,->
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFWYBWJFS@JOJU
<,->
#UGHFU@WBSJBCMF
(@# <->
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
TVNNBSZ
UGTVNNBSZIJTUPHSBN
XFJHIU 8
UGTVNNBSZIJTUPHSBN
XFJHIU 8
UGTVNNBSZIJTUPHSBN
CJBTFT #
UGTVNNBSZIJTUPHSBN
CJBTFT #
GDUGOOSFMV
UGNBUNVM
98
#
GDUGNBUNVM
GD8
#
QSPCUGOOTJHNPJE
GD
SFUVSOQSPC
4. Define the discriminator for input X. In principle, this is very similar to the
generator. The main difference is that if the parameter reuse is true, then we call
TDPQFSFVTF@WBSJBCMFT
to trigger a reuse. Then we define two dense layers.
The first layer uses a matrix W1 of dimensions [+, ,] with a bias B1 of
dimension [,], and it is based on the standard multiplication of X by W1.
This result is added to B1 and passed to a RELU activation function for getting
the result fc1. The second one uses a matrix W2 of the dimension [,, -]
with a bias B2 of dimension [-], and it is based on the standard multiplication
of fc1 by W2. This result is added to B2 and passed to a sigmoid function:
EFGEJTDSJNJOBUPS
9SFVTF'BMTF
XJUIUGWBSJBCMF@TDPQF
EJTDSJNJOBUPS
JGSFVTF
UGHFU@WBSJBCMF@TDPQF
SFVTF@WBSJBCMFT
+
,
-
8UGHFU@WBSJBCMF
%@8 <+,>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFWYBWJFS@JOJU
<+,>
#UGHFU@WBSJBCMF
%@# <,>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
8UGHFU@WBSJBCMF
%@8 <,->
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFWYBWJFS@JOJU
<,->
#UGHFU@WBSJBCMF
%@# <->
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
TVNNBSZ
[ 400 ]
Generative Models and CapsNet Chapter 11
UGTVNNBSZIJTUPHSBN
XFJHIU 8
UGTVNNBSZIJTUPHSBN
XFJHIU 8
UGTVNNBSZIJTUPHSBN
CJBTFT #
UGTVNNBSZIJTUPHSBN
CJBTFT #
GDUGOOSFMV
UGNBUNVM
98
#
MPHJUTUGNBUNVM
GD8
#
QSPCUGOOTJHNPJE
MPHJUT
SFUVSOQSPCMPHJUT
5. Now let's define some useful additional functions. First, we import a bunch of
standard modules:
JNQPSUUFOTPSGMPXBTUG
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
JNQPSUNBUQMPUMJCHSJETQFDBTHSJETQFD
JNQPSUPT
JNQPSUBSHQBSTF
6. Then we read the data from the MNIST dataset and define an auxiliary function
for plotting samples:
EFGSFBE@EBUB
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUBPOF@IPU5SVF
SFUVSONOJTU
EFGQMPU
TBNQMFT
GJHQMUGJHVSF
GJHTJ[F
HTHSJETQFD(SJE4QFD
HTVQEBUF
XTQBDFITQBDF
GPSJTBNQMFJOFOVNFSBUF
TBNQMFT
BYQMUTVCQMPU
HT<J>
QMUBYJT
PGG
BYTFU@YUJDLMBCFMT
<>
BYTFU@ZUJDLMBCFMT
<>
BYTFU@BTQFDU
FRVBM
QMUJNTIPX
TBNQMFSFTIBQF
DNBQ (SFZT@S
SFUVSOGJH
7. Now let's define the training function. First, let's read the MNIST data, and then
define a matrix X of shape 28 x 28 with one channel for a standard MNIST
handwritten character. Then let's define a z noise vector of size 100a common
choice proposed in the seminal GANs paper. The next step is to call the generator
on z and assign the result to G. After that, we pass X to the discriminator without
reuse. Then we pass the forged/fake G result to the discriminator, reusing the
[ 401 ]
Generative Models and CapsNet Chapter 11
learned weight. One important aspect of this is how we chose the loss function
for the discriminator, which is a sum of two cross entropies: one for real
characters, where all the real MNIST characters have a label set to one, and one
for forged characters, where all the forged characters have a label set to zero. The
discriminator and the generator run in an alternate sequence for 100,000 steps.
Every 500 steps, a sample is drawn from the learned distribution for printing
what that generator has learned so far. This is what defines a new epoch, and the
results are shown in the next section. Let's see the code snippet which
implements what we just described.
EFGUSBJO
MPHEJSCBUDI@TJ[F
GSPNNPEFM@GDJNQPSUEJTDSJNJOBUPSHFOFSBUPS
NOJTUSFBE@EBUB
XJUIUGWBSJBCMF@TDPQF
QMBDFIPMEFS
3BXJNBHF
9UGQMBDFIPMEFS
UGGMPBU</POF>
UGTVNNBSZJNBHF
SBXJNBHF UGSFTIBQF
9<>
/PJTF
[UGQMBDFIPMEFS
UGGMPBU</POF>OPJTF
UGTVNNBSZIJTUPHSBN
/PJTF [
XJUIUGWBSJBCMF@TDPQF
("/
(HFOFSBUPS
[
%@SFBM%@SFBM@MPHJUTEJTDSJNJOBUPS
9SFVTF'BMTF
%@GBLF%@GBLF@MPHJUTEJTDSJNJOBUPS
(SFVTF5SVF
UGTVNNBSZJNBHF
HFOFSBUFEJNBHF UGSFTIBQF
(<>
XJUIUGWBSJBCMF@TDPQF
1SFEJDUJPO
UGTVNNBSZIJTUPHSBN
SFBM %@SFBM
UGTVNNBSZIJTUPHSBN
GBLF %@GBLF
XJUIUGWBSJBCMF@TDPQF
%@MPTT
E@MPTT@SFBMUGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUT%@SFBM@MPHJUTMBCFMTUGPOFT@MJLF
%@SFBM@MPHJUT
E@MPTT@GBLFUGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUT%@GBLF@MPHJUTMBCFMTUG[FSPT@MJLF
%@GBLF@MPHJUT
E@MPTTE@MPTT@SFBM
E@MPTT@GBLF
UGTVNNBSZTDBMBS
E@MPTT@SFBM E@MPTT@SFBM
UGTVNNBSZTDBMBS
E@MPTT@GBLF E@MPTT@GBLF
UGTVNNBSZTDBMBS
E@MPTT E@MPTT
XJUIUGOBNF@TDPQF
(@MPTT
H@MPTTUGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUT%@GBLF@MPHJUTMBCFMTUGPOFT@MJLF
%@GBLF@MPHJUT
[ 402 ]
Generative Models and CapsNet Chapter 11
UGTVNNBSZTDBMBS
H@MPTT H@MPTT
UWBSUGUSBJOBCMF@WBSJBCMFT
EWBS<WBSGPSWBSJOUWBSJG EJTDSJNJOBUPS JOWBSOBNF>
HWBS<WBSGPSWBSJOUWBSJG HFOFSBUPS JOWBSOBNF>
XJUIUGOBNF@TDPQF
USBJO
E@USBJO@TUFQUGUSBJO"EBN0QUJNJ[FS
NJOJNJ[F
E@MPTT
WBS@MJTUEWBS
H@USBJO@TUFQUGUSBJO"EBN0QUJNJ[FS
NJOJNJ[F
H@MPTT
WBS@MJTUHWBS
TFTTUG4FTTJPO
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFTTSVO
JOJU
NFSHFE@TVNNBSZUGTVNNBSZNFSHF@BMM
XSJUFSUGTVNNBSZ'JMF8SJUFS
UNQNOJTU
MPHEJS
XSJUFSBEE@HSBQI
TFTTHSBQI
OVN@JNH
JGOPUPTQBUIFYJTUT
PVUQVU
PTNBLFEJST
PVUQVU
GPSJJOSBOHF
CBUDI@9@NOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
CBUDI@OPJTFOQSBOEPNVOJGPSN
<CBUDI@TJ[F>
JGJ
TBNQMFTTFTTSVO
(GFFE@EJDU\[OQSBOEPNVOJGPSN
<
>^
GJHQMPU
TBNQMFT
QMUTBWFGJH
PVUQVUTQOH TUS
OVN@JNH[GJMM
CCPY@JODIFT UJHIU
OVN@JNH
QMUDMPTF
GJH
@E@MPTT@QSJOUTFTTSVO
<E@USBJO@TUFQE@MPTT>
GFFE@EJDU\9CBUDI@9[CBUDI@OPJTF^
@H@MPTT@QSJOUTFTTSVO
<H@USBJO@TUFQH@MPTT>
GFFE@EJDU\[CBUDI@OPJTF^
JGJ
TTFTTSVO
NFSHFE@TVNNBSZGFFE@EJDU\9CBUDI@9[CBUDI@OPJTF^
XSJUFSBEE@TVNNBSZ
TJ
QSJOU
FQPDIEH@MPTTGE@MPTTG
JH@MPTT@QSJOUE@MPTT@QSJOU
JG@@OBNF@@ @@NBJO@@
QBSTFSBSHQBSTF"SHVNFOU1BSTFS
EFTDSJQUJPO 5SBJOWBOJMB("/VTJOH
GVMMZDPOOFDUFEMBZFSTOFUXPSLT
QBSTFSBEE@BSHVNFOU
MPHEJS UZQFTUSEFGBVMU IFMQ MPHEJSGPS
5FOTPSCPBSEHJWFBTUSJOH
QBSTFSBEE@BSHVNFOU
CBUDI@TJ[F UZQFJOUEFGBVMUIFMQ CBUDI
[ 403 ]
Generative Models and CapsNet Chapter 11
TJ[FHJWFBJOU
BSHTQBSTFSQBSTF@BSHT
USBJO
MPHEJSBSHTMPHEJSCBUDI@TJ[FBSHTCBUDI@TJ[F
How it works...
At each epoch, the generator makes a number of predictions (it creates forged MNIST
images) and the discriminator tries to learn how to produce forged images after mixing the
prediction with real MNIST images. After 32 epochs, the generator learns to forge this set of
handwritten numbers. No one has programmed the machine to write, but it has learned
how to write numbers that are indistinguishable from the ones written by humans. Note
that training GANs can be very difficult because it is necessary to find the equilibrium
between two players. If you are interested in the topic, I'd suggest you have a look at the
series of tricks collected by practitioners (IUUQTHJUIVCDPNTPVNJUIHBOIBDLT).
Let's look at a number of practical examples for different epochs to understand how the
machine will learn to improve its writing process:
[ 404 ]
Generative Models and CapsNet Chapter 11
'ZCORNGQHHQTIGF/0+56NKMGEJCTCEVGTUYKVJC)#0
Getting ready
This recipe is based on the code available at IUUQTHJUIVCDPN5FOHEB)BO("/
5FOTPS'MPX
How to do it...
We proceed with the recipe as follows:
HJUDMPOFIUUQTHJUIVCDPN5FOHEB)BO("/5FOTPS'MPX
[ 405 ]
Generative Models and CapsNet Chapter 11
EFGYBWJFS@JOJU
TJ[F
JO@EJNTJ[F<>
YBWJFS@TUEEFWUGTRSU
JO@EJN
SFUVSOUGSBOEPN@OPSNBM
TIBQFTJ[FTUEEFWYBWJFS@TUEEFW
SFUVSOYBWJFS@TUEEFW
3. Define a convolutional operation for the given input x, weight w, bias b, and
given stride. Our code uses the standard UGOODPOWE
module. Note that
we use 'SAME' padding, as defined in Chapter 4:
EFGDPOW
YXCTUSJEFOBNF
XJUIUGWBSJBCMF@TDPQF
DPOW
UGTVNNBSZIJTUPHSBN
XFJHIU X
UGTVNNBSZIJTUPHSBN
CJBTFT C
SFUVSOUGOODPOWE
Y
GJMUFSX
TUSJEFT<TUSJEFTUSJEF>
QBEEJOH 4".&
OBNFOBNF
C
4. Define a de-convolutional operation for the given input x, weight w, bias b, and
given stride. Our code uses the standard UGOODPOWE@USBOTQPTF
module. Again, we use 4".& padding.
EFGEFDPOW
YXCTIBQFTUSJEFOBNF
XJUIUGWBSJBCMF@TDPQF
EFDPOW
UGTVNNBSZIJTUPHSBN
XFJHIU X
UGTVNNBSZIJTUPHSBN
CJBTFT C
SFUVSOUGOODPOWE@USBOTQPTF
Y
GJMUFSX
PVUQVU@TIBQFTIBQF
TUSJEFT<TUSJEFTUSJEF>
QBEEJOH 4".&
OBNFOBNF
C
EFGMSFMV
YBMQIB
XJUIUGWBSJBCMF@TDPQF
MFBLZ3F-6
SFUVSOUGNBYJNVN
YBMQIBY
[ 406 ]
Generative Models and CapsNet Chapter 11
6. Define the generator. First, we define a fully connected layer with input size 100
(an arbitrary size for Z, the initial noise used by our generator). The fully
connected layer consists of a matrix W1 with the dimension [100, 7*7*256],
initialized according to a normal distribution, and a bias B1 with the dimension
[7*7*256]. The layer uses ReLu as an activation function. After the fully connected
layer, the generator applies two deconvolutional operations, deconv1 and
deconv2, both with stride=2. After the first deconv1 operation is done, results are
batch normalized. Note that the second deconvolutional operation is preceded by
dropout, with a probability of 40percent. The last stage is a sigmoid used as non-
linear activation as reported in the below code snippet:
EFGHFOFSBUPS
9CBUDI@TJ[F
XJUIUGWBSJBCMF@TDPQF
HFOFSBUPS
,
-
.
8UGHFU@WBSJBCMF
(@8 <,>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFW
#UGHFU@WBSJBCMF
(@# <,>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
8UGHFU@WBSJBCMF
(@8 <.,>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFW
#UGHFU@WBSJBCMF
(@# <.>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
8UGHFU@WBSJBCMF
(@8 <.>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFW
#UGHFU@WBSJBCMF
(@# <>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
9MSFMV
UGNBUNVM
98
#
9UGSFTIBQF
9<CBUDI@TJ[F,>
EFDPOWEFDPOW
98#TIBQF<CBUDI@TJ[F.>TUSJEF
OBNF EFDPOW
COUGDPOUSJCMBZFSTCBUDI@OPSN
EFDPOW
EFDPOWEFDPOW
UGOOESPQPVU
MSFMV
CO8#
TIBQF<CBUDI@TJ[F>TUSJEFOBNF EFDPOW
99UGSFTIBQF
EFDPOW<> SFTIBQF
SFUVSOUGOOTJHNPJE
99
7. Define the discriminator. As in the previous recipe, if the parameter reuse is true,
then we call TDPQFSFVTF@WBSJBCMFT
to trigger a reuse. The discriminator
uses two convolutional layers. The first one is followed by batch normalization
while the second one is preceded by a dropout with a probability of 40 percent
and followed again by a batch normalization step. After that, we have a dense
layer with the activation function ReLU followed by another dense layer with the
activation function based on the sigmoid:
[ 407 ]
Generative Models and CapsNet Chapter 11
EFGEJTDSJNJOBUPS
9SFVTF'BMTF
XJUIUGWBSJBCMF@TDPQF
EJTDSJNJOBUPS
JGSFVTF
UGHFU@WBSJBCMF@TDPQF
SFVTF@WBSJBCMFT
,
.
/
8UGHFU@WBSJBCMF
%@8 <,>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFW
#UGHFU@WBSJBCMF
%@# <,>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
8UGHFU@WBSJBCMF
%@8 <,.>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFW
#UGHFU@WBSJBCMF
%@# <.>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
8UGHFU@WBSJBCMF
%@8 <./>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFW
#UGHFU@WBSJBCMF
%@# </>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
8UGHFU@WBSJBCMF
%@8 </>
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFW
#UGHFU@WBSJBCMF
%@# <>
JOJUJBMJ[FSUGDPOTUBOU@JOJUJBMJ[FS
9UGSFTIBQF
9<> SFTIBQF
DPOWDPOW
98#TUSJEFOBNF DPOW
COUGDPOUSJCMBZFSTCBUDI@OPSN
DPOW
DPOWDPOW
UGOOESPQPVU
MSFMV
CO8#TUSJEF
OBNF DPOW
COUGDPOUSJCMBZFSTCBUDI@OPSN
DPOW
GMBUUGSFTIBQF
UGOOESPQPVU
MSFMV
CO<.>
OBNF GMBU
EFOTFMSFMV
UGNBUNVM
GMBU8
#
MPHJUTUGNBUNVM
EFOTF8
#
QSPCUGOOTJHNPJE
MPHJUT
SFUVSOQSPCMPHJUT
8. Then we read the data from the MNIST dataset, and define an auxiliary function
for plotting samples:
JNQPSUOVNQZBTOQ
JNQPSUNBUQMPUMJCQZQMPUBTQMU
JNQPSUNBUQMPUMJCHSJETQFDBTHSJETQFD
JNQPSUPT
JNQPSUBSHQBSTF
EFGSFBE@EBUB
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
[ 408 ]
Generative Models and CapsNet Chapter 11
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUBPOF@IPU5SVF
SFUVSONOJTU
EFGQMPU
TBNQMFT
GJHQMUGJHVSF
GJHTJ[F
HTHSJETQFD(SJE4QFD
HTVQEBUF
XTQBDFITQBDF
GPSJTBNQMFJOFOVNFSBUF
TBNQMFT
BYQMUTVCQMPU
HT<J>
QMUBYJT
PGG
BYTFU@YUJDLMBCFMT
<>
BYTFU@ZUJDLMBCFMT
<>
BYTFU@BTQFDU
FRVBM
QMUJNTIPX
TBNQMFSFTIBQF
DNBQ (SFZT@S
SFUVSOGJH
9. Now let's define the training function. First, let's read the MNIST data, and then
define a matrix X of shape 28 x 28 with one channel for a standard MNIST
handwritten character. Then let's define az noise vector of size 100a common
choice proposed in the seminal GANs paper. The next step is to call the generator
on z and assign the result to G. After that, we pass X to the discriminator without
reuse. Then we pass the forged/fake G result to the discriminator, reusing the
learned weight. One important aspect of this is how we chose the loss function
for the discriminator, which is a sum of two cross entropies: one for real
characters where all the real MNIST characters have a label set to one, and one for
forgetting characters where all the forged characters have a label set to zero. The
discriminator and the generator run in an alternate sequence for 100,000 steps.
Every 500 steps, a sample is drawn from the learned distribution for printing
what that generator has learned so far. This is what defines a new epoch, and the
results are shown during the next section. The training function code snippet is
reported below
EFGUSBJO
MPHEJSCBUDI@TJ[F
GSPNNPEFM@DPOWJNQPSUEJTDSJNJOBUPSHFOFSBUPS
NOJTUSFBE@EBUB
XJUIUGWBSJBCMF@TDPQF
QMBDFIPMEFS
3BXJNBHF
9UGQMBDFIPMEFS
UGGMPBU</POF>
UGTVNNBSZJNBHF
SBXJNBHF UGSFTIBQF
9<>
/PJTF
[UGQMBDFIPMEFS
UGGMPBU</POF>OPJTF
UGTVNNBSZIJTUPHSBN
/PJTF [
XJUIUGWBSJBCMF@TDPQF
("/
(HFOFSBUPS
[CBUDI@TJ[F
%@SFBM%@SFBM@MPHJUTEJTDSJNJOBUPS
9SFVTF'BMTF
[ 409 ]
Generative Models and CapsNet Chapter 11
%@GBLF%@GBLF@MPHJUTEJTDSJNJOBUPS
(SFVTF5SVF
UGTVNNBSZJNBHF
HFOFSBUFEJNBHF UGSFTIBQF
(<>
XJUIUGWBSJBCMF@TDPQF
1SFEJDUJPO
UGTVNNBSZIJTUPHSBN
SFBM %@SFBM
UGTVNNBSZIJTUPHSBN
GBLF %@GBLF
XJUIUGWBSJBCMF@TDPQF
%@MPTT
E@MPTT@SFBMUGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUT%@SFBM@MPHJUTMBCFMTUGPOFT@MJLF
%@SFBM@MPHJUT
E@MPTT@GBLFUGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUT%@GBLF@MPHJUTMBCFMTUG[FSPT@MJLF
%@GBLF@MPHJUT
E@MPTTE@MPTT@SFBM
E@MPTT@GBLF
UGTVNNBSZTDBMBS
E@MPTT@SFBM E@MPTT@SFBM
UGTVNNBSZTDBMBS
E@MPTT@GBLF E@MPTT@GBLF
UGTVNNBSZTDBMBS
E@MPTT E@MPTT
XJUIUGOBNF@TDPQF
(@MPTT
H@MPTTUGSFEVDF@NFBO
UGOOTJHNPJE@DSPTT@FOUSPQZ@XJUI@MPHJUT
MPHJUT%@GBLF@MPHJUTMBCFMTUGPOFT@MJLF
%@GBLF@MPHJUT
UGTVNNBSZTDBMBS
H@MPTT H@MPTT
UWBSUGUSBJOBCMF@WBSJBCMFT
EWBS<WBSGPSWBSJOUWBSJG EJTDSJNJOBUPS JOWBSOBNF>
HWBS<WBSGPSWBSJOUWBSJG HFOFSBUPS JOWBSOBNF>
XJUIUGOBNF@TDPQF
USBJO
E@USBJO@TUFQUGUSBJO"EBN0QUJNJ[FS
NJOJNJ[F
E@MPTTWBS@MJTUEWBS
H@USBJO@TUFQUGUSBJO"EBN0QUJNJ[FS
NJOJNJ[F
H@MPTTWBS@MJTUHWBS
TFTTUG4FTTJPO
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
TFTTSVO
JOJU
NFSHFE@TVNNBSZUGTVNNBSZNFSHF@BMM
XSJUFSUGTVNNBSZ'JMF8SJUFS
UNQ
HBO@DPOW@
MPHEJS
XSJUFSBEE@HSBQI
TFTTHSBQI
OVN@JNH
JGOPUPTQBUIFYJTUT
PVUQVU
PTNBLFEJST
PVUQVU
GPSJJOSBOHF
CBUDI@9@NOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
CBUDI@OPJTFOQSBOEPNVOJGPSN
<CBUDI@TJ[F>
JGJ
TBNQMFTTFTTSVO
(GFFE@EJDU\[OQSBOEPNVOJGPSN
<
>^
[ 410 ]
Generative Models and CapsNet Chapter 11
GJHQMPU
TBNQMFT
QMUTBWFGJH
PVUQVUTQOH TUS
OVN@JNH[GJMM
CCPY@JODIFT UJHIU
OVN@JNH
QMUDMPTF
GJH
@E@MPTT@QSJOUTFTTSVO
<E@USBJO@TUFQE@MPTT>
GFFE@EJDU\9CBUDI@9[CBUDI@OPJTF^
@H@MPTT@QSJOUTFTTSVO
<H@USBJO@TUFQH@MPTT>
GFFE@EJDU\[CBUDI@OPJTF^
JGJ
TTFTTSVO
NFSHFE@TVNNBSZGFFE@EJDU\9CBUDI@9[CBUDI@OPJTF^
XSJUFSBEE@TVNNBSZ
TJ
QSJOU
FQPDIEH@MPTTGE@MPTTG
JH@MPTT@QSJOUE@MPTT@QSJOU
JG@@OBNF@@ @@NBJO@@
QBSTFSBSHQBSTF"SHVNFOU1BSTFS
EFTDSJQUJPO 5SBJOWBOJMB("/VTJOH
DPOWPMVUJPOBMOFUXPSLT
QBSTFSBEE@BSHVNFOU
MPHEJS UZQFTUSEFGBVMU IFMQ MPHEJSGPS
5FOTPSCPBSEHJWFBTUSJOH
QBSTFSBEE@BSHVNFOU
CBUDI@TJ[F UZQFJOUEFGBVMUIFMQ CBUDI
TJ[FHJWFBJOU
BSHTQBSTFSQBSTF@BSHT
USBJO
MPHEJSBSHTMPHEJSCBUDI@TJ[FBSHTCBUDI@TJ[F
How it works...
Using CNN together with GANs results in a faster capacity to learn. Let's look at a number
of practical examples of different epochs to understand how the machine will learn to
improve its writing process. For instance, compare the results achieved after four iterations
in the following recipe with the results achieved after four iterations in the previous recipe.
Do you see the difference? I wish that I could learn this art myself!
[ 411 ]
Generative Models and CapsNet Chapter 11
'ZCORNGQHHQTIGF/0+56NKMGYKVJ&%)#0
[ 412 ]
Generative Models and CapsNet Chapter 11
In recent years, supervised learning with convolutional networks (CNNs) has seen huge
adoption in computer vision applications. Comparatively, unsupervised learning with
CNNs has received less attention. In this work we hope to help bridge the gap between the
success of CNNs for supervised learning and unsupervised learning. We introduce a class
of CNNs called deep convolutional generative adversarial networks (DCGANs), that have
certain architectural constraints, and demonstrate that they are a strong candidate for
unsupervised learning. Training on various image datasets, we show convincing evidence
that our deep convolutional adversarial pair learns a hierarchy of representations from
object parts to scenes in both the generator and discriminator. Additionally, we use the
learned features for novel tasks - demonstrating their applicability as general image
representations.
Note that the generator has the architecture represented in the following diagram:
Note that in the package there are changes with respect to the original paper to avoid the
fast convergence of D (discriminator) network, G (generator) network is updated twice for
[ 413 ]
Generative Models and CapsNet Chapter 11
Getting ready
This recipe is based on the code available at IUUQTHJUIVCDPNDBSQFEN%$("/
UFOTPSGMPX
How to do it...
We proceed with the recipe as follows:
HJUDMPOFIUUQTHJUIVCDPNDBSQFEN%$("/UFOTPSGMPX
QZUIPOEPXOMPBEQZNOJTUDFMFC"
QZUIPONBJOQZEBUBTFUDFMFC"JOQVU@IFJHIUUSBJODSPQ
QZUIPONBJOQZEBUBTFUDFMFC"JOQVU@IFJHIUDSPQ
5. Alternatively, you can use your own dataset by doing the following:
NLEJSEBUB%"5"4&5@/".&
BEEJNBHFTUPEBUB%"5"4&5@/".&
QZUIPONBJOQZEBUBTFU%"5"4&5@/".&USBJO
QZUIPONBJOQZEBUBTFU%"5"4&5@/".&
FYBNQMF
QZUIPONBJOQZEBUBTFUFZFTJOQVU@GOBNF@QBUUFSO@DSPQQFEQOH
USBJO
How it works...
The generator learns how to produce forged images of celebrities and the discriminator
learns how to distinguish the forged images from the real ones. Each epoch in the two
[ 414 ]
Generative Models and CapsNet Chapter 11
networks competes to improve and reduce losses. The first five epochs are reported in the
following table:
Epoch 0 Epoch 1
Epoch 2 Epoch 3
[ 415 ]
Generative Models and CapsNet Chapter 11
Epoch 4 Epoch 5
'ZCORNGQHHQTIGFEGNGDTKVKGUYKVJC&%)#0
There's more...
Content-aware fill is a tool used by photographers to fill in unwanted or missing parts of
images. The paper Semantic Image Inpainting with Perceptual and Contextual
Losses, by Raymond A. Yeh, Chen Chen, Teck Yian Lim, Alexander G. Schwing, Mark
Hasegawa-Johnson, Minh N., 2016 uses a DCGAN for image completion, and it learns how
to fill in parts of images.
[ 416 ]
Generative Models and CapsNet Chapter 11
figures based on the distribution they have been trained on. VAEs allow one to set complex
priors in the latent and thus learn powerful latent representations.
The following diagram describes a VAE. The Encoder network q (z|x) approximates the
true but intractable posterior distribution p(z|x), where x is the input to the VAE and z is the
latent representation. The decoder network pq(x|z) takes the d-dimensional latent variables
(also called latent space) as its input and generate new images following g the same
and the output of the decoder network samples x|z from x|z ~ :
'ZCORNGQH'PEQFGT&GEQFGTHQT#WVQGPEQFGTU
Getting ready...
Now that we have the basic architecture of VAEs, the question arises how can they be
trained, since the maximum likelihood of the training data and posterior density are
[ 417 ]
Generative Models and CapsNet Chapter 11
intractable? The network is trained by maximizing the lower bound of the log data
likelihood. Thus, the loss term consists of two components: generation loss, which is
obtained from the decoder network through sampling, and the KL divergence term, also
called the latent loss.
Generation loss ensures that the image generated by the decoder and the image used to
train the network is the same, and latent loss ensures that the posterior distribution q (z|x)
is close to the prior pq(z). Since the encoder uses Gaussian distribution for sampling, the
latent loss measures how closely the latent variables match a unit Gaussian.
Once the VAE is trained, we can use only the decoder network to generate new images.
How to do it...
The code here is based on the paper Autoencoding Variational Bayes by Kingma and
Welling (IUUQTBSYJWPSHQEGQEG), and is adapted from GitHub: IUUQT
KNFU[FOHJUIVCJPWBFIUNM .
1. The first step is as always importing the necessary modules. For this recipe we
will require Numpy, Matplolib, and TensorFlow:
JNQPSUOVNQZBTOQ
JNQPSUUFOTPSGMPXBTUG
JNQPSUNBUQMPUMJCQZQMPUBTQMU
NBUQMPUMJCJOMJOF
DMBTT7BSJBUJPOBM"VUPFODPEFS
PCKFDU
EFG@@JOJU@@
TFMGOFUXPSL@BSDIJUFDUVSFUSBOTGFS@GDUUGOOTPGUQMVT
MFBSOJOH@SBUFCBUDI@TJ[F
TFMGOFUXPSL@BSDIJUFDUVSFOFUXPSL@BSDIJUFDUVSF
TFMGUSBOTGFS@GDUUSBOTGFS@GDU
TFMGMFBSOJOH@SBUFMFBSOJOH@SBUF
[ 418 ]
Generative Models and CapsNet Chapter 11
TFMGCBUDI@TJ[FCBUDI@TJ[F
1MBDFIPMEFSGPSUIFJOQVU
TFMGYUGQMBDFIPMEFS
UGGMPBU</POF
OFUXPSL@BSDIJUFDUVSF<O@JOQVU>>
%FGJOFXFJHIUTBOECJBTFT
OFUXPSL@XFJHIUT
TFMG@JOJUJBMJ[F@XFJHIUT
TFMGOFUXPSL@BSDIJUFDUVSF
$SFBUFBVUPFODPEFSOFUXPSL
6TF&ODPEFS/FUXPSLUPEFUFSNJOFNFBOBOE
MPHWBSJBODFPG(BVTTJBOEJTUSJCVUJPOJOMBUFOU
TQBDF
TFMG[@NFBOTFMG[@MPH@TJHNB@TR=
TFMG@FODPEFS@OFUXPSL
OFUXPSL@XFJHIUT<XFJHIUT@FODPEFS>
OFUXPSL@XFJHIUT<CJBTFT@FODPEFS>
%SBXPOFTBNQMF[GSPN(BVTTJBOEJTUSJCVUJPO
O@[TFMGOFUXPSL@BSDIJUFDUVSF<O@[>
FQTUGSBOEPN@OPSNBM
TFMGCBUDI@TJ[FO@[
EUZQFUGGMPBU
[NV
TJHNBFQTJMPO
TFMG[
UGBEE
TFMG[@NFBOUGNVMUJQMZ
UGTRSU
UGFYQ
TFMG[@MPH@TJHNB@TRFQT
6TF%FDPEFSOFUXPSLUPEFUFSNJOFNFBOPG
#FSOPVMMJEJTUSJCVUJPOPGSFDPOTUSVDUFEJOQVU
TFMGY@SFDPOTUS@NFBO=
TFMG@EFDPEFS@OFUXPSL
OFUXPSL@XFJHIUT<XFJHIUT@EFDPEFS>
OFUXPSL@XFJHIUT<CJBTFT@EFDPEFS>
%FGJOFMPTTGVODUJPOCBTFEWBSJBUJPOBMVQQFSCPVOEBOE
DPSSFTQPOEJOHPQUJNJ[FS
EFGJOFHFOFSBUJPOMPTT
HFOFSBUJPO@MPTT=
UGSFEVDF@TVN
TFMGYUGMPH
F
TFMGY@SFDPOTUS@NFBO
TFMGYUGMPH
F
TFMGY@SFDPOTUS@NFBO
MBUFOU@MPTTUGSFEVDF@TVN
TFMG[@MPH@TJHNB@TR
UGTRVBSF
TFMG[@NFBOUGFYQ
TFMG[@MPH@TJHNB@TR
TFMGDPTUUGSFEVDF@NFBO
HFOFSBUJPO@MPTT
MBUFOU@MPTT
BWFSBHFPWFSCBUDI
%FGJOFUIFPQUJNJ[FS
TFMGPQUJNJ[FS=
UGUSBJO"EBN0QUJNJ[FS
MFBSOJOH@SBUFTFMGMFBSOJOH@SBUFNJOJNJ[F
TFMGDPTU
*OJUJBMJ[JOHUIFUFOTPSGMPXWBSJBCMFT
JOJUUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
-BVODIUIFTFTTJPO
TFMGTFTTUG*OUFSBDUJWF4FTTJPO
TFMGTFTTSVO
JOJU
EFG@JOJUJBMJ[F@XFJHIUT
TFMGO@IJEEFO@SFDPH@O@IJEEFO@SFDPH@
O@IJEEFO@HFOFS@O@IJEEFO@HFOFS@
O@JOQVUO@[
[ 419 ]
Generative Models and CapsNet Chapter 11
JOJUJBMJ[FSUGDPOUSJCMBZFSTYBWJFS@JOJUJBMJ[FS
BMM@XFJHIUTEJDU
BMM@XFJHIUT< XFJHIUT@FODPEFS >\
I UG7BSJBCMF
JOJUJBMJ[FS
TIBQF
O@JOQVUO@IJEEFO@SFDPH@
I UG7BSJBCMF
JOJUJBMJ[FS
TIBQF
O@IJEEFO@SFDPH@
O@IJEEFO@SFDPH@
PVU@NFBO UG7BSJBCMF
JOJUJBMJ[FS
TIBQF
O@IJEEFO@SFDPH@O@[
PVU@MPH@TJHNB UG7BSJBCMF
JOJUJBMJ[FS
TIBQF
O@IJEEFO@SFDPH@
O@[^
BMM@XFJHIUT< CJBTFT@FODPEFS >\
C UG7BSJBCMF
UG[FSPT
<O@IJEEFO@SFDPH@>EUZQFUGGMPBU
C UG7BSJBCMF
UG[FSPT
<O@IJEEFO@SFDPH@>EUZQFUGGMPBU
PVU@NFBO UG7BSJBCMF
UG[FSPT
<O@[>EUZQFUGGMPBU
PVU@MPH@TJHNB UG7BSJBCMF
UG[FSPT
<O@[>EUZQFUGGMPBU^
3. We build the encoder network and the decoder network. The first layer of the
Encoder network is taking the input and generating a reduced latent
representation of the input. The second layer maps the input to a Gaussian
distribution. The network learns these transformations:
EFG@FODPEFS@OFUXPSL
TFMGXFJHIUTCJBTFT
(FOFSBUFQSPCBCJMJTUJDFODPEFS
SFDPHOJUJPOOFUXPSLXIJDI
NBQTJOQVUTPOUPBOPSNBMEJTUSJCVUJPOJOMBUFOUTQBDF
5IFUSBOTGPSNBUJPOJTQBSBNFUSJ[FEBOEDBOCFMFBSOFE
MBZFS@TFMGUSBOTGFS@GDU
UGBEE
UGNBUNVM
TFMGYXFJHIUT< I >
CJBTFT< C >
MBZFS@TFMGUSBOTGFS@GDU
UGBEE
UGNBUNVM
MBZFS@XFJHIUT< I >
CJBTFT< C >
[@NFBOUGBEE
UGNBUNVM
MBZFS@XFJHIUT< PVU@NFBO >
CJBTFT< PVU@NFBO >
[@MPH@TJHNB@TR=
[ 420 ]
Generative Models and CapsNet Chapter 11
UGBEE
UGNBUNVM
MBZFS@XFJHIUT< PVU@MPH@TJHNB >
CJBTFT< PVU@MPH@TJHNB >
SFUVSO
[@NFBO[@MPH@TJHNB@TR
EFG@EFDPEFS@OFUXPSL
TFMGXFJHIUTCJBTFT
(FOFSBUFQSPCBCJMJTUJDEFDPEFS
EFDPEFSOFUXPSLXIJDI
NBQTQPJOUTJOMBUFOUTQBDFPOUPB#FSOPVMMJEJTUSJCVUJPOJOEBUB
TQBDF
5IFUSBOTGPSNBUJPOJTQBSBNFUSJ[FEBOEDBOCFMFBSOFE
MBZFS@TFMGUSBOTGFS@GDU
UGBEE
UGNBUNVM
TFMG[XFJHIUT< I >
CJBTFT< C >
MBZFS@TFMGUSBOTGFS@GDU
UGBEE
UGNBUNVM
MBZFS@XFJHIUT< I >
CJBTFT< C >
Y@SFDPOTUS@NFBO=
UGOOTJHNPJE
UGBEE
UGNBUNVM
MBZFS@XFJHIUT< PVU@NFBO >
CJBTFT< PVU@NFBO >
SFUVSOY@SFDPOTUS@NFBO
EFGGJU
TFMG9
PQUDPTUTFMGTFTTSVO
TFMGPQUJNJ[FSTFMGDPTU
GFFE@EJDU\TFMGY9^
SFUVSODPTU
EFGHFOFSBUF
TFMG[@NV/POF
(FOFSBUFEBUBCZTBNQMJOHGSPNMBUFOUTQBDF
*G[@NVJTOPU/POFEBUBGPSUIJTQPJOUJOMBUFOUTQBDFJT
HFOFSBUFE0UIFSXJTF[@NVJTESBXOGSPNQSJPSJOMBUFOU
TQBDF
JG[@NVJT/POF
[@NVOQSBOEPNOPSNBM
TJ[FTFMGOFUXPSL@BSDIJUFDUVSF<O@[>
/PUF5IJTNBQTUPNFBOPGEJTUSJCVUJPOXFDPVMEBMUFSOBUJWFMZ
TBNQMFGSPN(BVTTJBOEJTUSJCVUJPO
SFUVSOTFMGTFTTSVO
TFMGY@SFDPOTUS@NFBO
GFFE@EJDU\TFMG[[@NV^
EFGSFDPOTUSVDU
TFMG9
6TF7"&UPSFDPOTUSVDUHJWFOEBUB
SFUVSOTFMGTFTTSVO
TFMGY@SFDPOTUS@NFBO
GFFE@EJDU\TFMGY9^
5. Once the VAE class is done, we define a function train, which uses the VAE class
object and trains it for a given data.
[ 421 ]
Generative Models and CapsNet Chapter 11
EFGUSBJO
OFUXPSL@BSDIJUFDUVSFMFBSOJOH@SBUF
CBUDI@TJ[FUSBJOJOH@FQPDITEJTQMBZ@TUFQ
WBF7BSJBUJPOBM"VUPFODPEFS
OFUXPSL@BSDIJUFDUVSF
MFBSOJOH@SBUFMFBSOJOH@SBUF
CBUDI@TJ[FCBUDI@TJ[F
5SBJOJOHDZDMF
GPSFQPDIJOSBOHF
USBJOJOH@FQPDIT
BWH@DPTU
UPUBM@CBUDIJOU
O@TBNQMFTCBUDI@TJ[F
-PPQPWFSBMMCBUDIFT
GPSJJOSBOHF
UPUBM@CBUDI
CBUDI@YT@NOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
'JUUSBJOJOHVTJOHCBUDIEBUB
DPTUWBFGJU
CBUDI@YT
$PNQVUFBWFSBHFMPTT
BWH@DPTU
DPTUO@TBNQMFTCBUDI@TJ[F
%JTQMBZMPHTQFSFQPDITUFQ
JGFQPDIEJTQMBZ@TUFQ
QSJOU
&QPDI E
FQPDI
DPTU\G^GPSNBU
BWH@DPTU
SFUVSOWBF
6. Let us now use the VAE class and train function. We use the VAE for our favorite
MNIST dataset:
-PBE./*45EBUBJOBGPSNBUTVJUFEGPSUFOTPSGMPX
5IFTDSJQUJOQVU@EBUBJTBWBJMBCMFVOEFSUIJT63-
IUUQTSBXHJUIVCVTFSDPOUFOUDPNUFOTPSGMPXUFOTPSGMPXNBTUFSUFOTPSGMPX
FYBNQMFTUVUPSJBMTNOJTUJOQVU@EBUBQZ
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB POF@IPU5SVF
O@TBNQMFTNOJTUUSBJOOVN@FYBNQMFT
OFUXPSL@BSDIJUFDUVSF=
EJDU
O@IJEEFO@SFDPH@TUMBZFSFODPEFSOFVSPOT
O@IJEEFO@SFDPH@OEMBZFSFODPEFSOFVSPOT
O@IJEEFO@HFOFS@TUMBZFSEFDPEFSOFVSPOT
O@IJEEFO@HFOFS@OEMBZFSEFDPEFSOFVSPOT
O@JOQVU./*45EBUBJOQVU
JNHTIBQF
O@[EJNFOTJPOBMJUZPGMBUFOUTQBDF
WBFUSBJO
OFUXPSL@BSDIJUFDUVSFUSBJOJOH@FQPDIT
8. Let us now see if the VAE really reconstructs the input or not. The output shows
[ 422 ]
Generative Models and CapsNet Chapter 11
that digits are indeed reconstructed, and since we have used a 2D latent space,
there is a significant blurring of the images:
Y@TBNQMFNOJTUUFTUOFYU@CBUDI
<>
Y@SFDPOTUSVDUWBFSFDPOTUSVDU
Y@TBNQMF
QMUGJHVSF
GJHTJ[F
GPSJJOSBOHF
QMUTVCQMPU
J
QMUJNTIPX
Y@TBNQMF<J>SFTIBQF
WNJOWNBYDNBQHSBZ
QMUUJUMF
5FTUJOQVU
QMUDPMPSCBS
QMUTVCQMPU
J
QMUJNTIPX
Y@SFDPOTUSVDU<J>SFTIBQF
WNJOWNBYDNBQHSBZ
QMUUJUMF
3FDPOTUSVDUJPO
QMUDPMPSCBS
QMUUJHIU@MBZPVU
[ 423 ]
Generative Models and CapsNet Chapter 11
[ 424 ]
Generative Models and CapsNet Chapter 11
[ 425 ]
Generative Models and CapsNet Chapter 11
#PGZCORNGQH/0+56TGEQPUVTWEVGFEJCTCEVGTU
9. The following are the samples of handwritten digits generated using the trained
VAE:
OYOZ
Y@WBMVFTOQMJOTQBDF
OY
Z@WBMVFTOQMJOTQBDF
OZ
DBOWBTOQFNQUZ
OZOY
GPSJZJJOFOVNFSBUF
Y@WBMVFT
GPSKYJJOFOVNFSBUF
Z@WBMVFT
[@NVOQBSSBZ
<<YJZJ>>WBFCBUDI@TJ[F
Y@NFBOWBFHFOFSBUF
[@NV
DBOWBT<
OYJ
OYJK
K
>Y@NFBO<>SFTIBQF
QMUGJHVSF
GJHTJ[F
9J:JOQNFTIHSJE
Y@WBMVFTZ@WBMVFT
QMUJNTIPX
DBOWBTPSJHJOVQQFSDNBQHSBZ
QMUUJHIU@MBZPVU
[ 426 ]
Generative Models and CapsNet Chapter 11
#TCPIGQH/0+56NKMGEJCTCEVGTUIGPGTCVGFD[CWVQGPEQFGTU
How it works...
VAE learns to reconstruct and at the same time generate new images. The generated images
[ 427 ]
Generative Models and CapsNet Chapter 11
are dependent upon the latent space. The images generated have the same distribution as
the dataset they are trained on.
We can also see the data in the latent space by defining a transform function within the
VariationalAutoencoder class:
EFGUSBOTGPSN
TFMG9
5SBOTGPSNEBUBCZNBQQJOHJUJOUPUIFMBUFOUTQBDF
/PUF5IJTNBQTUPNFBOPGEJTUSJCVUJPOXFDPVMEBMUFSOBUJWFMZ
TBNQMFGSPN(BVTTJBOEJTUSJCVUJPO
SFUVSOTFMGTFTTSVO
TFMG[@NFBOGFFE@EJDU\TFMGY9^
The latent representation of the MNIST dataset using the transform function is as follows:
There's More...
The generated images of the VAE depend on the latent space dimensions. The blurring
reduces the dimension of the latent space is increased. The reconstructed images for 5-d, 10-
[ 428 ]
Generative Models and CapsNet Chapter 11
See also...
The paper by Kingma and Welling is the seminal paper in this field. They go through the
complete building thought process along with its elegant maths. For anyone interested in
VAEs, this is a must read: IUUQTBSYJWPSHQEGQEG .
[ 429 ]
Generative Models and CapsNet Chapter 11
Encoders: IUUQTBSYJWPSHQEGQEG.
The Github link contains another implementation of VAEs, along with the reproduction
of the images from the Kingma and Welling paper: IUUQTHJUIVCDPNIXBMTVLMFF
UFOTPSGMPXNOJTU7"&.
So what is the problem with CNNs? In CNNs each layer understands an image at a
progressive level of granularity. As we discussed in multiple recipes, the first layer will
most likely recognize straight lines or simple curves and edges, while subsequent layers
will start to understand more complex shapes such as rectangles and complex forms such as
human faces.
Now, one critical operation used for CNNs is pooling. Pooling aims to create the positional
invariance and it is generally used after each CNN layer to make any problem
computationally tractable. However, pooling introduces a significant problem because it
forces us to lose all the positional data. This is not good. Think about a face: it consists in
two eyes, a mouth, and a nose, and what is important is that there is a spatial relationship
between these parts (the mouth is below the nose, which is typically below the eyes).
Indeed, Hinton said:
The pooling operation used in convolutional neural networks is a big mistake and the fact
that it works so well is a disaster.
So what is new with Capsule Networks? According to the authors, our brain has modules
[ 430 ]
Generative Models and CapsNet Chapter 11
called capsules, and each capsule specializes in handling particular types of information. In
particular, there are capsules that work well for understanding the concept of position, the
concept of size, the concept of orientation, the concept of deformation, the textures, and so
on and so forth. In addition to that, the authors suggest that our brain has particularly
efficient mechanisms for dynamically routing each piece of information to the capsule,
which is considered best suited for handling a particular type of information.
So, the main difference between CNN and CapsNets is that with a CNN you keep adding
layers for creating a deep network, while with CapsNet you nest a neural layer inside
another. A capsule is a group of neurons and introduces more structure in the net; it
produces a vector to signal the existence of an entity in the image. In particular, Hinton uses
the length of the activity vector to represent the probability that the entity exists, and its
orientation to represent the instantiation parameters. When multiple predictions agree, a
higher-level capsule becomes active. For each possible parent, the capsule produces an
additional prediction vector.
Now a second innovation comes in place: we will use dynamic routing across capsules and
will no longer use the raw idea of pooling. A lower-level capsule prefers to send its output
to the higher-level capsule, and the activity vectors have a big scalar product, with the
prediction coming from the lower-level capsule. The parent with the largest scalar
prediction vector product increases the capsule bond. All the other parents decrease their
bond. In other words, the idea is that if a higher-level capsule agrees with a lower level one,
then it will ask to send more information of that type. If there is no agreement, it will ask to
send less of them. This dynamic routing using the agreement method is superior to the
current mechanisms such as max-pooling and, according to Hinton, routing is ultimately a
way to parse the image. Indeed, Max-pooling is ignoring anything but the largest value,
while dynamic routing selectively propagates information according to the agreement
between lower layers and upper layers.
A third difference is that a new nonlinear activation function has been introduced. Instead
of adding a squashing function to each layer as in CNN, CapsNet adds a squashing
function to a nested set of layers. The nonlinear activation function is represented in the
following figure and it is called the squashing function (equation 1):
5SWCUJKPIHWPEVKQPCUUGGPKP*KPVQP
UUGOKPCNRCRGT
Moreover, Hinton and others show that a discriminatively trained, multi-layer capsule
[ 431 ]
Generative Models and CapsNet Chapter 11
The paper Dynamic Routing Between Capsules shows us simple CapsNet architecture:
#UKORNG%CRU0GVCTEJKVGEVWTG
The architecture is shallow, with only two convolutional layers and one fully connected
layer. Conv1 has 256, 9 e 9 convolution kernels with a stride of 1, and ReLU activation. The
role of this layer is to convert pixel intensities to the activities of local feature detectors that
are then used as input to the primary capsules. PrimaryCapsules is a convolutional capsule
layer with 32 channels; each primary capsule contains eight convolutional units with a 9 e 9
kernel and a stride of 2. In total, PrimaryCapsules has [32, 6, 6] capsule outputs (each output
is an 8D vector) and each capsule in the [6, 6] grid is sharing its weight with each other. The
final layer (DigitCaps) has one 16D capsule per digit class and each one of these capsules
receives input from all the other capsules in the layer below. Routing happens only between
two consecutive capsule layers (such as PrimaryCapsules and DigitCaps).
Getting ready
This recipe is based on the code available at IUUQTHJUIVCDPNEFCBSLP$BQT/FU
5FOTPSGMPX, which in its turn is based on the code located at IUUQTHJUIVCDPN
OBUVSPNJDT$BQT/FU5FOTPSGMPXHJU
How to do it...
Here is how is how we proceed with the recipe:
[ 432 ]
Generative Models and CapsNet Chapter 11
HJUDMPOFIUUQTHJUIVCDPNOBUVSPNJDT$BQT/FU5FOTPSGMPXHJU
DE$BQT/FU5FOTPSGMPX
NLEJSQEBUBNOJTU
XHFUD1EBUBNOJTU==
IUUQZBOOMFDVODPNFYECNOJTU\USBJOJNBHFTJEYVCZUFH[USBJOMBCFMTJ
EYVCZUFH[ULJNBHFTJEYVCZUFH[ULMBCFMTJEYVCZUFH[^
HVO[JQEBUBNOJTUH[
QZUIPONBJOQZ
4. Let's see the code used for defining a capsule. Each capsule takes a 4D tensor as
input and returns a 4D tensor. It is possible to define a capsule as either a fully
connected network (the DigiCaps) or a convolutional one (the Primary Capsules).
Note that Primary is a collection of ConvNets and after them the nonlinear
squashing function is applied. Primary capsules will communicate with DigiCaps
via dynamic routing:
DBQT-BZFSQZ
JNQPSUOVNQZBTOQ
JNQPSUUFOTPSGMPXBTUG
GSPNDPOGJHJNQPSUDGH
FQTJMPOF
DMBTT$BQT-BZFS
PCKFDU
$BQTVMFMBZFS
"SHT
JOQVU"%UFOTPS
OVN@PVUQVUTUIFOVNCFSPGDBQTVMFJOUIJTMBZFS
WFD@MFOJOUFHFSUIFMFOHUIPGUIFPVUQVUWFDUPSPGBDBQTVMF
MBZFS@UZQFTUSJOHPOFPG '$ PS$0/7UIFUZQFPGUIJTMBZFS
GVMMZDPOOFDUFEPSDPOWPMVUJPOGPSUIFGVUVSFFYQBOTJPODBQBCJMJUZ
XJUI@SPVUJOHCPPMFBOUIJTDBQTVMFJTSPVUJOHXJUIUIF
MPXFSMFWFMMBZFSDBQTVMF
3FUVSOT
"%UFOTPS
EFG@@JOJU@@
TFMGOVN@PVUQVUTWFD@MFOXJUI@SPVUJOH5SVF
MBZFS@UZQF '$
TFMGOVN@PVUQVUTOVN@PVUQVUT
[ 433 ]
Generative Models and CapsNet Chapter 11
TFMGWFD@MFOWFD@MFO
TFMGXJUI@SPVUJOHXJUI@SPVUJOH
TFMGMBZFS@UZQFMBZFS@UZQF
EFG@@DBMM@@
TFMGJOQVULFSOFM@TJ[F/POFTUSJEF/POF
5IFQBSBNFUFST LFSOFM@TJ[F BOE TUSJEF XJMMCFVTFEXIJMF MBZFS@UZQF
FRVBM $0/7
JGTFMGMBZFS@UZQF $0/7
TFMGLFSOFM@TJ[FLFSOFM@TJ[F
TFMGTUSJEFTUSJEF
JGOPUTFMGXJUI@SPVUJOH
UIF1SJNBSZ$BQTMBZFSBDPOWPMVUJPOBMMBZFS
JOQVU<CBUDI@TJ[F>
BTTFSUJOQVUHFU@TIBQF
<DGHCBUDI@TJ[F>
DBQTVMFT<>
GPSJJOSBOHF
TFMGWFD@MFO
FBDIDBQTVMFJ<CBUDI@TJ[F>
XJUIUGWBSJBCMF@TDPQF
$POW6OJU@
TUS
J
DBQT@JUGDPOUSJCMBZFSTDPOWE
JOQVUTFMGOVN@PVUQVUT
TFMGLFSOFM@TJ[FTFMGTUSJEF
QBEEJOH7"-*%
DBQT@JUGSFTIBQF
DBQT@JTIBQF
DGHCBUDI@TJ[F
DBQTVMFTBQQFOE
DBQT@J
BTTFSUDBQTVMFT<>HFU@TIBQF
<DGHCBUDI@TJ[F>
<CBUDI@TJ[F>
DBQTVMFTUGDPODBU
DBQTVMFTBYJT
DBQTVMFTTRVBTI
DBQTVMFT
BTTFSUDBQTVMFTHFU@TIBQF
<DGHCBUDI@TJ[F>
SFUVSO
DBQTVMFT
JGTFMGMBZFS@UZQF '$
JGTFMGXJUI@SPVUJOH
UIF%JHJU$BQTMBZFSBGVMMZDPOOFDUFEMBZFS
3FTIBQFUIFJOQVUJOUP<CBUDI@TJ[F>
TFMGJOQVUUGSFTIBQF
JOQVUTIBQF
DGHCBUDI@TJ[F
JOQVUTIBQF<>WBMVF
XJUIUGWBSJBCMF@TDPQF
SPVUJOH
C@*+<OVN@DBQT@MOVN@DBQT@M@QMVT@>
C@*+UGDPOTUBOU
OQ[FSPT
<JOQVUTIBQF<>WBMVF
TFMGOVN@PVUQVUT>EUZQFOQGMPBU
DBQTVMFTSPVUJOH
TFMGJOQVUC@*+
DBQTVMFTUGTRVFF[F
DBQTVMFTBYJT
SFUVSO
DBQTVMFT
[ 434 ]
Generative Models and CapsNet Chapter 11
5. The routing algorithm is described in the paper Dynamic Routing Between Capsules
and the relevant section of the paper has been explained, together with the
definition of Equation 2 and Equation 3. The goal of the routing algorithm is to
pass information from lower layer capsules into higher-level ones and understand
where there is agreement. The agreement is computed by simply using a scalar
product between the current output vj of each capsule j in the layer above, and
[ 435 ]
Generative Models and CapsNet Chapter 11
The following method implements the steps described in Procedure 1 in the preceding
images. Note that the input is a 4D tensor from 1,152 capsules in the layer l. The output
is a Tensor of shape <CBUDI@TJ[FMFOHUI
W@K> representing the
vector output `v_j` of capsule j in the layer l+1:
EFGSPVUJOH
JOQVUC@*+
5IFSPVUJOHBMHPSJUIN
"SHT
JOQVU"5FOTPSXJUI<CBUDI@TJ[FOVN@DBQT@MMFOHUI
V@J>
TIBQFOVN@DBQT@MNFBOJOHUIFOVNCFSPGDBQTVMFJOUIFMBZFSM
3FUVSOT
"5FOTPSPGTIBQF<CBUDI@TJ[FOVN@DBQT@M@QMVT@MFOHUI
W@K>
SFQSFTFOUJOHUIFWFDUPSPVUQVUAW@KAJOUIFMBZFSM
/PUFT
V@JSFQSFTFOUTUIFWFDUPSPVUQVUPGDBQTVMFJJOUIFMBZFSMBOE
W@KUIFWFDUPSPVUQVUPGDBQTVMFKJOUIFMBZFSM
8<OVN@DBQT@KOVN@DBQT@JMFO@V@JMFO@W@K>
8UGHFU@WBSJBCMF
8FJHIU TIBQF
EUZQFUGGMPBU
JOJUJBMJ[FSUGSBOEPN@OPSNBM@JOJUJBMJ[FS
TUEEFWDGHTUEEFW
&RDBMDV@IBU
EPUJMJOHGPSJOQVUBOE8CFGPSFNBUNVM
JOQVU <CBUDI@TJ[F>
8 <CBUDI@TJ[F>
JOQVUUGUJMF
JOQVU<>
8UGUJMF
8<DGHCBUDI@TJ[F>
[ 436 ]
Generative Models and CapsNet Chapter 11
BTTFSUJOQVUHFU@TIBQF
<DGHCBUDI@TJ[F>
JOMBTUEJNT
<>5Y<> <> <CBUDI@TJ[F>
V@IBUUGNBUNVM
8JOQVUUSBOTQPTF@B5SVF
BTTFSUV@IBUHFU@TIBQF
<DGHCBUDI@TJ[F>
MJOFGPSSJUFSBUJPOTEP
GPSS@JUFSJOSBOHF
DGHJUFS@SPVUJOH
XJUIUGWBSJBCMF@TDPQF
JUFS@
TUS
S@JUFS
MJOF
<>
D@*+UGOOTPGUNBY
C@*+EJN
D@*+UGUJMF
D@*+<DGHCBUDI@TJ[F>
BTTFSUD@*+HFU@TIBQF
<DGHCBUDI@TJ[F>
MJOF
XFJHIUJOHV@IBUXJUID@*+FMFNFOUXJTFJOUIFMBTUUXPEJNT
<CBUDI@TJ[F>
T@+UGNVMUJQMZ
D@*+V@IBU
UIFOTVNJOUIFTFDPOEEJNSFTVMUJOHJO<CBUDI@TJ[F>
T@+UGSFEVDF@TVN
T@+BYJTLFFQ@EJNT5SVF
BTTFSUT@+HFU@TIBQF
<DGHCBUDI@TJ[F
MJOF
TRVBTIVTJOH&R
W@+TRVBTI
T@+
BTTFSUW@+HFU@TIBQF
<DGHCBUDI@TJ[F>
MJOF
SFTIBQFUJMFW@KGSPN<CBUDI@TJ[F>UP<CBUDI@TJ[F
>
UIFONBUNVMJOUIFMBTUUPXEJN<>5Y<> <>
SFEVDFNFBOJOUIF
CBUDI@TJ[FEJNSFTVMUJOHJO<>
W@+@UJMFEUGUJMF
W@+<>
V@QSPEVDF@WUGNBUNVM
V@IBUW@+@UJMFEUSBOTQPTF@B5SVF
BTTFSUV@QSPEVDF@WHFU@TIBQF
<DGHCBUDI@TJ[F>
C@*+
UGSFEVDF@TVN
V@QSPEVDF@WBYJTLFFQ@EJNT5SVF
SFUVSO
W@+
6. Now let us review the nonlinear activation squashing function. The input is a 4D
vector with the shape <CBUDI@TJ[FOVN@DBQTWFD@MFO> and the output
is a 4-D tensor with the same shape as a vector but squashed in the third and
fourth dimensions. Given a vector input, the goal is to compute the value
represented in the Equation 1 which is shown as follows:
[ 437 ]
Generative Models and CapsNet Chapter 11
EFGTRVBTI
WFDUPS
4RVBTIJOHGVODUJPODPSSFTQPOEJOHUP&R
"SHT
WFDUPS"%UFOTPSXJUITIBQF<CBUDI@TJ[FOVN@DBQTWFD@MFO>
3FUVSOT
"%UFOTPSXJUIUIFTBNFTIBQFBTWFDUPSCVUTRVBTIFEJOSEBOEUI
EJNFOTJPOT
WFD@TRVBSFE@OPSNUGSFEVDF@TVN
UGTRVBSF
WFDUPSLFFQ@EJNT5SVF
TDBMBS@GBDUPSWFD@TRVBSFE@OPSN
WFD@TRVBSFE@OPSN
UGTRSU
WFD@TRVBSFE@OPSN
FQTJMPO
WFD@TRVBTIFETDBMBS@GBDUPSWFDUPSFMFNFOUXJTF
SFUVSO
WFD@TRVBTIFE
7. During the previous steps, we have defined what a capsule is, the dynamic
routing algorithm among capsules, and the nonlinear squashing function. Now
we can define the proper CapsNet. A loss function is built for training and the
Adam Optimizer is chosen. The method CVJME@BSDI
defines the CapsNet
represented in the following figure:
We use an additional reconstruction loss to encourage the digit capsules to encode the
instantiation parameters of the input digit. During training, we mask out all but the
activity vector of the correct digit capsule.
[ 438 ]
Generative Models and CapsNet Chapter 11
The output of the digit capsule is fed into a decoder consisting of three fully connected
layers that model the pixel intensities as described in Fig. 2. We minimize the sum of
squared differences between the output of the logistic units and the pixel intensities. We
scale down this reconstruction loss by 0.0005 so that it does not dominate the margin loss
during training. The method CVJME@BSDI
implemented as follows is also used for
creating the Decoder:
DBQT/FUQZ
JNQPSUUFOTPSGMPXBTUG
GSPNDPOGJHJNQPSUDGH
GSPNVUJMTJNQPSUHFU@CBUDI@EBUB
GSPNDBQT-BZFSJNQPSU$BQT-BZFS
FQTJMPOF
DMBTT$BQT/FU
PCKFDU
EFG@@JOJU@@
TFMGJT@USBJOJOH5SVF
TFMGHSBQIUG(SBQI
XJUITFMGHSBQIBT@EFGBVMU
JGJT@USBJOJOH
TFMG9TFMGMBCFMTHFU@CBUDI@EBUB
TFMG:UGPOF@IPU
TFMGMBCFMTEFQUIBYJT
EUZQFUGGMPBU
TFMGCVJME@BSDI
TFMGMPTT
TFMG@TVNNBSZ
U@WBSTUGUSBJOBCMF@WBSJBCMFT
TFMGHMPCBM@TUFQUG7BSJBCMF
OBNF HMPCBM@TUFQ
USBJOBCMF'BMTF
[ 439 ]
Generative Models and CapsNet Chapter 11
TFMGPQUJNJ[FSUGUSBJO"EBN0QUJNJ[FS
TFMGUSBJO@PQTFMGPQUJNJ[FSNJOJNJ[F
TFMGUPUBM@MPTT
HMPCBM@TUFQTFMGHMPCBM@TUFQWBS@MJTUU@WBST
FMJGDGHNBTL@XJUI@Z
TFMG9UGQMBDFIPMEFS
UGGMPBU
TIBQF
DGHCBUDI@TJ[F
TFMG:UGQMBDFIPMEFS
UGGMPBUTIBQF
DGHCBUDI@TJ[F
TFMGCVJME@BSDI
FMTF
TFMG9UGQMBDFIPMEFS
UGGMPBU
TIBQF
DGHCBUDI@TJ[F
TFMGCVJME@BSDI
UGMPHHJOHJOGP
4FUUJOHVQUIFNBJOTUSVDUVSF
EFGCVJME@BSDI
TFMG
XJUIUGWBSJBCMF@TDPQF
$POW@MBZFS
$POW<CBUDI@TJ[F>
DPOWUGDPOUSJCMBZFSTDPOWE
TFMG9OVN@PVUQVUT
LFSOFM@TJ[FTUSJEF
QBEEJOH 7"-*%
BTTFSUDPOWHFU@TIBQF
<DGHCBUDI@TJ[F>1SJNBSZ
$BQTVMFTMBZFSSFUVSO<CBUDI@TJ[F>
XJUIUGWBSJBCMF@TDPQF
1SJNBSZ$BQT@MBZFS
QSJNBSZ$BQT$BQT-BZFS
OVN@PVUQVUTWFD@MFO
XJUI@SPVUJOH'BMTFMBZFS@UZQF $0/7
DBQTQSJNBSZ$BQT
DPOWLFSOFM@TJ[FTUSJEF
BTTFSUDBQTHFU@TIBQF
<DGHCBUDI@TJ[F>
%JHJU$BQTMBZFSSFUVSO<CBUDI@TJ[F>
XJUIUGWBSJBCMF@TDPQF
%JHJU$BQT@MBZFS
EJHJU$BQT$BQT-BZFS
OVN@PVUQVUTWFD@MFOXJUI@SPVUJOH5SVF
MBZFS@UZQF '$
TFMGDBQTEJHJU$BQT
DBQT
%FDPEFSTUSVDUVSFJO'JH
%PNBTLJOHIPX
XJUIUGWBSJBCMF@TDPQF
.BTLJOH
BDBMD]]W@D]]UIFOEPTPGUNBY
]]W@D]]
<CBUDI@TJ[F> <CBUDI@TJ[F>
TFMGW@MFOHUIUGTRSU
UGSFEVDF@TVN
UGTRVBSF
TFMGDBQT
BYJTLFFQ@EJNT5SVF
FQTJMPO
TFMGTPGUNBY@WUGOOTPGUNBY
TFMGW@MFOHUIEJN
BTTFSUTFMGTPGUNBY@WHFU@TIBQF
<DGHCBUDI@TJ[F>
CQJDLPVUUIFJOEFYPGNBYTPGUNBYWBMPGUIFDBQT
<CBUDI@TJ[F> <CBUDI@TJ[F>
JOEFY
TFMGBSHNBY@JEYUGUP@JOU
UGBSHNBY
TFMGTPGUNBY@WBYJT
[ 440 ]
Generative Models and CapsNet Chapter 11
BTTFSUTFMGBSHNBY@JEYHFU@TIBQF
<DGHCBUDI@TJ[F>
TFMGBSHNBY@JEYUGSFTIBQF
TFMGBSHNBY@JEYTIBQF
DGHCBUDI@TJ[F
.FUIPE
JGOPUDGHNBTL@XJUI@Z
DJOEFYJOH
*U TOPUFBTZUPVOEFSTUBOEUIFJOEFYJOHQSPDFTTXJUIBSHNBY@JEY
BTXFBSFEJNBOJNBM
NBTLFE@W<>
GPSCBUDI@TJ[FJOSBOHF
DGHCBUDI@TJ[F
WTFMGDBQT<CBUDI@TJ[F><TFMGBSHNBY@JEY<CBUDI@TJ[F>>
NBTLFE@WBQQFOE
UGSFTIBQF
WTIBQF
TFMGNBTLFE@WUGDPODBU
NBTLFE@WBYJT
BTTFSUTFMGNBTLFE@WHFU@TIBQF
<DGHCBUDI@TJ[F>
.FUIPENBTLJOHXJUIUSVFMBCFMEFGBVMUNPEF
FMTF
TFMGNBTLFE@WUGNBUNVM
UGTRVFF[F
TFMGDBQTUGSFTIBQF
TFMG:
USBOTQPTF@B5SVF
TFMGW@MFOHUIUGTRSU
UGSFEVDF@TVN
UGTRVBSF
TFMGDBQTBYJT
LFFQ@EJNT5SVF
FQTJMPO
3FDPOTUSVDUUIF./*45JNBHFTXJUI'$MBZFST
<CBUDI@TJ[F> <CBUDI@TJ[F> <CBUDI@TJ[F>
XJUIUGWBSJBCMF@TDPQF
%FDPEFS
WFDUPS@KUGSFTIBQF
TFMGNBTLFE@WTIBQF
DGHCBUDI@TJ[F
GDUGDPOUSJCMBZFSTGVMMZ@DPOOFDUFE
WFDUPS@KOVN@PVUQVUT
BTTFSUGDHFU@TIBQF
<DGHCBUDI@TJ[F>
GDUGDPOUSJCMBZFSTGVMMZ@DPOOFDUFE
GDOVN@PVUQVUT
BTTFSUGDHFU@TIBQF
<DGHCBUDI@TJ[F>
TFMGEFDPEFEUGDPOUSJCMBZFSTGVMMZ@DPOOFDUFE
GDOVN@PVUQVUT
BDUJWBUJPO@GOUGTJHNPJE
8. Another important part defined in the paper is the margin loss function. This is
explained in the snippet quote from the paper below (Equation 4) and
implemented in the loss(..) method which consists of three losses, the margin loss,
the reconstruction loss and the total loss:
[ 441 ]
Generative Models and CapsNet Chapter 11
EFGMPTT
TFMG
5IFNBSHJOMPTT
<CBUDI@TJ[F>
NBY@MNBY
N@QMVT]]W@D]]?
NBY@MUGTRVBSF
UGNBYJNVN
DGHN@QMVTTFMGW@MFOHUI
NBY@SNBY
]]W@D]]N@NJOVT?
NBY@SUGTRVBSF
UGNBYJNVN
TFMGW@MFOHUIDGHN@NJOVT
BTTFSUNBY@MHFU@TIBQF
<DGHCBUDI@TJ[F>
SFTIBQF<CBUDI@TJ[F> <CBUDI@TJ[F>
NBY@MUGSFTIBQF
NBY@MTIBQF
DGHCBUDI@TJ[F
NBY@SUGSFTIBQF
NBY@STIBQF
DGHCBUDI@TJ[F
DBMD5@D<CBUDI@TJ[F>
5@DTFMG:
<CBUDI@TJ[F>FMFNFOUXJTFNVMUJQMZ
-@D5@DNBY@M
DGHMBNCEB@WBM
5@DNBY@S
TFMGNBSHJO@MPTTUGSFEVDF@NFBO
UGSFEVDF@TVN
-@DBYJT
5IFSFDPOTUSVDUJPOMPTT
PSHJOUGSFTIBQF
TFMG9TIBQF
DGHCBUDI@TJ[F
TRVBSFEUGTRVBSF
TFMGEFDPEFEPSHJO
TFMGSFDPOTUSVDUJPO@FSSUGSFEVDF@NFBO
TRVBSFE
5PUBMMPTT
5IFQBQFSVTFTTVNPGTRVBSFEFSSPSBTSFDPOTUSVDUJPOFSSPSCVUXF
IBWFVTFESFEVDF@NFBOJOA5IFSFDPOTUSVDUJPOMPTTAUPDBMDVMBUF
NFBOTRVBSFEFSSPS*OPSEFSUPLFFQJOMJOFXJUIUIFQBQFSUIF
SFHVMBSJ[BUJPOTDBMFTIPVMECF
TFMGUPUBM@MPTTTFMGNBSHJO@MPTT
DGHSFHVMBSJ[BUJPO@TDBMF
TFMGSFDPOTUSVDUJPO@FSS
4VNNBSZ
EFG@TVNNBSZ
TFMG
[ 442 ]
Generative Models and CapsNet Chapter 11
USBJO@TVNNBSZ<>
USBJO@TVNNBSZBQQFOE
UGTVNNBSZTDBMBS
USBJONBSHJO@MPTT
TFMGNBSHJO@MPTTUSBJO@TVNNBSZBQQFOE
UGTVNNBSZTDBMBS
USBJOSFDPOTUSVDU
JPO@MPTT TFMGSFDPOTUSVDUJPO@FSS
USBJO@TVNNBSZBQQFOE
UGTVNNBSZTDBMBS
USBJOUPUBM@MPTT
TFMGUPUBM@MPTT
SFDPO@JNHUGSFTIBQF
TFMGEFDPEFETIBQF
DGHCBUDI@TJ[F
USBJO@TVNNBSZBQQFOE
UGTVNNBSZJNBHF
SFDPOTUSVDUJPO@JNH SFDPO@JNH
DPSSFDU@QSFEJDUJPOUGFRVBM
UGUP@JOU
TFMGMBCFMTTFMGBSHNBY@JEY
TFMGCBUDI@BDDVSBDZUGSFEVDF@TVN
UGDBTU
DPSSFDU@QSFEJDUJPO
UGGMPBU
TFMGUFTU@BDDUGQMBDFIPMEFS@XJUI@EFGBVMU
UGDPOTUBOU
TIBQF<>
UFTU@TVNNBSZ<>
UFTU@TVNNBSZBQQFOE
UGTVNNBSZTDBMBS
UFTUBDDVSBDZ TFMGUFTU@BDD
TFMGUSBJO@TVNNBSZUGTVNNBSZNFSHF
USBJO@TVNNBSZ
TFMGUFTU@TVNNBSZUGTVNNBSZNFSHF
UFTU@TVNNBSZ
How it works...
CapsNets are very different from state-of-the-art deep learning networks. Instead of adding
more layers and making the network deeper, CapsNets use a shallow network where
capsule layers are nested inside other layers. Each capsule specializes in detecting a specific
entity in an image, and a dynamic routing mechanism is used to send the detected entity to
parents layers. With CNNs you have to consider thousands of images from many different
perspectives in order to recognize an object from different angles. Hinton believes the
redundancies in the layers will allow capsule networks to identify objects from multiple
angles and in different scenarios with less data that is typically used by CNNs. Let's
examine the network as shown by tensorboad:
[ 443 ]
Generative Models and CapsNet Chapter 11
#PGZCORNGQH%CRU0GVCUFGaPGFKPVJGEQFGCPFUJQYPD[VGPUQTDQCTF
The results are impressive as shown in the following diagram taken from the seminal paper.
CapsNet has a low test error (0.25 percent) on a three-layer network previously only
achieved in deeper networks. The baseline is a standard CNN with three convolutional
layers of 256,256128 channels. Each has 5 x 5 kernels and a stride of 1. The last
convolutional layers are followed by two fully connected layers of size 328,192. The last
fully connected layer is connected with dropout to a 10 class softmax layer with cross-
entropy losses:
[ 444 ]
Generative Models and CapsNet Chapter 11
Let's examine the reduction of margin loss, reconstruction loss, and total loss:
[ 445 ]
Generative Models and CapsNet Chapter 11
Let's also examine the increase in accuracy; after 500 iterations it achieves 92 percent and
98.46 percent in 3,500 iterations:
[ 446 ]
Generative Models and CapsNet Chapter 11
>
Iteration Accuracy
500 0.922776442308
1000 0.959735576923
1500 0.971955128205
2000 0.978365384615
2500 0.981770833333
3000 0.983473557692
3500 0.984675480769
'ZCORNGUQHKPETGCUGKPCEEWTCE[HQT%CRU0GV
There's more...
CapsNets work very well on MNIST but there is a lot of research work to be done in terms
of understanding if the same impressive results can be achieved on other datasets such as
CIFAR, or on more general collections of images. If you are interested to know more please
have a look at the following:
[ 447 ]
Generative Models and CapsNet Chapter 11
BMUFSOBUJWFUPUSBEJUJPOBMOFVSBMOFUXPSLT
Geoffrey Hinton talks about what is wrong with convolutional neural nets: IUUQTXXX
ZPVUVCFDPNXBUDIWS5BX'X6WO-&GFBUVSFZPVUVCF
[ 448 ]
Distributed TensorFlow and
12
Cloud Deep Learning
In this chapter, we will discuss using distributed TensorFlow and Cloud deep learning. We
will present a number of recipes for:
Introduction
Each TensorFlow computation is described in terms of a graph. This allows a natural degree
of flexibility in the structure and the placement of operations that can be split across
distributed nodes of computation. The graph can be split into multiple subgraphs that are
assigned to different nodes in a cluster of servers.
Distributed TensorFlow and Cloud Deep Learning Chapter 12
I strongly suggest the reader have a look to the paper Large Scale Distributed Deep
Networks Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V.
Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and
Andrew Y. Ng. NIPS, 2012, IUUQTSFTFBSDIHPPHMFDPNBSDIJWFMBSHF@EFFQ@
OFUXPSLT@OJQTIUNM
One key result of the paper is to prove that it is possible to run distributed stochastic
gradient descent (SDG) where multiple nodes are working in parallel on data-shards and
update independently and asynchronously the gradient by sending updates to a parameter
server. Quoting the abstract of the paper:
This is well explained by the following picture taken from the paper itself:
#PGZCORNGQHFKUVTKDWVGFITCFKGPVFGUEGPVYKVJCRCTCOGVGTUGTXGTCUVCMGPHTQOIUUQTSFTFBSDIHPPHMFDPNBSDIJWFMBSHF@
EFFQ@OFUXPSLT@OJQTIUNM
Another document you should read is the white paper TensorFlow: Large-Scale Machine
Learning on Heterogeneous Distributed Systems Martn Abadi, and others November,
2015, IUUQEPXOMPBEUFOTPSGMPXPSHQBQFSXIJUFQBQFSQEG
[ 450 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
#PGZCORNGQH6GPUQT(NQYITCRJCUVCMGPHTQOIUUQEPXOMPBEUFOTPSGMPXPSHQBQFSXIJUFQBQFSQEG
[ 451 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
A graph can be partitioned across multiple nodes by having local computations and by
transparently adding to the graph remote communication nodes when needed. This is well
explained in the following figure which is still taken from the paper mentioned earlier:
#PGZCORNGQHFKUVTKDWVGF6GPUQT(NQYITCRJEQORWVCVKQPCUVCMGPHTQOIUUQEPXOMPBEUFOTPSGMPXPSHQBQFS
XIJUFQBQFSQEG
Gradient descent and all the major optimizer algorithms can be computed in either a
centralized way (left side of the figure below) or in a distributed way (right side). The latter
involves a master process which talks with multiple workers provisioning both GPUs and
CPUs:
#PGZCORNGQHUKPINGOCEJKPGCPFFKUVTKDWVGFU[UVGOUVTWEVWTGCUVCMGPHTQO#PGZCORNGQHFKUVTKDWVGF6GPUQT(NQYITCRJEQORWVCVKQPCUVCMGPHTQOIUUQ
EPXOMPBEUFOTPSGMPXPSHQBQFSXIJUFQBQFSQEG
[ 452 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
Distributed computations can be either synchronous ( all the workers are updating at the
same time the gradient on sharded data) or asynchronous (the update did not happen at the
same time). The latter typically allows higher scalability with larger graph computations
still working pretty well in terms of convergence to an optimal solution. Again the pictures
are taken from the TensorFlow white paper and I strongly encourage the interested reader
to have a look to this paper if you want to know more:
#PGZCORNGQHU[PEJTQPQWUCPFCU[PEJTQPQWUFCVCRCTCNNGNVTCKPKPI
[ 453 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
Getting ready
The first step is to install a version of TensorFlow that supports GPUs. The official
TensorFlow Installation Instruction is your starting point IUUQTXXXUFOTPSGMPXPSH .
Remember that you need to have an environment supporting GPUs either via CUDA or
CuDNN.
How to do it...
We proceed with the recipe as follows:
JNQPSUTZT
JNQPSUOVNQZBTOQ
JNQPSUUFOTPSGMPXBTUG
GSPNEBUFUJNFJNQPSUEBUFUJNF
2. Get from command line the type of processing unit that you desire to use (either
HQV or DQV)
EFWJDF@OBNFTZTBSHW<>$IPPTFEFWJDFGSPNDNEMJOF0QUJPOTHQVPS
DQV
TIBQF
JOU
TZTBSHW<>JOU
TZTBSHW<>
JGEFWJDF@OBNFHQV
EFWJDF@OBNFHQV
FMTF
EFWJDF@OBNFDQV
3. Execute the matrix multiplication either on GPU or on CPU. The key instruction
is with UGEFWJDF
EFWJDF@OBNF. It creates a new context manager, telling
TensorFlow to perform those actions on either the GPU or the CPU
XJUIUGEFWJDF
EFWJDF@OBNF
SBOEPN@NBUSJYUGSBOEPN@VOJGPSN
TIBQFTIBQFNJOWBMNBYWBM
EPU@PQFSBUJPOUGNBUNVM
SBOEPN@NBUSJYUGUSBOTQPTF
SBOEPN@NBUSJY
TVN@PQFSBUJPOUGSFEVDF@TVN
EPU@PQFSBUJPO
TUBSU5JNFEBUFUJNFOPX
XJUIUG4FTTJPO
DPOGJHUG$POGJH1SPUP
MPH@EFWJDF@QMBDFNFOU5SVFBT
TFTTJPO
SFTVMUTFTTJPOSVO
TVN@PQFSBUJPO
QSJOU
SFTVMU
[ 454 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
4. Print some debug timing just to verify what is the difference between CPU and GPU
QSJOU
4IBQFTIBQF%FWJDFEFWJDF@OBNF
QSJOU
5JNFUBLFOEBUFUJNFOPX
TUBSU5JNF
How it works...
This recipe explains how to assign TensorFlow computations either to CPUs or to GPUs.
The code is pretty simple and it will be used as a basis for the next recipe.
Getting ready
This recipe is inspired by a good blog posting written by Neil Tenenholtz and available
online: IUUQTDMJOEBUTDJDPNCMPHEJTUSJCVUFEUFOTPSGMPX
How to do it...
We proceed with the recipe as follows:
1. Consider this piece of code which runs a matrix multiplication on a single GPU.
TJOHMF(16
CBTFMJOF
JNQPSUUFOTPSGMPXBTUG
QMBDFUIFJOJUJBMEBUBPOUIFDQV
XJUIUGEFWJDF
DQV
JOQVU@EBUBUG7BSJBCMF
<<>
<>
<>
<>>
CUG7BSJBCMF
<<><><>>
DPNQVUFUIFSFTVMUPOUIFUIHQV
XJUIUGEFWJDF
HQV
PVUQVUUGNBUNVM
JOQVU@EBUBC
[ 455 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
DSFBUFBTFTTJPOBOESVO
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
UGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
QSJOUTFTTSVO
PVUQVU
2. Partition the code with in graph replication as in the following snippet between 2
different GPUs. Note that the CPU is acting as the master node distributing the
graph and collecting the final results.
JOHSBQISFQMJDBUJPO
JNQPSUUFOTPSGMPXBTUG
OVN@HQVT
QMBDFUIFJOJUJBMEBUBPOUIFDQV
XJUIUGEFWJDF
DQV
JOQVU@EBUBUG7BSJBCMF
<<>
<>
<>
<>>
CUG7BSJBCMF
<<><><>>
TQMJUUIFEBUBJOUPDIVOLTGPSFBDIHQV
JOQVUTUGTQMJU
JOQVU@EBUBOVN@HQVT
PVUQVUT<>
MPPQPWFSBWBJMBCMFHQVTBOEQBTTJOQVUEBUB
GPSJJOSBOHF
OVN@HQVT
XJUIUGEFWJDF
HQV
TUS
J
PVUQVUTBQQFOE
UGNBUNVM
JOQVUT<J>C
NFSHFUIFSFTVMUTPGUIFEFWJDFT
XJUIUGEFWJDF
DQV
PVUQVUUGDPODBU
PVUQVUTBYJT
DSFBUFBTFTTJPOBOESVO
XJUIUG4FTTJPO
BTTFTT
TFTTSVO
UGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
QSJOUTFTTSVO
PVUQVU
[ 456 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
How it works...
This is a very simple recipe where the graph is split in two parts by the CPU acting as
master and distributed to two GPUs acting as distributed workers. The result of the
computation is collected back to the CPU.
Getting ready
Again, this recipe is inspired by a good blog posting written by Neil Tenenholtz and
available online: IUUQTDMJOEBUTDJDPNCMPHEJTUSJCVUFEUFOTPSGMPX
How to do it...
We proceed with the recipe as follows:
1. Consider this piece of code where we specify the cluster architecture with one
master running on 192.168.1.1:1111 and two workers running on 192.168.1.2:1111
and 192.168.1.3:1111 respectively.
JNQPSUTZT
JNQPSUUFOTPSGMPXBTUG
TQFDJGZUIFDMVTUFS TBSDIJUFDUVSF
DMVTUFSUGUSBJO$MVTUFS4QFD
\ QT < >
XPSLFS <
>
^
[ 457 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
QBSTFDPNNBOEMJOFUPTQFDJGZNBDIJOF
KPC@UZQFTZTBSHW<>KPCUZQFXPSLFSPSQT
UBTL@JEYTZTBSHW<>JOEFYKPCJOUIFXPSLFSPSQTMJTU
BTEFGJOFEJOUIF$MVTUFS4QFD
3. Run the training server where given a cluster, we bless each computational with a
role (either worker or ps), and an id.
DSFBUF5FOTPS'MPX4FSWFS5IJTJTIPXUIFNBDIJOFTDPNNVOJDBUF
TFSWFSUGUSBJO4FSWFS
DMVTUFSKPC@OBNFKPC@UZQFUBTL@JOEFYUBTL@JEY
QBSBNFUFSTFSWFSJTVQEBUFECZSFNPUFDMJFOUT
XJMMOPUQSPDFFECFZPOEUIJTJGTUBUFNFOU
JGKPC@UZQF QT
TFSWFSKPJO
FMTF
XPSLFSTPOMZ
XJUIUGEFWJDF
UGUSBJOSFQMJDB@EFWJDF@TFUUFS
XJUIUG4FTTJPO
TFSWFSUBSHFU
USBJOZPVSNPEFMIFSF
[ 458 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
How it works...
In this recipe, we have seen how to create a cluster with multiple computation nodes. A
node can be either playing the role of a Parameter server or playing the role of a worker.
In both cases the code executed is the same but the execution of the code is different
according to parameters collected from the command line. The parameter server only needs
to wait until the workers send updates. Note that
UGUSBJOSFQMJDB@EFWJDF@TFUUFS
takes the role of automatically assigning
operations to available devices, while UGUSBJO$MVTUFS4QFD
is used for cluster
setup.
There is more...
An example of distributed training for MNIST is available online on IUUQTHJUIVCDPN
JTDIMBHEJTUSJCVUFEUFOTPSGMPXFYBNQMFCMPCNBTUFSFYBNQMFQZ
In addition, note that You can decide to have more than one parameter server for efficiency
reasons. Using parameters the server can provide better network utilization, and it allows to
scale models to more parallel machines. It is possible to allocate more than one parameter
server. The interested reader can have a look to IUUQTXXXUFOTPSGMPXPSHEFQMPZ
EJTUSJCVUFE
Getting ready
This recipe is based on the previous one. So it might be convenient to read them in order.
[ 459 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
How to do it...
We proceed with the recipe as follows:
1. Import a few standard modules and define the TensorFlow cluster where the
computation is run. Then start a server for a specific task
JNQPSUUFOTPSGMPXBTUG
JNQPSUTZT
JNQPSUUJNF
DMVTUFSTQFDJGJDBUJPO
QBSBNFUFS@TFSWFST<QD>
XPSLFST<QD
QD
QD>
DMVTUFSUGUSBJO$MVTUFS4QFD
\QTQBSBNFUFS@TFSWFSTXPSLFSXPSLFST^
JOQVUGMBHT
UGBQQGMBHT%&'*/&@TUSJOH
KPC@OBNF&JUIFS QT PS XPSLFS
UGBQQGMBHT%&'*/&@JOUFHFS
UBTL@JOEFY*OEFYPGUBTLXJUIJOUIF
KPC'-"(4UGBQQGMBHT'-"(4
TUBSUBTFSWFSGPSBTQFDJGJDUBTL
TFSWFSUGUSBJO4FSWFS
DMVTUFS
KPC@OBNF'-"(4KPC@OBNF
UBTL@JOEFY'-"(4UBTL@JOEFY
2. Read MNIST data and define the hyperparameters used for training
DPOGJH
CBUDI@TJ[F
MFBSOJOH@SBUF
USBJOJOH@FQPDIT
MPHT@QBUIUNQNOJTU
MPBENOJTUEBUBTFU
GSPNUFOTPSGMPXFYBNQMFTUVUPSJBMTNOJTUJNQPSUJOQVU@EBUB
NOJTUJOQVU@EBUBSFBE@EBUB@TFUT
./*45@EBUB POF@IPU5SVF
3. Check if your role is Parameter Server or Worker. If worker then define a simple
dense neural network, define an optimizer, and the metric used for evaluating the
classifier (for example accuracy).
JG'-"(4KPC@OBNFQT
TFSWFSKPJO
FMJG'-"(4KPC@OBNFXPSLFS
#FUXFFOHSBQISFQMJDBUJPO
XJUIUGEFWJDF
UGUSBJOSFQMJDB@EFWJDF@TFUUFS
[ 460 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
XPSLFS@EFWJDFKPCXPSLFSUBTLE'-"(4UBTL@JOEFY
DMVTUFSDMVTUFS
DPVOUUIFOVNCFSPGVQEBUFT
HMPCBM@TUFQUGHFU@WBSJBCMF
HMPCBM@TUFQ <>JOJUJBMJ[FS
UGDPOTUBOU@JOJUJBMJ[FS
USBJOBCMF'BMTF
JOQVUJNBHFT
XJUIUGOBNF@TDPQF
JOQVU
/POF CBUDITJ[FDBOCFBOZTJ[F GMBUUFOFENOJTUJNBHF
YUGQMBDFIPMEFS
UGGMPBUTIBQF</POF>OBNFYJOQVU
UBSHFUPVUQVUDMBTTFT
Z@UGQMBDFIPMEFS
UGGMPBUTIBQF</POF>OBNFZJOQVU
NPEFMQBSBNFUFSTXJMMDIBOHFEVSJOHUSBJOJOHTPXFVTFUG7BSJBCMF
UGTFU@SBOEPN@TFFE
XJUIUGOBNF@TDPQF
XFJHIUT
8UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
8UG7BSJBCMF
UGSBOEPN@OPSNBM
<>
CJBT
XJUIUGOBNF@TDPQF
CJBTFT
CUG7BSJBCMF
UG[FSPT
<>
CUG7BSJBCMF
UG[FSPT
<>
JNQMFNFOUNPEFM
XJUIUGOBNF@TDPQF
TPGUNBY
ZJTPVSQSFEJDUJPO
[UGBEE
UGNBUNVM
Y8C
BUGOOTJHNPJE
[
[UGBEE
UGNBUNVM
B8C
ZUGOOTPGUNBY
[
TQFDJGZDPTUGVODUJPO
XJUIUGOBNF@TDPQF
DSPTT@FOUSPQZ
UIJTJTPVSDPTU
DSPTT@FOUSPQZUGSFEVDF@NFBO
UGSFEVDF@TVN
Z@UGMPH
ZSFEVDUJPO@JOEJDFT<>
TQFDJGZPQUJNJ[FS
XJUIUGOBNF@TDPQF
USBJO
PQUJNJ[FSJTBOPQFSBUJPOXIJDIXFDBOFYFDVUFJOBTFTTJPO
HSBE@PQUGUSBJO(SBEJFOU%FTDFOU0QUJNJ[FS
MFBSOJOH@SBUF
USBJO@PQHSBE@PQNJOJNJ[F
DSPTT@FOUSPQZHMPCBM@TUFQHMPCBM@TUFQ
XJUIUGOBNF@TDPQF
"DDVSBDZ
BDDVSBDZ
DPSSFDU@QSFEJDUJPOUGFRVBM
UGBSHNBY
ZUGBSHNBY
Z@
[ 461 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
BDDVSBDZUGSFEVDF@NFBO
UGDBTU
DPSSFDU@QSFEJDUJPOUGGMPBU
DSFBUFBTVNNBSZGPSPVSDPTUBOEBDDVSBDZ
UGTVNNBSZTDBMBS
DPTUDSPTT@FOUSPQZ
UGTVNNBSZTDBMBS
BDDVSBDZBDDVSBDZ
NFSHFBMMTVNNBSJFTJOUPBTJOHMFPQFSBUJPOXIJDIXFDBOFYFDVUFJOB
TFTTJPO
TVNNBSZ@PQUGTVNNBSZNFSHF@BMM
JOJU@PQUGHMPCBM@WBSJBCMFT@JOJUJBMJ[FS
QSJOU
7BSJBCMFTJOJUJBMJ[FE
4. Start a supervisor which acts as a Chief machine for the distributed setting. The
chief is the worker machine which manages all the rest of the cluster. The session
is maintained by the chief and the key instruction is TW
UGUSBJO4VQFSWJTPS
JT@DIJFG
'-"(4UBTL@JOEFY. Also, with
QSFQBSF@PS@XBJU@GPS@TFTTJPO
TFSWFSUBSHFU the supervisor will wait
for the model to be ready for use. Note that each worker will take care of different
batched models and the final model is then available for the chief.
TWUGUSBJO4VQFSWJTPS
JT@DIJFG
'-"(4UBTL@JOEFY
CFHJO@UJNFUJNFUJNF
GSFRVFODZ
XJUITWQSFQBSF@PS@XBJU@GPS@TFTTJPO
TFSWFSUBSHFUBTTFTT
DSFBUFMPHXSJUFSPCKFDU
UIJTXJMMMPHPOFWFSZNBDIJOF
XSJUFSUGTVNNBSZ'JMF8SJUFS
MPHT@QBUIHSBQIUGHFU@EFGBVMU@HSBQI
QFSGPSNUSBJOJOHDZDMFT
TUBSU@UJNFUJNFUJNF
GPSFQPDIJOSBOHF
USBJOJOH@FQPDIT
OVNCFSPGCBUDIFTJOPOFFQPDI
CBUDI@DPVOUJOU
NOJTUUSBJOOVN@FYBNQMFTCBUDI@TJ[F
DPVOU
GPSJJOSBOHF
CBUDI@DPVOU
CBUDI@YCBUDI@ZNOJTUUSBJOOFYU@CBUDI
CBUDI@TJ[F
QFSGPSNUIFPQFSBUJPOTXFEFGJOFEFBSMJFSPOCBUDI
@DPTUTVNNBSZTUFQTFTTSVO
<USBJO@PQDSPTT@FOUSPQZTVNNBSZ@PQHMPCBM@TUFQ>
GFFE@EJDU\YCBUDI@YZ@CBUDI@Z^
XSJUFSBEE@TVNNBSZ
TVNNBSZTUFQ
DPVOU
JGDPVOUGSFRVFODZPSJ
CBUDI@DPVOU
FMBQTFE@UJNFUJNFUJNF
TUBSU@UJNF
TUBSU@UJNFUJNFUJNF
QSJOU
4UFQE
TUFQ
&QPDIE
FQPDI
#BUDIEPGE
J
CBUDI@DPVOU
$PTUGDPTU
[ 462 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
"WH5JNFGNTGMPBU
FMBQTFE@UJNFGSFRVFODZ
DPVOU
QSJOU
5FTU"DDVSBDZGTFTTSVO
BDDVSBDZGFFE@EJDU\Y
NOJTUUFTUJNBHFTZ@NOJTUUFTUMBCFMT^
QSJOU
5PUBM5JNFGTGMPBU
UJNFUJNF
CFHJO@UJNF
QSJOU
'JOBM$PTUGDPTU
TWTUPQ
QSJOU
EPOF
How it works...
This recipe describes an example of distributed MNIST classifier. In this example,
TensorFlow allows us to define a cluster of three machines. One acts as parameter server
and two more machines are used as workers working on separate batches of the training
data.
Getting ready
We will use Docker and will assume that you are familiar with the system. If not, please
make sure that you have a look to IUUQTXXXEPDLFSDPN and install it. What we are
going to do is to build a version of TF Serving.
[ 463 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
How to do it...
Here is how we proceed with the recipe:
EPDLFSCVJMEQVMMU64&3UFOTPSGMPXTFSWJOHEFWFMG%PDLFSGJMFEFWFM
EPDLFSSVOJU64&3UFOTPSGMPXTFSWJOHEFWFM
HJUDMPOFSFDVSTFTVCNPEVMFTIUUQTHJUIVCDPNUFOTPSGMPXTFSWJOH
DETFSWJOHUFOTPSGMPX
DPOGJHVSF
DE
CB[FMUFTUUFOTPSGMPX@TFSWJOH
5. Now let's see an example of saving a model so that the Server can save it. This
step is inspired by an example used for building a MNIST trainer and serving the
model (see IUUQTHJUIVCDPNUFOTPSGMPXTFSWJOHCMPCNBTUFS
UFOTPSGMPX@TFSWJOHFYBNQMFNOJTU@TBWFE@NPEFMQZ ). The first step is to
import the builder as saved_model_builder. Then the bulk of the work is done by
the 4BWFE.PEFM#VJMEFS
which saves a snapshot of the trained model to
reliable storage. Note that here FYQPSU@QBUI is /tmp/mnist_model/
GSPNUFOTPSGMPXQZUIPOTBWFE@NPEFMJNQPSUCVJMEFSBTTBWFE@NPEFM@CVJMEFS
FYQPSU@QBUI@CBTFTZTBSHW<>
FYQPSU@QBUIPTQBUIKPJO
DPNQBUBT@CZUFT
FYQPSU@QBUI@CBTF
DPNQBUBT@CZUFT
TUS
'-"(4NPEFM@WFSTJPO
QSJOU &YQPSUJOHUSBJOFENPEFMUP FYQPSU@QBUI
CVJMEFSTBWFE@NPEFM@CVJMEFS4BWFE.PEFM#VJMEFS
FYQPSU@QBUI
CVJMEFSBEE@NFUB@HSBQI@BOE@WBSJBCMFT
TFTT<UBH@DPOTUBOUT4&37*/(>
TJHOBUVSF@EFG@NBQ\
QSFEJDU@JNBHFT
QSFEJDUJPO@TJHOBUVSF
TJHOBUVSF@DPOTUBOUT%&'"6-5@4&37*/(@4*(/"563&@%&'@,&:
DMBTTJGJDBUJPO@TJHOBUVSF
[ 464 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
^
MFHBDZ@JOJU@PQMFHBDZ@JOJU@PQ
CVJMEFSTBWF
UFOTPSGMPX@NPEFM@TFSWFSQPSUNPEFM@OBNFNOJTU
NPEFM@CBTF@QBUIUNQNOJTU@NPEFM
How it works...
In February 2016 Google released TensorFlow Serving (IUUQTXXXUFOTPSGMPXPSH
TFSWJOH), a high-performance serving system for machine learned models, designed for
production environments. In August 2017, there are over 800 projects within Google using
TensorFlow Serving in production.
There is more...
TensoFlow Serving is a very versatile piece of software and in this recipe, we have just
scratched the surface of the potential uses. If you have interest in knowing more about
advanced functionalities. such as running in large batches, or loading models dynamically,
you should have a look to IUUQTHJUIVCDPNUFOTPSGMPXTFSWJOHCMPCNBTUFS
UFOTPSGMPX@TFSWJOHHEPDTFSWJOH@BEWBODFENE
Getting ready
It would be good to have a look to how GCP works in IUUQTDMPVEHPPHMFDPN . Note
that GCP offers $300 free credit to get started with any GCP product. In addition, there are
free usage limits on some products for eligible customers, during and after the free trial.
(Offer is subject to change, see IUUQTDMPVEHPPHMFDPNGSFF).
[ 465 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
How to do it...
Here is how we proceed with the recipe:
1. Create a new Google Cloud project from the Web console IUUQTQBOUIFPO
HPPHMFDPNDMPVESFTPVSDFNBOBHFS
[ 466 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
2. Enable billing for this project by selecting the related voice on the left bar of the
console. Then enable the Compute Engine and Cloud Machine Learning APIs
for the project:
4. From the console run the following commands for configuring the zone where
the computation will be executed, for downloading the sample code and for
creating the VMs used for running the code. Finally connect to the machine:
HDMPVEDPOGJHTFUDPNQVUF[POFVTFBTUD
HDMPVEDPOGJHTFUQSPKFDU<:063@130+&$5@*%>
HJUDMPOFIUUQTHJUIVCDPN(PPHMF$MPVE1MBUGPSNDMPVENMEJTUNOJTUFYBNQMF
DEDMPVENMEJTUNOJTUFYBNQMF
HDMPVEDPNQVUFJOTUBODFTDSFBUFUFNQMBUFJOTUBODF=
JNBHFQSPKFDUVCVOUVPTDMPVE=
JNBHFGBNJMZVCVOUVMUT=
CPPUEJTLTJ[F(#=
NBDIJOFUZQFOTUBOEBSE
HDMPVEDPNQVUFTTIUFNQMBUFJOTUBODF
[ 467 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
5. Now, after logging into the machine we need to setup the environment by
installing PIP and TensorFlow with these commands.
TVEPBQUHFUVQEBUF
TVEPBQUHFUZVQHSBEF=
TVEPBQUHFUJOTUBMMZQZUIPOQJQQZUIPOEFW
TVEPQJQJOTUBMMUFOTPSGMPX
TVEPQJQJOTUBMMVQHSBEFUFOTPSGMPX
#6$,&5NOJTU3"/%0.3"/%0.
HTVUJMNCDSFHJPOBMMVTFBTUHT\#6$,&5^
TVEPTDSJQUTDSFBUF@SFDPSETQZ
HTVUJMDQUNQEBUBUSBJOUGSFDPSETHT\#6$,&5^EBUB
HTVUJMDQUNQEBUBUFTUUGSFDPSETHT\#6$,&5^EBUB
7. We are now going to create multiple workers (worker-0, worker-1) which are
clones of the initial template-instance machine. We don't want the machines to
delete the disks when they are turned off so that's why we have the first
command.
HDMPVEDPNQVUFJOTUBODFTTFUEJTLBVUPEFMFUFUFNQMBUFJOTUBODF=
EJTLUFNQMBUFJOTUBODFOPBVUPEFMFUF
HDMPVEDPNQVUFJOTUBODFTEFMFUFUFNQMBUFJOTUBODF
HDMPVEDPNQVUFJNBHFTDSFBUFUFNQMBUFJNBHF=
TPVSDFEJTLUFNQMBUFJOTUBODF
HDMPVEDPNQVUFJOTUBODFTDSFBUF=
NBTUFSXPSLFSXPSLFSQT=
JNBHFUFNQMBUFJNBHF=
NBDIJOFUZQFOTUBOEBSE=
TDPQFTEFGBVMUTUPSBHFSX
TDSJQUTTUBSUUSBJOJOHTIHT\#6$,&5^
[ 468 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
How it works...
The demo script moves the code to each VM and starts the distributed computation. The
two workers are running in parallel on the same MNIST data shared on a common storage
bucket. When the computation is concluded the script will print the location of the trained
model.
There is more...
If we don't want to manage TensorFlow than we can decide to use the managed version run
by Google on your behalf. This is the CloudML service described during the next recipe. In
addition if you decide t use GPUs with CloudML then this URL is a good starting point
IUUQTDMPVEHPPHMFDPNNMFOHJOFEPDTVTJOHHQVT
Getting ready
Here we assume that you have already created a Cloud Platform Project, enabling billing
for your project, and enable the Google Compute Engine and the Cloud Machine Learning
APIs. These steps are similar to the ones described in the previous recipes. This recipe is
inspired by the MNIST training code available in IUUQTDMPVEHPPHMFDPNNMFOHJOF
EPDTEJTUSJCVUFEUFOTPSGMPXNOJTUDMPVEEBUBMBC.
[ 469 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
How to do it...
We proceed with running distributed TensorFlow on Google CloudML:
130+&$5@*%
HDMPVEDPOGJHMJTUQSPKFDUGPSNBUWBMVF
DPSFQSPKFDU
#6$,&5\130+&$5@*%^NM
HTVUJMNCDSFHJPOBMMVTDFOUSBMHT\#6$,&5^
TDSJQUTDSFBUF@SFDPSETQZ
HTVUJMDQUNQEBUBUSBJOUGSFDPSETHT\#6$,&5^EBUB
HTVUJMDQUNQEBUBUFTUUGSFDPSETHT\#6$,&5^EBUB
3. Submitting a training job is a very easy: we can simply invoke the training step
with CloudML engine. In this example, the trainer code runs for 1000 iterations in
the region us-central1. The input data is taken from a storage bucket and the
output bucket will be submitted to a different storage bucket.
+0#@/".&KPC@
EBUF
:NE@).4
HDMPVENMFOHJOFKPCTTVCNJUUSBJOJOH\+0#@/".&^=
QBDLBHFQBUIUSBJOFS=
NPEVMFOBNFUSBJOFSUBTL=
TUBHJOHCVDLFUHT\#6$,&5^=
KPCEJSHT\#6$,&5^\+0#@/".&^=
SVOUJNFWFSTJPO=
SFHJPOVTDFOUSBM=
DPOGJHDPOGJHDPOGJHZBNM=
=
EBUB@EJSHT\#6$,&5^EBUB=
PVUQVU@EJSHT\#6$,&5^\+0#@/".&^=
USBJO@TUFQT
4. If you want You can control the training process by accessing the CloudML
console in IUUQTQBOUIFPOHPPHMFDPNNMFOHJOF
5. Once the training is concluded, it is possible to serve the model directly from
CloudML
.0%&-@/".&./*45
HDMPVENMFOHJOFNPEFMTDSFBUFSFHJPOTVTDFOUSBM\.0%&-@/".&^
7&34*0/@/".&W
[ 470 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
03*(*/
HTVUJMMTHT\#6$,&5^\+0#@/".&^FYQPSU4FSWP]UBJM
HDMPVENMFOHJOFWFSTJPOTDSFBUF=
PSJHJO\03*(*/^=
NPEFM\.0%&-@/".&^=
\7&34*0/@/".&^
HDMPVENMFOHJOFWFSTJPOTTFUEFGBVMUNPEFM\.0%&-@/".&^\7&34*0/@/".&^
6. Once the model is served online it is possible to access the server and make a
prediction. The request.json is created by using the script NBLF@SFRVFTUQZ
which reads data from MNIST, performs a one hot encoding and then write the
features with a well-formatted json schema.
HDMPVENMFOHJOFQSFEJDUNPEFM\.0%&-@/".&^KTPOJOTUBODFT
SFRVFTUKTPO
How it works...
CloudML is a convenient solution for using a managed version of TensorFlow run by
Google. Instead of directly taking care of the infrastructure and operations, it is possible to
focus directly on developing machine learning models.
There is more...
One very cool feature of CloudML is the capacity of auto-tuning the hyperparameters
contained in your model by running multiple trials in parallel. This provides you optimized
values for hyperparameters, which maximizes your model's predictive accuracy. If you are
interested in knowing more, then it might be beneficial to have a look at IUUQTDMPVE
HPPHMFDPNNMFOHJOFEPDTIZQFSQBSBNFUFSUVOJOHPWFSWJFX
[ 471 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
Getting ready
As the first step we need to have an Azure account: if you do not already have one, you can
create one for free here: IUUQTB[VSFNJDSPTPGUDPNFOVTTFSWJDFTCBUDIBJ. The
Azure offers $200 credit for 30 days to new users. This recipe will follow the example
provided by Microsoft Azure to run MNIST on two GPUs using TensorFlow distributed,
the relevant codes are available on Github: IUUQTHJUIVCDPN"[VSFCBUDITIJQZBSE
USFFNBTUFSSFDJQFT5FOTPS'MPX%JTUSJCVUFE
How to do it...
We proceed with the recipe as follows:
1. The first step would be to install Azure CLI. The details for installing Azure CLI
on different OS platforms can be obtained from here IUUQTEPDTNJDSPTPGU
DPNFOVTDMJB[VSFJOTUBMMB[VSFDMJWJFXB[VSFDMJMBUFTU.
2. Before creating a cluster you would require to login in Azure, using command B[
MPHJO. It will generate a code and provide you with a website address, where
you will be asked to verify your credentials, once all the steps on the website are
over, you will be asked to close, and your az credentials will be verified.
3. Configure the default location, create and configure a resource group.
B[HSPVQDSFBUFOBNFNZ3FTPVSDF(SPVQMPDBUJPOFBTUVT
B[DPOGJHVSFEFGBVMUTHSPVQNZ3FTPVSDF(SPVQ
B[DPOGJHVSFEFGBVMUTMPDBUJPOFBTUVT
4. Next, we will require creating a storage using <az storage account create>
command, and set the environmental variables depending upon your OS, details
about environment variables and their values are given at IUUQTEPDT
NJDSPTPGUDPNFOVTB[VSFCBUDIBJRVJDLTUBSUDMJ
[ 472 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
XHFU
IUUQTCBUDIBJTBNQMFTCMPCDPSFXJOEPXTOFUTBNQMFTNOJTU@EBUBTFU@PSJHJOB
M[JQTU5"";TF5"";TQSMTW
TSCTJH2D3"[T9*1PFJP9VUL-19*S)+0Q)+MQQ4S*%*%0
NOJTU@EBUBTFU@PSJHJOBM[JQ
VO[JQNOJTU@EBUBTFU@PSJHJOBM[JQ
XHFU
IUUQTSBXHJUIVCVTFSDPOUFOUDPN"[VSF#BUDI"*NBTUFSSFDJQFT5FOTPS'MPX
5FOTPS'MPX(16
%JTUSJCVUFENOJTU@SFQMJDBQZUPLFO"D;[SDQ+(%)$6[T$ZKM8J,7/G#V%ELRXLT;E1S
X"%%0NOJTU@SFQMJDBQZ
7. Next create an Azure File Share, where you upload the downloaded MNIST
dataset and NOJTU@SFQMJDBQZ files.
B[TUPSBHFTIBSFDSFBUFOBNFCBUDIBJTBNQMF
B[TUPSBHFEJSFDUPSZDSFBUFTIBSFOBNFCBUDIBJTBNQMFOBNFNOJTU@EBUBTFU
B[TUPSBHFGJMFVQMPBETIBSFOBNFCBUDIBJTBNQMFTPVSDFULJNBHFT
JEYVCZUFH[QBUINOJTU@EBUBTFU
B[TUPSBHFGJMFVQMPBETIBSFOBNFCBUDIBJTBNQMFTPVSDFULMBCFMT
JEYVCZUFH[QBUINOJTU@EBUBTFU
B[TUPSBHFGJMFVQMPBETIBSFOBNFCBUDIBJTBNQMFTPVSDFUSBJOJNBHFT
JEYVCZUFH[QBUINOJTU@EBUBTFU
B[TUPSBHFGJMFVQMPBETIBSFOBNFCBUDIBJTBNQMFTPVSDFUSBJOMBCFMT
JEYVCZUFH[QBUINOJTU@EBUBTFU
B[TUPSBHFEJSFDUPSZDSFBUFTIBSFOBNFCBUDIBJTBNQMFOBNF
UFOTPSGMPX@TBNQMFT
B[TUPSBHFGJMFVQMPBETIBSFOBNFCBUDIBJTBNQMFTPVSDFNOJTU@SFQMJDBQZ
QBUIUFOTPSGMPX@TBNQMFT
8. Now we create a cluster, for this recipe, the cluster consists of two GPU nodes of
standard_NC6 size, with standard Ubuntu LTS or Ubuntu DVSM. The cluster can
be created using Azure CLI command:
For Linux:
B[CBUDIBJDMVTUFSDSFBUFOODJ6CVOUV%47.T4UBOEBSE@/$NJO
NBYBGTOBNFCBUDIBJTBNQMFBGTNPVOUQBUIFYUFSOBMV64&3L
_TTIJE@STBQVC
[ 473 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
For Windows:
B[CBUDIBJDMVTUFSDSFBUFOODJ6CVOUV%47.T4UBOEBSE@/$NJO
NBYBGTOBNFCBUDIBJTBNQMFBGTNPVOUQBUIFYUFSOBMVVTFS@OBNF Q
QBTTXPSE
\
QSPQFSUJFT\
OPEF$PVOU
UFOTPS'MPX4FUUJOHT\
QBSBNFUFS4FSWFS$PVOU
XPSLFS$PVOU
QZUIPO4DSJQU'JMF1BUI";@#"5$)"*@*/165@4$3*15NOJTU@SFQMJDBQZ
NBTUFS$PNNBOE-JOF"SHTKPC@OBNFXPSLFSOVN@HQVT
QT@IPTUT";@#"5$)"*@14@)0454XPSLFS@IPTUT";@#"5$)"*@803,&3@)0454
UBTL@JOEFY";@#"5$)"*@5"4,@*/%&9EBUB@EJS";@#"5$)"*@*/165@%"5"4&5
PVUQVU@EJS";@#"5$)"*@[email protected]%&-
XPSLFS$PNNBOE-JOF"SHTKPC@OBNFXPSLFSOVN@HQVT
QT@IPTUT";@#"5$)"*@14@)0454XPSLFS@IPTUT";@#"5$)"*@803,&3@)0454
UBTL@JOEFY";@#"5$)"*@5"4,@*/%&9EBUB@EJS";@#"5$)"*@*/165@%"5"4&5
PVUQVU@EJS";@#"5$)"*@[email protected]%&-
QBSBNFUFS4FSWFS$PNNBOE-JOF"SHTKPC@OBNFQTOVN@HQVT
QT@IPTUT";@#"5$)"*@14@)0454XPSLFS@IPTUT";@#"5$)"*@803,&3@)0454
UBTL@JOEFY";@#"5$)"*@5"4,@*/%&9EBUB@EJS";@#"5$)"*@*/165@%"5"4&5
PVUQVU@EJS";@#"5$)"*@[email protected]%&-
^
TUE0VU&SS1BUI1SFGJY";@#"5$)"*@.06/5@3005FYUFSOBM
JOQVU%JSFDUPSJFT<\
JE%"5"4&5
QBUI";@#"5$)"*@.06/5@3005FYUFSOBMNOJTU@EBUBTFU
^\
JE4$3*15
QBUI";@#"5$)"*@.06/5@3005FYUFSOBMUFOTPSGMPX@TBNQMFT
^>
PVUQVU%JSFDUPSJFT<\
JE.0%&-
QBUI1SFGJY";@#"5$)"*@.06/5@3005FYUFSOBM
QBUI4VGGJY.PEFMT
^>
DPOUBJOFS4FUUJOHT\
JNBHF4PVSDF3FHJTUSZ\
JNBHFUFOTPSGMPXUFOTPSGMPXHQV
^
^
^
[ 474 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
B[CBUDIBJKPCDSFBUFOEJTUJCVUFE@UFOTPSGMPXDMVTUFSOBNFODD
KPCKTPO
How it works...
Batch AI manages the resources by itself, you just need to specify the job, the location of
inputs and where to store the outputs. If during the execution of the job you want to see the
results, you can use the command:
B[CBUDIBJKPCTUSFBNGJMFKPCOBNFNZKPCPVUQVUEJSFDUPSZJETUEPVUFSS
OBNFTUEFSSUYU
Once the job is over, you delete both the job and the cluster using B[CBUDIBJKPC
EFMFUF and B[CBUDIBJDMVTUFSEFMFUF commands.
There's more...
Above we learned how to use Microsoft Azure Batch AI for Distributed TensorFlow using
Azure Command Line tool. We can do the same using Jupyter Notebook too. This will
involve setting the Azure Active Directory, and making a new App registration. The details
can be had at this link: IUUQTEPDTNJDSPTPGUDPNFOVTB[VSFB[VSFSFTPVSDF
NBOBHFSSFTPVSDFHSPVQDSFBUFTFSWJDFQSJODJQBMQPSUBM.
Azure BatchAI works with other AI deep learning libraries too, we suggest that you should
go through the BatchAI Github for more details: IUUQTHJUIVCDPN"[VSF#BUDI"*.
[ 475 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
You can see that Amazon AWS provides a host of services, but here we are concerned with
using Amazon AWS for Deep learning.
[ 476 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
The GPUs are available only in P2 instance creation, and are not available by default, to get
this service one has to raise a ticket for increasing resources via AWS support, the support is
available in the top right corner, once you go to support, you will see a button Create case,
choose the button and make following selections:
You can choose any region offering p2.xlarge instance. The new limit value decides the
maximum number of the instances you can use, choose the number carefully because the
service is not free and costs roughly $1 per hour. It takes about 24 hours for AWS to
respond to the request.
Getting ready
Now that you have the AWS account and p2.xlarge instances available to you, you are
ready to create your own instance from EC2 dashboard. The first step is selecting the
machine image, as of now, Amazon offers special machine images preloaded with Deep
learning libraries you can select the Ubuntu or Linux version. Next, choose the GPU to
compute instance type.
[ 477 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
You can review and launch the instance with default parameters, or you can configure the
settings, choose the storage and configure the security group. Configuring the security
group is important, by default SSH security group is set, but if you want to use Jupyter
Notebook, you need to add a custom security group for port 8888, you have the option to
choose the source from which you will log in to the instance, there are three possibilities
Custom, Anywhere or My IP:
Finally, when you launch the instance, you will be asked to create a key pair, which allows
you to login to the particular instance, you will need to create one key-pair and download
the respective QFN file, this you will use for logging later.
How to do it...
1. The first step will be connecting to the instance, you can do it either through ssh
using the command line, or via browser. We use CLI.
[ 478 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
DINPETFDVSJUZLFZQBJSGJMFOBNF QFN
3. Next use the the command below to SSH the instance. The exact address will be
available on the dashboard, when you select connect:
TTIJTFDVSJUZLFZQBJSGJMFOBNF QFNVCVOUV!FD
9999999999999DPNQVUFBNB[POBXTDPN
4. The machine instance that we have chosen already contains all the deep learning
libraries including TensorFlow and so we don't need to install anything:
[ 479 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
5. Each folder contains a readme file which describes how to use the corresponding
library:
How it works...
You can run the codes that you have learned on the instance we have created. Once the
work is over do not forget to exit, and from dashboard stop the instance. More details about
pricing and usage is available here: IUUQTBXTBNB[PODPNEPDVNFOUBUJPOFD
[ 480 ]
Distributed TensorFlow and Cloud Deep Learning Chapter 12
There is more...
A large number of docker images and machine images with preconfigured libraries and
APIs are available at AWS marketplace. To launch the jupyter notebook , at command line
use KVQZUFSOPUFCPPLJQOPCSPXTFS . This will result in an output
as shown in the following:
Copy/paste this URL into your browser when you connect for the first time to login with a
token:
http://0.0.0.0:8888/?token=3156e...
You copy and paste the URL on your browser and you are ready to go.
[ 481 ]
Learning to Learn with AutoML
13
(Meta-Learning)
The success of deep learning has immensely facilitated the work of feature engineering.
Indeed, traditional machine learning depended very much on the selection of the right set
of features, and very frequently, this step was more important that the selection of a
particular learning algorithm. Deep learning has changed this scenario; creating a right
model is still very important but nowadays networks are less sensitive to the selection of a
particular set of feature and are much more able to auto-select the features that really
matter.
Instead, the introduction of deep learning has increased the focus on the selection of the
right neural network architecture. This means that progressively the interest of researchers
is shifting from feature engineering to network engineering. AutoML (Meta Learning) is an
emerging research topic which aims at auto-selecting the most efficient neural network for a
given learning task. In other words, AutoML represents a set of methodologies for learning
how to learn efficiently. Consider for instance the tasks of Machine Translation, or Image
Recognition, or Game playing. Typically, the models are manually designed by a team of
engineers, data scientist, and domain experts. If you consider that a typical 10-layer
network can have ~1010 candidate network, you understand how expensive, error prone,
and ultimately sub-optimal the process can be.
Learning to Learn with AutoML (Meta-Learning) Chapter 13
For instance, a controller recurrent neural network can sample a convolutional network.
The controller can predict many hyper-parameters such as filter height, filter width, stride
height, stride width, and the number of filters for one layer and then can repeat. Every
prediction can be carried out by a softmax classifier and then fed into the next RNN time
step as input. This is well expressed by the following images taken from Neural Architecture
Search with Reinforcement Learning, Barret Zoph, Quoc V. Le, IUUQTBSYJWPSHBCT
:
[ 483 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
Besides that, parallelism can be used for optimizing the parameters of the controller RNN.
Quoc Le & Barret Zoph proposed to adopt a parameter-server scheme where we have a
parameter server of S shards, that store the shared parameters for K controller replicas. Each
controller replica samples m different child architectures that are trained in parallel as
illustrated in the following images, taken from Neural Architecture Search with Reinforcement
Learning, Barret Zoph, Quoc V. Le, IUUQTBSYJWPSHBCT:
[ 484 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
Quoc and Barret applied AutoML techniques for Neural Architecture Search to the Penn
Treebank dataset (IUUQTFOXJLJQFEJBPSHXJLJ5SFFCBOL), a well-known benchmark
for language modeling. Their results improve the manually designed networks currently
considered the state-of-the-art. In particular, they achieve a test set perplexity of 62.4 on the
Penn Treebank, which is 3.6 perplexity better than the previous state-of-the-art model.
Similarly, on the CIFAR-10 dataset (IUUQTXXXDTUPSPOUPFEV_LSJ[DJGBSIUNM),
starting from scratch, the method can design a novel network architecture that rivals the
best human-invented architecture in terms of test set accuracy. The proposed CIFAR-10
model achieves a test error rate of 3.65, which is 0.09 percent better and 1.05x faster than the
previous state-of-the-art model that used a similar architectural scheme.
Meta-learning blocks
In Learning Transferable Architectures for Scalable Image Recognition, Barret Zoph, Vijay
Vasudevan, Jonathon Shlens, Quoc V. Le, 2017 IUUQTBSYJWPSHBCT.
propose to learn an architectural building block on a small dataset that can be transferred to
a large dataset. The authors propose to search for the best convolutional layer (or cell) on
the CIFAR-10 dataset and then apply this learned cell to the ImageNet dataset by stacking
together more copies of this cell, each with their own parameters. Precisely, all
convolutional networks are made of convolutional layers (or cells) with identical structures
but different weights. Searching for the best convolutional architectures is therefore reduced
to searching for the best cell structures, which is faster more likely to generalize to other
problems. Although the cell is not learned directly on ImageNet, an architecture
constructed from the best learned cell achieves, among the published work, state-of-the-art
accuracy of 82.7 percent top-1 and 96.2 percent top-5 on ImageNet. The model is 1.2 percent
better in top-1 accuracy than the best human-invented architectures while having 9 billion
fewer FLOPSa reduction of 28% from the previous state of the art model. What is also
important to notice is that the model learned with RNN+RL (Recurrent Neural Networks +
Reinforcement Learning) is beating the baseline represented by Random Search (RL) as
shown in the figure taken from the paper. In the mean performance of the top-5 and top-25
models identified in RL versus RS, RL is always winning:
[ 485 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Chelsea Finn, Pieter
Abbeel, Sergey Levine, 2017, IUUQTBSYJWPSHBCT proposes a model-
agnostic approach names MAML, compatible with any model trained with gradient descent
and applicable to a variety of different learning problems, including classification,
regression, and reinforcement learning. The goal of meta-learning is to train a model on a
variety of learning tasks, such that it can solve new learning tasks using only a small
number of training samples. The meta-learner aims at finding an initialization that rapidly
adapts to various problems quickly (in a small number of steps) and efficiently (using only
a few examples). A model represented by a parametrized function f with parameters .
When adapting to a new task Ti, the model's parameters become i~. In MAML, the
updated parameter vector i' is computed using one or more gradient descent updates on
task Ti.
[ 486 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
For example, when using one gradient update, i~ = LTi (f) where LTi is the loss
function for the task T and is a meta-learning parameter. The MAML algorithm is
reported in this figure:
[ 487 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
At the end of 2017, the topic of AutoML (or meta-learning) is an active research topic that
aims at auto-selecting the most efficient neural network for a given learning task. The goal
is to learn how to efficiently and automatically design networks that, in turn, can learn
specific tasks or can adapt to new tasks. The main problem is that designing a network
cannot simply be described with a differentiable loss function and therefore traditional
optimization techniques cannot simply be adopted for meta-learning. Therefore a few
solutions have been proposed including the idea of having a controller recurrent network
(RNN) and a reward policy based on reinforcement learning and the idea of having a
model-agnostic meta-Learning. Both the approaches are very promising but there is
certainly still a lot of room for research.
So, if you are interested in a hot topic, this Learning to learn for deep learning is certainly a
space to consider as your next job.
Google proposed the adoption of RNNs for the controller in Using Machine
Learning to Explore Neural Network Architecture; Quoc Le & Barret Zoph,
2017, IUUQTSFTFBSDIHPPHMFCMPHDPNVTJOHNBDIJOFMFBSOJOH
UPFYQMPSFIUNM.
Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le,
IUUQTBSYJWPSHBCT, is a seminal paper proving more details
about Google's approach. However, RNNs are not the only option.
Large-Scale Evolution of Image Classifiers, Esteban Real, Sherry Moore, Andrew
Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, Alex Kurakin,
2017, IUUQTBSYJWPSHBCT, proposed to use evolutionary
computing where genetic intuitive mutation operations are explored to generate
new candidate networks.
[ 488 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
Learning Transferable Architectures for Scalable Image Recognition, Barret Zoph, Vijay
Vasudevan, Jonathon Shlens, Quoc V. Le, IUUQTBSYJWPSHBCT,
proposed the idea of cells learnt on CIFAR and used for improving ImageNet
classification.
Building A.I. That Can Build A.I.: Google and others, fighting for a small pool of
researchers, are looking for automated ways to deal with a shortage of artificial
intelligence experts., The New York Times, IUUQTXXXOZUJNFTDPN
UFDIOPMPHZNBDIJOFMFBSOJOHBSUJGJDJBMJOUFMMJHFODFBJIUNM.
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, Chelsea Finn,
Pieter Abbeel, Sergey Levine, 2017, IUUQTBSYJWPSHBCT.
Learning to Learn by Gradient Descent by Gradient Descent, Marcin Andrychowicz,
Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul,
Brendan Shillingford, Nando de Freitas, IUUQTBSYJWPSHBCT,
shows how the design of an optimization algorithm can be cast as a learning
problem, allowing the algorithm to learn to exploit structure in the problems of
interest in an automatic way. The LSMT learned algorithms outperform hand-
designed competitors on the tasks for which they are trained, and also generalize
well to new tasks with similar structure. The code for this algorithm is available
on GitHub at IUUQTHJUIVCDPNEFFQNJOEMFBSOJOHUPMFBSO.
Siamese Network
Siamese Networks are a special type of neural networks introduced by Yann LeCun and his
colleagues in NIPS 1994 (IUUQXXXXPSMETDJFOUJGJDDPNEPJQEG
4). The basic idea behind them is that like the 'Siamese Twins', the
network consists of two different Neural Networks, both sharing the same architecture and
weights.
[ 489 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
During the training phase the pair-network is presented with a training pair (X1, X2), where
the two inputs are different yet similar, for example, X1 = He is smart, and X2 = He is a wise
man. The two neural networks will produce two different results; the combined network
can be thought of as a scalar energy function measuring the similarity between the training
pair (X1, X2), defined as:
The goal of the Siamese network is that the energy between the training-pair (X1, X2) should
be less than energy between any other imposter pair (X1, ).
During the training phase the network is given as input a training pair, and a label
i th
associated with it, telling it is a genuine pair or an impostor pair: (X1, X2, Y) is the i training
sample. The contrastive Loss function is calculated:
[ 490 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
Besides these, Siamese networks have also been explored for face verification/recognition
(IUUQTHJUIVCDPNIBSWFZTMBTI'BDJBM4JNJMBSJUZXJUI4JBNFTF/FUXPSLTJO
1ZUPSDI). They have been used for Question-Answering (IUUQTBSYJWPSHQEG
WQEG).
[ 491 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
SVOQZ: It contains the basic wrapper to perform training. It uses the Gradient Descent
algorithm to minimize the contrastive loss.
JOGFSFODFQZ: This contains the Siamese Class which defines a 3-layered fully
connected network. The similarity between the output of two networks in the code is
Euclidean. The partial generative loss and the partial imposter Loss are used to then to
calculate the Contrastive loss.
[ 492 ]
Learning to Learn with AutoML (Meta-Learning) Chapter 13
You can see that same (labeled) digits embedded together in the 2D space.
Here, using Tensorflow, deep Siamese LSTM networks are trained to capture
phrase/sentence similarity using character embeddings.
[ 493 ]
TensorFlow Processing Units
14
Google services such as Google Search (RankBrain), Street View, Google Photos, and
Google Translate have one thing in common: they all use Googlebs Tensor Processing Unit,
or TPU, for their computations.
You might be thinking what is a TPU and what is so great about these services? All these
services use state-of-the-art machine learning algorithms in the background, and these
algorithms involve large computations. TPUs help to accelerate the neural network
computations involved. Even AlphaGo, the deep learning program that defeated Lee Sedol
in the game of Go, was powered by TPUs. So let us see what exactly a TPU is.
The first generation of TPUs is, as of now, targeting inference, that is, use of an already
trained model. The training of DNNs, which usually takes more time, was still done on
CPUs and GPUs. The second-generation TPUs announced in a May 2017 blog
post
IUUQTXXXCMPHHPPHMFUPQJDTHPPHMFDMPVEHPPHMFDMPVEPGGFSUQVTNBDIJO
FMFBSOJOH) can both train and infer machine learning models.
Components of TPUs
In all the deep learning models covered in this book, irrespective of the learning paradigm,
three basic calculations were necessary: multiplication, addition, and application of an
activation function.
TensorFlow Processing Units Chapter 14
The first two components are part of matrix multiplication: the weight matrix W needs to be
multiplied to input matrix X, generally expressed as WTX; matrix multiplication is
computationally expensive on a CPU, and though a GPU parallelizes the operation, still
there is scope for improvement.
The TPU has a 65,536 8-bit integer matrix multiplier unit (MXU) that gives a peak
throughput of 92 TOPS. The major difference between GPU and TPU multiplication is that
GPUs contain floating point multipliers, while TPUs contain 8-bit integer multipliers. TPUs
also contain a Unified Buffer (UB), 24 MB of SRAM that works as registers, and an
Activation Unit (AU), which contains hardwired activation functions.
Each TPU also has an off-chip 8-GiB DRAM pool, called weight memory. It has a four-stage
pipeline and executes CISC instructions. TPUs, as of now, consist of six neural networks:
two MLPs, two CNNs, and two LSTMs.
The TPU is programmed with the help of high-level instructions; some of the instructions
used to program TPUs are the following:
Google has created an API stack to facilitate TPU programming; it translates the API calls
from Tensorflow graphs into TPU instructions.
[ 495 ]
TensorFlow Processing Units Chapter 14
Advantages of TPUs
The first and foremost advantage that TPUs offer over GPUs and CPUs is performance.
Google compared the TPU performance to a server-class Intel Haswell CPU and a NVIDIA
K80 GPU running the benchmark code representative of 95% of the inference workload; it
found that the TPU is 15-30 times faster than both NVIDIA's GPU and Intel's CPU.
The second important parameter is the power consumption. It is important to reduce power
consumption since it has a twofold energy benefit: not only does it reduce the amount of
power consumed, it also saves power by reducing the cooling cost to dissipate the heat
generated while processing. The TPU/CPU per watt performance shows a 30-80 times
improvement over other CPU and GPU configurations.
Another advantage of TPUs is their minimal and deterministic design since they have to
execute only one task at a time.
As compared to CPUs and GPUs, the single-threaded TPU has none of the sophisticated
microarchitectural features that consume transistors and energy to improve the average
case but not the 99th-percentile case: no caches, branch prediction, out-of-order execution,
multiprocessing, speculative prefetching, address coalescing, multithreading, context
switching, and so forth. Minimalism is a virtue of domain-specific processors.
Accessing TPUs
Google has decided not to sell TPUs directly to others; instead, the TPUs will be made
available via the Google cloud platform: Cloud TPU Alpha (IUUQTDMPVEHPPHMFDPN
UQV). Cloud TPU Alpha will offer up to 180 teraflops of computing performance and 64 GB
of ultra-high bandwidth memory. Users will be able to connect to these Cloud TPUs from
custom virtual machines.
[ 496 ]
TensorFlow Processing Units Chapter 14
Google has also decided to make a cluster of 1,000 cloud TPUs available free of charge to
machine learning researchers around the world, with the aim of accelerating the pace of
open machine learning research. Access to it will be given to selected individuals for limited
amounts of computing time; individuals can sign up using this link: IUUQTTFSWJDFT
HPPHMFDPNGCGPSNTUQVTJHOVQ. According to the Google Blog:
"Since the main goal of the TensorFlow Research Cloud is to benefit the open machine
learning research community as a whole, successful applicants will be expected to do the
following:
Share their TFRC-supported research with the world through peer-reviewed publications,
open-source code, blog posts, or other open media
Share concrete, constructive feedback with Google to help us improve the TFRC program
and the underlying Cloud TPU platform over time.
Imagine a future in which ML acceleration is abundant and develop new kinds of machine
learning models in anticipation of that future"
Resources on TPUs
Norman P. Jouppi and others, In-datacenter performance analysis of a tensor
processing unit, arXiv preprint arXiv:1704.04760 (2017). In this paper, the authors
compare the TPU with a server-class Intel Haswell CPU and an NVIDIA k80
GPU. The paper benchmarks TPUs performance as compared to CPUs and K80
GPU.
This Google Blog explains the TPU and its working in simple terms at: IUUQT
DMPVEHPPHMFDPNCMPHCJHEBUBBOJOEFQUIMPPLBUHPPHMFT
GJSTUUFOTPSQSPDFTTJOHVOJUUQV
[ 497 ]
Index
Overcomplete
A Undercomplete
accelerated linear algebra (XLA) AutoML (Meta Learning)
reference , about
used, for enhancing computational performance blocks
, novel tasks
activation functions with recurrent networks
hyperbolic tangent activation function with reinforcement learning
Linear activation function Azure BatchAI
Rectified linear units (ReLU) reference
reference Azure CLI
Sigmoidal activation function reference
Softmax activation function
threshold activation function: B
using
backpropagation (BPN) algorithm
activity of the neuron
about
Amazon AWS
gradients, calculating ,
Distributed TensorFlow, running ,
reference
reference
starting with
Anaconda
backward pass ,
reference
batch gradient descent
Android Studio
Bayesian optimization
installation link
Bazel
Android
about
examples ,
installation link
reference
reference
TensorFlow mobile, installing
Boston house prices
AoT (Ahead of Time)
predicting ,
AoT Compilation
Boston housing price dataset
application-specific integrated circuit (ASIC)
reference
Artificial Neural Networks (ANNs)
Atari game
playing, with deep Q network (DQN) , ,
C
Central Processing Unit (CPU)
autoassociators versus Graphical Processing Unit (GPU)
autoencoders CIFAR-10 classification
about ConvNet, creating
Cloud TPU Alpha
reference D
CNN data flow graph
about using
average pooling data preprocessing
in TensorFlow data
local receptive fields feeding
mathematical example filename queue
max-pooling operator reading
pooling layers reading, from files
shared weights and bias training/Learning
summarizing DBN
components, deep learning used, for emotion detection , ,
CIFAR10 DCGANs
dataset MNIST images, forging with
ImageNET decoder
MNIST Deep Belief Networks (DBN)
WORDNET deep learning
YouTube-8M TensorFlow
compression Deep neural networks (DNNs) ,
about deep Q network (DQN)
reference laying Atari game , ,
computational graph used, for playing Atari game
connectionist models deep-siamese-text-similarity example
constants reference
about DeepDream network
working with , , creating ,
Contrastive Divergence (CD) reference
ConvNet denoising autoencoder
creating, for CIFAR-10 classification , about
creating, for handwritten MNIST numbers working
classification , , dependent variable
creating, for Sentiment Analysis Diabolo networks
used, for generating music Distributed TensorFlow MNIST classifier
convolution reference
Convolutional Autoencoders (CAE) training ,
about Distributed TensorFlow
working multiple GPUs and one CPU
Convolutional Neural Networks (CNN) , , multiple servers ,
reference
CPU/GPU devices running, on Amazon AWS ,
invoking running, on Google Cloud (GCP) with Compute
CSV files Engine ,
reading from running, on Google CloudML
running, on Microsoft Azure ,
[ 499 ]
DNN-based problems gradient descent algorithms
Python packages gradient descent optimizer
Docker reference links
reference using, in TensorFlow
Dockerfile.devel greedy wise training
download link
H
E handwritten MNIST numbers
eigen decomposition or singular value classification, by creating ConvNet , ,
decomposition (SVD)
elbow method hidden layer
embeddings hyperbolic tangent activation function ,
Estimators Hyperopt
reference reference
execution of graph hyperparameters
Exploration-Exploitation trade-off optimization, reference
tuning ,
F
feedforward networks I
filter image classification
forward pass , Inception, using
fully connected layers ResNet, using
VGGNet, using
G Xception, using
GANs image recognition
MNIST images, forging with , about
Gated Recurrent Units (GRUs) reference
Generative Adversarial Networks (GANs) ImageNet for the Large Scale Visual Recognition
about , Challenge (ILSVRC)
applications ImageNET
reference reference
gesture recognition images
Google Cloud (GCP) questions, answering , ,
Distributed TensorFlow, running with Compute Inception
Engine , image classification
reference used, for image classification
Google CloudML InceptionV3 Net
Distributed TensorFlow, running used, for transfer learning ,
feature independent variable
reference , integrated development environment (IDE)
Google research blog iPhone
reference TensorFlow mobile, installing ,
GPUs
working with
[ 500 ]
using, in MNIST classifier
J MNIST classifier
JIT (Just In Time) MLP, using
JIT Compilation MNIST dataset
Logistic regression, performing ,
K reference , ,
k-means clustering MNIST images
about firging, with DCGANs
reference forging, with simple GANs ,
working , MNIST
Keras reference
reference mobile devices
using TensorFlow graph, optimizing ,
Kohonen networks TensorFlow graph, profiling ,
Kullback-Leiber (KL) TensorFlow graph, transforming ,
model
L defining
L1 regularization evaluating
L2 Regularization MovieLens
learning with a critic reference
Linear activation function , multi-layered perceptron
Linear regression multiple linear regression
about performing, on Boston house price estimation ,
reference
Logistic regression
about N
performing, on MNIST dataset , natural language processing (NLP) ,
reference neural network agent
Long Short Term Memory (LSTM) building, for playing Pac-Man ,
loss functions NSynth
selecting , used, for generating music
M O
macOS object localization
TensorFlow mobile, installing , about
matrix manipulation reference
performing, TensorFlow used , OpenAI Gym
McCulloch Pitts (MCP) about ,
reference reference ,
mean square error (MSE) , using ,
Microsoft Azure optical character recognition
Distributed TensorFlow, running ,
MLP
used, for function approximation ,
[ 501 ]
reference
P reader
padding recommender system
Peephole LSTM RBM, using ,
Perceptron Rectified linear units (ReLU)
placeholders Recurrent Neural Networks (RNNs) , ,
about regression
working with , about
policy gradients L2 Regularization
AlphaGo Zero Linear regression
used, to play Pong game , , Logistic regression
Pong game regularization
playing, policy gradients used , , regularization
pre-built Deep Learning models reinforcement learning (RL) , ,
recycling, for extracting features ReLU activation function
preloaded data ResNet
pretrained VGG16 net used, for image classification ,
used, for transfer learning , Restricted Boltzmann Machine (RBM)
principal component analysis (PCA) about
about using ,
using , RFHO
principal components reference
Python packages
for DNN-based problems S
H5fs
self-organized maps (SOM)
Keras
about
Matplolib
using , ,
Numpy
Sentiment Analysis
OS
ConvNet, creating ,
Pandas
seq2seq RNN
PythonMagick
about
seaborn
used, for neural machine translation
TFlearn
shallow neural networks
PythonMagick
Siamese Network
reference
about
applications
Q MNIST, working example
Q learning reference
used, for balancing Cart-Pole wiki , , Sigmoid activation function ,
, simple linear regression
web links performing, on Boston house price estimation ,
,
R single layer perceptron
random tensors about
RBM using
[ 502 ]
Softmax activation function , features
sparse autoencoder for deep learning
about gradient descent optimizer, using
working , Hello world program ,
speech recognition installation, weblinks
Stacked Autoencoder (Deep Autoencoders) installing ,
about , matrix manipulations, performing ,
working , program structure ,
Stacked RBMs reference , ,
sum of squared error (SSE) distance working with
supervised learning Text-to-Speech (TTS) systems
TF Classify
T TF Detect
temporal credit assignment problem TF Speech
tensor processing units (TPUs) TF Stylize
TensorFlow 1.x tf_upgrade.py file
migrating, to TensorFlow 0.x limitations
TensorFlow computation Threshold activation function ,
TensorFlow graph transfer learning
optimizing, for mobile devices , InceptionV3 Net, using ,
profiling, for mobile devices , pretrained VGG16 net, using ,
transforming, for mobile devices , reference
TensorFlow mobile translation
installing, for Android transposed convolution layers
installing, for iPhone ,
installing, for macOS , U
TensorFlow Processing Units (TPU) components unsupervised learning
matrix multiplier unit (MXU)
Unified Buffer (UB) V
Activation Unit (AU) vanilla autoencoder
SRAM about
TensorFlow Processing Units (TPU) using ,
about variables
accessing about
advantages saving
components working with
reference variations, gradient descent
resources, URL mini-batch gradient descent
TensorFlow Serving Stochastic gradient descent
and Docker, working with , vanilla gradient descent
TensorFlow vectors
advanced functionalities sequence, operating on
and cloud VGG pre-built network
data types, reference inspecting , ,
examples ,
[ 503 ]
VGG16 Winner take all units (WTU)
used, for image classification WORDNET
VGG19 reference
used, for image classification
used, for style transfer for image repainting , X
, , Xception
VGGNet image classification
used, for image classification used, for image classification
videos Xcode
classifying, with pre-trained nets reference
W Y
WaveNet YouTube-8M
used, for generating music , reference
web cloudshell
URL Z
weights
zeroth layer