Lecture 22

Lecture22
April 7, 2021
1 Lecture 22
1.1 ID 5059
Tom Kelsey - April 2020
Chapter 11 – Training Deep Neural Networks
This notebook contains all the sample code and solutions to the exercises in chapter 11.
Run in Google Colab
2 Setup
First, let’s import a few common modules, ensure MatplotLib plots figures inline and prepare a
function to save the figures. We also check that Python 3.5 or later is installed (although Python
2.x may work, it is deprecated so we strongly recommend you use Python 3 instead), as well as
Scikit-Learn �0.20 and TensorFlow �2.0.
[1]: # Python �3.5 is required
import sys
assert sys.version_info >= (3, 5)
# Scikit-Learn �0.20 is required

import sklearn
assert sklearn.__version__ >= "0.20"
try:
# %tensorflow_version only exists in Colab.
%tensorflow_version 2.x
except Exception:
pass
# TensorFlow �2.0 is required

import tensorflow as tf
from tensorflow import keras
assert tf.__version__ >= "2.0"
%load_ext tensorboard
1
# Common imports
import numpy as np
import os
# to make this notebook's output stable across runs

np.random.seed(42)
# To plot pretty figures

%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)
# Where to save the figures

PROJECT_ROOT_DIR = "."
CHAPTER_ID = "deep"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)
def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):

path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
print("Saving figure", fig_id)
if tight_layout:
plt.tight_layout()
plt.savefig(path, format=fig_extension, dpi=resolution)
INFO:tensorflow:Enabling eager execution

INFO:tensorflow:Enabling v2 tensorshape
INFO:tensorflow:Enabling resource variables
INFO:tensorflow:Enabling tensor equality
INFO:tensorflow:Enabling control flow v2
3 Vanishing/Exploding Gradients Problem

[2]: def logit(z):
return 1 / (1 + np.exp(-z))
def derivative(f, z, eps=0.000001):

return (f(z + eps) - f(z - eps))/(2 * eps)
z = np.linspace(-5, 5, 200)
plt.figure(figsize=(11,4))
plt.subplot(121)
2
plt.plot(z, logit(z), "g--", linewidth=2, label="Sigmoid")
plt.grid(True)
plt.legend(loc="center right", fontsize=14)
plt.title("Activation function", fontsize=14)
plt.axis([-5, 5, -1.2, 1.2])
plt.subplot(122)
plt.plot(z, derivative(logit, z), "r--", linewidth=2, label="Sigmoid")
plt.grid(True)
plt.title("Derivative", fontsize=14)
plt.axis([-5, 5, -0.2, 1.2])
save_fig("activation_function_plot")
plt.show()
Saving figure activation_function_plot
• The derivative values are close to zero for large inputs to the sigmoid activation function
– the inputs are weighted sums from potentially many nodes, so can have large absolute
value
• For a shallow (i.e. few layers) network this is not a big problem
• For deep networks the gradients can be so small that learning effectively stops
• For back propagation the gradients from each layer are multiplied
– as a consequence of the chain rule
• Multiplying small numbers by small numbers soon gives very small numbers
• This is the vanishing gradient problem
• We consider three common approaches to resolving this:
– make sure that the initial weights have a suitable size
– change the activation function(s)
– apply batch normalisation to ensure that the weighted-sum inputs are small…
– …so that the gradients are always relatively large
3
[3]: z = np.linspace(-5, 5, 200)
plt.plot([-5, 5], [0, 0], 'k-')

plt.plot([-5, 5], [1, 1], 'k--')
plt.plot([0, 0], [-0.2, 1.2], 'k-')
plt.plot([-5, 5], [-3/4, 7/4], 'g--')
plt.plot(z, logit(z), "b-", linewidth=2)
props = dict(facecolor='black', shrink=0.1)
plt.annotate('Saturating', xytext=(3.5, 0.7), xy=(5, 1), arrowprops=props,␣
,→fontsize=14, ha="center")
plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props,␣

plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props,␣

plt.grid(True)
plt.title("Sigmoid activation function", fontsize=14)
plt.axis([-5, 5, -0.2, 1.2])
save_fig("sigmoid_saturation_plot")
plt.show()
Saving figure sigmoid_saturation_plot
4
3.1 Xavier and He Initialization
• Xavier initialization works better for layers with sigmoid/tanh activation
• The idea is to select weights from a Gaussian random distribution
– instead of uniform random from [−1, 1]
• He initialization works better for layers with ReLu activation
– small modification of the Gaussian approach
• There are many other initialisation options
• Which one (if any) is best depends on the data and the NN design
[4]: [name for name in dir(keras.initializers) if not name.startswith("_")]
[4]: ['Constant',
'GlorotNormal',
'GlorotUniform',
'HeNormal',
'HeUniform',
'Identity',
'Initializer',
'LecunNormal',
'LecunUniform',
'Ones',
'Orthogonal',
'RandomNormal',
'RandomUniform',
'TruncatedNormal',
'VarianceScaling',
'Zeros',
'constant',
'deserialize',
'get',
'glorot_normal',
'glorot_uniform',
'he_normal',
'he_uniform',
'identity',
'lecun_normal',
'lecun_uniform',
'ones',
'orthogonal',
'random_normal',
'random_uniform',
'serialize',
'truncated_normal',
'variance_scaling',
'zeros']
[5]: keras.layers.Dense(10, activation="relu", kernel_initializer="he_normal")
5
[5]: <tensorflow.python.keras.layers.core.Dense at 0x1896b8f70>
[6]: init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg',

distribution='uniform')
keras.layers.Dense(10, activation="relu", kernel_initializer=init)
[6]: <tensorflow.python.keras.layers.core.Dense at 0x105868640>
3.2 Nonsaturating Activation Functions

• Saturating means “restricted to a small range”
3.2.1 Leaky ReLU
[7]: def leaky_relu(z, alpha=0.01):

return np.maximum(alpha*z, z)
[8]: plt.plot(z, leaky_relu(z, 0.05), "b-", linewidth=2)

plt.plot([-5, 5], [0, 0], 'k-')
plt.plot([0, 0], [-0.5, 4.2], 'k-')
plt.grid(True)
props = dict(facecolor='black', shrink=0.1)
plt.annotate('Leak', xytext=(-3.5, 0.5), xy=(-5, -0.2), arrowprops=props,␣
plt.title("Leaky ReLU activation function", fontsize=14)

plt.axis([-5, 5, -0.5, 4.2])
save_fig("leaky_relu_plot")
plt.show()
Saving figure leaky_relu_plot
6
[9]: [m for m in dir(keras.activations) if not m.startswith("_")]
[9]: ['deserialize',
'elu',
'exponential',
'gelu',
'get',
'hard_sigmoid',
'linear',
'relu',
'selu',
'serialize',
'sigmoid',
'softmax',
'softplus',
'softsign',
'swish',
'tanh']
[10]: [m for m in dir(keras.layers) if "relu" in m.lower()]
[10]: ['LeakyReLU', 'PReLU', 'ReLU', 'ThresholdedReLU']
Let’s train a neural network on Fashion MNIST using the Leaky ReLU:
7
[11]: (X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.
,→load_data()
X_train_full = X_train_full / 255.0

X_test = X_test / 255.0
X_valid, X_train = X_train_full[:5000], X_train_full[5000:]
y_valid, y_train = y_train_full[:5000], y_train_full[5000:]
[12]: tf.random.set_seed(42)
np.random.seed(42)
model = keras.models.Sequential([
keras.layers.Flatten(input_shape=[28, 28]),
keras.layers.Dense(300, kernel_initializer="he_normal"),
keras.layers.LeakyReLU(),
keras.layers.LeakyReLU(),
keras.layers.Dense(10, activation="softmax")
])
[13]: model.compile(loss="sparse_categorical_crossentropy",
optimizer=keras.optimizers.SGD(learning_rate=1e-3),
metrics=["accuracy"])
[14]: %%time
history = model.fit(X_train, y_train, epochs=10,
validation_data=(X_valid, y_valid))
Epoch 1/10
1719/1719 [==============================] - 2s 1ms/step - loss: 1.6314 -
accuracy: 0.5054 - val_loss: 0.8886 - val_accuracy: 0.7160
Epoch 2/10
1719/1719 [==============================] - 2s 968us/step - loss: 0.8416 -
Epoch 3/10
1719/1719 [==============================] - 2s 959us/step - loss: 0.7053 -
Epoch 4/10
1719/1719 [==============================] - 2s 955us/step - loss: 0.6325 -
Epoch 5/10
1719/1719 [==============================] - 2s 961us/step - loss: 0.5992 -
Epoch 6/10
1719/1719 [==============================] - 2s 959us/step - loss: 0.5624 -
Epoch 7/10
1719/1719 [==============================] - 2s 965us/step - loss: 0.5379 -
8
Epoch 8/10
1719/1719 [==============================] - 2s 959us/step - loss: 0.5152 -
Epoch 9/10
1719/1719 [==============================] - 2s 961us/step - loss: 0.5100 -
Epoch 10/10
1719/1719 [==============================] - 2s 965us/step - loss: 0.4918 -
CPU times: user 40.9 s, sys: 9.87 s, total: 50.8 s
Wall time: 17.5 s
Now let’s try PReLU:
np.random.seed(42)
keras.layers.PReLU(),
keras.layers.PReLU(),
])
[18]: %%time
Epoch 1/10
1719/1719 [==============================] - 2s 1ms/step - loss: 1.6969 -
Epoch 2/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.8706 -
Epoch 3/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.7211 -
Epoch 4/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.6447 -
Epoch 5/10
9
1719/1719 [==============================] - 2s 1ms/step - loss: 0.6077 -
Epoch 6/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5692 -
Epoch 7/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5427 -
Epoch 8/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5193 -
Epoch 9/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5128 -
Epoch 10/10
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4940 -
Wall time: 19.6 s
3.2.2 ELU
[19]: def elu(z, alpha=1):
return np.where(z < 0, alpha * (np.exp(z) - 1), z)
[20]: plt.plot(z, elu(z), "b-", linewidth=2)

plt.plot([-5, 5], [0, 0], 'k-')
plt.plot([-5, 5], [-1, -1], 'k--')
plt.plot([0, 0], [-2.2, 3.2], 'k-')
plt.grid(True)
plt.title(r"ELU activation function ($\alpha=1$)", fontsize=14)
plt.axis([-5, 5, -2.2, 3.2])
save_fig("elu_plot")
plt.show()
Saving figure elu_plot
10
Implementing ELU in TensorFlow is trivial, just specify the activation function when building each
layer:
[21]: keras.layers.Dense(10, activation="elu")
[21]: <tensorflow.python.keras.layers.core.Dense at 0x10583cb80>
3.2.3 SELU
This activation function was proposed in this great paper by Günter Klambauer, Thomas Un-
terthiner and Andreas Mayr, published in June 2017. During training, a neural network composed
exclusively of a stack of dense layers using the SELU activation function and LeCun initialization
will self-normalize: the output of each layer will tend to preserve the same mean and variance
during training, which solves the vanishing/exploding gradients problem. As a result, this activa-
tion function outperforms the other activation functions very significantly for such neural nets, so
you should really try it out. Unfortunately, the self-normalizing property of the SELU activation
function is easily broken: you cannot use �1 or �2 regularization, regular dropout, max-norm, skip
connections or other non-sequential topologies (so recurrent neural networks won’t self-normalize).
However, in practice it works quite well with sequential CNNs. If you break self-normalization,
SELU will not necessarily outperform other activation functions.
[22]: from scipy.special import erfc
# alpha and scale to self normalize with mean 0 and standard deviation 1
11
# (see equation 14 in the paper):
alpha_0_1 = -np.sqrt(2 / np.pi) / (erfc(1/np.sqrt(2)) * np.exp(1/2) - 1)
scale_0_1 = (1 - erfc(1 / np.sqrt(2)) * np.sqrt(np.e)) * np.sqrt(2 * np.pi) *␣
,→(2 * erfc(np.sqrt(2))*np.e**2 + np.pi*erfc(1/np.sqrt(2))**2*np.e - 2*(2+np.
,→pi)*erfc(1/np.sqrt(2))*np.sqrt(np.e)+np.pi+2)**(-1/2)
[23]: def selu(z, scale=scale_0_1, alpha=alpha_0_1):

return scale * elu(z, alpha)
[24]: plt.plot(z, selu(z), "b-", linewidth=2)

plt.plot([-5, 5], [0, 0], 'k-')
plt.plot([-5, 5], [-1.758, -1.758], 'k--')
plt.plot([0, 0], [-2.2, 3.2], 'k-')
plt.grid(True)
plt.title("SELU activation function", fontsize=14)
plt.axis([-5, 5, -2.2, 3.2])
save_fig("selu_plot")
plt.show()
Saving figure selu_plot
By default, the SELU hyperparameters (scale and alpha) are tuned in such a way that the mean
output of each neuron remains close to 0, and the standard deviation remains close to 1 (assuming
12
the inputs are standardized with mean 0 and standard deviation 1 too). Using this activation
function, even a 1,000 layer deep neural network preserves roughly mean 0 and standard deviation
1 across all layers, avoiding the exploding/vanishing gradients problem:
[25]: np.random.seed(42)
Z = np.random.normal(size=(500, 100)) # standardized inputs
for layer in range(1000):
W = np.random.normal(size=(100, 100), scale=np.sqrt(1 / 100)) # LeCun␣
,→initialization
Z = selu(np.dot(Z, W))
means = np.mean(Z, axis=0).mean()
stds = np.std(Z, axis=0).mean()
if layer % 100 == 0:
print("Layer {}: mean {:.2f}, std deviation {:.2f}".format(layer,␣
,→means, stds))
Layer 0: mean -0.00, std deviation 1.00

Layer 100: mean 0.02, std deviation 0.96
Using SELU is easy:
[26]: keras.layers.Dense(10, activation="selu",
kernel_initializer="lecun_normal")
[26]: <tensorflow.python.keras.layers.core.Dense at 0x1895f0670>
Let’s create a neural net for Fashion MNIST with 100 hidden layers, using the SELU activation
function:
tf.random.set_seed(42)
[28]: model = keras.models.Sequential()

model.add(keras.layers.Flatten(input_shape=[28, 28]))
model.add(keras.layers.Dense(300, activation="selu",
kernel_initializer="lecun_normal"))
model.add(keras.layers.Dense(100, activation="selu",
kernel_initializer="lecun_normal"))
model.add(keras.layers.Dense(10, activation="softmax"))
13
Now let’s train it. Do not forget to scale the inputs to mean 0 and standard deviation 1:
[30]: pixel_means = X_train.mean(axis=0, keepdims=True)
pixel_stds = X_train.std(axis=0, keepdims=True)
X_train_scaled = (X_train - pixel_means) / pixel_stds
X_valid_scaled = (X_valid - pixel_means) / pixel_stds
X_test_scaled = (X_test - pixel_means) / pixel_stds
[31]: %%time
history = model.fit(X_train_scaled, y_train, epochs=5,
validation_data=(X_valid_scaled, y_valid))
Epoch 1/5
1719/1719 [==============================] - 13s 7ms/step - loss: 1.5628 -
Epoch 2/5
1719/1719 [==============================] - 11s 6ms/step - loss: 0.8384 -
Epoch 3/5
1719/1719 [==============================] - 11s 6ms/step - loss: 0.7810 -
Epoch 4/5
1719/1719 [==============================] - 11s 6ms/step - loss: 0.6288 -
Epoch 5/5
1719/1719 [==============================] - 11s 6ms/step - loss: 0.5652 -
CPU times: user 1min 55s, sys: 15.4 s, total: 2min 10s
Wall time: 56.7 s
Now look at what happens if we try to use the ReLU activation function instead:
[34]: model = keras.models.Sequential()

model.add(keras.layers.Flatten(input_shape=[28, 28]))
model.add(keras.layers.Dense(300, activation="relu",␣
,→kernel_initializer="he_normal"))

model.add(keras.layers.Dense(100, activation="relu",␣
,→kernel_initializer="he_normal"))
14
[36]: %%time
history = model.fit(X_train_scaled, y_train, epochs=5,
Epoch 1/5
1719/1719 [==============================] - 11s 6ms/step - loss: 2.1142 -
Epoch 2/5
1719/1719 [==============================] - 9s 5ms/step - loss: 1.3122 -
Epoch 3/5
1719/1719 [==============================] - 9s 5ms/step - loss: 1.0267 -
Epoch 4/5
1719/1719 [==============================] - 9s 5ms/step - loss: 1.0195 -
Epoch 5/5
1719/1719 [==============================] - 9s 5ms/step - loss: 0.8486 -
Wall time: 48.7 s
Not great at all, we suffered from the vanishing/exploding gradients problem.
4 Batch Normalization
[37]: model = keras.models.Sequential([
keras.layers.BatchNormalization(),
keras.layers.Dense(300, activation="relu"),
keras.layers.Dense(100, activation="relu"),
])
[38]: model.summary()
Model: "sequential_6"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten_6 (Flatten) (None, 784) 0
15
_________________________________________________________________
batch_normalization (BatchNo (None, 784) 3136
_________________________________________________________________
dense_316 (Dense) (None, 300) 235500
_________________________________________________________________
batch_normalization_1 (Batch (None, 300) 1200
_________________________________________________________________
dense_317 (Dense) (None, 100) 30100
_________________________________________________________________
_________________________________________________________________
=================================================================
Total params: 271,346
Trainable params: 268,978
Non-trainable params: 2,368
_________________________________________________________________
[39]: bn1 = model.layers[1]

[(var.name, var.trainable) for var in bn1.variables]
[39]: [('batch_normalization/gamma:0', True),

('batch_normalization/beta:0', True),
('batch_normalization/moving_mean:0', False),
('batch_normalization/moving_variance:0', False)]
[40]: bn1.updates
/usr/local/lib/python3.9/site-
packages/tensorflow/python/keras/engine/base_layer.py:1331: UserWarning:
`layer.updates` will be removed in a future version. This property should not be
used in TensorFlow 2.0, as `updates` are applied automatically.
warnings.warn('`layer.updates` will be removed in a future version. '
[40]: []
[42]: %%time
Epoch 1/10
1719/1719 [==============================] - 3s 2ms/step - loss: 1.2126 -
Epoch 2/10
16
1719/1719 [==============================] - 3s 2ms/step - loss: 0.5990 -
Epoch 3/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.5274 -
Epoch 4/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4860 -
Epoch 5/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4661 -
Epoch 6/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4360 -
Epoch 7/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4218 -
Epoch 8/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4085 -
Epoch 9/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4045 -
Epoch 10/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.3894 -
Wall time: 28.8 s
Sometimes applying BN before the activation function works better (there’s a debate on this topic).
Moreover, the layer before a BatchNormalization layer does not need to have bias terms, since
the BatchNormalization layer some as well, it would be a waste of parameters, so you can set
use_bias=False when creating those layers:
keras.layers.Dense(300, use_bias=False),
keras.layers.Activation("relu"),
keras.layers.Dense(100, use_bias=False),
keras.layers.Activation("relu"),
])
17
[45]: %%time
Epoch 1/10
1719/1719 [==============================] - 3s 2ms/step - loss: 1.4225 -
Epoch 2/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.7177 -
Epoch 3/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.6161 -
Epoch 4/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.5588 -
Epoch 5/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.5307 -
Epoch 6/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4971 -
Epoch 7/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4773 -
Epoch 8/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4597 -
Epoch 9/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4525 -
Epoch 10/10
1719/1719 [==============================] - 3s 2ms/step - loss: 0.4334 -
CPU times: user 1min 14s, sys: 15 s, total: 1min 29s
Wall time: 28.5 s
4.1 Gradient Clipping

• Gradients can become too large as well as too small
• This is the exploding gradient problem
– often caused by trying to avoid a vanishing gradient!
• A simple approach is to check error gardient values against a threshold value and clip them
(i.e. set to that threshold value) if the threshold is exceeded
• Another approach (that we don’t look at here) is weight normalisation
– use ℓ1 or ℓ2 norm penalisation of weight values
18
All Keras optimizers accept clipnorm or clipvalue arguments:
[46]: optimizer = keras.optimizers.SGD(clipvalue=1.0)
[47]: optimizer = keras.optimizers.SGD(clipnorm=1.0)
4.2 Reusing Pretrained Layers

4.2.1 Reusing a Keras model
Let’s split the fashion MNIST training set in two: * X_train_A: all images of all items except for
sandals and shirts (classes 5 and 6). * X_train_B: a much smaller training set of just the first 200
images of sandals or shirts.
The validation set and the test set are also split this way, but without restricting the number of
images.
We will train a model on set A (classification task with 8 classes), and try to reuse it to tackle
set B (binary classification). We hope to transfer a little bit of knowledge from task A to task B,
since classes in set A (sneakers, ankle boots, coats, t-shirts, etc.) are somewhat similar to classes in
set B (sandals and shirts). However, since we are using Dense layers, only patterns that occur at
the same location can be reused (in contrast, convolutional layers will transfer much better, since
learned patterns can be detected anywhere on the image, as we will see in the CNN chapter).
[48]: def split_dataset(X, y):

y_5_or_6 = (y == 5) | (y == 6) # sandals or shirts
y_A = y[~y_5_or_6]
y_A[y_A > 6] -= 2 # class indices 7, 8, 9 should be moved to 5, 6, 7
y_B = (y[y_5_or_6] == 6).astype(np.float32) # binary classification task:␣
,→is it a shirt (class 6)?
return ((X[~y_5_or_6], y_A),

(X[y_5_or_6], y_B))
(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)

(X_valid_A, y_valid_A), (X_valid_B, y_valid_B) = split_dataset(X_valid, y_valid)
(X_test_A, y_test_A), (X_test_B, y_test_B) = split_dataset(X_test, y_test)
X_train_B = X_train_B[:200]
y_train_B = y_train_B[:200]
[49]: X_train_A.shape
[49]: (43986, 28, 28)
[50]: X_train_B.shape
[50]: (200, 28, 28)
[51]: y_train_A[:30]
19
[51]: array([4, 0, 5, 7, 7, 7, 4, 4, 3, 4, 0, 1, 6, 3, 4, 3, 2, 6, 5, 3, 4, 5,
1, 3, 4, 2, 0, 6, 7, 1], dtype=uint8)
[52]: y_train_B[:30]
[52]: array([1., 1., 0., 0., 0., 0., 1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0.,
0., 0., 1., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1.], dtype=float32)
np.random.seed(42)
[54]: model_A = keras.models.Sequential()

model_A.add(keras.layers.Flatten(input_shape=[28, 28]))
for n_hidden in (300, 100, 50, 50, 50):
model_A.add(keras.layers.Dense(n_hidden, activation="selu"))
model_A.add(keras.layers.Dense(8, activation="softmax"))
[55]: model_A.compile(loss="sparse_categorical_crossentropy",
[56]: %%time
history = model_A.fit(X_train_A, y_train_A, epochs=20,
validation_data=(X_valid_A, y_valid_A))
Epoch 1/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.9249 -
Epoch 2/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.3651 -
Epoch 3/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.3182 -
Epoch 4/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.3049 -
Epoch 5/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2804 -
Epoch 6/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2701 -
Epoch 7/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2626 -
Epoch 8/20
20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2609 -
Epoch 9/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2558 -
Epoch 10/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2512 -
Epoch 11/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2431 -
Epoch 12/20
1375/1375 [==============================] - 1s 1ms/step - loss: 0.2422 -
Epoch 13/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2360 -
Epoch 14/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2266 -
Epoch 15/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2225 -
Epoch 16/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2262 -
Epoch 17/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2191 -
Epoch 18/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2171 -
Epoch 19/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2180 -
Epoch 20/20
1375/1375 [==============================] - 2s 1ms/step - loss: 0.2112 -
Wall time: 30.7 s
[57]: model_A.save("my_model_A.h5")
[58]: model_B = keras.models.Sequential()

model_B.add(keras.layers.Flatten(input_shape=[28, 28]))
for n_hidden in (300, 100, 50, 50, 50):
model_B.add(keras.layers.Dense(n_hidden, activation="selu"))
21
model_B.add(keras.layers.Dense(1, activation="sigmoid"))
[59]: model_B.compile(loss="binary_crossentropy",
[60]: %%time
history = model_B.fit(X_train_B, y_train_B, epochs=20,
validation_data=(X_valid_B, y_valid_B))
Epoch 1/20
7/7 [==============================] - 0s 22ms/step - loss: 1.0360 - accuracy:
0.4975 - val_loss: 0.6314 - val_accuracy: 0.6004
Epoch 2/20
Epoch 3/20
Epoch 4/20
Epoch 5/20
Epoch 6/20
Epoch 7/20
Epoch 8/20
Epoch 9/20
Epoch 10/20
Epoch 11/20
Epoch 12/20
Epoch 13/20
22
Epoch 14/20
Epoch 15/20
Epoch 16/20
Epoch 17/20
Epoch 18/20
Epoch 19/20
Epoch 20/20
CPU times: user 1.87 s, sys: 261 ms, total: 2.14 s
Wall time: 1.27 s
[61]: model.summary()
_________________________________________________________________
=================================================================
_________________________________________________________________
_________________________________________________________________
dense_319 (Dense) (None, 300) 235200
_________________________________________________________________
_________________________________________________________________
activation (Activation) (None, 300) 0
_________________________________________________________________
_________________________________________________________________
_________________________________________________________________
activation_1 (Activation) (None, 100) 0
_________________________________________________________________
23
=================================================================
Non-trainable params: 2,368
_________________________________________________________________
[62]: model_A = keras.models.load_model("my_model_A.h5")

model_B_on_A = keras.models.Sequential(model_A.layers[:-1])
model_B_on_A.add(keras.layers.Dense(1, activation="sigmoid"))
[63]: model_A_clone = keras.models.clone_model(model_A)

model_A_clone.set_weights(model_A.get_weights())
[64]: for layer in model_B_on_A.layers[:-1]:

layer.trainable = False
model_B_on_A.compile(loss="binary_crossentropy",
[65]: %%time
history = model_B_on_A.fit(X_train_B, y_train_B, epochs=4,
for layer in model_B_on_A.layers[:-1]:

layer.trainable = True
model_B_on_A.compile(loss="binary_crossentropy",
optimizer=keras.optimizers.SGD(lr=1e-3),
history = model_B_on_A.fit(X_train_B, y_train_B, epochs=16,
Epoch 1/4
Epoch 2/4
Epoch 3/4
Epoch 4/4
Epoch 1/16
24
packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:374: UserWarning:
The `lr` argument is deprecated, use `learning_rate` instead.
warnings.warn(
Epoch 2/16
Epoch 3/16
Epoch 4/16
Epoch 5/16
Epoch 6/16
Epoch 7/16
Epoch 8/16
Epoch 9/16
Epoch 10/16
Epoch 11/16
Epoch 12/16
Epoch 13/16
Epoch 14/16
Epoch 15/16
25
Epoch 16/16
CPU times: user 2.24 s, sys: 273 ms, total: 2.52 s
Wall time: 1.63 s
So, what’s the final verdict?
[66]: model_B.evaluate(X_test_B, y_test_B)
63/63 [==============================] - 0s 761us/step - loss: 0.1408 -

accuracy: 0.9705
[66]: [0.1408407837152481, 0.9704999923706055]
[67]: model_B_on_A.evaluate(X_test_B, y_test_B)
63/63 [==============================] - 0s 759us/step - loss: 0.0683 -

accuracy: 0.9930
[67]: [0.0683266744017601, 0.9929999709129333]
Great! We got quite a bit of transfer: the error rate dropped by a factor of 4!
[68]: (100 - 96.95) / (100 - 99.25)
[68]: 4.066666666666663
5 Faster Optimizers
• The details are beyond the scope of the module
• Optimizers should
– be efficient
– avoid saddle points
– avoid local minima
• These desirable features are often mutually exclusive
• The design and empirical evaluation of optimizers is a large area of research
5.1 Momentum optimization

[69]: optimizer = keras.optimizers.SGD(learning_rate=0.001, momentum=0.9)
26
5.2 Nesterov Accelerated Gradient
[70]: optimizer = keras.optimizers.SGD(learning_rate=0.001, momentum=0.9,␣
,→nesterov=True)
5.3 AdaGrad
[71]: optimizer = keras.optimizers.Adagrad(learning_rate=0.001)
5.4 RMSProp
[72]: optimizer = keras.optimizers.RMSprop(learning_rate=0.001, rho=0.9)
5.5 Adam Optimization

[73]: optimizer = keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999)
5.6 Adamax Optimization

[74]: optimizer = keras.optimizers.Adamax(learning_rate=0.001, beta_1=0.9, beta_2=0.
,→999)
5.7 Nadam Optimization

[75]: optimizer = keras.optimizers.Nadam(learning_rate=0.001, beta_1=0.9, beta_2=0.
,→999)
5.8 Learning Rate Scheduling

• The default in Keras is constant rate
• Using a schedule can speed up learning and avoid gradient issues
• Adaptive learning rates are also possible
• We’ll look at performance scheduling
– the idea is to choose a rate that seems suitable for the current state of weight learning
5.8.1 Power Scheduling

lr = lr0 / (1 + steps / s)**c * Keras uses c=1 and s = 1 / decay
[76]: optimizer = keras.optimizers.SGD(learning_rate=0.01, decay=1e-4)

keras.layers.Dense(300, activation="selu",␣
,→kernel_initializer="lecun_normal"),
27
kernel_initializer="lecun_normal"),
,→
])
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer,␣
,→metrics=["accuracy"])
[78]: %%time
n_epochs = 25
history = model.fit(X_train_scaled, y_train, epochs=n_epochs,
Epoch 1/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5980 -
Epoch 2/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.3831 -
Epoch 3/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.3492 -
Epoch 4/25
1719/1719 [==============================] - 2s 979us/step - loss: 0.3277 -
Epoch 5/25
1719/1719 [==============================] - 2s 979us/step - loss: 0.3173 -
Epoch 6/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.2923 -
Epoch 7/25
1719/1719 [==============================] - 2s 998us/step - loss: 0.2871 -
Epoch 8/25
1719/1719 [==============================] - 2s 990us/step - loss: 0.2721 -
Epoch 9/25
1719/1719 [==============================] - 2s 981us/step - loss: 0.2729 -
Epoch 10/25
1719/1719 [==============================] - 2s 984us/step - loss: 0.2585 -
Epoch 11/25
1719/1719 [==============================] - 2s 993us/step - loss: 0.2529 -
Epoch 12/25
1719/1719 [==============================] - 2s 999us/step - loss: 0.2485 -
28
Epoch 13/25
1719/1719 [==============================] - 2s 980us/step - loss: 0.2419 -
Epoch 14/25
1719/1719 [==============================] - 2s 989us/step - loss: 0.2372 -
Epoch 15/25
1719/1719 [==============================] - 2s 983us/step - loss: 0.2363 -
Epoch 16/25
1719/1719 [==============================] - 2s 980us/step - loss: 0.2310 -
Epoch 17/25
1719/1719 [==============================] - 2s 979us/step - loss: 0.2235 -
Epoch 18/25
1719/1719 [==============================] - 2s 995us/step - loss: 0.2248 -
Epoch 19/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.2235 -
Epoch 20/25
1719/1719 [==============================] - 2s 993us/step - loss: 0.2228 -
Epoch 21/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.2194 -
Epoch 22/25
1719/1719 [==============================] - 2s 984us/step - loss: 0.2164 -
Epoch 23/25
1719/1719 [==============================] - 2s 998us/step - loss: 0.2130 -
Epoch 24/25
1719/1719 [==============================] - 2s 987us/step - loss: 0.2077 -
Epoch 25/25
1719/1719 [==============================] - 2s 986us/step - loss: 0.2103 -
Wall time: 42.9 s
[79]: learning_rate = 0.01

decay = 1e-4
batch_size = 32
n_steps_per_epoch = len(X_train) // batch_size
29
epochs = np.arange(n_epochs)
lrs = learning_rate / (1 + decay * epochs * n_steps_per_epoch)
plt.plot(epochs, lrs, "o-")

plt.axis([0, n_epochs - 1, 0, 0.01])
plt.xlabel("Epoch")
plt.ylabel("Learning Rate")
plt.title("Power Scheduling", fontsize=14)
plt.grid(True)
plt.show()
5.8.2 Exponential Scheduling

lr = lr0 * 0.1**(epoch / s)
[80]: def exponential_decay_fn(epoch):
return 0.01 * 0.1**(epoch / 20)
[81]: def exponential_decay(lr0, s):

def exponential_decay_fn(epoch):
return lr0 * 0.1**(epoch / s)
return exponential_decay_fn
30
exponential_decay_fn = exponential_decay(lr0=0.01, s=20)

])
model.compile(loss="sparse_categorical_crossentropy", optimizer="nadam",␣
n_epochs = 25
[83]: %%time
lr_scheduler = keras.callbacks.LearningRateScheduler(exponential_decay_fn)
validation_data=(X_valid_scaled, y_valid),
callbacks=[lr_scheduler])
Epoch 1/25
1719/1719 [==============================] - 4s 2ms/step - loss: 1.1350 -
Epoch 2/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.6934 -
Epoch 3/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.5910 -
Epoch 4/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.5455 -
Epoch 5/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.5145 -
Epoch 6/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.4288 -
Epoch 7/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.4253 -
Epoch 8/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3863 -
Epoch 9/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3644 -
31
Epoch 10/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.3240 -
Epoch 11/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2990 -
Epoch 12/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2759 -
Epoch 13/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2609 -
Epoch 14/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2503 -
Epoch 15/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2353 -
Epoch 16/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.2139 -
Epoch 17/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.1895 -
Epoch 18/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.1777 -
Epoch 19/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1704 -
Epoch 20/25
1719/1719 [==============================] - 4s 3ms/step - loss: 0.1638 -
Epoch 21/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.1519 -
Epoch 22/25
1719/1719 [==============================] - 4s 2ms/step - loss: 0.1432 -
Epoch 23/25
1719/1719 [==============================] - 4s 3ms/step - loss: 0.1336 -
Epoch 24/25
1719/1719 [==============================] - 4s 3ms/step - loss: 0.1253 -
Epoch 25/25
1719/1719 [==============================] - 4s 3ms/step - loss: 0.1209 -
32
CPU times: user 6min 9s, sys: 2min 39s, total: 8min 48s
Wall time: 1min 43s
[84]: plt.plot(history.epoch, history.history["lr"], "o-")

plt.axis([0, n_epochs - 1, 0, 0.011])
plt.xlabel("Epoch")
plt.title("Exponential Scheduling", fontsize=14)
plt.grid(True)
plt.show()
The schedule function can take the current learning rate as a second argument:
[85]: def exponential_decay_fn(epoch, lr):
return lr * 0.1**(1 / 20)
If you want to update the learning rate at each iteration rather than at each epoch, you must write
your own callback class:
[86]: K = keras.backend
class ExponentialDecay(keras.callbacks.Callback):
def __init__(self, s=40000):
super().__init__()
33
self.s = s
def on_batch_begin(self, batch, logs=None):

# Note: the `batch` argument is reset at each epoch
lr = K.get_value(self.model.optimizer.lr)
K.set_value(self.model.optimizer.lr, lr * 0.1**(1 / s))
def on_epoch_end(self, epoch, logs=None):

logs = logs or {}
logs['lr'] = K.get_value(self.model.optimizer.lr)
])
lr0 = 0.01
optimizer = keras.optimizers.Nadam(lr=lr0)
n_epochs = 25
warnings.warn(
[87]: s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)

exp_decay = ExponentialDecay(s)
callbacks=[exp_decay])
Epoch 1/25
1719/1719 [==============================] - 5s 3ms/step - loss: 1.1034 -
Epoch 2/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.6554 -
Epoch 3/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.5847 -
Epoch 4/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.5607 -
34
Epoch 5/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.5004 -
Epoch 6/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.3996 -
Epoch 7/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.3862 -
Epoch 8/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.3457 -
Epoch 9/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.3308 -
Epoch 10/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.3019 -
Epoch 11/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.2712 -
Epoch 12/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.2563 -
Epoch 13/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.2395 -
Epoch 14/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.2223 -
Epoch 15/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.2076 -
Epoch 16/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1905 -
Epoch 17/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1715 -
Epoch 18/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1606 -
Epoch 19/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1533 -
Epoch 20/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1421 -
35
Epoch 21/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1351 -
Epoch 22/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1233 -
Epoch 23/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1144 -
Epoch 24/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1069 -
Epoch 25/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.1043 -
[88]: n_steps = n_epochs * len(X_train) // 32

steps = np.arange(n_steps)
lrs = lr0 * 0.1**(steps / s)
[89]: plt.plot(steps, lrs, "-", linewidth=2)

plt.axis([0, n_steps - 1, 0, lr0 * 1.1])
plt.xlabel("Batch")
plt.title("Exponential Scheduling (per batch)", fontsize=14)
plt.grid(True)
plt.show()
36
5.8.3 Piecewise Constant Scheduling
[90]: def piecewise_constant_fn(epoch):

if epoch < 5:
return 0.01
elif epoch < 15:
return 0.005
else:
return 0.001
[91]: def piecewise_constant(boundaries, values):

boundaries = np.array([0] + boundaries)
values = np.array(values)
def piecewise_constant_fn(epoch):
return values[np.argmax(boundaries > epoch) - 1]
return piecewise_constant_fn
piecewise_constant_fn = piecewise_constant([5, 15], [0.01, 0.005, 0.001])
[92]: %%time
lr_scheduler = keras.callbacks.LearningRateScheduler(piecewise_constant_fn)
37
])
n_epochs = 25

Epoch 1/25
1719/1719 [==============================] - 6s 3ms/step - loss: 1.1657 -
Epoch 2/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.7137 -
Epoch 3/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.8323 -
Epoch 4/25
1719/1719 [==============================] - 5s 3ms/step - loss: 1.0637 -
Epoch 5/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.9993 -
Epoch 6/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.9075 -
Epoch 7/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.7330 -
Epoch 8/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.6619 -
Epoch 9/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.6794 -
Epoch 10/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.6079 -
Epoch 11/25
38
1719/1719 [==============================] - 6s 3ms/step - loss: 0.5980 -
Epoch 12/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.6234 -
Epoch 13/25
1719/1719 [==============================] - 6s 4ms/step - loss: 0.5759 -
Epoch 14/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.6123 -
Epoch 15/25
1719/1719 [==============================] - 6s 4ms/step - loss: 0.5889 -
Epoch 16/25
1719/1719 [==============================] - 6s 4ms/step - loss: 0.4829 -
Epoch 17/25
1719/1719 [==============================] - 6s 4ms/step - loss: 0.3853 -
Epoch 18/25
1719/1719 [==============================] - 7s 4ms/step - loss: 0.3593 -
Epoch 19/25
1719/1719 [==============================] - 7s 4ms/step - loss: 0.3320 -
Epoch 20/25
1719/1719 [==============================] - 7s 4ms/step - loss: 0.3187 -
Epoch 21/25
1719/1719 [==============================] - 5s 3ms/step - loss: 0.3030 -
Epoch 22/25
1719/1719 [==============================] - 6s 3ms/step - loss: 0.2926 -
Epoch 23/25
1719/1719 [==============================] - 7s 4ms/step - loss: 0.2748 -
Epoch 24/25
1719/1719 [==============================] - 7s 4ms/step - loss: 0.2653 -
Epoch 25/25
1719/1719 [==============================] - 7s 4ms/step - loss: 0.2589 -
CPU times: user 6min 17s, sys: 3min 24s, total: 9min 42s
Wall time: 2min 27s
39
[93]: plt.plot(history.epoch, [piecewise_constant_fn(epoch) for epoch in history.
,→epoch], "o-")
plt.axis([0, n_epochs - 1, 0, 0.011])

plt.xlabel("Epoch")
plt.title("Piecewise Constant Scheduling", fontsize=14)
plt.grid(True)
plt.show()
5.8.4 Performance Scheduling
np.random.seed(42)
[95]: %%time
lr_scheduler = keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)
40
])
optimizer = keras.optimizers.SGD(learning_rate=0.02, momentum=0.9)
n_epochs = 25
Epoch 1/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.7109 -
Epoch 2/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4888 -
Epoch 3/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4976 -
Epoch 4/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4981 -
Epoch 5/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5240 -
Epoch 6/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5144 -
Epoch 7/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.3364 -
Epoch 8/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.2494 -
Epoch 9/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.2369 -
Epoch 10/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.2164 -
Epoch 11/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.1965 -
Epoch 12/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.1906 -
41
Epoch 13/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.1834 -
Epoch 14/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.1747 -
Epoch 15/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.1758 -
Epoch 16/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.1220 -
Epoch 17/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0952 -
Epoch 18/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0900 -
Epoch 19/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0824 -
Epoch 20/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0761 -
Epoch 21/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0621 -
Epoch 22/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0533 -
Epoch 23/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0505 -
Epoch 24/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0453 -
Epoch 25/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.0428 -
CPU times: user 2min 2s, sys: 31 s, total: 2min 33s
Wall time: 47.9 s
[96]: plt.plot(history.epoch, history.history["lr"], "bo-")

plt.xlabel("Epoch")
plt.ylabel("Learning Rate", color='b')
plt.tick_params('y', colors='b')
42
plt.gca().set_xlim(0, n_epochs - 1)
plt.grid(True)
ax2 = plt.gca().twinx()
ax2.plot(history.epoch, history.history["val_loss"], "r^-")
ax2.set_ylabel('Validation Loss', color='r')
ax2.tick_params('y', colors='r')
plt.title("Reduce LR on Plateau", fontsize=14)

plt.show()
5.8.5 tf.keras schedulers

class ExponentialDecay: A LearningRateSchedule that uses an exponential decay schedule.
class InverseTimeDecay: A LearningRateSchedule that uses an inverse time decay schedule.
class LearningRateSchedule: A serializable learning rate decay schedule.
class PiecewiseConstantDecay: A LearningRateSchedule that uses a piecewise constant decay sched-
ule.
class PolynomialDecay: A LearningRateSchedule that uses a polynomial decay schedule.
[97]: %%time
43
])
s = 20 * len(X_train) // 32 # number of steps in 20 epochs (batch size = 32)
learning_rate = keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)
optimizer = keras.optimizers.SGD(learning_rate)
n_epochs = 25
Epoch 1/25
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5995 -
Epoch 2/25
1719/1719 [==============================] - 2s 984us/step - loss: 0.3890 -
Epoch 3/25
1719/1719 [==============================] - 2s 994us/step - loss: 0.3531 -
Epoch 4/25
1719/1719 [==============================] - 2s 987us/step - loss: 0.3297 -
Epoch 5/25
1719/1719 [==============================] - 2s 993us/step - loss: 0.3177 -
Epoch 6/25
1719/1719 [==============================] - 2s 988us/step - loss: 0.2929 -
Epoch 7/25
1719/1719 [==============================] - 2s 992us/step - loss: 0.2854 -
Epoch 8/25
1719/1719 [==============================] - 2s 988us/step - loss: 0.2713 -
Epoch 9/25
1719/1719 [==============================] - 2s 995us/step - loss: 0.2714 -
Epoch 10/25
1719/1719 [==============================] - 2s 996us/step - loss: 0.2570 -
Epoch 11/25
1719/1719 [==============================] - 2s 983us/step - loss: 0.2501 -
44
Epoch 12/25
1719/1719 [==============================] - 2s 978us/step - loss: 0.2452 -
Epoch 13/25
1719/1719 [==============================] - 2s 984us/step - loss: 0.2409 -
Epoch 14/25
1719/1719 [==============================] - 2s 977us/step - loss: 0.2379 -
Epoch 15/25
1719/1719 [==============================] - 2s 977us/step - loss: 0.2377 -
Epoch 16/25
1719/1719 [==============================] - 2s 992us/step - loss: 0.2317 -
Epoch 17/25
1719/1719 [==============================] - 2s 990us/step - loss: 0.2265 -
Epoch 18/25
1719/1719 [==============================] - 2s 978us/step - loss: 0.2284 -
Epoch 19/25
1719/1719 [==============================] - 2s 981us/step - loss: 0.2285 -
Epoch 20/25
1719/1719 [==============================] - 2s 991us/step - loss: 0.2287 -
Epoch 21/25
1719/1719 [==============================] - 2s 991us/step - loss: 0.2265 -
Epoch 22/25
1719/1719 [==============================] - 2s 993us/step - loss: 0.2257 -
Epoch 23/25
1719/1719 [==============================] - 2s 985us/step - loss: 0.2222 -
Epoch 24/25
1719/1719 [==============================] - 2s 986us/step - loss: 0.2181 -
Epoch 25/25
1719/1719 [==============================] - 2s 996us/step - loss: 0.2222 -
Wall time: 42.8 s
For piecewise constant scheduling, try this:
45
[98]: learning_rate = keras.optimizers.schedules.PiecewiseConstantDecay(
boundaries=[5. * n_steps_per_epoch, 15. * n_steps_per_epoch],
values=[0.01, 0.005, 0.001])
6 Avoiding Overfitting Through Regularization

6.1 ℓ1 and ℓ2 regularization
[99]: layer = keras.layers.Dense(100, activation="elu",
kernel_initializer="he_normal",
kernel_regularizer=keras.regularizers.l2(0.01))
# or l1(0.1) for �1 regularization with a factor or 0.1
# or l1_l2(0.1, 0.01) for both �1 and �2 regularization, with factors 0.1 and 0.
,→01 respectively
[100]: %%time
keras.layers.Dense(300, activation="elu",
kernel_regularizer=keras.regularizers.l2(0.01)),
keras.layers.Dense(100, activation="elu",
kernel_regularizer=keras.regularizers.l2(0.01)),
keras.layers.Dense(10, activation="softmax",
])
n_epochs = 2
Epoch 1/2
1719/1719 [==============================] - 6s 3ms/step - loss: 3.1835 -
Epoch 2/2
1719/1719 [==============================] - 6s 3ms/step - loss: 0.7294 -
Wall time: 12.4 s
[101]: %%time
from functools import partial
RegularizedDense = partial(keras.layers.Dense,
46
activation="elu",
RegularizedDense(300),
RegularizedDense(100),
RegularizedDense(10, activation="softmax")
])
n_epochs = 2
Epoch 1/2
1719/1719 [==============================] - 6s 3ms/step - loss: 3.3555 -
Epoch 2/2
1719/1719 [==============================] - 5s 3ms/step - loss: 0.7282 -
Wall time: 11.6 s
6.2 Dropout
• During training, randomly zero some of the values of the input and/or weighted sums with
probability p
• The idea is to reduce overfitting
• It would be great if we could average the predictions from thousands of neural nets
– as done in Random Forests
• But this would be too expensive computationally
• So we mimic the randomness introduced in ensemble methods by dropping values here and
there
• As always, its use is data, architecture and human expert dependent
[103]: %%time
keras.layers.Dropout(rate=0.2),
keras.layers.Dense(300, activation="elu", kernel_initializer="he_normal"),
keras.layers.Dense(100, activation="elu", kernel_initializer="he_normal"),
])
47
n_epochs = 2
Epoch 1/2
1719/1719 [==============================] - 7s 4ms/step - loss: 0.7216 -
Epoch 2/2
1719/1719 [==============================] - 6s 4ms/step - loss: 0.4274 -
Wall time: 13.3 s
6.3 Alpha Dropout

• Keep mean and variance of inputs to their original values, in order to ensure the self-
normalizing property, even after a dropout event
np.random.seed(42)
[105]: %%time
keras.layers.AlphaDropout(rate=0.2),
])
optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)
n_epochs = 20
warnings.warn(
Epoch 1/20
48
1719/1719 [==============================] - 3s 2ms/step - loss: 0.8023 -
Epoch 2/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.5664 -
Epoch 3/20
1719/1719 [==============================] - 3s 1ms/step - loss: 0.5263 -
Epoch 4/20
1719/1719 [==============================] - 3s 1ms/step - loss: 0.5120 -
Epoch 5/20
1719/1719 [==============================] - 3s 1ms/step - loss: 0.5069 -
Epoch 6/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4792 -
Epoch 7/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4726 -
Epoch 8/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4569 -
Epoch 9/20
1719/1719 [==============================] - 3s 1ms/step - loss: 0.4627 -
Epoch 10/20
1719/1719 [==============================] - 3s 1ms/step - loss: 0.4542 -
Epoch 11/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4471 -
Epoch 12/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4415 -
Epoch 13/20
1719/1719 [==============================] - 3s 1ms/step - loss: 0.4337 -
Epoch 14/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4319 -
Epoch 15/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4311 -
Epoch 16/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4261 -
Epoch 17/20
49
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4213 -
Epoch 18/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4347 -
Epoch 19/20
1719/1719 [==============================] - 3s 1ms/step - loss: 0.4276 -
Epoch 20/20
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4199 -
Wall time: 50.4 s
[106]: model.evaluate(X_test_scaled, y_test)
313/313 [==============================] - 0s 665us/step - loss: 0.4833 -

accuracy: 0.8587
[106]: [0.48326751589775085, 0.8586999773979187]
[107]: model.evaluate(X_train_scaled, y_train)
1719/1719 [==============================] - 1s 669us/step - loss: 0.3603 -

accuracy: 0.8821
[107]: [0.36031341552734375, 0.8821091055870056]
[108]: history = model.fit(X_train_scaled, y_train)
1719/1719 [==============================] - 2s 1ms/step - loss: 0.4219 -

accuracy: 0.8437
6.4 MC Dropout
• For Monte Carlo dropout, the dropout is applied at both training and test time
• Justification is Bayesian statistics
np.random.seed(42)
[110]: y_probas = np.stack([model(X_test_scaled, training=True)

for sample in range(100)])
y_proba = y_probas.mean(axis=0)
y_std = y_probas.std(axis=0)
[111]: np.round(model.predict(X_test_scaled[:1]), 2)
50
[111]: array([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]],
dtype=float32)
[112]: np.round(y_probas[:, :1], 2)
[112]: array([[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.45, 0. , 0.54]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.93, 0. , 0.05]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0. , 0. , 0.98]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.12, 0. , 0.87]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.48, 0. , 0.52]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.42, 0. , 0.57]],
[[0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.39, 0. , 0.54]],
[[0. , 0. , 0. , 0. , 0. , 0.1 , 0. , 0.14, 0. , 0.76]],
[[0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.03, 0. , 0.9 ]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.99]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.11, 0. , 0.88]],
[[0. , 0. , 0. , 0. , 0. , 0.04, 0. , 0.14, 0. , 0.82]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.18, 0. , 0.81]],
[[0. , 0. , 0. , 0. , 0. , 0.14, 0. , 0.1 , 0. , 0.75]],
[[0. , 0. , 0. , 0. , 0. , 0.06, 0. , 0.15, 0. , 0.79]],
[[0. , 0. , 0. , 0. , 0. , 0.12, 0. , 0.07, 0. , 0.81]],
[[0. , 0. , 0. , 0. , 0. , 0.04, 0. , 0. , 0. , 0.96]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.54, 0. , 0.44]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.5 , 0. , 0.49]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.02, 0. , 0.96]],
[[0. , 0. , 0. , 0. , 0. , 0.95, 0. , 0. , 0. , 0.05]],
51
[[0. , 0. , 0. , 0. , 0. , 0.03, 0. , 0.04, 0. , 0.93]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. ]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.1 , 0. , 0.9 ]],
[[0. , 0. , 0. , 0. , 0. , 0.04, 0. , 0.06, 0. , 0.89]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.74, 0. , 0.23]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.06, 0. , 0.92]],
[[0. , 0. , 0. , 0. , 0. , 0.13, 0. , 0.22, 0. , 0.65]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.04, 0. , 0.94]],
[[0. , 0. , 0. , 0. , 0. , 0.28, 0. , 0.46, 0. , 0.26]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. ]],
[[0. , 0. , 0. , 0. , 0. , 0.35, 0. , 0.03, 0. , 0.62]],
[[0. , 0. , 0. , 0. , 0. , 0.11, 0. , 0.83, 0. , 0.06]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.16, 0. , 0.81]],
[[0. , 0. , 0. , 0. , 0. , 0.05, 0. , 0.74, 0. , 0.22]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.98]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.2 , 0. , 0.78]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.22, 0. , 0.76]],
[[0. , 0. , 0. , 0. , 0. , 0.73, 0. , 0.02, 0. , 0.24]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.92]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.98]],
[[0. , 0. , 0. , 0. , 0. , 0.26, 0. , 0.22, 0. , 0.52]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.99]],
52
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.98]],
[[0. , 0. , 0. , 0. , 0. , 0.3 , 0. , 0.16, 0. , 0.54]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.29, 0. , 0.71]],
[[0. , 0. , 0. , 0. , 0. , 0.03, 0. , 0.2 , 0. , 0.77]],
[[0. , 0. , 0. , 0. , 0. , 0.16, 0. , 0.35, 0. , 0.49]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.7 , 0. , 0.28]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]],
[[0. , 0. , 0. , 0. , 0. , 0.06, 0. , 0.07, 0. , 0.87]],
[[0. , 0. , 0. , 0. , 0. , 0.06, 0. , 0. , 0. , 0.94]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.08, 0. , 0.92]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.27, 0. , 0.73]],
[[0. , 0. , 0. , 0. , 0. , 0.03, 0. , 0.2 , 0. , 0.77]],
[[0. , 0. , 0. , 0. , 0. , 0.03, 0. , 0.15, 0. , 0.83]],
[[0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.09, 0. , 0.85]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.02, 0. , 0.97]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.11, 0. , 0.89]],
[[0. , 0. , 0. , 0. , 0. , 0.05, 0. , 0.02, 0. , 0.93]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.22, 0. , 0.76]],
[[0. , 0. , 0. , 0. , 0. , 0.3 , 0. , 0.59, 0. , 0.11]],
[[0. , 0. , 0. , 0. , 0. , 0.41, 0. , 0.38, 0. , 0.22]],
[[0. , 0. , 0. , 0. , 0. , 0.05, 0. , 0.3 , 0. , 0.64]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]],
53
[[0. , 0. , 0. , 0. , 0. , 0.03, 0. , 0.29, 0. , 0.69]],
[[0. , 0. , 0. , 0. , 0. , 0.1 , 0. , 0.54, 0. , 0.35]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. ]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.1 , 0. , 0.89]],
[[0. , 0. , 0. , 0. , 0. , 0.04, 0. , 0.13, 0. , 0.83]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.23, 0. , 0.76]],
[[0. , 0. , 0. , 0. , 0. , 0.08, 0. , 0.22, 0. , 0.7 ]],
[[0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.01, 0. , 0.92]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.1 , 0. , 0.88]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.6 , 0. , 0.39]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.39, 0. , 0.6 ]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.93]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.15, 0. , 0.85]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.04, 0. , 0.94]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. ]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.99]],
[[0. , 0. , 0. , 0. , 0. , 0.03, 0. , 0.04, 0. , 0.93]],
[[0. , 0. , 0. , 0. , 0. , 0.05, 0. , 0.48, 0. , 0.47]],
[[0. , 0. , 0. , 0. , 0. , 0.88, 0. , 0.06, 0. , 0.06]],
[[0. , 0. , 0. , 0. , 0. , 0.21, 0. , 0.05, 0. , 0.75]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.04, 0. , 0.95]],
[[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.98]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.04, 0. , 0.96]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.06, 0. , 0.93]],
54
[[0. , 0. , 0. , 0. , 0. , 0.06, 0. , 0.19, 0. , 0.75]],
[[0. , 0. , 0. , 0. , 0. , 0.15, 0. , 0.14, 0. , 0.71]],
[[0. , 0. , 0. , 0. , 0. , 0.02, 0. , 0.14, 0. , 0.85]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.01, 0. , 0.98]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.24, 0. , 0.76]],
[[0. , 0. , 0. , 0. , 0. , 0.01, 0. , 0.06, 0. , 0.93]],
[[0. , 0. , 0. , 0. , 0. , 0.11, 0. , 0.15, 0. , 0.75]],
[[0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.07, 0. , 0.87]]],

dtype=float32)
[113]: np.round(y_proba[:1], 2)
[113]: array([[0. , 0. , 0. , 0. , 0. , 0.07, 0. , 0.18, 0. , 0.75]],

dtype=float32)
[114]: y_std = y_probas.std(axis=0)

np.round(y_std[:1], 2)
[114]: array([[0. , 0. , 0. , 0. , 0. , 0.16, 0. , 0.21, 0. , 0.26]],

dtype=float32)
[115]: y_pred = np.argmax(y_proba, axis=1)
[116]: accuracy = np.sum(y_pred == y_test) / len(y_test)

accuracy
[116]: 0.8673
[117]: class MCDropout(keras.layers.Dropout):

def call(self, inputs):
return super().call(inputs, training=True)
class MCAlphaDropout(keras.layers.AlphaDropout):
np.random.seed(42)
55
[119]: mc_model = keras.models.Sequential([
MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout)␣
,→else layer
for layer in model.layers

])
[120]: mc_model.summary()
_________________________________________________________________
=================================================================
_________________________________________________________________
mc_alpha_dropout (MCAlphaDro (None, 784) 0
_________________________________________________________________
dense_366 (Dense) (None, 300) 235500
_________________________________________________________________
mc_alpha_dropout_1 (MCAlphaD (None, 300) 0
_________________________________________________________________
_________________________________________________________________
mc_alpha_dropout_2 (MCAlphaD (None, 100) 0
_________________________________________________________________
=================================================================
Non-trainable params: 0
_________________________________________________________________
[121]: optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True)

mc_model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer,␣
[122]: mc_model.set_weights(model.get_weights())
Now we can use the model with MC Dropout:

[123]: np.round(np.mean([mc_model.predict(X_test_scaled[:1]) for sample in␣
,→range(100)], axis=0), 2)
[123]: array([[0. , 0. , 0. , 0. , 0. , 0.11, 0. , 0.21, 0. , 0.68]],

dtype=float32)
56
7 Exercises
7.1 1. to 7.
See appendix A.
7.2 8. Deep Learning on CIFAR10

7.2.1 a.
Exercise: Build a DNN with 20 hidden layers of 100 neurons each (that’s too many, but it’s the
point of this exercise). Use He initialization and the ELU activation function.
[ ]: keras.backend.clear_session()
np.random.seed(42)
model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))
for _ in range(20):
model.add(keras.layers.Dense(100,
activation="elu",
kernel_initializer="he_normal"))
7.2.2 b.
Exercise: Using Nadam optimization and early stopping, train the network on the CIFAR10 dataset.
You can load it with keras.datasets.cifar10.load_data(). The dataset is composed of 60,000
32 × 32–pixel color images (50,000 for training, 10,000 for testing) with 10 classes, so you’ll need
a softmax output layer with 10 neurons. Remember to search for the right learning rate each time
you change the model’s architecture or hyperparameters.
Let’s add the output layer to the model:
[ ]: model.add(keras.layers.Dense(10, activation="softmax"))
Let’s use a Nadam optimizer with a learning rate of 5e-5. I tried learning rates 1e-5, 3e-5, 1e-4, 3e-4,
1e-3, 3e-3 and 1e-2, and I compared their learning curves for 10 epochs each (using the TensorBoard
callback, below). The learning rates 3e-5 and 1e-4 were pretty good, so I tried 5e-5, which turned
out to be slightly better.
[ ]: optimizer = keras.optimizers.Nadam(lr=5e-5)
model.compile(loss="sparse_categorical_crossentropy",
optimizer=optimizer,
Let’s load the CIFAR10 dataset. We also want to use early stopping, so we need a validation set.
Let’s use the first 5,000 images of the original training set as the validation set:
57
[ ]: (X_train_full, y_train_full), (X_test, y_test) = keras.datasets.cifar10.
,→load_data()
X_train = X_train_full[5000:]
y_train = y_train_full[5000:]
X_valid = X_train_full[:5000]
y_valid = y_train_full[:5000]
Now we can create the callbacks we need and train the model:
[ ]: early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)
model_checkpoint_cb = keras.callbacks.ModelCheckpoint("my_cifar10_model.h5",␣
,→save_best_only=True)
run_index = 1 # increment every time you train the model

run_logdir = os.path.join(os.curdir, "my_cifar10_logs", "run_{:03d}".
,→format(run_index))
tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)
callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]
[ ]: %tensorboard --logdir=./my_cifar10_logs --port=6006
[ ]: model.fit(X_train, y_train, epochs=100,

validation_data=(X_valid, y_valid),
callbacks=callbacks)
[ ]: model = keras.models.load_model("my_cifar10_model.h5")
model.evaluate(X_valid, y_valid)
The model with the lowest validation loss gets about 47% accuracy on the validation set. It took
39 epochs to reach the lowest validation loss, with roughly 10 seconds per epoch on my laptop
(without a GPU). Let’s see if we can improve performance using Batch Normalization.
7.2.3 c.
Exercise: Now try adding Batch Normalization and compare the learning curves: Is it converging
faster than before? Does it produce a better model? How does it affect training speed?
The code below is very similar to the code above, with a few changes:
• I added a BN layer after every Dense layer (before the activation function), except for the
output layer. I also added a BN layer before the first hidden layer.
• I changed the learning rate to 5e-4. I experimented with 1e-5, 3e-5, 5e-5, 1e-4, 3e-4, 5e-4,
1e-3 and 3e-3, and I chose the one with the best validation performance after 20 epochs.
• I renamed the run directories to run_bn_* and the model file name to
my_cifar10_bn_model.h5.
np.random.seed(42)
58
model.add(keras.layers.BatchNormalization())
for _ in range(20):
model.add(keras.layers.Dense(100, kernel_initializer="he_normal"))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Activation("elu"))
optimizer = keras.optimizers.Nadam(lr=5e-4)
early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)
model_checkpoint_cb = keras.callbacks.ModelCheckpoint("my_cifar10_bn_model.h5",␣
,→save_best_only=True)

run_logdir = os.path.join(os.curdir, "my_cifar10_logs", "run_bn_{:03d}".
model.fit(X_train, y_train, epochs=100,

validation_data=(X_valid, y_valid),
model = keras.models.load_model("my_cifar10_bn_model.h5")
model.evaluate(X_valid, y_valid)
• Is the model converging faster than before? Much faster! The previous model took 39 epochs
to reach the lowest validation loss, while the new model with BN took 18 epochs. That’s
more than twice as fast as the previous model. The BN layers stabilized training and allowed
us to use a much larger learning rate, so convergence was faster.
• Does BN produce a better model? Yes! The final model is also much better, with 55% accuracy
instead of 47%. It’s still not a very good model, but at least it’s much better than before (a
Convolutional Neural Network would do much better, but that’s a different topic, see chapter
14).
• How does BN affect training speed? Although the model converged twice as fast, each epoch
took about 16s instead of 10s, because of the extra computations required by the BN layers.
So overall, although the number of epochs was reduced by 50%, the training time (wall time)
was shortened by 30%. Which is still pretty significant!
7.2.4 d.
Exercise: Try replacing Batch Normalization with SELU, and make the necessary adjustements to
ensure the network self-normalizes (i.e., standardize the input features, use LeCun normal initial-
59
ization, make sure the DNN contains only a sequence of dense layers, etc.).
np.random.seed(42)
for _ in range(20):
kernel_initializer="lecun_normal",
activation="selu"))
model_checkpoint_cb = keras.callbacks.ModelCheckpoint("my_cifar10_selu_model.
,→h5", save_best_only=True)

run_logdir = os.path.join(os.curdir, "my_cifar10_logs", "run_selu_{:03d}".
X_means = X_train.mean(axis=0)
X_stds = X_train.std(axis=0)
X_train_scaled = (X_train - X_means) / X_stds
X_valid_scaled = (X_valid - X_means) / X_stds
X_test_scaled = (X_test - X_means) / X_stds
model.fit(X_train_scaled, y_train, epochs=100,

model = keras.models.load_model("my_cifar10_selu_model.h5")
model.evaluate(X_valid_scaled, y_valid)
[ ]: model = keras.models.load_model("my_cifar10_selu_model.h5")
We get 51.4% accuracy, which is better than the original model, but not quite as good as the model
using batch normalization. Moreover, it took 13 epochs to reach the best model, which is much
faster than both the original model and the BN model, plus each epoch took only 10 seconds, just
like the original model. So it’s by far the fastest model to train (both in terms of epochs and wall
60
time).
7.2.5 e.
Exercise: Try regularizing the model with alpha dropout. Then, without retraining your model, see
if you can achieve better accuracy using MC Dropout.
np.random.seed(42)
for _ in range(20):
activation="selu"))
model.add(keras.layers.AlphaDropout(rate=0.1))
model_checkpoint_cb = keras.callbacks.
,→ModelCheckpoint("my_cifar10_alpha_dropout_model.h5", save_best_only=True)

run_logdir = os.path.join(os.curdir, "my_cifar10_logs", "run_alpha_dropout_{:
,→03d}".format(run_index))
X_means = X_train.mean(axis=0)
X_stds = X_train.std(axis=0)
X_train_scaled = (X_train - X_means) / X_stds
X_valid_scaled = (X_valid - X_means) / X_stds
X_test_scaled = (X_test - X_means) / X_stds
model.fit(X_train_scaled, y_train, epochs=100,

model = keras.models.load_model("my_cifar10_alpha_dropout_model.h5")
61
The model reaches 50.8% accuracy on the validation set. That’s very slightly worse than without
dropout (51.4%). With an extensive hyperparameter search, it might be possible to do better (I
tried dropout rates of 5%, 10%, 20% and 40%, and learning rates 1e-4, 3e-4, 5e-4, and 1e-3), but
probably not much better in this case.
Let’s use MC Dropout now. We will need the MCAlphaDropout class we used earlier, so let’s just
copy it here for convenience:
[ ]: class MCAlphaDropout(keras.layers.AlphaDropout):
Now let’s create a new model, identical to the one we just trained (with the same weights), but
with MCAlphaDropout dropout layers instead of AlphaDropout layers:
[ ]: mc_model = keras.models.Sequential([
MCAlphaDropout(layer.rate) if isinstance(layer, keras.layers.AlphaDropout)␣
,→else layer
for layer in model.layers

])
Then let’s add a couple utility functions. The first will run the model many times (10 by default) and
it will return the mean predicted class probabilities. The second will use these mean probabilities
to predict the most likely class for each instance:
[ ]: def mc_dropout_predict_probas(mc_model, X, n_samples=10):
Y_probas = [mc_model.predict(X) for sample in range(n_samples)]
return np.mean(Y_probas, axis=0)
def mc_dropout_predict_classes(mc_model, X, n_samples=10):

Y_probas = mc_dropout_predict_probas(mc_model, X, n_samples)
return np.argmax(Y_probas, axis=1)
Now let’s make predictions for all the instances in the validation set, and compute the accuracy:
np.random.seed(42)
y_pred = mc_dropout_predict_classes(mc_model, X_valid_scaled)

accuracy = np.mean(y_pred == y_valid[:, 0])
accuracy
We only get virtually no accuracy improvement in this case (from 50.8% to 50.9%).
So the best model we got in this exercise is the Batch Normalization model.
62
7.2.6 f.
Exercise: Retrain your model using 1cycle scheduling and see if it improves training speed and
model accuracy.
np.random.seed(42)
for _ in range(20):
activation="selu"))
optimizer = keras.optimizers.SGD(lr=1e-3)
[ ]: batch_size = 128
rates, losses = find_learning_rate(model, X_train_scaled, y_train, epochs=1,␣
,→batch_size=batch_size)
plot_lr_vs_loss(rates, losses)
plt.axis([min(rates), max(rates), min(losses), (losses[0] + min(losses)) / 1.4])
np.random.seed(42)
for _ in range(20):
activation="selu"))
optimizer = keras.optimizers.SGD(lr=1e-2)
63
[ ]: n_epochs = 15
onecycle = OneCycleScheduler(len(X_train_scaled) // batch_size * n_epochs,␣
,→max_rate=0.05)
history = model.fit(X_train_scaled, y_train, epochs=n_epochs,␣

,→batch_size=batch_size,
callbacks=[onecycle])
One cycle allowed us to train the model in just 15 epochs, each taking only 3 seconds (thanks to the
larger batch size). This is over 3 times faster than the fastest model we trained so far. Moreover,
we improved the model’s performance (from 50.8% to 52.8%). The batch normalized model reaches
a slightly better performance, but it’s much slower to train.
[ ]:
64

Lecture 22

Uploaded by

Copyright:

Available Formats

Lecture 22

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 22

Uploaded by

Copyright:

Available Formats

Lecture22

# Scikit-Learn �0.20 is required

# TensorFlow �2.0 is required

# to make this notebook's output stable across runs

# To plot pretty figures

# Where to save the figures

def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):

INFO:tensorflow:Enabling eager execution

3 Vanishing/Exploding Gradients Problem

def derivative(f, z, eps=0.000001):

Saving figure activation_function_plot

plt.plot([-5, 5], [0, 0], 'k-')

plt.annotate('Saturating', xytext=(-3.5, 0.3), xy=(-5, 0), arrowprops=props,␣

plt.annotate('Linear', xytext=(2, 0.2), xy=(0, 0.5), arrowprops=props,␣

Saving figure sigmoid_saturation_plot

[4]: [name for name in dir(keras.initializers) if not name.startswith("_")]

[5]: keras.layers.Dense(10, activation="relu", kernel_initializer="he_normal")

[6]: init = keras.initializers.VarianceScaling(scale=2., mode='fan_avg',

[6]: <tensorflow.python.keras.layers.core.Dense at 0x105868640>

3.2 Nonsaturating Activation Functions

3.2.1 Leaky ReLU

[7]: def leaky_relu(z, alpha=0.01):

[8]: plt.plot(z, leaky_relu(z, 0.05), "b-", linewidth=2)

plt.title("Leaky ReLU activation function", fontsize=14)

Saving figure leaky_relu_plot

[10]: [m for m in dir(keras.layers) if "relu" in m.lower()]

[10]: ['LeakyReLU', 'PReLU', 'ReLU', 'ThresholdedReLU']

X_train_full = X_train_full / 255.0

[20]: plt.plot(z, elu(z), "b-", linewidth=2)

Saving figure elu_plot

[21]: <tensorflow.python.keras.layers.core.Dense at 0x10583cb80>

[23]: def selu(z, scale=scale_0_1, alpha=alpha_0_1):

[24]: plt.plot(z, selu(z), "b-", linewidth=2)

Saving figure selu_plot

Layer 0: mean -0.00, std deviation 1.00

[26]: <tensorflow.python.keras.layers.core.Dense at 0x1895f0670>

[28]: model = keras.models.Sequential()

[34]: model = keras.models.Sequential()

for layer in range(99):

[39]: bn1 = model.layers[1]

[39]: [('batch_normalization/gamma:0', True),

4.1 Gradient Clipping

[47]: optimizer = keras.optimizers.SGD(clipnorm=1.0)

4.2 Reusing Pretrained Layers

[48]: def split_dataset(X, y):

return ((X[~y_5_or_6], y_A),

(X_train_A, y_train_A), (X_train_B, y_train_B) = split_dataset(X_train, y_train)

[49]: (43986, 28, 28)

[50]: (200, 28, 28)

[54]: model_A = keras.models.Sequential()

[58]: model_B = keras.models.Sequential()

[62]: model_A = keras.models.load_model("my_model_A.h5")

[63]: model_A_clone = keras.models.clone_model(model_A)

[64]: for layer in model_B_on_A.layers[:-1]:

for layer in model_B_on_A.layers[:-1]:

63/63 [==============================] - 0s 761us/step - loss: 0.1408 -

[66]: [0.1408407837152481, 0.9704999923706055]

[67]: model_B_on_A.evaluate(X_test_B, y_test_B)