KGiSL Institute of Technology

(Approved by AICTE, New Delhi; Affiliated to Anna University, Chennai)

Recognized by UGC, Accredited by NBA (IT)
365, KGiSL Campus, Thudiyalur Road, Saravanampatti, Coimbatore – 641035.

Department of Electronics and Communication and Engineering

Name of the Faculty : Dr.KAMALAKANNAN
• Subject Code & Name : CCS355 - NEURAL NETWORKS AND DEEP
LEARNING
Department : Electronics and Communication Engineering

Year & Semester : III / VI

Academic Year : 2023-24

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

 UNIT III : THIRD-GENERATION NEURAL NETWORKS

Contents:-
➢ Spiking Neural Networks
➢ Convolutional Neural Networks
➢ Deep Learning Neural Networks
➢ Extreme Learning Machine Model
➢ Convolutional Neural Networks:
➢ The Convolution Operation
➢ Motivation – Pooling – Variants of the basic Convolution Function
➢ Structured Outputs
➢ Data Types
➢ Efficient Convolution Algorithms
➢ Neuroscientific Basis
➢ Applications: Computer Vision, Image Generation, Image Compression
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
Spiking Neural Networks

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

Recall

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

Cont…

Problem :
React only when
receiving a pulse

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
Leaky Integrate-and-Fire (LIF)

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

Leaky Integrate-and-Fire (LIF)

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

Convolutional Neural Networks (CNN)
• What is a Convolutional Neural Network (CNN)?
• A Convolutional Neural Network (CNN), also known as ConvNet, is a
specialized type of deep learning algorithm mainly designed for tasks
that necessitate object recognition, including image classification,
detection, and segmentation. CNNs are employed in a variety of
practical scenarios, such as autonomous vehicles, security camera
systems, and others.

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

 Convolutional Neural Networks (CNN)
• How CNNs Work
• A convolutional neural network can have tens or
hundreds of layers that each learn to detect different
features of an image. Filters are applied to each training
image at different resolutions, and the output of each
convolved image is used as the input to the next layer.
The filters can start as very simple features, such as
brightness and edges, and increase in complexity to
features that uniquely define the object.
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
Feature Learning, Layers, and Classification
• A CNN is composed of an input layer, an output layer, and many
hidden layers in between.

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

Shared Weights and Biases
• Unlike a traditional neural network, a CNN has shared
weights and bias values, which are the same for all
hidden neurons in a given layer.
• This means that all hidden neurons are detecting the
same feature, such as an edge or a blob, in different
regions of the image. This makes the network tolerant to
translation of objects in an image. For example, a
network trained to recognize cars will be able to do so
wherever the car is in the image.
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
Why CNNs Matter ?
Medical Imaging: CNNs can examine thousands of pathology
reports to visually detect the presence or absence of cancer cells
in images.
Audio Processing: Keyword detection can be used in any
device with a microphone to detect when a certain word or phrase
is spoken (“Hey Siri!”). CNNs can accurately learn and detect the
keyword while ignoring all other phrases regardless of the
environment.
Object Detection: Automated driving relies on CNNs to
accurately detect the presence of a sign or other object and make
decisions based on the output.
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
Cont…

Synthetic Data Generation: Using Generative Adversarial Networks

(GANs), new images can be produced for use in deep learning
applications including face recognition and automated driving.

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

 The importance of CNNs
• There are several reasons why CNNs are important in the modern world, as
highlighted below:
• CNNs are distinguished from classic machine learning algorithms such
as SVMs and decision trees by their ability to autonomously extract features at a
large scale, bypassing the need for manual feature engineering and thereby
enhancing efficiency.
• The convolutional layers grant CNNs their translation-invariant characteristics,
empowering them to identify and extract patterns and features from data
irrespective of variations in position, orientation, scale, or translation.
• A variety of pre-trained CNN architectures, including VGG-16, ResNet50,
Inceptionv3, and EfficientNet, have demonstrated top-tier performance. These
models can be adapted to new tasks with relatively little data through a process
known as fine-tuning.
• Beyond image classification tasks, CNNs are versatile and can be applied to a
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
• Inspiration Behind CNN and Parallels With The Human Visual System
• Convolutional neural networks were inspired by the layered
architecture of the human visual cortex, and below are some key
similarities and differences:

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

 lustration of the
correspondence
between the areas
associated with the
primary visual cortex
and the layers in a
convolutional neural
network (source)

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

• Hierarchical architecture: Both CNNs and the visual cortex have a
hierarchical structure, with simple features extracted in early layers and
more complex features built up in deeper layers. This allows increasingly
sophisticated representations of visual inputs.
• Local connectivity: Neurons in the visual cortex only connect to a local
region of the input, not the entire visual field. Similarly, the neurons in a
CNN layer are only connected to a local region of the input volume through
the convolution operation. This local connectivity enables efficiency.
• Translation invariance: Visual cortex neurons can detect features
regardless of their location in the visual field. Pooling layers in a CNN
provide a degree of translation invariance by summarizing local features.
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
• Multiple feature maps: At each stage of visual processing, there are
many different feature maps extracted. CNNs mimic this through
multiple filter maps in each convolution layer.
• Non-linearity: Neurons in the visual cortex exhibit non-linear
response properties. CNNs achieve non-linearity through activation
functions like ReLU applied after each convolution.
• CNNs mimic the human visual system but are simpler, lacking its
complex feedback mechanisms and relying on supervised learning
rather than unsupervised, driving advances in computer vision
despite these differences.

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

 Key Components of a CNN
• The convolutional neural network is made of four main
parts.
• But how do CNNs Learn with those parts?
• They help the CNNs mimic how the human brain
operates to recognize patterns and features in images:
• Convolutional layers
• Rectified Linear Unit (ReLU for short)
• Pooling layers
• Fully connected layers
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
• These layers perform operations that alter the data with the intent of
learning features specific to the data. Three of the most common
layers are convolution, activation or ReLU, and pooling.
• Convolution puts the input images through a set of convolutional
filters, each of which activates certain features from the images.
• Rectified linear unit (ReLU) allows for faster and more effective
training by mapping negative values to zero and maintaining positive
values. This is sometimes referred to as activation, because only the
activated features are carried forward into the next layer.
• Pooling simplifies the output by performing nonlinear downsampling,
reducing the number of parameters that the network needs to learn.
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
• It uses M filters, which are basically feature extractors that extract features like
edges, corner and so on. Following are the layers [INPUT-CONV-RELU-POOL-
FC] that are used to construct Convolutional neural networks (CNNs)−
• INPUT− As the name implies, this layer holds the raw pixel values. Raw pixel
values mean the data of the image as it is. Example, INPUT [64×64×3] is a 3-
channeled RGB image of width-64, height-64 and depth-3.
• CONV− This layer is one of the building blocks of CNNs as most of the
computation is done in this layer. Example - if we use 6 filters on the above
mentioned INPUT [64×64×3], this may result in the volume [64×64×6].
• RELU−Also called rectified linear unit layer, that applies an activation function to
the output of previous layer. In other manner, a non-linearity would be added to
the network by RELU.
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
• POOL− This layer, i.e. Pooling layer is one other building block of CNNs.
The main task of this layer is down-sampling, which means it operates
independently on every slice of the input and resizes it spatially.
• FC− It is called Fully Connected layer or more specifically the output
layer. It is used to compute output class score and the resulting output
is volume of the size 1*1*L where L is the number corresponding to
class score.

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

 Creating CNN structure
• We have seen the architecture and the basics of CNN, now we are going to
building convolutional network using CNTK. Here, we will first see how to
put together the structure of the CNN and then we will look at how to train
the parameters of it.
• At last we’ll see, how we can improve the neural network by changing its
structure with various different layer setups. We are going to use MNIST
image dataset.
• So, first let’s create a CNN structure. Generally, when we build a CNN for
recognizing patterns in images, we do the following−
• We use a combination of convolution and pooling layers.
• One or more hidden layer at the end of the network.
• At last, we finish the network with a softmax layer for classification
purpose. CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
CCS355/NN&DL/III ECE/VI SEM/KG-KiTE
 Practical Applications of CNNs
• Convolutional Neural Networks have revolutionized the field of
computer vision, leading to significant advancements in many
real-world applications. Below are a few examples of how they
are applied.

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE

CCS355/NN&DL/III ECE/VI SEM/KG-KiTE