2017project Paper
2017project Paper
2017 International Conference on Computation of Power, Energy, Information and Communication (ICCPEIC)
Abstract— The learning aids for hearing and speech disabled American Sign Language substantially facilitates
people exist but the usage of these aids are limited. The proposed communication in the hearing impaired community. However,
system would be a real time system wherein live sign gestures would there are only ~250,000-500,000 speakers which limits the
be processed using image processing.Then classifiers would be number of people that they can communicate with [7].In order
used to differentiate various signs and the translated output would
to diminish this obstacle and to enable better communication,
be displaying text . Machine Learning algorithms will be used to
train on the data set. The purpose of the system is to improve the we would like to propose an ASL recognition system that uses
existing system in this area in terms of response time and accuracy Convolutional Neural Networks to translate a user’s ASL
with the use of efficient algorithms, high quality data sets and signs into text in real time.
better sensors. The existing systems have been able to recognize
gestures with high latency as it uses only image processing. In our
project we aim to develop a cognitive system which would be
responsive and robust so as to be used in day to day applications by
hearing and speech disabled people.
Keywords-
ML-Machine Learning
SL-Supervised learning,
NN-Neural Networks,
ASL- American Sign Language,
AUSLAN-Australian Sign Language,
CNN-Convolutional Neural Networks,
SVM- Support Vector Machine. Figure 1. Sign chart for the American Sign Language (ASL)[7]
the input of the user’s sign language video,classifies the (1) takes the mean loss for each training example, xi , to
actions for each letter and then tries to come up with the most produce full softmax loss.
accurate words. The factors such as lighting conditions and
sign language border detection are considered while designing B. Recognizing Hand Gestures
the project.They attained a validation accuracy of nearly 98% Hand gestures can be recognized using several methods .Two
with five letters and 74% with ten letters. of the most widely used methods are: Haar Cascades and
Hardie Cate,Fahim Dalvi and Zeshan Hussain[2] use Neural Networks. Haar Cascades were initially used for facial
machine learning techniques for temporal classification, detection and are very easily transferrable to hand gesture
specifically the multivariate case.case. Although the results recognition.
obtained are not very high, we believe that a more efficient
implementation of the algorithms can yield bigger and more C. Neural Networks
complex models that will perform well.This system uses the
University of South wales dataset which has 95 unique signs. Neural Networks are inspired by the biological arrangement
It suggests the use of baseline SVM for high quality data and of processing elements called neurons in the brain.These
Temporal classification techniques for lower quality data. It neurons enable parallel processing of computational tasks.This
also considers implementing a custom LSTM model that enables Neural networks to solve complex problems of pattern
removes the assumptions of the technique that do not apply to recognition better than procedural algorithms.CNNs are neural
the data. A device called as Myo armband is also used to networks in which the response of the neuron can be
collect new data .They achieved an accuracy of 78.6% on a calculated by a convolution operation.The initial layer of CNN
high quality dataset. can be used for matching images with respect to a fixed
template.The subsequent layer can then be used for detecting
variations of the identified image for improved accuracy and
for generating patterns of a pattern.
III. APPROACH AND METHODS
Convolutional Neural Networks (CNNs) are machine D. Haar Cascade
learning algorithms that have seen a great success as it handles It is a computer vision learning approach in which we train a
a variety of tasks related to processing videos and images. cascade function on many positive and negative images.It can
Like other machine learning algorithms, CNNs seek to therefore be used to detect objects from other images. Initially,
optimize some objective function, specifically the loss the algorithm is trained for facial recognition and is trained
function. CNNs have seen a rapid improvement in image with many images.When the model is trained, features could
classification with many proposed models like GoogleNet, then be extracted.Haar Cascades can be implemented with
AlexNet giving an accuracy almost near to human perception. OpenCV library[9].
The main cause of the recent improvement in CNNs has been
due to the ImageNet Large Scale Visual Recognition
Competition(ILSVRC). For image processing we propose to
use OpenCV library [9] along with TensorFlow [11] and
Keras [10] which will be used for training the classifier. For
other mathematical calculations we may use the NumPy
Array [13] in Python [12]. The various approaches we
considered are explained in the subsequent paragraphs. Figure 2. Features of Haar Cascade[9]
A. Softmax Regression
Softmax regression is a generalization of logistic regression E. Deep Learning
to the case where we want to handle multiple classes.Softmax Deep learning is used to mask the levels of abstraction in a
regression allows us to handle y(i) ߳ {1,…,N} where N is the machine learning algorithm.It contains a set of hidden layers
number of classes. each using the output of the previous layer for better feature
extraction and pattern recognition.This is specially used for
unsupervised learning of a large unclassified dataset.We have
implemented deep learning using the Keras [10] deep learning
library in python [12] which contains an important numerical
library TensorFlow [11].
2017 International Conference on Computation of Power, Energy, Information and Communication (ICCPEIC)
IV. ARCHITECTURAL DIAGRAM images of different hand gestures. All hand gestures are a part
of the American sign language.
VI. IMPLEMENTATION
A. Dataset Preprocessing
The starting point for implementation will be first
preprocessing the dataset in the format accepted by the CNN
models. Our current dataset does not have a aspect ratio of 1:1
and hence have random width and height. We will be first
resizing the images in the dataset to a size of 256x256 so as to
attain an aspect ratio of 1:1 which is the accepted image
format for the input layer of the CNN. To resize the images in
the dataset, we will be using the imagemagick tool, an open
source image manipulation library. Currently, Our dataset also
Figure 3. Architecture diagram for our proposed system contains the images just for the right hand. To make our
application compatible with the left hand too, we will be
The architecture depicts that it is a real time sign language flipping the existing images horizontally.
recognition system and also the methods for image processing
is mentioned in the paper.The proposed image dataset is the B. Model of the CNN
Massey University dataset because it contains an exhaustive The proposed system will be using a CNN model which will
set of data. consist of an input layer, followed by a set of two
convolutional layers each backed up by a max pooling layer
V. DATASET and further followed by two fully connected layers. For
The various datasets we considered are as follows: varying the learning rate of the CNN during the training of the
model, we will be using the Stochastic Gradient Descent
A. Australian Sign Language signs (High Quality) Data Set (SGD) optimizer. SGD reduces cost of training the dataset and
it leads to fast convergence. For calculating the final
The above mentioned data consists of samples of Auslan
probabilities of the recognized classes, we will be using the
signs. In this dataset [4] there 95 Auslan signs with 27
softmax regression in the last Fully Connected layer of our
examples each which were captured from a native Auslan
CNN. We will use the Sequential API provided by Keras [10]
signer using very good-quality position trackers.
to build the above foresaid model. In the Sequential API, we
will take a bottom up approach and start from the Input Layer.
B. Massey University Dataset The further layers in our CNN model will be stacked one upon
The images are all png in RGB mode, with the hands the other until we reach the last layer in our model which is
segmented by colour and with black background (0,0,0). the Fully Connected layer. A Diagrammatic Representation of
our CNN Model is given in Figure 5 and Figure 6.
C. University of Texas Arlington Dataset:
Video dataset [5] containing short sequences of hand gestures
, it consists of multiple frames.
D. Image Processing
The final step will be to process the images from the webcam
in real time and extract the hand image using the image
extraction API provided by the OpenCV [9]. To get the image
of hand, we will employ various techniques such as
Background Subtraction, Convex Hull detection and counting
the defects in the resulting Convex Hull. Before giving the
real time image to the trained CNN model for classification,
we will be performing the same preprocessing steps on it just
as we do it for the training data. We will be displaying three
windows to the user which will enable the user to position his
hand properly so that a proper image of the hand is taken. One
of them will also display the largest recognized contour in the
real time image.
E. Transfer Learning
A further step in our project will be to fine tune already
established models like GoogleNet, AlexNet, VGGNet,
CaffeNet which have been the previous winners in the
ILSVRC challenge. They have been trained on the 2012
ILSVRC dataset. As our dataset is markedly different from
ImageNet data, we will vary the model weights and learning
rate in the internal layers of these models and modify the
output layer to match the input size of our training data.
2017 International Conference on Computation of Power, Energy, Information and Communication (ICCPEIC)
VII. CONCLUSION
We are going to implement sign language recognition using
concepts of machine learning and image processing.After our
application is implemented it would benefit the NGOs and
various organizations which involve people with special needs
as it will be a real-time system for hand gesture identification.
The proposed system can also be further carried forward to
implement Indian Sign Language recognition which is much
more complex as it involves two hand gestures. Also the
response time of the system can be reduced with better camera
and graphics support.
ACKNOWLEDGEMENT
REFERENCES