Professional Documents
Culture Documents
Data Analytics Concepts Techniques and A PDF
Data Analytics Concepts Techniques and A PDF
Data Analytics
Concepts, Techniques, and Applications
Edited by
Mohiuddin Ahmed and Al-Sakib Khan Pathan
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all material or the consequences of their use. The authors
and publishers have attempted to trace the copyright holders of all material reproduced in this
publication and apologize to copyright holders if permission to publish in this form has not been
obtained. If any copyright material has not been acknowledged please write and let us know so we
may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information
storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.
copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC),
222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that
provides licenses and registration for a variety of users. For organizations that have been granted a
photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Acknowledgments...............................................................................................ix
Preface.................................................................................................................xi
List of Contributors........................................................................................... xv
vii
viii ◾ Contents
Index�����������������������������������������������������������������������������������������������������������417
Acknowledgments
I am grateful to the Almighty Allah for blessing me with the opportunity to work
on this book. It is my first time as a book editor and I express my sincere gratitude
to Al-Sakib Khan Pathan for guiding me throughout the process. The book e diting
journey enhanced my patience, communication, and tenacity. I am thankful to
all the contributors, critics, and the publishing team. Last but not least, my very
best wishes for my family members whose support and encouragement contributed
significantly to the completion of this book.
Mohiuddin Ahmed
Centre for Cyber Security and Games
Canberra Institute of Technology, Australia
ix
Preface
Introduction
Big data is a term for datasets that are so large or complex that traditional data
processing applications are inadequate for them. The significance of big data has
been recognized very recently and there are various opinions on its definition.
In layman’s terms, big data reflects the datasets that cannot be perceived, acquired,
managed, and processed by the traditional information technology (IT) and
software/hardware tools in an efficient manner. Communities such as s cientific
and technological enterprises, research scholars, data analysts, and technical
practitioners have different definitions of big data. Due to a large amount of data
arriving at a fast speed, a new set of efficient data analysis techniques are required.
In addition to this, the term data science has gained a lot of attention from both the
academic research community and the industry. Therefore, data analytics becomes
an essential component for any organization. For instance, if we consider health
care, financial trading, Internet of Things (IoT) smart cities, or cyber-physical sys-
tems, one can find the role of data analytics. However, with these diverse applica-
tion domains, new research challenges are also arising. In this context, this book on
data analytics will provide a broader picture of the concepts, techniques, applica-
tions, and open research directions in this area. In addition, the book is expected to
serve as a single source of reference for acquiring knowledge on the emerging trends
and a pplications of data analytics.
xi
xii ◾ Preface
Section I contains six chapters that cover the fundamental concepts of data
analytics. These chapters reflect the important knowledge areas, such as machine
learning, regression, clustering, information retrieval, and graph analysis. Section II
has six chapters that cover the major techniques of data analytics, such as t ransition
from regular database to big data, big graph analysis tools and techniques, and
game theoretical approaches for big data analysis. The rest of the chapters in this
section cover topics that lead to newer research domains, i.e., project management,
Preface ◾ xiii
Industry 4.0, and dark data. These topics are considered as the emerging trends in
data analytics. Section III is dedicated to the applications of data analytics in differ-
ent domains, such as education, traffic offenses, sports data visualization and, last
but not the least, two interesting chapters on cybersecurity for big data analytics
with specific focus on the health care sector.
xv
xvi ◾ List of Contributors
Tamanna Motahar is a lecturer at the electrical and computer engineering depart-
ment of North South University. She completed her master’s degree in electri-
cal engineering from the University of Alberta, Canada. She graduated summa
cum laude with a BSc in computer engineering from the American International
University, Bangladesh. Her high-school graduation is from Mymensingh Girls’
Cadet College, Bangladesh. Her interdisciplinary research works are based on opti-
cal electromagnetics and nano optics. Her current research areas include Internet
of Things, human–computer interaction, and big data. She is taking Junior Design
classes for computer science engineering students in the North South University
and is also mentoring several groups for their technical projects. Her email address
is [email protected].
Asma Noor graduated in 2017 from Charles Sturt University with a master’s degree
in information technology, specializing in networking and system analysis. In 2011,
she obtained her master’s degree in international relations from the University of
Balochistan, Pakistan. In 2007, she obtained a bachelor’s degree in business adminis-
tration from Iqra University, Pakistan, specializing in marketing. Her hobbies include
reading and writing. Her interests in the field of information technology include cloud
computing, information security, Internet of Things, and big data analytics.
xxii ◾ List of Contributors
Ahsanur Rahman is an assistant professor of the electrical and computer engi-
neering department at the North South University. He completed his PhD
in computer science from Virginia Tech, Blacksburg, VA. Before that he has
worked as a lecturer in the computer science and engineering department at
the American International University Bangladesh. He obtained his bachelor’s
degree in computer science and engineering from Bangladesh University of
Engineering and Technology. His research interest lies in the areas of compu-
tational systems biology, graph algorithms, hypergraphs, and big data. He is
List of Contributors ◾ xxiii
mentoring several groups of NSU for their research projects. His email address
is [email protected].
An Introduction to
Machine Learning
Mark A. Norrie
icare
Contents
1.1 A Definition of Machine Learning................................................................3
1.1.1 Supervised or Unsupervised?..............................................................4
1.2 A rtificial Intelligence.....................................................................................5
1.2.1 Th e First AI Winter...........................................................................8
1.3 ML and Statistics..........................................................................................9
1.3.1 R ediscovery of ML..........................................................................13
1.4 Critical Events: A Timeline.........................................................................14
1.5 Types of ML................................................................................................19
1.5.1 Supervised Learning........................................................................19
1.5.2 Unsupervised Learning....................................................................20
1.5.3 Semisupervised Learning.................................................................21
1.5.4 R einforcement Learning..................................................................21
1.6 Summary....................................................................................................22
1.7 Glossary......................................................................................................23
References............................................................................................................29
3
4 ◾ Data Analytics
1.2 Artificial Intelligence
The two main disciplines involved predate computers by quite a margin. They are
artificial intelligence (AI) and classical statistics. Interest in AI extends historically
all the way back to classical antiquity. Once the Greeks convincingly demonstrated
that thought and reason are basic physical processes, it was hoped that machines
could also be built to demonstrate thought, reason, and intelligence.
One of the key milestones in the development of AI came from the well-known pio-
neer of information systems, Alan Turing. In 1950, Turing suggested that it should be
possible to create a learning machine that could learn and become artificially intelligent.
A couple of years after this, people began to write programs to play games
with humans, such as checkers on early IBM computers (rather large computers
at that, as the transistor hadn’t been invented yet, and Integrated Circuits (ICs)
and Complementary metal-oxide-semiconductors (CMOS) were so far ahead in
the future that nobody had given the matter much thought).
Leading thinkers, such as John von Neumann, long advocated thinking of (and
designing) computer and information systems architectures based on an under-
standing of the anatomy and physiology of the brain, particularly its massively
parallel nature. In the Silliman lectures of 1956, which the dying von Neumann
was too ill to give, this approach is set out in detail. The lectures were published
posthumously as The Computer and the Brain, (1958) by Yale University Press, New
Haven and London [1].
In the year 1957, Frank Rosenblatt, a research psychologist, invented the
Perceptron, a single layer neural network while working at the Cornell Aeronautical
Laboratory. In Figure 1.2, we can see Dr. Rosenblatt’s original Perceptron design.
6 ◾ Data Analytics
LEARNS BY DOING
In the first fifty trials, the machine made no distinction between them.
It then started registering a Q for the left squares and O for the right squares.
8 ◾ Data Analytics
Dr. Rosenblatt said he could explain why the machine learned only in highly
technical terms. But he said the computer had undergone a self-induced
change in the wiring diagram.
The first Perceptron will have about 1,000 electronic association cells
receiving electrical Impulses from an eye-like scanning device with 400
photo-cells. The human brain has 10,000,000.000 responsive cells, including
100,000,000 connections with the eyes.
Interestingly, the IBM 704 computer they were using was state-of-the-art and would
be worth over $17M USD in today’s money. Five months later, in the December
6th issue of the New Yorker an article appeared where Dr Rosenblatt was also
interviewed [3]. The article was imaginatively entitled Rival. He was quoted saying
similar things to what had appeared in the earlier New York Times article. This was
the first true hype around AI. This hype when it finally came off the rails was also
the cause of the first AI Winter.
There is absolutely no doubt that the Perceptron was an astonishing intellectual
and engineering achievement. For the first time ever, a non-human agent (and non-
biological for that matter) could actually demonstrate learning by trial and error.
Rosenblatt published a paper (1958) [4] and a detailed report, later published as
a book on the Perceptron and used this in his lecture classes for a number of years
(1961, 1962) [5,6].
these days), and they also pulled the plug on major research funding. In the 1960s,
enormous amounts of money had been given to various AI researchers (Minsky,
Rosenblatt, etc.) pretty much to spend as they liked.
In 1969, the Mansfield Amendment was passed as part of the Defense
Authorization Act of 1970. This required the DARPA to restrict its support for basic
research to only those things that were directly related to military functions and
operational requirements. All major funding for basic AI research was withdrawn.
While it is true that there had been a number of high profile failures, in hindsight
it can be seen that the optimism was partly correct. Today almost all the advanced
technologies have elements of AI within them, as Ray Kurzweil has remarked in his
2006 book The Singularity Is Near: “many thousands of AI a pplications are deeply
embedded in the infrastructure of every industry.” [9]
To Kurzweil, the AI winter has ended, but there have been two very highly sig-
nificant ones, the first from 1974 to 1980 and the second from 1987 to 1993. There
have also been a host of minor incidents or episodes, e.g., the failure of machine
translation in 1966 and the abandonment of connectionism in 1970.
It is still the case, apparently, that venture capitalists as well as government
officials get a bit nervous at the suggestion of AI, and so most of the AI research
undertook a rebranding exercise. They also refocused research on smaller, more
tractable problems. This reminds one of what Dennis Ritchie said about why he
invented the UNIX operating system–to concentrate on doing one thing at a time
and doing it well.
écarter le moins possible. This sentence translates to English as “This shows that the
least squares method demonstrates the center around which the measured values
from the experiment are distributed so as to deviate as little as possible.”
Gauss also published on this method in 1809 [12]. A controversy erupted (simi-
lar to the one between Newton and Leibnitz about the primacy of invention of
the calculus). It turns out that although it is highly likely that Gauss knew about
this method before Legendre did, he didn’t publish on it or talk about it widely,
so Legendre can be rightly regarded as the inventor of the least squares method
(Stigler, 1981) [13].
Stigler also states: “When Gauss did publish on least squares, he went far beyond
Legendre in both conceptual and technical development, linking the method to
probability and providing algorithms for the computation of estimates.”
As further evidence of this, Gauss’s 1821 paper included a version of the Gauss–
Markov theorem [14].
The next person in the story of least squares regression is Francis Galton. In
1886, he published Regression towards Mediocrity in Hereditary Stature, which was
the first use of the term regression [15]. He defined the difference between the height
of a child and the mean height as a deviate.
Galton defined the law of regression as: “The height deviate of the offspring
is on average two thirds of the height deviate of its mid-parentage” where mid-
parentage refers to the average height of the two parents. Mediocrity is the old term
for average. Interestingly, it has taken on a vernacular meaning today as substan-
dard; however, the correct meaning is average. For Galton, regression was only ever
meaningful in the biological context.
His work was further extended by Udny Yule (1897) [16] and Karl Pearson
(1903) [17]. Yule’s 1897 paper on the theory of correlation introduced the term
variables for numerical quantities, since he said their magnitude varies. He also used
the term correlation and wanted this to be used instead of causal relation, presaging
the long debate that correlation does not imply causation. In some cases, this debate
still rages today.
Yule and Pearson’s work specified that linear regression required the variables to
be distributed in a Gaussian (or normal) manner.
They assumed that the joint distribution of both the independent and depen-
dent variables was Gaussian; however, this assumption was weakened by Fisher, in
1922, in his paper: The goodness of fit of regression formulae, and the distribution of
regression coefficients [19].
Fisher assumed that the dependent variable was Gaussian, but that the joint
distribution needn’t be. This harkens back to the thoughts that Gauss was express-
ing a century earlier.
Another major advance was made by John Nelder and Robert Wedderburn
in 1972, when they published the seminal paper “Generalised Linear Models” in
the Journal of the Royal Statistical Society [20]. They developed a class of general-
ized linear models, which included the normal, poisson, binomial, and gamma
distributions.
The link function allowed a model with linear and nonlinear components. A
maximum likelihood procedure was demonstrated to fit the models. This is the way
we do logistic regression nowadays, which is the single most important statistical
procedure employed by actuaries and in modern industry (particularly the finance
industry).
The multiple linear regression model is given by
where
yi is the dependent variable for the ith observation
β 0 is the Y intercept
β1, β2,…, βk are the population partial regression coefficients
x1i, x 2i,…,xki are the observed values of the independent variables x1, x 2,..., xk
and k = 1, 2, 3,..., K are the explanatory variables
The model written in vector form is shown as
y = Xθ + ε (1.2)
E ( ε ) = 0 (1.3)
This embodies the assumption that the εj are all uncorrelated, i.e., they have zero
means and the same variance σ2.
12 ◾ Data Analytics
µj = ∑x θ ,
i =1
ij i j = 1, 2,…, n (1.4)
2
∑j
yj −
∑
i
xijθi (1.5)
which is the sum of squares (Eq. 1.5) in the least squares method [21].
Figure 1.4 graphically demonstrates a line of best fit, i.e., the line which mini-
mizes the squared distance between all the points and the line. This is the basis of
linear regression. The slope of the line corresponds to the regression coefficients and
the intercept is the first term in our model above. Note that this corresponds to the
well-known equation of a straight line: y = ax + b, where the slope is given by a and
b is the intercept.
Regression methods have been a continuing part of statistics and, even today,
they are being used and extended. Modern extensions include things such as ridge
regression and lasso regression.
1.3.1 Rediscovery of ML
In a sense to take up the slack, expert systems came into vogue during the first
AI Winter. By the end of it, they were waning in popularity and are hardly used
these days.
After about a 15-year hiatus, ML was revived with the invention of backpropa-
gation. This took the Perceptron model to a new level. This helped break the logjam
where it was assumed that neural nets would never amount to anything. The key
paper in 1986 was by David Rumelhart, Geoff Hinton, and Ronald J Williams [22].
It was entitled “Learning Representations by Back-Propagating Errors” and was
published in Nature, the world’s most prestigious scientific journal.
In Figure 1.5, the essential architecture and the process of backpropagation
are shown. The network topology has three layers (there may be more than this,
but three layers is a fairly common arrangement). The weights propagate forward
through the layers. Each node in the network has two units: a summing function
(for the weights) and an activation unit that uses a nonlinear transfer function.
Common transfer functions include the log-sigmoid which outputs 0 to 1 and the
tan-sigmoid which outputs –1 to 1.
The weights are rolled up to predict y, and this is compared with the actual
target vector z. The difference is known as the error signal, and it is transmitted
back through the layers, which cause re-weighting that runs forward for another
comparison. The algorithm proceeds to find an optimal value for the output by
means of gradient descent. Sometimes, local minima in the error surface can
disrupt the process of finding a global minimum, but if the number of nodes is
increased, this problem resolves, according to Rumelhart, Hinton, and Williams
(1986).
Research on neural nets exploded. Three years after that, Kur Hornik, Maxwell
Stinchcombe, and Halber White published Multilayer Feedforward Networks Are
Universal Approximators [23]. As they said, “backpropagation is the most common
neural net model used today it overcomes the limitations of the Perceptron by using
a combination of a hidden layer and a sigmoid function.”
Development in neural nets has expanded enormously, and today the current
interest is in deep belief networks, which are neural networks with multiple hid-
den layers and are topologically quite complex. They have enjoyed recent success
in allowing images to be classified accurately as well as in many other use cases.
Today, it is fair to say that Google makes an extensive use of neural nets (almost
exclusively) and they have also open-sourced such important technologies as
TensorFlow and Google Sling, which are natural language frame semantics parsers
of immense power.
1945 von Neumann A draft report that John von Neumann wrote on Eckert and Mauchly’s EDVAC proposal was
architecture widely circulated and became the basis for the von Neumann architecture. All digital
electronic computers today use this architecture [24].
1950 Turing test In a famous paper, “Computing Machinery and Intelligence” published in the journal, Mind,
Alan Turing introduces what he calls The Imitation Game [25]. Later on, this becomes a more
generalized proposition which in effect states that if a person were unable to tell a
1952 Computer checkers Arthur Samuel created a program to play checkers at IBM’s Poughkeepsie Laboratory.
programs
1957 Perceptron Frank Rosenblatt invents the Perceptron. This is the first time that a machine can be said to
learn something. This invention creates a massive amount of excitement and is widely
covered in influential media (The New York Times, The New Yorker magazine, and many
others).
Some of the claims made are quite extraordinary examples of hyperbole and this will be the
direct cause of the first AI Winter.
(Continued)
16 ◾ Data Analytics
Table 1.1 (Continued) Critical Events: A Timeline
Year Event Details
1969 Perceptrons Marvin Minsky and Seymour Papert publish their book Perceptrons, which outlines a
number of limitations to the use of Perceptrons, most notably their inability to execute an
XOR function. The XOR, or exclusive OR, is a logical operation that outputs as true only
when the inputs differ (one being true, the other false).
This is in practice a severe limitation for case or conditional logic in programming. This
work was seen by many as the major trigger in bringing about the AI Winter.
1973 Lighthill report James Lighthill is commissioned by the British Science Research Council. He published this
report as “Artificial Intelligence: A General Survey” in 1973.
Lighthill was extremely (and unfairly) pessimistic about what AI research has achieved to
date. For example, he stated: “in no part of the field have discoveries made so far produced
the major impact that was then promised.”
However, he also failed to understand that the main purpose of AI research is “studying
the structure of information and the structure of problem solving processes
independently of applications and independently of its realization in animals or
humans.” (John McCarthy, 1974).
The Lighthill report was flawed; however the damage had been done, and after this, the first
AI Winter (which went for at least 15 years) began.
(Continued)
Table 1.1 (Continued) Critical Events: A Timeline
Year Event Details
1986 Backpropagation It is described by David Rumelhart, Geoff Hinton, and Ronald J. Williams, in a paper in
Nature entitled “Learning Representations by Back-Propagating Errors.”
In effect, they designed a system that involves repeated adjustment of weights of
connections in a hidden layer until they have minimized the overall difference between
the actual output vector and the desired output vector.
1995 Random forests Tin Kam Ho published the first paper describing a random forest [26]. In the abstract, it was
stated that decision trees are attractive classifiers because of their computational
efficiency, but that traditional trees often can’t be grown to an arbitrary complexity
because generalization starts to break down. In other words, overfitting occurs.
A stochastic modeling method was devised where multiple trees are built in randomly
selected subspaces of the overall feature space. These multiple trees can then be
combined to give a stable estimator.
The algorithm was demonstrated on the NIST handwriting data.
(Continued)
18 ◾ Data Analytics
Table 1.1 (Continued) Critical Events: A Timeline
Year Event Details
1995 Support vector Corinna Cortes and Vladimir Vapnik published the paper establishing support vector
machines machines [27].
In the paper, they gave a history, starting with Fisher’s linear discriminant function and then
moving on to Rosenblatt’s Perceptron and the recently discovered backpropagation
algorithm. They proposed a support vector network as a new learning machine for two-
group classification problems.
The key idea is that input vectors are nonlinearly mapped to a high dimensional feature
space, where a linear hyperplane can be constructed that gives maximum separation of the
groups. Using dot products as well as the very clever kernel trick, the solution becomes
computationally feasible. SVMs that are known these days give very good performance
and generalizability. They concluded the paper by demonstrating success with the NIST
character recognition data.
1997 Machine defeats chess Garry Kasparov is defeated by IBM’s Deep Blue chess machine.
grand master
2011 Machine wins at IBM’s Watson defeats two human experts in a Jeopardy! Competition. The system is
Jeopardy! extremely sophisticated and uses a hybrid combination of information retrieval, natural
language processing, and machine learning.
2015 Machine beats a human An expert human Go player is defeated by Google’s AlphaGo. This system is based on a
Go champion. combination of tree search and machine learning.
An Introduction to Machine Learning ◾ 19
1.5 Types of ML
There are basically four important types of ML. They are supervised learning,
semisupervised learning, unsupervised learning, and reinforcement learning.
Essentially in supervised learning a known target or label is modeled. The pur-
pose of algorithms that are used in supervised learning is to predict the likelihood
of a specific event or outcome.
A typical example of SL would be a consumer credit risk model employed by a bank.
Unsupervised learning is where the data are investigated for particular pat-
terns or structure, a common example of this is cluster analysis, commonly used
in market segmentation operations. Unsupervised learning is commonly known to
statisticians as a hypothesis-generating technique.
A typical example of UL is given by market segmentation. This has been found
to be especially valuable for marketing campaigns.
Semisupervised learning falls between supervised and unsupervised learn-
ing, as one might expect. There are a variety of approaches, and one commonly
taken is to use unsupervised techniques to ascertain a class structure and then
switch to supervised learning to verify the structure/target. This process can
occur iteratively.
A typical example of SSL would be the use of a hidden Markov model or HMM,
together with a Baum-Welch learning procedure in speech recognition [28].
Reinforcement learning is characterized by agent-based processes that use
current knowledge to make a decision where the aim is to maximize a reward.
They proceed in discrete time intervals and usually involve Markov processes.
An example of RL would be teaching a computer to play backgammon and
other games.
1.5.1 Supervised Learning
Supervised learning is the most prevalent form of ML. The key point in supervised
learning is that you have a known target or label. The modeling process involves pre-
diction of the new label or target, based on the characteristics of the dependent vari-
ables in the training set. In a sense, supervised learning is like regression in statistics. It
involves the estimation of new possible cases from known, understood targets or labels.
Most ML uses supervised learning techniques. Essentially, we have an input
vector with the goal of approximating the mapping function so well that we can
predict the output with a high degree of precision.
The two subtypes of supervised machines are classification and regression.
Classification is when the output variable is a category that is binary, ternary, or a
higher number. Either way it is categorical (grouped outcome). The point of regres-
sion is the output variable which is a real number, such as cost or height. Good
examples of supervised ML techniques include linear regression, random forests,
and support vector machines.
Let’s look at the different types of supervised learning algorithms.
20 ◾ Data Analytics
1.5.2 Unsupervised Learning
By contrast, unsupervised learning is a technique in which you have inputs that
have no specific output. The goal is to understand the structure or groupings in the
data. There are two subtypes of unsupervised learning: clustering and association.
Clustering is used to discover inherent groupings in the data; usually confirmatory
discriminant analysis is required as a follow-up to determine the validity of the
discovered structure.
Association rules are about co-occurrence of different attributes. Recommender sys-
tems are usually based on this type of analysis. Past purchase co-occurrences are used to
suggest appropriate additional purchases to a customer. Amazon uses such techniques
very heavily. The other term for association rules is market basket analysis. This has
been used extensively in retail settings, particularly in store layout and redesign.
scikit-learn unsupervised learning algorithms include the following:
1. Gaussian mixture models
2. Manifold learning
An Introduction to Machine Learning ◾ 21
3. Clustering
4. Biclustering
5. Decomposing signals in components (matrix factorization problems)
6. Covariance estimation
7. Novelty and outlier detection
8. Density estimation
9. Neural network models (unsupervised)
1.5.3 Semisupervised Learning
Semisupervised learning is a hybrid of the two techniques: supervised and unsuper-
vised learning. It is commonly used where it is impractical or too expensive to label
all of the input data so we only have some of it labeled.
One approach to semisupervised learning involves using the known labeled data
and projecting it into the other data attributes and then testing to see how good this
is. The second approach involves using unsupervised learning to discover structure,
then validating how closely the assigned structure matches with the existing labeled
data. Both approaches converge on a similar outcome.
Support vector clustering can be used for semisupervised learning. This is often
used in industrial applications either when data is not labeled or when only some
data is labeled as a preprocessing operation for a classification pass. Another good
example of semisupervised learning is in the use of HMM, which is commonly
employed with a Baum-Welch learning procedure. This is used in both speech
recognition and bioinformatics.
1.5.4 Reinforcement Learning
In reinforcement learning the most important thing is characterization of the learning
problem. A key difference between reinforcement learning and supervised learning
is that outcomes are not determined, but the system learns from its own experience.
Two competing processes are set up, commonly referred to as exploitation: the
reapplication of a prior decision and exploration and the search through the solu-
tion space for a better performing decision function. These two operate against each
other in a classic trade-off, and the configuration of agents is important to optimize
both, otherwise convergence is unlikely.
22 ◾ Data Analytics
A key aspect of this learning method is the discrete time dimension. The major
enabling algorithmic process is a Markov decision process.
Currently, there is a tremendous interest in reinforcement learning. It can
be applied in numerous disciplines and contexts, e.g., control engineering, game
theory, and economics.
An example of reinforcement learning would be to set up a computer to play
backgammon or other games. Building elevator control is another use case.
1.6 Summary
We covered an introduction to both the nature and history of ML. The complex
and, sometimes, convoluted way in which things developed was briefly discussed,
especially the development of AI, which has a fascinating history of stops and starts,
i.e., the so-called AI winter where funding was literally cut off midstream. This is
not uncommon with other types of computer technology development, but it was
especially pronounced in the case of AI. Neural net methods are well established
in the mainstream these days and form the major part of what is being done at
Google, among other tech giants.
We looked at one of the most important methods in statistics, least squares
regression—how it came about historically and just how it works. Statistics and ML
have a long and intertwined history.
The critical timeline from 1945 (the dawn of the Computer and Atomic Age) to
2015 with the defeat of a Go master by a computer gives us a detailed understand-
ing of some of the broad sweep of history.
Lastly, we had a brief exposition on the various types of ML, including super-
vised, unsupervised, semisupervised, and reinforcement learning. Finally, some of
the algorithms available in the scikit-learn Python ML library were listed to give one
an idea of where to start the journey to learn how to apply these methods. An entire
and detailed book could be written on this topic, but hopefully the interested readers
will have gained some idea about where to start their search for further knowledge.
An Introduction to Machine Learning ◾ 23
1.7 Glossary
(Continued)
24 ◾ Data Analytics
(Continued)
An Introduction to Machine Learning ◾ 25
(Continued)
26 ◾ Data Analytics
(Continued)
An Introduction to Machine Learning ◾ 27
(Continued)
An Introduction to Machine Learning ◾ 29
References
1. Von Neumann, John (1958) The Computer & the Brain, Yale University Press, New Haven
and London.
2. Anonymous (1958) New Navy Device Learns by Doing; Psychologist Shows Embryo of
Computer Designed to Read and Grow Wiser, The New York Times, July 1958, 25.
30 ◾ Data Analytics
3. Mason, Harding, Stewart, D., Gill, Brendan (1958) The Talk of the Town: Rival, The New
Yorker, December 1958 Issue, pp. 44–45.
4. Rosenblatt, Frank (1958) The Perceptron: A probabilistic model for information storage
and organization in the brain, Psychological Review, Vol. 65, No. 6, pp. 386–408.
5. Rosenblatt, Frank (1961) Principles of Neurodynamics: Perceptrons and the Theory of Brain
Mechanisms, Cornell Aeronautical Laboratory, Inc., Cornell University, New York.
6. Rosenblatt, Frank (1962) Principles of Neurodynamics: Perceptrons and the Theory of Brain
Mechanisms, Spartan books, Washington, DC, pp. xvi, 616.
7. Minsky, Marvin and Papert, Seymour (1969) Perceptrons: An Introduction to Computational
Geometry, MIT Press, Cambridge, MA.
8. Lighthill, James (1973) Artificial Intelligence: A General Survey, Artificial Intelligence:
A Paper Symposium, Science Research Council.
9. Kurzweil, Ray (2006) The Singularity Is Near, Viking Press, New York.
10. Jordan, Michael Irwin (2014) Comments in Relation to Statistics and Machine Learning,
www.reddit.com/r/MachineLearning/comments/2fxi6v/ama_michael_i_jordan/; 09 Sep
2014.
11. Legendre, Adrien-Marie (1805, 1806) Nouvelles Méthodes pour la Détermination des
Orbites des Comètes, Courcier, Paris.
12. Gauss, Carl Friedrich (1809) Theoria Motus Corporum Coelestium in Sectionibus Conicis
Solem Ambientum, Friedrich Perthes and I.H. Besser, Hamburg.
13. Stigler, Stephen M. (1981) Gauss and the invention of least squares, The Annals of Statistics,
Vol. 9, No. 3, pp. 465–474.
14. Gauss, Carl Friedrich (1821) Theoria Combinationis Observationum Erroribus Minimis
Obnoxiae, H. Dieterich, Gottingen.
15. Galton, Francis (1886) Regression towards mediocrity in hereditary stature, The
Journal of the Anthropological Institute of Great Britain and Ireland, Vol. 15, No. 1886,
pp. 246–263.
16. Yule, George Udny (1897) On the theory of correlation, Journal of the Royal Statistical
Society, Vol. 60, No. 4, pp. 812–854.
17. Pearson, Karl (1903) The law of ancestral heredity, Biometrika, Vol. 2, No. 2, pp. 211–236.
18. Pearl, Raymond (1910) Letter Raymond Pearl letter to Karl Pearson, about disagreement on
hereditary theory and his removal as an editor of Biometrika (2/15/1910), www.dnalc.org/
view/12037-Raymond-Pearl-letter-to-Karl-Pearson-about-disagreement-on-hereditary-
theory-and-his-removal-as-an-editor-of-Biometrika-2-15-1910-.html.
19. Fisher, Ronald Aylmer (1922) The goodness of fit of regression formulae, and the distribu-
tion of regression coefficients, Journal of the Royal Statistical Society, Vol. 85, No. 4, pp.
597–612.
20. Nelder, John and Wedderburn, Robert William Maclagan (1972) Generalised linear mod-
els, Journal of the Royal Statistical Society. Series A (General), Vol. 135, No. 3, pp. 370–384.
21. Stuart, Alan and Ord, J. Keith (1991) Kendall’s advanced theory of statistics. Volume 2,
Classical Inference and Relationship, 5th edition, Edward Arnold, London.
22. Rumelhart, David; Hinton, Geoff and Williams, Ronald J. (1986) Learning representa-
tions by back-propagating errors, Nature, Vol. 323, pp. 533–536.
23. Hornik, Kur; Stinchcombe, Maxwell and White, Halber (1989) Multilayer feedforward
networks are universal approximators, Neural Networks, Vol. 2, pp. 359–366.
An Introduction to Machine Learning ◾ 31
24. Von Neumann, John (1945) First Draft of a Report on the EDVAC, Contract No.
W–670–ORD–4926 Between the United States Army Ordnance Department and
the University of Pennsylvania, Moore School of Electrical Engineering, University of
Pennsylvania, June 30, 1945.
25. Turing, Alan Mathison (1950) Computing machinery and intelligence, Mind, Vol. LIX.
No. 236, pp. 433–460.
26. Ho, Tin Kam (1995) Random Decision Forests, Proceedings of the Third International
Conference on Document Analysis and Recognition. Montreal, Quebec: IEEE. 1: 278–282.
27. Cortes, Corinna and Vapnik, Vladimir (1995) Support-vector Networks, Machine Learning,
Kluwer Academic Publishers, Dordrecht, 20 (3), pp. 273–297.
28. Baum, Leonard E., Petrie, Ted, Soules, George and Weiss, Norman (1970) A maximiza-
tion technique occurring in the statistical analysis of probabilistic functions of Markov
chains, The Annals of Mathematical Statistics, Vol. 41, No. 1, pp. 164–171.
29. Bellman, Richard (1957) A Markovian decision process, Journal of Mathematics and
Mechanics, Vol. 6, No. 5, pp. 679–684.
Chapter 2
Regression for
Data Analytics
M. Saiful Bari
Southeast University
Nanyang Technological University
Contents
2.1 I ntroduction............................................................................................... 34
2.1.1 Chapter Roadmap.......................................................................... 34
2.1.2 What Is Regression?.........................................................................35
2.2 Linear Regression........................................................................................35
2.2.1 Dataset Description.........................................................................36
2.2.2 Problem Definition..........................................................................36
2.2.2.1 To Wrap Up the Whole Thing...........................................39
2.2.3 Probabilistic Interpretation............................................................. 40
2.2.4 Optimization Method.................................................................... 44
2.2.5 Block Diagram................................................................................45
2.2.6 Overview of the Model................................................................... 46
2.3 Logistic Regression......................................................................................47
2.3.1 Problem Definition..........................................................................47
2.3.2 Logistic Function.............................................................................47
2.3.3 Probabilistic Interpretation..............................................................48
2.3.4 Optimization Method.....................................................................49
2.3.5 Overview of the Model....................................................................49
2.4 Problems of Regression................................................................................50
2.4.1 Underfitting and Overfitting...........................................................50
2.4.2 Outlier.............................................................................................50
33
34 ◾ Data Analytics
2.4.3 Hyper-Parameter.............................................................................51
2.5 Conclusion..................................................................................................52
References............................................................................................................53
2.1 Introduction
The world is getting smaller, but in the process of doing so, we are producing a lot
of data (by using data oriented technology, i.e. social network, e-commerce etc.).
Throughout the development of modern civilization, we can divide the whole pro-
cedure into multiple stages. Agriculture (around 1980), industrialization (around
1900–1960) and information technology (around 1960–2007) are the major foot-
prints of human civilization. Now we have entered into the fourth industrial revolu-
tion: artificial intelligence. In the next decade, the advancement of technology and
programming philosophy of the existing field will change drastically and we will get
much closer to the human level intelligence for a lot of tasks without actually telling a
computer what to do. It will be possible only through the amount of data that we are
producing in our day-to-day life. Philosophically, people will develop intelligent sys-
tems by learning from their previous stories and activities. To make a long story short,
people will be able to analyze the huge amount of data more systematically so that
they can take the correct decision based on statistical data. The stage of information
technology gives us a complete platform to gather the necessary data through Internet
of Things (IoT) devices and products. We will explore new methods in the coming
decades to gather and take decisions and solve some major problems of our society.
To be able to cope up with the change that is happening right now, we should
have proper knowledge of understanding data and analyze it so that we can take a
more educative guess, which will be so perfect that we cannot differentiate it from
that of a human. This art of exploration is known as data science. Analyzing big
data is more difficult in terms of technique and usability because of the need for
memory and computational power. To be able to do that, we need more sophisti-
cated algorithms that can also be implemented in real life.
Data science is not “rocket science.” In response to a bigger picture of the relevant
subject, we can safely assume it to be the analysis of pattern exploration in higher
dimension—the dimension that can only be visualized by vectors. Although data sci-
ence is full of attractive, mouthful names, the basic intuitions are so simple that anyone
can grasp the beauty of this so-called MAGIC, given a proper and focused direction.
2.1.1 Chapter Roadmap
This chapter aims to provide the fundamental idea of a data exploration methodol-
ogy, regression. Without a proper understanding of regression, we will not be able
to go further and explore more opportunities to play with the data. Data science is
nothing but exploiting the features of machine learning with proper visualization.
No matter what advanced model [1–3] we might develop in the future, the core of
Regression for Data Analytics ◾ 35
the model will lie in understanding regression. At the start of the chapter, we will
be learning about linear regression and its intuitions. We will be asking the ques-
tion “why?” instead of “how?” We will see some examples and try to understand
what is going on. Later on, we will develop some mathematical thinking based on
our intuitions and begin the question “how?” Finally, we will derive the complete
probabilistic analysis of linear regression. Then we will add a condition on the
linear regression model and see how we can derive logistic regression from linear
regression. We will also discuss the probable mathematical solution of optimization
techniques, though we will not go into details of their optimization algorithms.
2.1.2 What Is Regression?
Regression is the process of estimating a relationship between variables. An event can
be represented by statistical data where multiple variables participate to explain sub-
events. Sometimes these variables might have a correlation between them. Say, given
that you have sufficient knowledge about the perspective, if you know the age and
geographical area of a certain person, you may guess the height of that person in most
cases. If you consider age, geographic area, and height as variables, then you can corre-
late them to guess one value, given other values. Regression helps us to do such things.
In relevance to data science, we will look into linear regression and logistic
regression.
2.2 Linear Regression
When the variables are linearly dependent on each other, we can estimate one
variable from the other by linear regression. Assume X and Y are two variables.
If they are linearly dependent on each other, we can write
Y ∝X
(2.1)
Y = mX
If we write the equation in the form of an equation of a line (for proper mathemati-
cal analysis), then it will become
Y = mX + b (2.2)
So, the relation between X and Y can be defined by m and b. Now given enough sta-
tistical data, our objective will be finding a suitable m and b that support the statisti-
cal data most of the time. But why do we want to do that? Remember that we want to
learn from our actual data. Before going through the formal process, let’s have some
good intuitions about what types of problems we will be solving by linear regression.
A classic example of linear regression is housing price estimation [4]. Suppose
you have some detailed statistical information such as house size, door type, utilities,
36 ◾ Data Analytics
style, roof type, number of kitchens, and most importantly price. The variable we
want to predict is called label. In this case, the price of the house is the label and the
rest of the variables are called feature(s). So, house size, door type, utilities, style,
roof type, number of kitchens are features.
Let’s consider another interesting example. Suppose you have the honest data-
set of a student’s total daily average time of study in hours. Can you estimate the
student’s grade (in number) from the amount of study hours? So, here the student’s
grade is the label and the study hour is the feature.
Linear regression is a supervised problem. You must have the label on the data
for training. For housing price estimation, we need the price of the house and for
student grade estimation, we need the actual grades of the students.
2.2.1 Dataset Description
Let’s assume you have the relevant data. Now our challenge is to generate a function
f and feed the function with data in a proper way and expect some results. Let X be
our data and Y be the output. So, we can write
f : X → Y (2.3)
Our objective is to design this function f. Before designing such a function, let’s
introduce our dataset in a more definitive manner:
{ xi , yi }i =1
N
where
2.2.2 Problem Definition
Given a set of pair { xi , yi }i =1 where yi is the dependent variable against xi, can you
N
Figure 2.1 (a) Some points are drawn in the 2D coordinate system. The x-axis is
the feature and the y-axis is the response variable. (b) A regression line is drawn
based on the given points. (c–f) a new point (pointed as x) is added in each of the
figures, and based on the new points, a new regression line is drawn.
based on the correlation of ten points that are in Figure 2.1a. From Figure 2.1c–f,
each time a new point (showed as cross) is added, a new regression line is drawn.
Notice how the line changes direction while adding a new point in Figure 2.1c–f.
These four continuous figures help us to understand how the correlation is repre-
sented by the regression line. See Figure 2.1f more carefully. An outlier (strongly
oppose the correlation) point causes a bad regression line. We will see some tech-
niques at the end of the chapter to resolve the issue. For the time being, let’s make
38 ◾ Data Analytics
sure that we get an idea of how to represent the correlation of points with the
regression line.
For any arbitrary given x̂ , the value ŷ ( (hat) represents it’s not from the
dataset) on the line is the estimation. We call this line as the best fitted line (consid-
ering the given points). The line actually explains the best linear relationship for the
given data points. As we learn the line (How? There’s no need to think about that
now.) that shows the relationship between x and y of the data points, we can use
this relationship (equation of line) to generate ŷ for any given x̂ . From Figure 2.1,
now we understand that we have to draw some line (line → linear) to estimate some
points (as they have linear relation).
Assume we have higher dimensions of data. Like, for each x, it’s not one point.
There could be D points. Note that higher dimension does not mean a higher order
of relationship. Two D-dimensional points could have a linear relationship. Just
think about it a bit and don’t skip this if you have any confusion.
When you are expressing something in the higher dimension, the only good
mathematical tools we have are the vector and linear algebra operations. For the rest
of the explanation, we will be explaining everything in terms of vector. If you are
weak in the basics of linear algebra, now it is a good time for a recap.
If we express the equation of a line in linear algebra, it will be as follows:
T
w0 x0
w1 * x1
Y = = W T X (2.4)
w D x D
∈= Y − Yˆ (2.5)
where the superscript T represents the transpose operation and * represents the
matrix multiplication. X = [ x0 x1 x 2 … x D ] is the input of the model. You need
T
to find a vector W such that Y gets its original value in the dataset. In short, W actu-
ally relates X to Y. Yˆ is the predicted value (the value our model will generate) for
X and Y is the original label in the dataset. Assume ∈ is the residual error. Notice
W T *X is scalar. Now here is the trick. If we take a random matrix for W (as we
don’t know the relation between X and Y, we initialize it randomly), the probabil-
ity will be very low that W T *X can estimate the correct Yˆ that is close to Y. It will
predict something greater or lesser than our expected output Y. Mathematically,
Yˆ Y or Yˆ Y . So, what can we do? We can just add an error ∈term to make the
prediction correct. Let’s write the equation:
Yˆ = W T *X + ∈ (2.6)
Regression for Data Analytics ◾ 39
Yˆ = W T *X + ∈ (2.7)
For a random initialization of W, our residual error ∈will be much higher. We need
to change (iteratively or algebraically) W in such a way so that the residual error ∈
gets lower. We can only calculate the residual error ∈ when we have a W (guessed
or calculated):
∈= Yˆ − W T *X (2.8)
Now intuitively, it’s quite clear that we need to optimize the residual error ∈or some
form of the residual error. For analyzing which function of ∈needs to be optimized,
let’s interpret the model by probability distribution.
Figure 2.2 Crosses are the predicted values on the regression line. The line from
the given points to cross shows the distance between the predicted and given
points.
40 ◾ Data Analytics
2.2.3 Probabilistic Interpretation
For a data scientist, the main objectives are to implement and visualize the model cleverly
so that one may get the chance to extract useful information from the predictive model.
For that a basic mathematical understanding of the model is the building block. For
readers’ suitability, the mathematical explanation in this chapter is derived in a simple X
manner and in details without skipping any c alculation. The readers are encouraged to
reason every mathematical form of explanation from this chapter as these mathematical
understandings will be the building blocks for the more advanced models.
There are two types of probabilistic model:
1.
Generative model: A generative model asks the question—which category
is most likely to generate the data? This interpretation works well only with
large amounts of data. It calculates the complete probability model, perform-
ing inference condition on the data and model.
2.
Discriminative model: A discriminative model doesn’t care about how data
is generated, but it only classifies the data. Its elegance is simplicity. Compared
to the generative approach, most of the time discriminative model can work
with small amounts of data.
For the problem discussed previously, we can decide that we need a discriminative
probabilistic model. As we are not working on data generation, only the discrimina-
tive model will suffice.
We can represent the problem as p ( y | x ) that implies,
what is the probability of outcome y given an input x?
We assume that residual error ∈ follows normal distribution ( ) with zero
mean (µ = 0) and constant variance (σ), which implies the conditional probability
of y given x:
( )
∈~ µ = 0, σ 2 (2.9)
( {
p y | x, W , σ 2 }) = ( y | µ ( x ) , σ 2
( x )) (2.10)
Here, σ and W are the parameters of the model. σ is constant (constant noise).
For linear regression, we assume that µ ( x ) = W T x . Suppose the input is one-
dimensional. Then we define μ(x) as
µ ( x ) = w1x + w0 = W T x (2.11)
where
W T = [ w0 w1 ]
1
x=
x
Regression for Data Analytics ◾ 41
mentation of Eq. (2.2). Note that x is a vector and x is a scalar. The conversion of
x to x is called basis function expansion. With basis function expansion, we can
make a linear model nonlinear. Basis function is generally written as ϕ(x). General
basis function is
φ ( x,d ) = 1x x 2 … x d
Figure 2.3 A regression line is drawn. Based on Eq. (2.10), each point on the
line is calculated based on the zero mean of the normal distribution of the points
around it.
42 ◾ Data Analytics
Inductive bias is the assumption that your model takes into account before learning/
computation. The total likelihood is the product of the likelihood for each point yi
p ( y ,θ ) = ∏ p ( y ,θ ) (2.12)
i
i
p ( y ,θ ) = l (θ ) = ∑ log ( p ( y ,θ )) (2.13)
i
i
1
1 − ( x − µ )T Σ −1 ( x − µ )
p ( x, θ ) = ( µ ,σ ) = e 2 (2.14)
( )
d
(2π det Σ )
d d 1
log p ( x,θ ) = − log ( 2π ) − log ( det Σ ) − ( x − µ ) Σ −1 ( x − µ ) (2.15)
T
2 2 2
As we are working on 1D data points (two values in x), our p.d.f for normal
distribution will be
1 − 2σ 2 ( y −W T x )
1 2
(
µ ,σ 2
) =
2πσ 2
e (2.16)
(
l (θ ) log p ( D|θ ) = ) ∑ i =1
p ( yi | x i , θ ) = ∑ log ( N ( µ,σ ))
i =1
2
− 2 ( yi −W T x i )
N
1 1 2
l (θ ) ∑
i =1
log
2πσ
2
e 2σ
N 1
−l (θ )
2
(
log 2πσ 2 + 2 RSS (W )
2σ
)
Regression for Data Analytics ◾ 43
N N
∑ ( y − W x ) = ∑ ∈ (2.17)
2
RSS (W ) T 2
i i
i =1 i =1
Here, RSS means residual sum of squares. This is also called Sum of squared errors
(SSE). If we take the mean, it will be
1
MSE = SSE (2.18)
N
MSE is the mean squared error or l2 norm. This is considered as the loss of the model
(remember ∈; We finally get the function of ∈ that must be reduced/optimized
according to Eq. (2.8)).
For the discriminative model, we calculate the maximum likelihood estimation
(MLE) θˆ to estimate the value. Minimizing error is equivalent to maximizing like-
lihood as the likelihood defines how likely it is for y to be selected for a given x,
which means that we can also calculate the value of the parameter θ for which the
loss will be minimized. According to Eq. (2.17), parameter θ is W .
µ
( )
θˆ arg max log ( µ , σ 2 ) = arg max l (θ )
θ
Note that for a better model, we need to reduce the residual error ∈ according to
Eq. (2.5). Reducing ∈is the same as maximizing likelihood. Maximizing likelihood
is the same as maximizing log likelihood and minimizing negative log likelihood.
If we can convert the same problem to a minimization problem by taking
negative log-likelihood,
θˆ arg min ( −l (θ ))
θ
N N
∑( ) ∑∈
2
θˆ = arg min
2
yi − W T x i = (2.20)
w
i =1 i =1
Remember at the end of Section 2.2.2 that we need to optimize the residual
error. Equation (2.20) is the final form of the equation that we need to optimize.
44 ◾ Data Analytics
This equation also works for a higher dimension. You may need to tweak the size of
W according to the dimension of x, for a higher dimension.
After optimizing the loss function, we will have such a representation of W that
maps the data X to Y. Actually, W will contain such a value for which if you take
a new example that is not in the dataset, W will be able to estimate the value of
y based on its experience (training process) with the dataset. The accuracy of the
linear regression will depend on the size of the dataset and the design of the model.
In a real-life scenario, linear regression is not the feasible solution. You may need
to try nonlinear regression or extend the concept of linear regression to further
advanced models.
2.2.4 Optimization Method
For the calculation of arg min part, we need an optimization method. There are
two types of optimization methods for linear regression.
1.
Iterative method: Use some learning algorithms such as gradient descent,
stochastic gradient descent [5], Adam [6], and Newton’s optimization
method.
2.
Closed-form solution: Use vector algebra to derive the solution.
The discussion of the learning algorithms such as stochastic gradient descent, Adam
etc. is out of the scope of this chapter. In this chapter, we will be discussing the
closed form solution. For the closed-form solution, we will be doing some algebraic
derivation and we will try to minimize negative log-likelihood by vector algebra.
From Eq. (2.15), taking negative log-likelihood,
d d 1
−l (θ ) = log ( 2π ) + log(det Σ ) + ( x − µ ) Σ −1 ( x − µ ) (2.21)
T
2 2 2
d d
For our case, log ( 2π ) + log ( det Σ ) and Σ−1 terms are constant. So, we can
2 2
ignore these terms. Remember, for our model, µ ( x ) = W T x . Our reduced negative
log-likelihood (NLL) term will be (In Eq. [2.21], x is the random variable. In our
problem, y is the random variable that we want to predict. Equation [2.21] is just
an example of normal distribution.):
−l (θ ) = NLL (W ) = ( y − xW )( y − xW ) as W T x = xW
With some vector algebra, we can easily simplify the equation as follows:
NLL (W ) = ( y − xW ) ( y − xW )
T
Regression for Data Analytics ◾ 45
(
= yT − ( xW )
T
) ( y − xW ) as ( A + B ) T
= AT + BT
(
= yT − W T xT ) ( y − xW ) as ( AB ) T
= BT AT
= yT y − yT xW − W T xT y + W T xT W
= yT y − 2W T xT y + W T xT xW as AT ( BC ) = ( BC ) A
T
NLL ( w ) = yT y − 2W T xT y + W T xT xW (2.22)
θˆ = arg min ( yT y − 2W T xT y + W T xT xW )
W
δ
NLL (W ) = −2xT y + xT xW
δw
For minimization according to calculus,
−2xT y + xT xW = 0
( ( ) x y)
W = 2 xT x
−1 T
θˆ = (( x x ) x y )
T −1 T
(2.23)
2.2.5 Block Diagram
Block diagram is a way of representing the model in a brief way. In this diagram,
the variables that are open to access by the model are shaded.
Step 1: At the beginning of the experiment, feature X is observed. We assume
W is the transformation matrix such that multiplying W T with X, we get Y. But
W is unknown (initialized randomly). So, at this stage, W is hidden (Figure 2.4).
Step 2: Together with X and Y, we train the model to estimate the best fitting.
At this point, Y is observed and W is converging to its best solution (Figure 2.5).
Step 3: Now with trained W and a given X̂ , we can estimate any Yˆ . As W is
trained, we can access W. At this stage, there’s no need to access Y (Figure 2.6).
46 ◾ Data Analytics
Figure 2.6 Now after training W, we can predict Y for any given X.
2.3 Logistic Regression
Linear regression is a regression model where the dependent variable (DV) is
continuous. Logistic regression or logit regression or logit model is a regression
model where the dependent variable (DV) is categorical.
Suppose you have the last 5 years’ daily weather information (if it is rainy or
not) of a specific geographic region. Can you estimate if tomorrow’s weather will be
rainy or not (yes or no question)?
Another example will be, suppose you have the honest dataset of a student’s
total daily average time of study. Can you predict if the student will pass the exam
or not from his/her amount of study hours (yes or no question)?
Logistic regression is also a supervised problem. You need to have the label of
the data about which you want to predict.
2.3.1 Problem Definition
Given a set of pairs { xi , yi }i =1 where yi is the dependent binary or dichotomous
N
variable against xi, can you predict the category ŷ (either 0 or 1) for an arbitrary
given x̂ ?
Logistic regression predicts either 0 or 1. The probability distribution of linear
regression is Gaussian distribution. We will change the distribution to the Bernoulli
distribution as the Bernoulli distribution is discrete and it categorizes the data as
either 0 or 1.
We will implement the Bernoulli feature by using sigmoid function. It is also known
as logistic or logit function.
2.3.2 Logistic Function
The equation of sigmoid/logistic/logit function is
1 eη
sigmoid (η ) = (2.25)
−η
1 + e 1 + eη
The term sigmoid means S-shaped squashing function. Look at Figure 2.7.
From the figure, you may see that all the outputs of the sigmoid function are
between 0 and 1.
0 ≤ sigmoid ( x ) ≤ 1
sigmoid ( − inf ) = 0
sigmoid ( 0 ) = 0.5
sigmoid ( inf ) = 1
yˆ ∈{0,1}
2.3.3 Probabilistic Interpretation
Logistic regression is a discriminative model. The parameter descriptions are given
in the text that follows:
As logistic regression follows the Bernoulli distribution, the log-likelihood will
be the probability mass function (p.m.f) of the Bernoulli distribution.
p ( y|x,W ) = Ber ( y | µ ( x ))
φ ( x ) = x = [ 1 x ]
Regression for Data Analytics ◾ 49
where μ is the linear function of x and ϕ(x) is the basis function. As we apply the
logit/logistic function over the equation of linear regression, we call this procedure
as logistic regression.
The MLE derivation for the loss function is given below. The derivation is simi-
lar to linear regression. The only difference is the use of the Bernoulli distribution
instead of the Gaussian distribution.
N N
( )
l (θ ) log p ( |θ ) = ∑ log ( p ( y |x ,θ )) = ∑ log Ber ( y | sigmoid (W x ))
i =1
i i
i =1
T
where yi ∈{0,1}. This is also called cross entropy, which is the loss function.
We need to optimize this function.
2.3.4 Optimization Method
We cannot write down the MLE optimization of logistic regression in closed form.
So, we will have to use an iterative method such as gradient descent and Newton’s
optimization method. These iterative methods of learning will be discussed later.
yi ∈{0,1}
◾◾ Training:
– Iterative method: using a convex optimizer such as gradient descent and
Newton’s optimization method.
50 ◾ Data Analytics
The basic operation of linear and logistic regression is very similar. The difference
is the assumption or inductive bias on the probability distribution which derives
the loss function. A good example of this could be, suppose you have a huge col-
lection of labeled images of digits (0–9) [7]. Given a new image of a digit, can you
predict the digit number in it? Observe carefully, here the number of categorical
values is 10. So, is this a logistic regression? No, this is called softmax regression
or multinomial regression. Here we use multinomial distribution as the likelihood
function to predict the outcomes. The whole procedure of designing such a model
is the same, except that in place of sigmoid function, another specialized function
named softmax (for multinomial distribution) is used.
softmax ( x ) : R K → [0,1]
K
e xi
softmax ( x )i = for i = 1, 2,…, k (2.29)
∑
K
e xk
k =1
2.4 Problems of Regression
Regression is a very naïve model that is heavily dependent on data. The perfor-
mance of regression depends on how the statistical variables are correlated and,
more importantly, on the amount and quality of data. Here, we discuss some of the
specific problems of regression.
2.4.2 Outlier
Outliers are the points that oppose the correlation of the points in the dataset
(Figure 2.9). The cost function of regression is the square difference of the real value
and predicted value. We can see from Figure 2.2 that our objective is to reduce the
Regression for Data Analytics ◾ 51
Figure 2.8 The same data distribution is mapped by three different degrees of
curves. The Degree 1 curve is a straight line that cannot fit the data properly.
The Degree 15 curve perfectly fits the data, but it’s accuracy in predicting a
value correctly will be very low. But the Degree 4 curve can properly fit the data
and more accurately predict the value for some unknown data that is not in the
training/given data [8].
length of the vertical segments. But one outlier dominates the total length of the
vertical line, thus creating a bad regression curve. To attack this type of a problem,
we change the loss function a bit. The changed loss function is called Huber loss.
For calculating Huber loss, a threshold distance is taken into account. If any seg-
ment length crosses the threshold, we assume it as an outlier and take the absolute
value of the output instead of squaring. Other than that, we take half of the regular
square distance. The loss function of Huber loss is
1
2
( y − f ( x ))2 for y − f ( x ) ≤ δ
Lδ ( y , f ( x )) = (2.30)
1
δ y − f ( x ) − δ 2 otherwise
2
where δ is the threshold.
2.4.3 Hyper-Parameter
Hyper-parameters are the parameters whose values are set before the learning process
begins. In Eq. (2.4), we took a D × 1 dimensional weight matrix. Before the learning
procedure begins, you must fix the value of D. D is the number of parameters that you
will be learning by the learning algorithm. If D is higher, the computation gets higher.
If D is smaller, then we may not be able to learn the correlation between the points
properly. To solve a specific problem, we may need to check different values of D for a
better model (you may have to change other parameters of the model also). There are
also some algorithms for hyper-parameter optimization [11,12].
52 ◾ Data Analytics
Figure 2.9 (a) A set of points with their regression line. (b) An outlier is added
(marked by cross) in the graph and a regression line is drawn.
2.5 Conclusion
The main context of this chapter is to realize how different kinds of probabilis-
tic analysis lead to different kinds of regression. However, readers should also
grow strong intuitions about the applicability and about the types of problems
that regression can solve. In the era of deep learning, linear regression and logistic
regression are hardly used as a single unit. These two are considered the most pre-
liminary model of machine learning. Most advanced discriminative models (that
perform far better than these two models) actually consist of one of these two basic
models. These two are like the smallest LEGO® bricks among a lot of complex
problems. Before going to the next chapters, the readers are requested to make their
concepts clear of linear regression and logistic regression.
Regression for Data Analytics ◾ 53
References
1. Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton. ImageNet Classification with Deep
Convolutional Neural Networks, Advances in neural information processing systems,
2012.
2. Sepp Hochreiter, Jurgen Schmidhuber. Long short-term memory, Neural Computation
9(8):1735–1780, 1997.
3. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, Yoshua Bengio. Generative Adversarial Nets. Advances in
neural information processing systems, 2672–2680, 2014.
4. Dean De Cock. Ames, Iowa: Alternative to the Boston housing data as an end of
semester regression project, Journal of Statistics Education 19(3), 2011. doi = 10.1080/
10691898.2011.11889627.
5. Léon Bottou. Proceedings of COMPSTAT’2010, 177–186, 2014.
6. Diederik Kingma, Jimmy Ba. Adam: A method for stochastic optimization, International
Conference on Learning Representation, San Diego, CA, 2015.
7. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning
applied to document recognition, Proceedings of the IEEE 86(11):2278–2324, 1998.
8. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand
Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg,
Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu
Perrot, Edouard Duchesnay. Scikit-learn: Machine learning in {P}ython, Journal of Machine
Learning Research 12:2825–2830, 2011.
9. Sandra Lach Arlinghaus. PHB Practical Handbook of Curve Fitting. CRC Press, 1994,
249pp. Persistent URL (URI): http://hdl.handle.net/2027.42/58759.
10. William M. Kolb. Curve Fitting for Programmable Calculators. Syntec, Incorporated,
1984.
11. James Bergstra, Yoshua Bengio. Random search for hyper-parameter optimization,
Journal of Machine Learning Research 13:281–305, 2012.
12. James Bergstra, Remi Bardenet, Yoshua Bengio, Balazs Kégl. Algorithms for
Hyper-Parameter Optimization, Advances in neural information processing systems,
2546–2554, 2011.
Chapter 3
Big Data-Appropriate
Clustering via Stochastic
Approximation and
Gaussian Mixture Models
Hien D. Nguyen
La Trobe University
Andrew Thomas Jones
University of Queensland
Contents
3.1 Introduction................................................................................................56
3.1.1 Chapter Roadmap...........................................................................58
3.2 Stochastic Approximation Algorithm..........................................................58
3.2.1 C onvergence Result.........................................................................59
3.3 Gaussian Mixture Model............................................................................60
3.4 A n SAA for Maximum Likelihood Estimation of GMMs...........................61
3.5 Simulation Results......................................................................................63
3.5.1 P ractical Considerations................................................................. 64
3.5.2 S imulating Gaussian Mixture Models.............................................65
3.5.3 C omparisons....................................................................................65
3.5.4 Results............................................................................................ 66
3.6 MNIST Application....................................................................................68
3.6.1 Data Preprocessing..........................................................................68
3.6.2 Results.............................................................................................69
3.7 Conclusions.................................................................................................70
References............................................................................................................70
55
56 ◾ Data Analytics
3.1 Introduction
Let Zn = {Zi }i =1 be an IID (independent and identically distributed) ran-
n
( )
dom sample of n ∈ observations, where Zi = Xi ,Yi , X i ∈ X ⊂ R p and
Yi ∈ q = {1,…, q } (i ∈[n ]). Here, p, q ∈ , where is the set of natural numbers
(zero exclusive), is the set of real numbers, and (⋅) is the matrix transposition
operator. We call X i and Yi the feature vector and label of the ith observation from
the sample Z n, respectively.
We suppose that the data generating process (DGP) of Z n is unknown, but that
each observation Zi is generated by the following mechanism:
(A1) First, generate Yi from a distribution with probabilities (Yi = y ) = π y > 0,
q
where ∑π y =1
y = 1.
feature vectors X = { X i }i =1, but not the corresponding set of labels Y = {Yi }i =1 .
n n n n
Each y ∈ q can be viewed as one of the q subpopulations of the DGP. We can
then propose to estimate the realized labels y n (corresponding to xn) by some set
y*n = { y ( x i )}i =1, where each y ( x i ) ∈ q is a function that estimates the label Yi.
n
Here, we seek labelings y*n that are optimal in some sense, with respect to a math-
ematical criterion, or that are sensible with respect to some scientific hypothesis or
theory. This is the problem of clustering (or vector quantization).
A model-based approach to clustering is one where the DGP that yields either
X n is modeled directly. This approach is advocated in the well-referenced books
[1,2]. A common and popular probabilistic model used when applying the model-
based clustering approach is the Gaussian mixture model (GMM).
GMMs are one of the earliest proposed techniques for clustering and they
date back to the famous work of [3]. Since then, GMM-based methods have been
among the most influential and successful methods for solving the aforemen-
tioned clustering problem. For examples regarding applications and implementa-
tions of GMMs, refer to the works and the results of [1,2,4,5]. In particular, we
note the great success of the mclust software package [6–9] for the R statistical
programming language [10], over many years. Unfortunately, when faced with
big data (which we define in the sequel), traditional software packages, such as
mclust, which rely on older computation paradigms, such as batch expectation-
maximization (EM; [11]) a lgorithms (cf. [12]), cannot be applied due to computa-
tional and storage limitations.
Big Data-Appropriate Clustering ◾ 57
We note that model-based clustering is not the only available paradigm for solv-
ing the problem of data clustering. Examples of other popular clustering paradigms
include centroid-based clustering, hierarchical clustering, and modal clustering.
Good and instructive references regarding clustering algorithms and their applica-
tion in general include [13–16]. For brevity of exposition, we shall forgo further
discussions of such alternative clustering paradigms.
There are many available notions of big data in the literature. We choose to
define the big data analysis context via the “three Vs” [17]. The three Vs stand for
variety, velocity, and volume, and each can be defined as follows:
Variety: The data arise from multiple subpopulations that exhibit heteroge-
neous behaviors with differing idiosyncrasies.
Velocity: The data are not available as a static batch, but rather arrives to the
analyst in a continuous stream or in chunks or mini-batches.
Volume: The data are large to the point that they cannot be contained within
the storage of a single computing unit and cannot be manipulated
upon by said computer, in their entirety.
In this chapter, we consider the clustering of data using GMMs, fitted via the
maximum likelihood (ML) criterion using a bespoke stochastic approximation
algorithm (SAA). The algorithm that we utilize is a specialization of the SAA that
[18] derived for the fitting of mixtures of exponential power family distributions,
a class of distributions that include the GMM as a special case. The original algo-
rithm was developed for the fitting of the aforementioned mixture distributions in
the context of medium-sized and low-dimensional biomedical data analytic prob-
lems; see also [19]. We shall demonstrate that the SAA of [18] is also applicable in
the big data setting. That is, we demonstrate that the derived SAA allows for the
rapid fitting of GMMs in a low-memory and stream-suitable manner.
Although a powerful paradigm, the presented specialization of the SAA of [18] only
allows for the estimation of spherical-covariance GMMs. Thus, it cannot estimate the
entire spectrum of covariance matrix specifications and restrictions that software pack-
ages, such as mclust, allow. However, it is well known that clusterings via a spherical-
covariance and homoskedastic GMM, estimated via the ML criterion, and clustering
via the k-means algorithm share many similarities in algorithmic construction and in
paradigm (cf. [20]). Thus, the presented SAA can instead be considered as comparable to
the k-means algorithm of [21] and its many big data variants, such as those of [22–26].
Along with the presentation of the SAA for fitting spherical-covariance GMMs,
we also briefly describe the theoretical framework behind the SAA paradigm of
[18]. This then allows us to present a discussion regarding the convergence of the
presented SAA algorithm. A set of numerical simulations are also presented to dem-
onstrate the capability of the SAA when applied to clustering problems of various
sizes and specifications. For a real-world demonstration of our methodology, we
then apply the SAA to fit a GMM for the clustering of data from the famous
MNIST handwritten digits dataset of [27].
58 ◾ Data Analytics
3.1.1 Chapter Roadmap
The rest of the chapter proceeds as follows: the theoretical framework surrounding
the SAA approach of [18] is briefly described in Section 3.2. The GMM is described
in Section 3.3, along with its usage in model-based clustering. The specialization of
the SAA of [18] for ML fitting of spherical-covariance GMMs is then presented in
Section 3.4. Reports of conducted simulation studies are presented in Section 3.5.
The SAA for GMM-based clustering is demonstrated via clustering of the MNIST
dataset in Section 3.6. Conclusions are drawn and future directions are discussed
in Section 3.7.
g ( ψ ) = W H ( ψ ; W ) , (3.1)
h ( ψ ) ≡ W ∇H ( ψ ; W ) = 0, (3.2)
( )
ψ (i +1) = ψ (i ) + γ i ∇H ψ (i ) ; Wi +1 , (3.3)
{ }
n
where ψ (i ) is a sequence of solutions that iterate with the index i and γi > 0
i =0
is a gain factor that will be explained in the sequel. Here ψ(0) is an initial value of
the parameter vector that is selected arbitrarily over the parameter space Ψ. The
n
problem can further be made concrete when the sequence W is replaced by some
observed stream of data w n = { w i }i =1.
n
Big Data-Appropriate Clustering ◾ 59
The iterative scheme (3.3) is generally referred to as an SAA for obtaining a root
of equation system (3.2). The study of SAAs date back to the early works of [28,29].
Good expositions regarding the modern theory of SAAs can be found in [30,31].
A recent review of the literature can be found in [32].
3.2.1 Convergence Result
The appeal of the SAA of [18] is that the conditions under which convergence can
be established are quite simple to state and verify. We first begin by supposing that
n
the sequence W consists of n IID observations drawn uniformly with replacement
from some finite (but potentially very large) set of N observations w N = { w j } j =1.
N
write h ( ψ ) = N −1 ∑ ∇H ( ψ ; w ).
j =1
j
Let ⋅ denote the usual Euclidean norm. Assume that there exists an objective
function of interest g ( ψ ) with stationary points that correspond to the elements of
{ }
the set Ψ * = ψ ∈Ψ : h ( ψ ) = 0 . Define the sets
{
Ψ 0 = ψ 0 ∈Ψ : h ( ψ ) is not continuous at ψ 0 , or g ( ψ ) is not differentiable at ψ 0 }
and
Ξ= {{ψ }
(i ) ∞
i =1
: there exists a closed subset Q of Ψ Ψ 0 such that ψ (i ) ∈Q ,
for all i ∈ . }
( ) ( )
Further, let Bi +1 = ∇H ψ (i ) ; Wi +1 − h ψ (i ) , so that the SAA scheme (3.3) can be
rewritten as
( )
ψ (i +1) = ψ (i ) + γ i h ψ (i ) + Bi +1 .
Make the following assumptions regarding the sequences {γ i }i =1 and {Bi }i =1 : (B1)
n n
n n
and (B4) supi Bi2 < ∞. Furthermore, make the following regularity assumption on
g ( ψ ) and h ( ψ ) : (C1) g ( ψ ) is continuous and differentiable, and for any ψ ∈Ψ,
60 ◾ Data Analytics
Theorem 1. If we assume that (B1)—(B4), (C1), and (C2) hold, then as n → ∞, we have
( ) ( )
limn →∞ dist ψ (n ) , Ψ * = 0 almost surely on Ξ, where dist ψ (n ) , Ψ * = inf ψ ∈Ψ *ψ (n ) − ψ .
We note that [18], in fact, states a more general result that replaces assumptions
(B1) with a criterion regarding the generalized gradient of g ( ψ ) in the sense of
[33]. This allows for the maximization or minimization of functions that may not
be continuously differentiable. As we do not require such a strong result, we have
chosen to simplify the assumptions for brevity of exposition.
1
φ ( x ; µ y , ∑ y ) = 2π ∑ y exp − ( x − µ y ) ∑ −y 1 ( x − µ y ) ,
−1/2
2
f ( x; θ) = ∑π φ ( x; µ , ∑ ) , (3.4)
y =1
y y y
where θ is a parameter vector that contains the unique elements of πy, μy, and Σy.
Suppose it is know that the DGP of Z n is a GMM, as described above, or can
be approximated by a GMM with the parameter vector θ 0 containing the unique
elements of the parameter components π 0y, μ0y, and Σ0y ( y ∈[q ]). Suppose that we
n
observe only the realizations xn of the feature vectors X . We can utilize the usual
maximum a posteriori (MAP) rule for model-based clustering in order to compute
n n
the estimated labels y* that best predict the labels Y . The MAP rule for mixture
models is to predict Yi (i ∈[n ]) by the label that yields the highest a posteriori prob-
ability based on Bayes’ rule (cf. [2]). For GMMs, the MAP rule for the prediction
of the label Yi based on the observed feature xi can be written as
Upon computing y (xi) for each i, we can concatenate the predicted label in order to
n
obtain the set of cluster allocations y* for the observed feature vectors xn.
Note that the application of rule (3.5) requires that we know the parameter
vector θ 0 and that we know the number of clusters or labels q. Using the SAA that
was described in Section 3.2, we shall demonstrate how an estimate for θ0 can be
obtained in the big data context. The determination of q is however outside of the
scope of this chapter.
There are numerous ways by which q can be determined. Recent results of [34]
suggests that the Bayesian information criterion of [35] asymptotically correctly
selects the true number of labels under generous regularity conditions. For a finite
number of observations, Maugis and Michel [36] utilized an oracle result of [37]
that can be combined with the computational technique of [38] in order to com-
pute an estimate of q with good penalized loss properties. The aforementioned
methods are all probability based. Yet other methods for choosing q exist outside
of such a setting. A good recent reference for such methods is [39]. We refer the
interested reader to [40] for an excellent summary on the topic of determining q in
the context of GMMs.
where θ now contains the elements of the parameter components πy, μy, and λy. The
parameter components λy control the volume of the confidence sets of each of the
respective mixture component PDFs.
n
Suppose that we are given a realization xn of X that arises from a DGP that
can be characterized by a GMM with a PDF of form (3.6) with the parameter
vector θ 0 or can be well-approximated by a GMM with a PDF of form (3.6) with
the parameter vector θ 0. We are then required to estimate the parameter vector θ 0
from the data in order to obtain the best possible fitting GMM. Traditionally, this
is achieved via ML.
62 ◾ Data Analytics
n q
λ y2
L (θ) =
1
n ∑ ∑
i =1
log
y =1
π yφ x n ; µ y , I . (3.7)
2
We then seek to obtain the ML estimator θ̂ θn, which can be defined as an appropri-
ate root of the score equation ∇L ( θ ) = 0. Here, we cannot define the ML estimator
purely as the maximizer of (3.7) since the objective function is highly multimodal
and lacks identifiability under permutation of the mixture components (cf. [42]).
A stochastic version of the ML estimation problem is to maximize
l ( θ ) = X l ( θ; X ) (3.8)
s ( θ ) ≡ X ∇l ( θ; X ) = 0. (3.9)
Unfortunately, solving for the roots of (3.9) is made difficult due to the restrictions on
the parameter space of θ. That is, although μy is unrestricted in p , π = (π 1 ,…, π g )
q
The parameter map that allows for simplification of the root finding process is Set
q
π y = exp (π *
y ) ∑ exp (π ) for y ∈q − 1 and π
j =1
*
y
*
q = 0. Further set λ y* = log ( λ y )
and µ = µ y (for consistency of notation), for each y ∈ q . The parameter vector
*
( )
y
θ*, which contains the components π *y ∈ , µ *y ∈ p, and λ y* ∈ >, y ∈ q maps
one-to-one between the space of real vectors in q ( 2 + p )−1 and the restricted space
of the original parameterization θ. We can now consider the simpler problem of
obtaining a root to the equation system
Big Data-Appropriate Clustering ◾ 63
( )
s * θ * ≡ X ∇ *l ( θ; X ) = 0 (3.10)
instead, where ∇ * is taken to mean partial differentiation with respect to the alter-
native parameterization θ*.
We can write the components of ∇ *l ( θ; x ) with respect to π *y as
λ y2
π yφ x ; µ y , I
2
∇ π* *y l ( θ; x ) = −π y,
f ( x; θ)
λ y2
π yφ x ; µ y , I
2 2
∇ *µ *y l ( θ; x ) =
f ( x; θ) λ y2
(x − µ y ),
and
λ y2
π yφ x ; µ y , I
2 2
∇ *λ *y l ( θ; x ) = 2 (x − µλ ) (x − µλ ) −
p ,
f ( x; θ) λ
y
for y ∈[q ]. Given a stream of data X n, we can apply scheme (3.3) for solving (3.10)
to obtain the SAA
( )
θ *(i +1) = θ *(i ) + γ i ∇ *l θ(i ) ; X i +1 , (3.11)
which can then be made concrete upon substitution of a realized sample xn in place
n
of X .
For a well-chosen sequence of gain factors {γ i }i =1, the SAA algorithm that is
n
defined by Eq. (3.11) can be shown to satisfy the regularity conditions of Theorem 1
(i.e., conditions (C1) and (C2) are fulfilled). Thus, the conclusion of Theorem 1
applies to the SAA sequence of parameters that is generated via Eq. (3.11).
3.5 Simulation Results
The SAA algorithm described in Section 3.4 is implemented in the Rstatistical
programming language [10], with particularly computationally intensive loops pro-
grammed in C and integrated into the core program via the Rcpp package of [43]. The
implementation can be freely obtained at https://github.com/andrewthomasjones/
64 ◾ Data Analytics
3.5.1 Practical Considerations
Although the SAA for fitting GMMs from Section 3.4 is theoretically sound, there
are some minor issues that require resolution in order for it to yield successful and
consistent outcomes. First, we require a choice for the sequence of gain factors
{γ i }in=1 that fulfill assumptions (B1)–(B3). In [18], the authors choose a sequence
of the form γ i = γ 0i0 max {i0 , i }, for i ∈ , where γ 0 , i0 > 0. Via some preliminary
experimentation, we have found that such sequences were not successful in our
applications.
In [44], the so-called search-then-converge sequence was suggested for SAA and
similar algorithms. The suggested sequence of gain factors {γ i }i =1 have the form
n
γ0
γi = ,
1 + i /i0
sequence, where γ i = log n /n for i ∈[n0 ], and γ i = γ 0 (1 + i /i0 ) for i ∈[n ] [n0 ],
with γ 0 = log n / n and i0 = n1. For the remainder of the paper, we utilize the initial-
ization scheme and gain factors that are described above.
3.5.3 Comparisons
For each scenario, we repeat the simulation ten times and compute the average time
taken using the proc.time() function from R. Additionally, we assess the quality of
the GMM clusterings via the well-known and popularly used adjusted-Rand index
(ARI) of [46], for comparison of the obtained clusterings to the true subpopulation
labels. The ARI is a measurement that takes value between −1 and 1. A value of 0 or
lower indicates a poor clustering performance, whereas a value close to 1 indicates
a high-quality clustering.
For comparisons, we assess the performance of the SAA-fitted GMM cluster-
ings against those of the k-means algorithm implemented via the kmeans() function
in the R with all default settings, except that we allow the algorithm to restart ten
times. We note that this is a batch algorithm and we should expect it to perform
better than our stream-based algorithm, since it is allowed to utilize the information
66 ◾ Data Analytics
from all of the data, simultaneously. We also compare our methodology to the
DSC_CluStream() function from the streamMOAR package [47], which is a port of
the two-stage stream k-means algorithm of [23], from [48]. We make a final note
that we do not compare our algorithm to any batch GMM methods as they are
generally significantly slower and thus are not applicable in our scenarios.
3.5.4 Results
The average timing results and the ARI results for the three tested algorithms are
reported in Table 3.1. We observe that the SAA for fitting GMMs is the fastest
among the three algorithms in all but two scenarios. Between the two configura-
tions for the burn-in ratio b, we find that b = 10 is faster than b = 5. On average,
the algorithm implemented with b = 10 requires approximately 88% the time of
the b = 5 implementation. Furthermore, and as expected, we also find that all
a lgorithms increase in computational time with increases in d, g, and n.
With regards to ARI, the SAA-fitted GMM outperforms or matches the perfor-
mance of the two comparison algorithms in all but three of the simulation scenar-
ios. This is particularly surprising when comparing to the batch k-means algorithm,
since it is not required to learn the clustering in a streamed manner but is able to
Table 3.1 Average (over Ten Repetition) Computation Time (in Seconds) and
ARI Results for the SAA-Fitted GMM Algorithm, the Batch k-Means Algorithm,
and the Stream k-Means Algorithm of [23] for Each Simulation Scenario
Batch Stream
D q N ω SAA (b = 5) SAA (b = 10) k-Means k-Means
Time (s)
5 5 1.00E+04 0.01 0.21 0.18 0.22 1.05
(Continued)
Big Data-Appropriate Clustering ◾ 67
Note: The bold values indicate the minimum computational time out of the
four methods that were assessed.
68 ◾ Data Analytics
construct the clustering from all of the data, simultaneously. Also interesting is the
fact that the GMM clustering outperforms the stream k-means clustering from
the DSC_CluStream() function in all but one of the problems. This is an excel-
lent outcome considering that the DSC_CluStream() function represents one of
the state-of-the-art algorithms for clustering streamed data. Lastly, we note that
there appears to be little predictable pattern regarding the setting of b with regard
to the performance outcome of the SAA-fitted GMM algorithm. As such, we can
recommend that b = 10 be used when choosing between 5 and 10, since it requires
lesser amount of data in order to initialize the algorithm.
3.6 MNIST Application
The MNIST dataset is a famous and publicly available machine learning dataset
that can be obtained from [27]. The dataset zn consists of n = 70,000 handwrit-
ten images of digits yi (i ∈[n ]) between 0 and 9 (coded as y ∈[10 ] for a total of
q = 10 classes), each rendered as a 28 × 28-pixel image of intensities x i, taking values
between 0 and 255. The numbers of images of digits in the dataset between 0 and
9 (in order) are 6903, 7877, 6990, 7141, 6824, 6313, 6876, 7293, 6825, and 6958.
We utilize the SAA algorithm for GMM-based clustering to cluster the
MNIST data using the feature vectors xn without knowledge of the digit labels
yn. We then utilize the labels yn in order to assess the quality of the clustering
outcome via the ARI, as we had done in Section 3.5. We further compare the
GMM clustering to those obtained via the the kmeans() function in the R as well
as the DSC_CluStream() function, in order to gain some insight into how our
method benchmarks against a standard batch methodology and a state-of-the-art
streamed data clustering algorithm. Clustering performances are once again mea-
sured with the ARI statistic and computation times are measured using the proc.
time() function.
3.6.1 Data Preprocessing
In the original format, each feature vector x i is p = 282 = 784 dimensional. Upon
inspection of the data, many of the pixels, or features, take value zero for all i ∈[ n ] .
As such, we can remove these zero features in order to decrease the dimensionality
of the data dramatically. Using the R data processing package caret [49], we can
remove these features by using the nearZeroVar() function.
Upon removal of the zero features, we are left with p(1) = 250 dimensional
{ }
n
feature vectors x (i1) . For each coordinate k ∈ p(1) of the feature vector x (i1)
i =1
( )
(denoted as xik ), we take the logarithm transformation xik( 2) = log xik(1) + 1 , in order
(1)
to mitigate the large spread on the original intensity scale. A principal compo-
nent analysis (PCA; see, for instance, [50]) is then conducted and the first five
Big Data-Appropriate Clustering ◾ 69
and ten principal components are used to form the reduced-dimensional datasets
{ } { }
n n
x [n5] = x [i 5] and x[n10] = x [i10] .
i =1 i =1
3.6.2 Results
Using the PCA reduced-dimensional datasets x [n5] and x [n10], we then perform clustering
using the GMM fitted by SAA with burn-in ratios set to b ∈{5,10} and q ∈{10, 20}.
Here, we perform analysis for q = 20 in order to capture the potential of heterogeneity
within the digit subpopulations. For example, some sevens may be crossed, whereas
others are not. Along with the GMM clusterings, we also perform clustering using
the kmeans() and the DSC_CluStream() functions in R, for the same values of q and
PCA reduced data x [n5] and x [n10]. Each clustering method and combination of settings
is repeated ten times. The average timing and ARI results are presented in Table 3.2.
From Table 3.2, we observe that the SAA for fitting GMMs is the fastest
among the compared methods in every scenario (when implemented with b = 10).
Furthermore, the GMM clusterings were able to match the performance of the
batch k-means algorithm with respect to ARI performance in all but one scenario.
This is a good result considering that the algorithm can be fitted iteratively and does
not require the entire dataset simultaneously.
Table 3.2 Average (over Ten Repetition) Computation Time (in Seconds) and
ARI Results for the SAA-Fitted GMM Algorithm, the Batch k-Means Algorithm,
and the Stream k-Means Algorithm of [23] for Clustering of the MNIST Data
Principal Batch Stream
Components Q SAA (b = 5) SAA (b = 10) k-Means k-Means
Time (s)
5 10 2.66 2.29 3.78 7.70
Note: The bold values indicate the minimum computational time out of the
four methods that were assessed.
70 ◾ Data Analytics
3.7 Conclusions
The big data context, which can be characterized by the three Vs, is a prevalent set-
ting for modern data analysis. Here, the Vs stand for variety, velocity, and volume,
and modern algorithms that operate with big data are required to be able to cope
with all three of these data characteristics.
In this chapter, we presented an algorithm for conducting model-based cluster-
ing, which utilizes an SAA to fit heterogeneous spherical-covariance GMMs. The
SAA framework allows the GMMs to be fitted to very large datasets that arrive to
the analyst in a stream and, thus, the algorithm is suitable in the big data context.
The clusterings produced by our algorithm can be directly comparable to other
streamed k-means algorithms such as that of [23].
We provided a brief introduction of the SAA framework for algorithm
construction and derived the algorithm for the ML estimation of GMMs within
the discussed framework. The derived algorithm can be proved convergent due to
the known results of [18].
To demonstrate the SAA algorithm GMM-based clustering, we perform a set of
numerical simulations and apply the algorithm to cluster the well-known MNIST
dataset. In both cases, the SAA-fitted GMM algorithm compared very favorably
against the state-of-the-art stream k-means algorithm of [23] as well as the tra-
ditional batch k-means algorithm, in terms of both clustering performance and
computation time.
The positive assessment of the algorithm leads to numerous avenues for future
study. For example, we can restrict the algorithm to fitted homogeneous Gaussian
components with equal component probabilities in order to produce a “soft” k-means
algorithm in the sense of [20]. Alternatively, we can expand upon the current work
and utilize the matrix derivative expressions of [51] in order to produce an SAA algo-
rithm for estimating heterogeneous non-spherical GMMs. Both of these directions
are potentially highly impactful and require further attention and investigation.
References
1. G.J. McLachlan and K.E. Basford, Mixture Models: Inference and Applications to Clustering.
New York: Marcel Dekker, 1988.
2. B.S. Everitt, S. Landau, M. Leese, and D. Stahl, Cluster Analysis. Chichester: Wiley, 2011.
3. K. Pearson, Contributions to the theory of mathematical evolution, Philosophical
Transactions of the Royal Society of London A, vol. 185, pp. 71–110, 1894.
4. J.D. Banfield and A.E. Raftery, Model-based Gaussian and non-Gaussian clustering,
Biometrics, vol. 49, pp. 803–821, 1993.
5. C. Fraley and A.E. Raftery, Model-based clustering, discriminant analysis, and density
estimation, Journal of the American Statistical Association, vol. 97, pp. 611–631, 2002.
6. C. Fraley and A.E. Raftery, MCLUST: Software for model-based cluster analysis, Journal
of Classification, vol. 16, pp. 297–306, 1999.
Big Data-Appropriate Clustering ◾ 71
7. C. Fraley and A.E. Raftery, Enhanced model-based clustering, density estimation, and
discriminant analysis software: MCLUST, Journal of Classification, vol. 20, pp. 263–286,
2003.
8. C. Fraley and A.E. Raftery, Model-based methods of classification: Using the mclust
software in chemometrics, Journal of Statistical Software, vol. 18, pp. 1–13, 2007.
9. L. Scrucca, M. Fop, T.B. Murphy, and A.E. Raftery, mclust 5: Clustering, classifica-
tion and density estimation using Gaussian finite mixture models, R Journal, vol. 8,
pp. 289–317, 2016.
10. R Core Team, R: A Language and Environment for Statistical Computing. Vienna: R Foundation
for Statistical Computing, 2016.
11. A.P. Dempster, N.M. Laird, and D.B. Rubin, Maximum likelihood from incomplete data
via the EM algorithm, Journal of the Royal Statistical Society Series B, vol. 39, pp. 1–38,
1977.
12. G.J. McLachlan and D. Peel, Finite Mixture Models. New York: Wiley, 2000.
13. J.K. Jain and R.C. Dubes, Algorithms for Clustering Data. Englewood Cliffs: Prentice
Hall, 1988.
14. L. Kaufman and P.J. Rousseeuw, Finding Groups in Data: An Introduction to Cluster
Analysis. New York: Wiley, 1990.
15. A.K. Jain, M.N. Murty, and P.J. Flynn, Data clustering: A review, ACM Computing
Surveys, vol. 31, pp. 264–323, 1999.
16. C. Hennig, M. Meila, F. Murtagh, and R. Rocci, Eds., Handbook of Cluster Analysis. Boca
Raton: CRC Press, 2016.
17. A. McAfee, E. Brynjolfsson, and T.H. Davenport, Big data: The management revolution,
Harvard Business Review, vol. 90, pp. 60–68, 2012.
18. J. Zhang and F. Liang, Convergence of stochastic approximation algorithms under irregu-
lar conditions, StatisticaNeerlandica, vol. 62, pp. 393–403, 2008.
19. F. Liang and J. Zhang, Estimating the false discovery rate using the stochastic approxima-
tion algorithm, Biometrika, vol. 95, pp. 961–977, 2008.
20. W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes:
The Art of Scientific Computing. Cambridge: Cambridge University Press, 2007.
21. J.A. Hartigan and M.A. Wong, Algorithm AS 136: A k-means clustering algorithm,
Journal of the Royal Statistical Society Series C, vol. 28, pp. 100–108, 1979.
22. L. Bottou and Y. Bengio, Convergence properties of the k-means algorithms, In Advances
in Neural Information Processing Systems, 1994.
23. C.C. Aggarwal, J. Han, J. Wang, and P.S. Yu, A framework for clustering evolving data
streams, In Proceedings of the 29th VLDB Conference, 2003.
24. N. Ailon, R. Jaiswal, and C. Monteleoni, Streaming k-means approximation, In Advances
in Neural Information Processing Systems, 2009.
25. D. Sculley, Web-scale k-means clustering, In Proceedings of the 19th International
Conference on World Wide Web, 2010.
26. M. Shindler, A. Wong, and A.W. Meyerson, Fast and accurate k-means for large datasets,
In Advances in Neural Information Processing Systems, 2011.
27. Y. LeCun, The MNIST Database of Handwritten Digits. 1998.
28. H. Robbins and S. Monro, A stochastic approximation method, Annals of Mathematical
Statistics, vol. 22, pp. 400–407, 1951.
29. J. Kiefer and J. Wolfowitz, Stochastic approximation of a regression function, Annals of
Mathematical Statistics, vol. 23, pp. 462–466, 1952.
30. H.F. Chen, Stochastic Approximation and Its Applications. London: Kluwer, 2002.
72 ◾ Data Analytics
31. H.J. Kushner and G. Yin, Stochastic Approximation and Recursive Algorithms and
Applications. New York: Springer, 2003.
32. M. Chau and M.C. Fu, An Overview of Stochastic Approximation, In Handbook of
Simulated Optimization, M.C. Fu, Ed. New York: Springer, 2015, pp. 149–178.
33. F.H. Clarke, Optimization and Nonsmooth Analysis. SIAM, 1990.
34. J.-P. Baudry, Estimation and model selection for model-based clustering with the con-
ditional classification likelihood, Electronic Journal of Statistics, vol. 9, pp. 1041–1077,
2015.
35. G. Schwarz, Estimating the dimensions of a model, Annals of Statistics, vol. 6, pp. 461–464,
1978.
36. C. Maugis and B. Michel, A non asymptotic penalized criterion for Gaussian model
selection, ESAIM: Probability and Statistics, vol. 15, pp. 41–68, 2011.
37. P. Massart, Concentration Inequalities and Model Selection. New York: Springer, 2007.
38. J.-P. Baudry, C. Maugis, and B. Michel, Slope heuristic: Overview and implementation,
Statistics and Computing, vol. 22, pp. 455–470, 2012.
39. B. Mirkin, Choosing the number of clusters, WIREs Data Mining and Knowledge
Discovery, vol. 252–260, 2011.
40. G.J. McLachlan and S. Rathnayake, On the number of components in a Gaussian mixture
model, WIREs Data Mining and Knowledge Discovery, vol. 4, pp. 341–355, 2014.
41. H. White, Maximum likelihood estimation of misspecified models, Econometrica, vol. 50,
pp. 1–25, 1982.
42. D.M. Titterington, A.F.M. Smith, and U.E. Makov, Statistical Analysis of Finite Mixture
Distributions. New York: Wiley, 1985.
43. D. Eddelbuettel, Seamless R and C++ Integration with Rcpp. New York: Springer, 2013.
44. C. Darken and J. Moody, Note on learning rate schedules for stochastic optimization, In
Advances in Neural Information Processing, 1991.
45. V. Melnykov and I. Melnykov, Initializing the EM algorithm in Gaussian mixture models
with an unknown number of components, Computational Statistics and Data Analysis, vol.
56, pp. 1381–1395, 2012.
46. L. Hubert and P. Arabie, Comparing partitions, Journal of Classification, vol. 2, pp.
193–218, 1985.
47. M. Hahsler, M. Bolanos, and J. Forrest, streamMOA: Interface for MOA Stream
Clustering Algorithms. 2015.
48. A. Bifet, G. Holmes, R. Kirkby, and B. Pfahringer, MOA: Massive online analysis, Journal
of Machine Learning Research, vol. 11, pp. 1601–1604, 2010.
49. M. Kuhn, Building predictive models in R using the caret package, Journal of Statistical
Software, vol. 28, pp. 1–26, 2008.
50. I.T. Jolliffe, Principal Component Analysis. New York: Springer, 2002.
51. O. Boldea and J.R. Magnus, Maximum likelihood estimation of the multivariate normal
mixture model, Journal of the American Statistical Association, vol. 104, pp. 1539–1549,
2009.
Chapter 4
Information Retrieval
Methods for Big Data
Analytics on Text
Abhay Kumar Bhadani and Ankur Narang
Yatra Online Pvt. Ltd
Contents
4.1 Introduction to Information Retrieval.........................................................74
4.2 Vector Space Models...................................................................................75
4.2.1 D ocument-Term Matrix...................................................................75
4.2.2 Distance Metrics............................................................................. 77
4.2.2.1 E uclidean Distance........................................................... 77
4.2.2.2 M ahalanobis Distance...................................................... 77
4.2.2.3 Jaccard Index.....................................................................78
4.2.2.4 Cosine Similarity...............................................................79
4.2.2.5 W ord Mover’s Distance.....................................................79
4.2.3 T erm Frequency–Inverse Document Frequency...............................80
4.3 Information Retrieval Approaches...............................................................82
4.3.1 L atent Semantic Analysis.................................................................82
4.3.2 w
ord2vec.........................................................................................83
4.3.2.1 CBOW Model...................................................................83
4.3.2.2 S kip-Gram Model............................................................ 84
4.3.3 f astText............................................................................................85
4.4 Walk-Through of IR: An Illustrative Example.............................................87
4.5 Applications................................................................................................92
73
74 ◾ Data Analytics
IR has many applications ranging from web search, searching emails, recom-
mending similar jobs, finding similar users, understanding trends, search polarity,
determining sentiments, and so on. However, despite rapid advancement in search
engines, there are numerous challenges that still exist which have the potential to
improve the quality of results and determine the document order in the fastest pos-
sible manner. These challenges are increasing day by day as a huge amount of text
is getting generated in various forms, such as blogs, digitization of libraries, com-
ments, and posts on social networks.
Recently, models developed on neural networks have become increasingly pop-
ular [3–5]. We will discuss different models that are based on neural networks as
well as linear models to understand its effectiveness in building IR systems.
One of the critical parts of IR lies in fetching relevant documents by captur-
ing the intent of the search phrase, based on which results can be retrieved and
displayed. In this chapter, we address some of the concerns related to captur-
ing the intent of the user and developing a context-based information retrieval
system.
The abovementioned sentences are broken down into three separate documents.
Here, we have just one sentence in one paragraph, but a paragraph can also have
more than one sentence.
doc1: “The quick brown fox jumps over the lazy dog.”
doc2: “The fox was red but dog was lazy.”
doc3: “The dog was brown, and the fox was red.”
The document-term matrix for the above corpus would look like Table 4.1.
Now, we have a document-term matrix for each document. Thus, we can also
say that now we have a vector representation for each document. The length of the
vector is equal to the size of the vocabulary in the corpus. In our example, the size
of vocabulary is 11, which is also the length of the vector for representing each
document.
Let’s assume that we have these example sentences stored in the underlying
database, which will be used to serve the search request raised by a query phrase.
Let us say that the query is “fox jumps over brown dog.”
First, the query will be converted to an equivalent vector of size 11 (Table 4.2).
Now, the task is to retrieve relevant documents based on some similarity score for
the given query.
Table 4.1 Document-Term Matrix
The quick brown fox jumps over lazy dog was red and
doc1 2 1 1 1 1 1 1 1 0 0 0
doc2 2 0 0 1 0 0 1 1 2 1 1
doc3 2 0 1 1 0 0 0 1 2 1 1
Information Retrieval Methods ◾ 77
Table 4.2 Document-Term Matrix for Query: “fox jumps over brown dog”
The quick brown fox jumps over lazy dog was red and
query 0 0 1 1 1 1 0 1 0 0 0
In order to perform this operation, we are required to use distance metrics that
will help in deciding the order of the documents retrieved, based on the relevance
of the searched phrase.
n −1
where x and y are two vectors used for computing the Euclidean distance.
into consideration the covariance between the features. And, further, if the covari-
ance matrix is equal to the identity matrix, then the Mahalanobis distance is like
computing the Euclidean distance between two vectors.
The Euclidean and Mahalanobis distance compute the magnitude of the feature
vector and, thus, portrays similar objects closer in a variable space. Mathematically,
the Mahalanobis distance is defined as follows:
−1
( X i − X j )T ∑( X − X ) (4.2)
i j
where Xi and Xj are the vector representations for i and j cases, whereas Σ is the
group variance–covariance matrix.
J ( X ,Y ) = | X ∩ Y | / | X ∪ Y | (4.3)
where X and Y are the two different sets for comparison and compute the similarity.
Let’s say we have two sentences: (A) “The quick brown fox jumps over the lazy
dog.” and (B) “The fox was red but dog was lazy.”
These sentences can be represented using the following set notations:
4.2.2.4 C
osine Similarity
Cosine similarity computes the normalized dot product of two vectors of equal
length. It helps to find the cosine of the angle between the two vectors. The mathe-
matical notation for the cosine similarity is given as Equation 4.4, and the extended
notation is given in Equation 4.5.
The cosine of 0° is 1 (it also means that there is no difference between the two
vectors and hence 0, which means the similarity between the two vectors computes
to 1), and it is less than 1 for any other angle. If the distance is closer to 1, the terms
are somewhat closer or similar in nature. On the other hand, if the distance is closer
to 0 (i.e., the two vectors are at 90°), it indicates that the terms are not related. Thus,
cosine similarity is a judgment of orientation and does not focus on magnitude.
x. y
cos( x , y ) = (4.4)
|| x |||| y ||
∑
n −1
x i yi
cos( x , y ) = i =0
(4.5)
∑ ∑
n −1 n −1
( xi )2 ( yi )2
i =0 i =0
It is particularly used in positive space and the outcome is bounded within [0, 1].
One of the primary reasons for the popularity of cosine similarity is that it can be
evaluated very efficiently, especially if the vectors are sparse in nature.
distance; two very different documents will have low similarity score and a large
distance. Document lengths have a high impact on the running time of WMD.
WMD is a method that allows us to assess the “distance” between two documents
in a meaningful way, even when they have no words in common [12,13]. It uses
word2vec/fastText vector embeddings of words.
N
idf t = log 1 + (4.6)
nt
where
IDFt 0.301 0.602 0.398 0.301 0.602 0.602 0.398 0.301 0.398 0.398 0.602
Table 4.4 TF–IDF Matrix
The quick brown fox jumps over lazy dog was red and
doc1 0.602 0.602 0.398 0.301 0.602 0.602 0.398 0.301 0.000 0.000 0.000
doc3 0.602 0.000 0.398 0.301 0.000 0.000 0.000 0.301 0.796 0.398 0.602
query 0.000 0.000 0.398 0.301 0.602 0.602 0.000 0.301 0.000 0.000 0.000
82 ◾ Data Analytics
SVD for information retrieval tasks. LSA is a special case where LSI is applied to
natural language processing and principal component analysis is a statistical tech-
nique that is an application of SVD.
LSA approach makes three claims: that semantic information can be captured
from a co-occurrence matrix of a document; one of the essential parts of this
approach is dimensionality reduction; and any documents or words can be repre-
sented as a point in a Euclidean space. It is a class of statistical models in which the
semantic properties of words and documents are expressed in terms of probabilistic
topics.
4.3.2 word2vec
As the name itself suggests, word2vec is a technique that converts words to vector
representation. Primarily, there are two approaches to compute the vectors: (a) con-
tinuous bag-of-words (CBOW) and (b) skip-gram model.
As we know, n-grams are a sequence of words occurring in an order. Though
it is logical to think of a sequence of words, however it has been shown that skip-
grams work equally good while computing the word embeddings. Skip-grams or
more specifically skip-n-grams consider the subsequence of words that may occur,
at most, within some k-neighborhood distance.
The probabilistic feedforward neural network language model proposed by
Vincent et al. [17] is the basis for designing word2vec [18]. It consists of input
layers, projection, hidden, and output layers. The N previous words are encoded
using 1-of-S coding, where S is the size of the vocabulary. The input layer is further
projected to a layer P that has dimensionality N * D, using a distributed projection
matrix. At a given time, only N inputs are active; hence, the composition of the
projection layer is a relatively computationally efficient operation.
4.3.2.2 S
kip-Gram Model
In the skip-gram model, we are given a corpus of words w and their contexts c. So,
we can model this phenomenon with conditional probabilities, i.e., p(w|c) and then
try to compute the probabilities for the predicted word. We also need to under-
stand that the prediction lies on the appropriateness of tuned parameters or hyper-
parameters, which is normally denoted by θ. So, to be precise, the conditional
probability equation is represented as p(w|c; θ).
Information Retrieval Methods ◾ 85
∏
max
arg p(c | w ;θ ) (4.7)
θ ( w ,c )∈S
Here, S is the set of possible words and context pairs that is built from the corpus.
The concept behind skip-gram is to train a model in the context of each word,
such that similar words end up having similar embeddings. The skip-gram model
in word2vec uses neural networks to estimate the hyper parameters θ by adjust-
ing the weights using backward-propagation. It models the conditional probability
p(w|c; θ) that uses soft-max classifier in the last layer of the neural networks.
e vc . vw
p( c | w ; θ ) = (4.8)
∑ c ′∈C
e v c ′ . vw
where (vc, vw) ∈ Rd are vector representations for c and w, respectively, and C is the
set of all available contexts. The parameters θ are vci, vwi for w ∈ V, c ∈ C, I ∈ 1, 2,...,
d (a total of |C|*|V|*d parameters).
While defining context in a sentence having n words w1,...,wn, contexts
of a word wi come from a window of size k in the neighborhood of the word:
C(w) = wi−k,…, wi−1, wi+1,…, wi+k, where k is a parameter.
At this point, we are ready to fine-tune the parameters such that the product
(4.4) is maximized. This product operation can be converted to summation if we
apply log on both sides for easier computation. An illustrative diagram of the skip-
gram model is shown on the right side of Figure 4.2.
Overall, word2vec treats each word in the corpus like an atomic entity and gen-
erates word-embedding for each word and context words. Thus, the smallest unit in
word2vec is a word. The main limitation of this approach is that it fails to generate
or classify words that do not exist in the corpus.
In Section 4.3.2, we will see how fastText overcomes this limitation by using
n-gram character as its basic unit for creating the word embeddings.
4.3.3 fastText
fastText is one of the recent advances in the area of NLP, which helps to generate
word embeddings, very similar to what word2vec does [9]. It is an open-source
library created by Facebook for carrying out their NLP activities, such as word
embeddings or text classification. However, the beauty of fastText lies in the fact
that it takes several orders of magnitude less time to train a model. fastText takes
less than ten minutes to train a model using a standard multicore CPU and can
classify 312K classes in less than a minute for half a million sentences.
Its performance is comparable and even better in few cases than deep learn-
ing approaches, such as word2vec and GloVe, in terms of accuracy, but it is many
orders of magnitude faster for training and evaluation. Thus, it helps in reducing
86 ◾ Data Analytics
the training time. The basic unit for fastText is character n-gram, whereas the basic
unit for word2vec is word or n-gram word.
It represents a text by a low-dimensional vector, which is an aggregation of
words appearing in the text. It treats each word as a composition of n-gram char-
acters. For example, we want to understand how orange is treated by fastText using
n-gram character. Here, there are two hyper-parameters that need to be set initially
Information Retrieval Methods ◾ 87
(i.e., n-gram min and n-gram max). Let’s say n-gram min is 3 and n-gram max is 7.
Then, orange will be represented as follows:
The beauty of this approach lies in the fact that it generates a decent word embed-
dings for words that occur rarely. This happens due to the fact that the character
n-grams used to construct the words share the vector representation of different
words. Hence, it is likely that the embeddings can still be acceptable.
This approach of character n-grams also helps in constructing words that were
not seen previously in the corpus. Thus, out-of-vocabulary problems are handled
using this approach.
In spite of some good points, it also comes with a price. Since it trains the model
on a character level or a character n-gram level, it ends up taking more time to
generate word embeddings compared to word2vec. So, we must be selecting the
hyper-parameters carefully, as the values of min and max have direct impact on
this time. One of the limitations of fastText is that it works better on small datasets.
Example 1
Let us consider the discussed example again.
doc1: “The quick brown fox jumps over the lazy dog.”
doc2: “The fox was red but dog was lazy.”
doc3: “The dog was brown, and the fox was red.”
We have learned the two basic and most popular distance metrics (Euclidean and
cosine) that can handle vector arithmetic and one advanced metric WMD. Now,
we will compute the scores and document retrieval order using Euclidean and
cosine metrics. Further, we will show a detailed example to show WMD.
The search query is: “fox jumps over brown dog”
Here, we discussed how to compute and retrieve documents, given a query
string using two approaches in Tables 4.5 and 4.6.
However, there are certain limitations with this approach. The document vector
representation we explained, is also popularly called BOW vector representation.
88 ◾ Data Analytics
doc1 2.4494897428 1
doc2 3.7416573868 3
doc3 3.4641016151 2
doc1 0.6741998625 1
doc2 0.2581988897 3
doc3 0.3872983346 2
In BOW, the basic assumptions about the data are that the order of words is not
important, but only the term frequency matters. So, we can say, BOW is a collec-
tion of unordered list of terms. One of the major limitations of this approach is
that it does not capture the context. It may fail to detect the difference between a
valid positive sentence and, maybe, a negative sentence composed of the same set
of words, or even sarcasm may go undetected. For Example:
“The quick brown fox jumps over the lazy dog” is, in this context, identical
to the document “The lazy brown dog jumps over the quick fox.” It is so because
both the sentences have the same vector representation (i.e., BOW representation).
It may be a good fit if we intend to find similarity in the topics, something
like grouping political news, sports news, scientific news, and so on. Now, we will
again compute both the metrics based on TF–IDF matrix (refer Table 4.4) to see
if there are any differences.
We see that the document retrieval order does not change in Tables 4.7 and
4.8, in this particular case. But, if we have a large corpus, then TF–IDF works bet-
ter compared to the document-term matrix. It is so because it penalizes frequently
occurring words and gives more weight for those words whose frequency is less.
doc1 0.1324138712 1
doc2 1.9478555502 3
doc3 1.5957805672 2
Information Retrieval Methods ◾ 89
doc1 0.7392715831 1
doc2 0.1437187361 3
doc3 0.2692925425 2
Example 2
Now, we will take another example shown on a real dataset of Amazon’s book
reviews. It is a rich dataset and comprises of real reviews written by several cus-
tomers based on their experience. Few researchers have used this dataset to address
some real challenges, such as answering product related queries and recommend-
ing products [20,21] The dataset can be downloaded from [19]. It consists of the
following fields.
In our case, we extracted “asin” and “reviewText” columns for training our
model. Word embedding is an approach where the words (in any language, i.e.,
English, Chinese, French, Hindi, etc.) are represented using dense vectors and
still maintain their relative meaning. We can also say that the word embedding is
a continuous vector space where semantically related words are mapped to nearby
points. It is an enhanced way of representing the words in a dense vector format
as compared with BOW (document-term matrix), which is sparse representation.
Dense vector representation of textual data is usually achieved using neural net-
works having certain hidden nodes trained over the underlying corpus. Once the
model is trained, it can be reused again and again for multiple purposes. The usage
can vary from finding synonyms, finding dissimilar words, machine translation,
and sentence generation to name a few.
A word embedding (W: words -> Rn) is a representation technique of mapping
words in some language to high-dimensional vectors (perhaps 100–300 dimen-
sions) by training a neural network.
For simplicity, we have trained our model having ten dimensions (though it
may not be desirable). The dense vector representation of a few initial words from
the review corpus that is trained using word2vec-CBOW approach is shown in
Table 4.9.
90 ◾ Data Analytics
Table 4.9 Dense Word Vector Representation Having Ten Dimensions Generated Using word2vec
book −0.17 −0.96 0.48 4.78 0.18 0.58 −0.14 −1.44 −0.56 2.17
read −0.96 −1.13 0.53 4.97 −0.72 1.57 1.52 −0.10 −1.68 4.78
story 0.21 0.52 −2.28 3.11 0.93 2.44 −2.88 −2.36 0.69 0.56
one 1.13 −1.10 −0.24 2.58 −0.47 0.46 −0.09 0.69 −0.80 2.28
like 0.73 −0.21 −0.46 1.70 1.51 −0.57 0.20 0.25 −3.70 1.57
characters 1.14 0.58 −2.40 3.17 2.59 1.62 −6.31 −2.91 0.08 0.17
books −0.11 −2.24 1.65 3.82 1.01 0.90 −0.24 −1.66 −0.14 4.49
would −0.45 1.31 1.85 3.90 0.03 −1.85 −1.48 −0.30 −1.78 2.53
good −0.71 0.23 −1.32 3.41 0.63 1.03 0.40 −1.42 −2.34 3.17
reading −1.65 −0.52 2.02 6.05 −1.03 1.76 0.52 0.09 −0.86 2.38
could −0.72 0.84 1.69 6.37 1.05 1.08 −3.81 0.62 0.01 1.61
really 0.74 0.17 −0.95 4.69 1.59 −0.62 −1.61 −0.93 −1.98 1.72
rest 1.74 −3.95 −0.76 5.12 0.62 1.48 0.93 −0.23 −2.20 0.44
great −0.37 0.11 −2.73 3.17 −0.09 2.67 0.30 −1.50 −0.64 4.01
love 2.08 0.96 −4.09 2.02 −1.96 −1.46 −1.34 −1.51 −1.13 3.30
Information Retrieval Methods ◾ 91
2.8175 This is a good story and story and easy read. I am a Grisham fan
and this did not disappoint.
2.9357 Awesome book I was very impressed with the story the plot was good
it made me cry hf oe nf pdd awww ggtt vgvbh cvcvg dfrw shyev
3.0640 This was a wonderful book, it was an easy read with a great story. I
am happy to own this book.
motivational
The results presented here are dependent on the trained corpus, and we may expect
a few changes in the results as the corpus changes. The quality of results is likely to
improve with the increase in the size of the dataset.
4.5 Applications
NLP is a special branch that falls under a broad application of artificial intelligence.
It has many implications for the ways that people and computers have started inter-
acting. NLP has the power to become an important technology that has the potential
to bridge the gap between human communication and digital data. A few applica-
tions that are very useful and are yet to mature are enlisted in Sections 4.5.1–4.5.8.
4.5.7 R
eplacing Medical Codes from Patient’s Prescription
Medical codes (commonly referred to as the International Classification of Disease
[ICD]) are a standard set of codes that are used by the insurance agencies to cross-
verify the procedures followed by the physicians or hospitals while serving a patient
during his/her visit. Accordingly, payments are released, if the procedures followed
lie under the policy. Converting patient’s prescriptions to medically coded docu-
ments requires human interference, in spite of the fact that these are a standard set
of practice that is followed in a standard way. Placing correct medical codes at the
right place is a challenging task, even for humans, and it still suffers from errors
leading to declination of the payment. Developing such solutions is the need for
processing huge amount of patient’s prescriptions.
4.5.8 W
riting Automatically/Text Generation/Poetry
Text generation is one of the ambitious projects that are being researched upon.
Writing or generating correct and meaningful sentences in a particular language
94 ◾ Data Analytics
is still a far-sighted dream. However, deep learning in NLP has proved to be quite
helpful.
4.6 Conclusions
In this chapter, we discussed the basics of vector space model and its need for min-
ing textual data for extracting meaningful insights in terms of information retrieval,
classification, or sentiment analysis. We introduced some basic concepts related to the
vector space model and how words are represented for deriving word embeddings.
Different distance metrics have been discussed which are used frequently. We dis-
cussed different approaches such as TF–IDF, word2vec, and fastText for representing
the words in dense vector notation. Toward the end of the chapter, several applications
of NLP and text analysis along with relevant examples were discussed. After reading
this chapter, the readers must have become aware of the nuances of text mining and
handling textual data.
References
1. Wikipedia. Information retrieval. https://en.wikipedia.org/wiki/Information_retrieval.
Accessed on 15th Feb, 2018.
2. C.D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval.
Cambridge University Press, New York, 2008.
3. A. Conneau, H. Schwenk, L. Barrault, and Y. Lecun. Very deep convolutional networks
for natural language processing. arXiv preprint, 2016.
4. Y. Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.
5. X. Zhang and Y. LeCun. Text understanding from scratch. arXiv preprint, 2015.
6. D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent dirichlet allocation. The Journal of machine
Learning Research, 3:993–1022, 2003.
7. T. Mikolov, I. Sutskever, K. Chen, Greg S. Corrado, and J. Dean. Distributed representa-
tions of words and phrases and their compositionality. In C.J.C. Burges, L. Bottou, M.
Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information
Processing Systems, 26, pp. 3111–3119. Curran Associates, Inc., Lake Tahoe, NV, 2013.
8. J. Pennington, R. Socher, and C.D. Manning. Glove: Global vectors for word represen-
tation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543,
2014.
9. A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov. Bag of tricks for efficient text clas-
sification. CoRR, abs/1607.01759, 2016.
10. G. Salton and M.J. McGill. Introduction to Modern Information Retrieval. McGraw-Hill,
Inc., New York, 1986.
11. G. Taguchi and R. Jugulum. The MahalanobisTaguchi Strategy: A Pattern Technology
System. John Wiley and Sons, Somerset, NJ, 2002.
12. M.J. Kusner, Y. Sun, N.I. Kolkin, and K.Q. Weinberger. From word embeddings to
document distances. In Proceedings of the 32nd International Conference on International
Conference on Machine Learning - Volume 37, ICML’15, pp. 957–966. JMLR.org, 2015.
Information Retrieval Methods ◾ 95
13. E. Levina and P.J. Bickel. The earth mover’s distance is the mallows distance: Some
insights from statistics. In ICCV, pp. 251–256, 2001.
14. S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman. Indexing by
latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–
407, 1990.
15. J.R. Bellegarda. Latent semantic mapping [information retrieval]. IEEE Signal Processing
Magazine, 22(5):70–80, 2005.
16. D. Kalman. A Singularly Valuable Decomposition: The SVD of a Matrix. The American
University, Washington, DC, 2002.
17. P. Vincent. Y. Bengio, and R. Ducharme. A neural probabilistic language model. Journal
of Machine Learning Research, 3:1137–1155, 2003.
18. T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representa-
tions in vector space. CoRR, abs/1301.3781, 2013.
19. Amazon. Amazon product data. http://snap.stanford.edu/data/amazon/productGraph/
categoryFiles/ reviews_Books_5.json.gz, 2016.
20. J. McAuley and A. Yang. Addressing complex and subjective product-related queries
with customer reviews. In Proceedings of the 25th International Conference on World Wide
Web, WWW ‘16, pp. 625–635, Republic and Canton of Geneva, Switzerland, 2016.
International World Wide Web Conferences Steering Committee.
21. R. He and J. McAuley. Ups and downs: Modeling the visual evolution of fashion trends
with one-class collaborative filtering. In Proceedings of the 25th International Conference on
World Wide Web, WWW ‘16, pp. 507–517, Republic and Canton of Geneva, Switzerland,
2016. International World Wide Web Conferences Steering Committee.
Chapter 5
Contents
5.1 Introduction................................................................................................98
5.1.1 Motivation and Challenges of Big Graph Analytics.........................98
5.1.2 Frameworks for Big Graph Analytics...............................................99
5.1.3 Organization and Goal of This Chapter...........................................99
5.2 Distributed Frameworks for Analyzing Big Static Graphs...........................99
5.2.1 Vertex-Centric Frameworks...........................................................100
5.2.1.1 Classification of Vertex-Centric Frameworks...................102
5.2.2 B lock-Centric Frameworks............................................................108
5.2.2.1 BLOGEL.........................................................................108
5.2.2.2 GIRAPH++..................................................................... 110
5.2.3 S ubgraph-Centric Frameworks...................................................... 111
5.2.3.1 NScale............................................................................. 111
5.2.4 Matrix-Based Frameworks.............................................................112
5.2.4.1 P EGUSUS.......................................................................112
5.2.5 DBMS-Based Frameworks............................................................. 114
5.2.5.1 Pregelix............................................................................ 114
5.2.5.2 DG-SPARQL.................................................................. 115
5.3 Single-Machine Frameworks for Analyzing Big Static Graphs.................. 116
5.4 Distributed Frameworks for Analyzing Big Dynamic Graphs................... 118
5.4.1 F rameworks for Analyzing Temporal Graphs................................ 119
5.4.1.1 DeltaGraph...................................................................... 119
5.4.1.2 Chronos...........................................................................120
5.4.2 F rameworks for Analyzing Streaming Graphs...............................120
97
98 ◾ Data Analytics
5.1 Introduction
Nowadays, people, devices, processes, and other entities are much more connected
with each other than in any other time of history. This connectivity stems from
the popularity of online social networks (such as Facebook, Twitter, Google+, and
LinkedIn), mobile communication networks, online forums, collaboration networks
(such as Stack Overflow, Quora, Ubuntu Forums, Academia.edu, and Mendeley),
wireless sensor networks, World Wide Web (WWW), and many other networks.
These networks often contain billions of entities (vertices) and/or connections
(edges) and as such have been aptly named as large-scale graphs [1] or simply, big
graphs [2,3]. An example of a big graph is the Twitter network which contains more
than 40 million users and more than a billion edges (leader–follower relations) [4].
Big graphs may have other characteristics other than their huge sizes. Specifically,
big graphs can be characterized by the three V ’s of big data1: volume, variety, and
velocity [5]. A big graph (i) must have high volume, i.e., it must contain numerous2
vertices and edges, (ii) it may contain a variety of vertices/edges (e.g., we can form a big
graph using Facebook data consisting of undirected edges representing friendship rela-
tions, directed edges representing who follows whose status updates, undirected edges
representing connections among the persons in the same Facebook group, directed
edges representing who liked/shared whose status, etc.), and (iii) it may have high
velocity, i.e., it may change its topology with time (from now on, we will call such
graphs big dynamic graphs, whereas graphs that don’t change with time will be called
big static graphs). Note that while high volume is a necessary feature of big graphs, hav-
ing a high variety or velocity is not required for a network to be called a big graph.3
on these types of frameworks. We divide these frameworks into the following types
based on their working principles: (i) vertex-centric frameworks, (ii) block-centric
frameworks, (iii) subgraph-centric frameworks, (iv) matrix-based frameworks, and
(v) DBMS (database management system)-based frameworks. We dedicate separate
sections to each of these types of frameworks as discussed next.
5.2.1 Vertex-Centric Frameworks
Vertex-centric frameworks enforce the programmers to “think like a vertex”
(TLAV).4 Specifically, programmers define a user-defined function (UDF),5 which
is to be executed on each vertex using only the “local data” available to that vertex,
i.e., data about itself and its adjacent vertices/edges. These programs are executed
for a certain number of iterations or until a predefined convergence criterion is sat-
isfied. These programs typically communicate with each other via message passing
or by accessing a shared memory. Thus, it is possible to execute these functions in
parallel, thereby yielding high scalability.
As an example, we illustrate the procedure (Algorithm 1) of Pregel [1], the pio-
neer vertex-centric graph processing framework. Pregel allows the programmers to
define a function to initialize the vertices (init function in line 2 of Algorithm 1)
as well as a UDF (compute function in line 5). It takes three inputs: a graph, maxi-
mum number of iterations, and a set of vertices (called the active vertices6) upon
which the UDF will be executed in the first iteration. After initializing the vertices,
Pregel iteratively executes the compute function on the set of active vertices (lines 4
and 5). While running the compute function, an active vertex may send messages
to its neighbors. At the end of each superstep, all the vertices which received some
message(s) form the new set of active vertices, upon which the compute UDF is run
in the next iteration. The program terminates when no vertex receives any message
in a superstep or when the number of iterations crosses the given threshold (line 3).
Inputs: G = (V, E): the input graph whose set of vertices is V and set of edges is E
MaxIteration: maximum number of iterations (set it to ∞ if not needed),
A: initial set of active vertices.
1. foreach v ∈V do
2. v.init()
3. while numIteration ≤ MaxIteration AND A ≠ ∅ do
4. foreach v ∈ A do in parallel
5. v.compute()
6. A ← set of vertices which received message(s) in this superstep
Big Graph Analytics ◾ 101
Figure 5.1 Illustration of how shortest paths from vertex a to other vertices are
computed in Pregel. Active vertices are shown in gray shade, active vertices that
send messages are shown as dashed circles, messages from one vertex to another
are shown as dashed arcs, and thick dashed arcs represent messages which
cause their end vertices to update their current values in the next superstep. The
value beside each edge in the initial graph (inset [a]) indicates its weight, whereas
the value beside a dashed arc in the other insets [b–f] indicates a message sent by
a vertex to its out-neighbors.
We will now explain how the compute() UDF of Pregel can be defined
(Algorithm 2) to calculate single-source shortest paths (SSSPs, in short) in a graph
(Figure 5.1). Just like Dijkstra’s algorithm, we set the initial value of each vertex to
∞. Then a message containing the value zero is sent to the source vertex (vertex a
in Figure 5.1) so that it becomes an active vertex and as such the compute UDF is
executed on it in the next (i.e., first) iteration. During the execution of the compute
UDF, an active vertex checks if the smallest of its received messages is less than its
current value (line 1 of Algorithm 2), and if so, it replaces its current value by the
smallest received message and then sends a message (containing the sum of its new
value and the edge weight) to each of its out-neighbors (lines 2–4 of Algorithm 2).
Upon completion of the compute() UDF, the active vertex is always automatically
deactivated in Pregel unless it receives a new message.
Each superstep of Algorithm 1 calls this function (line 5 of Algorithm 1) on
each of the current active vertices, which causes updates of their states (Figure
5.1b–g). This process terminates when there is no active vertex left. The values of
the vertices in the final graph give their SSSP distances (Figure 5.1h).
Figure 5.2 Computation of CCs of a graph using HashMin algorithm [8]. Active
vertices are shown in gray color, messages from one vertex to another are shown
as directed edges (arcs), and their associated messages are shown beside those.
The value inside each vertex represents its current value (equal to its ID in the ini-
tial graph). In insets (b–d), undirected edges are shown between two n eighboring
vertices only when no communication is happening between them.
Big Graph Analytics ◾ 103
communicate with each other, (iii) how the graph is partitioned among worker
machines, and (iv) which type of programming model is used in this framework.
We will briefly discuss these classifications as well as their relative advantages and
disadvantages below. For a more detailed discussion of these classifications, please
read the review paper written by Mccune et al. [10].
5.2.1.1.1 Partitioning Graph
Distributed vertex-centric frameworks typically partition the input graph first
and then assign those partitions to different worker machines7 before the actual
execution of the framework starts. For example, Pregel uses a hash function to
determine the worker machine to which a vertex will be assigned. It also allows
the user to replace this default partitioning algorithm by an algorithm of his/
her choice [1]. Different algorithms can be used to partition the graph. Among
them, hash-based algorithms and range-based algorithms (which assign a range
of vertex IDs to a certain worker) are the simplest to implement and they are the
fastest to execute as well. However, a lot of edges may be present between the par-
titions generated by these algorithms. As a result, the number of messages passed
between worker machines (during the actual execution) can potentially be very
high, which may hamper the total efficiency of the framework. This issue can be
solved by employing more advanced partitioning algorithms that try to minimize
the number of edges between partitions. Although rigorous partitioning algo-
rithms (such as METIS [11]) can reduce the communication cost during actual
execution, they also incur a huge preprocessing time, especially for big graphs.
Another disadvantage of these algorithms is that unlike hashing-based methods
(which can be used to quickly compute the ID of the worker machine that will
process a query vertex), these algorithms make it difficult for a message-sender
(active vertex) to find the location of a message-receiver. Distributed hash tables
or some other mechanisms can be used to solve this problem, but those come with
additional communication overhead [12].
A lot of research has been going on that attempt to design better partition-
ing algorithms. These algorithms can be classified into two categories: distributed
heuristics and streaming algorithms. The goal of distributed heuristics [13,14] is to
detect communities efficiently using distributed label propagation algorithm [15].
Communities are good candidates for graph partitions because communities tend
to have few edges between them. Streaming partitioning algorithms, on the other
hand, attempt to assign each vertex to a worker at a constant time. For e xample,
LDG [16], a prominent streaming partitioning algorithm, puts a vertex to a partition
that contains most of its neighbors. Streaming algorithms are typically implemented
on top of the graph loader so that vertices are partitioned while the graph is loaded
from the file-system to the main memories.
So far, we have discussed about algorithms for partitioning vertices among
workers, which is used as a preprocessing step in most vertex-centric frameworks.
104 ◾ Data Analytics
However, partitioning vertices may cause the problem of load imbalance due to the
skewed degree distribution of real-world networks. Specifically, real graphs often
follow power-law degree distribution, i.e., only a few vertices in these graphs have
very high degrees (such vertices are called hubs), whereas most vertices tend to have
moderate or low degrees. Thus, partitioning of vertices may yield some partitions
containing the hubs. Processing these partitions typically requires more resources
than processing other partitions8 does; thus, working loads of different workers
becomes imbalanced (this is called the load imbalance problem) [17].
Vertex-cut algorithms attempt to solve this problem by partitioning edges9
among the workers. Specifically, the data of a vertex (e.g., its adjacency list/matrix)
is split into multiple workers (Figure 5.3). This approach has been found to reduce
load imbalance [7] and therefore decrease the communication cost [10]. However,
this approach does not come without price. To be able to use vertex-cut partitioning,
a framework must adopt some mechanism (via message passing, for example) to
synchronize these workers, i.e., whenever the vertex data is changed in a worker,
this update must be reflected into its mirrors. Some frameworks that use vertex-cut
partitioning10 are PowerGraph [18], PowerLyra [19], and GraphX [20].
Even if we use a good partitioning algorithm before the actual execution of the
framework, load imbalance may still occur after some iterations of the framework
because the number of active vertices in a worker may be drastically different from
that in another worker. In order to solve this issue, Mizan [12], GPS [21], and some
other frameworks migrate active vertices from one worker to another during the
execution of the framework. This strategy is called dynamic repartitioning [10].
systems, one process communicates with another by sending messages to it, whereas
in shared memory systems, processes communicate with each other by reading/
writing to a memory shared by all of them (Figure 5.4).
Message passing systems are very common among distributed vertex-centric
frameworks. If both the sender and the receiver vertices are processed by the same
worker, then the messages are directly delivered to the recipient vertices. However,
if sender and receiver vertices are in different machines, then the message is sent to
the worker that possesses the recipient. Typically, these messages are sent in batches,
i.e., the sender worker stores its messages in an outgoing message buffer until the
number of messages in that buffer exceeds a threshold, at which point all the
messages are sent to their recipient workers. Sending messages in batches reduces
communication overhead [22]. Moreover, it allows a worker to group messages
by their destinations11 and to send each such group-message in one go (instead
of sending multiple messages). This technique reduces network traffic and there-
fore improves the network throughput [23]. Similarly, a buffer is also used in the
Figure 5.4 Two ways of communication among the UDFs running on the v ertices
of the graph of Figure 5.3(a) are illustrated. (a) In message passing system, a
process sends a message to another process, which activates the second process.
Here, process a is sending messages to processes b, c, d, and e. Each arc indicates a
message and the big arrow represents a batch message from worker 1 to worker 2.
Incoming message buffer of worker 1 and outgoing message buffer of worker 2
are not shown for simplicity. (b) In the shared memory system, a process (here
process a) writes its state to a shared memory, which are later read by some other
processes (here processes b, c, d, and e). Here, solid lines represent edges in the
graph and dashed gray lines represent read/write operations on shared memory.
106 ◾ Data Analytics
receiver side to store the incoming messages temporarily before they are delivered
to the corresponding receivers and are subsequently processed by the UDF running
on the recipient vertices.
In a shared memory system, processes do not have to deal with messages because
they communicate with each other by accessing a shared memory, as if they are
running in a single machine. In reality, the shared memory is distributed across
different machines. Since the data of the same vertex are either split (in case of vertex-
cut partitioning used by PowerGraph, for example.) or copied (in case of ordinary
partitioning used by GraphLab, for example.) among different workers, ensuring
data consistency among them is a big challenge for shared memory s ystems. For this
purpose, GraphLab uses pipelined distributed locking [24], whereas PowerGraph
uses parallel Chandy-Misra locking [25]. Both these approaches ensure consistency
via inter-machine communication.
Another challenge for shared memory frameworks is to handle race condi-
tions that may arise when the same memory location is accessed by multiple
processes. In such a case, memory consistency is typically ensured via mutual
exclusions, i.e., by ordering the memory accesses of processes in such a way that
no two processes work on the same memory location at the same time. In other
words, shared memory systems ensure that the schedule of memory accesses by
the processors has an equivalent sequential schedule. Such a schedule is called a
serializable schedule [10].
5.2.1.1.4 Programming Model
Different frameworks adopt different programming models depending on their
mode of execution, communication, and/or other design choices. For example,
Message Passing Pregel [1], GPS [21], Maiter [30], GiraphUC PowerSwitch
Pregel+ [17], [31], MOCgraph* [28],
Giraph [29], Mizan [32], GraphHP [34],
[12] GiraphAsync [33] Hieroglyph [35]
Scatter–gather X-Stream
Pregel adopts a single phase model, i.e., in each iteration, only one UDF, namely,
the compute() function, is executed on active vertices. Giraph, MOCgraph, Mizan,
and so on (see Table 5.2) also adopt similar strategies.
X-Stream adopts a programming model consisting of two phases: (i) scatter: in
this phase, an active vertex broadcasts its values to some other vertices (typically,
its neighbors) and (ii) gather: in this phase, an active vertex collects data from the
adjacent vertices and then uses them to update its own value. Such a model is
t ypically used by frameworks which read/write edge values [37].
PowerGraph, PowerLyra, and PowerSwitch use a programming model c onsisting
of three phases: (i) gather, (ii) apply, and (iii) scatter. The meanings of gather and
scatter phases remain the same as that of X-Stream; the apply phase combines the
data gathered in the previous phase. Such a programming model is called gather–
apply–scatter or GAS model, in short. Briefly, in each iteration of such a model, an
active vertex gathers data from its neighboring vertices and/or edges, aggregates
those data to update its own value (apply phase), and then scatters this updated value
across its neighbors [18].
5.2.2 Block-Centric Frameworks
Vertex-centric frameworks like Pregel try to achieve extreme parallelism, even
among the processes running on the vertices within the same worker. This goal
comes at the cost of high communication overhead. To solve this problem some
systems adopt sequential execution of the vertices within the same worker while
keeping execution of worker processes parallel. Following Yan et al. [7], we will
call them block-centric frameworks. In this section, we will discuss two such
frameworks: BLOGEL [38] and Giraph++ [39].
5.2.2.1 BLOGEL
BLOGEL considers the problem of skewed degree distribution where a few nodes
have very high degrees and most others have low/moderate degrees. BLOGEL
Big Graph Analytics ◾ 109
s upports three types of tasks: (i) computation on vertices, (ii) partitioning graph,
and (iii) computation on blocks. In the first two tasks, the message receiver (called
the worker vertex), ∈Γv ,13 is identified by hashing id(u) when a vertex v sends
message to its neighbor. On the other hand, in block-centric tasks, the vertices
are divided into blocks by a partitioning algorithm at first. These blocks are then
executed by the B-workers (in block-centric computation, a worker is called a
B-worker). The block of a vertex v is denoted by block(v) and the worker of a block
B is denoted by worker(B). An ID is then assigned to a vertex v represented as a
triplet: trip(v) = {id(v), block(v), worker(v)}. A B-worker will be able to get the worker
and the block of vertex v using its ID. A block B can reach any of its vertices. Also, a
block can send message to any other block B while worker(B) is available. Similarly,
it can send message to any vertex v if worker(v) is available. Each B-worker has two
message buffers: one is to receive/send messages to/from vertices and the other is for
block level messaging. A combiner is used to combine these two which is associated
with each buffer.
BLOGEL provides numerous predefined partitioning algorithms. It also sup-
ports user-defined partitioners. While partitioning, each vertex v is assigned to a
block, denoted by block(v). After partitioning, each block B is assigned to a worker,
denoted by worker(B), in such a way that load imbalance is minimized. The steps
followed for assigning workers to blocks are given below:
Step 1: At first, each partitioner divides the group of vertices into blocks and
sends the master the number of vertices of each block. The master collects
these numbers from the partitioners and uses these to compute an aggregated
number of vertices in blocks.
Step 2: Using a greedy algorithm, the master then computes the workload and
broadcasts the block-to-worker assignment to each block.
Step 3: According to the received assignment, each partitioner selects its worker
from its vertices and transmits this to all workers.
BLOGEL can be applied to solve different graph problems, such as for computing
PageRank of vertices. It has been found that PageRank runs faster (especially on
web-graphs14) in BLOGEL than in Pregel because the former benefits from initial-
izing PageRank values using the information of its blocks. In Blogel, calculation
of PageRank value is done in two steps. The first step is done in the block mode
and, in this step, each block B calculates the local PageRank of each vertex in B
from the single-machine PageRank algorithm. Then the BlockRank is computed
by combining all the PageRank results using vertex-centric PageRank algorithm,
and therefore, a weight is assigned to all outgoing edges. Then the BlockRank is
dispersed to outgoing neighbors in proportion to the assigned edge weights. Finally,
in the second step, the standard PageRank algorithm is applied for the total graph
using the current BlockRank.
110 ◾ Data Analytics
5.2.2.2 GIRAPH++
GIRAPH++ is a distributed graph processing platform, which exposes the partition
structure to the programmers so that they can exploit it via asynchronous com-
munication within the partition. It has been found that this technique makes it
more flexible and more efficient than the corresponding vertex-centric frameworks
because it has less communication overhead.
In this model, the graph is divided into multiple subgraphs. The vertices within
each subgraph are called internal vertices. The neighbors of internal vertices outside the
subgraph are called boundary vertices. Thus, internal vertices can be in only one sub-
graph, but boundary vertices can be in more than one subgraph. A partition has access
to all the information of its internal vertices, but it knows only the vertex numbers of
its boundary vertices. In GIRAPH++ boundary, vertices can only send messages to the
primary copies of the internal vertices since communication among the internal vertices
are cheap and spontaneous. State changes of boundary vertices have to wait for the next
superstep, while internal vertices can change their states at any point of time. The inter-
nal vertices can be active or inactive, whereas boundary vertices do not have such states.
Algorithm 3 shows how PageRank can be implemented in GIRAPH++. Here
each vertex has two components/fields: (i) its current PageRank score (v.pr in
Algorithm 3) and (ii) the iterative updates of its PageRank score computed from
other vertices within its partition, which is called its delta value (v.delta in Algorithm
3). Initially, these fields are initialized (lines 1–4 of Algorithm 3) with zeros (except
that the delta values of active internal vertices are initialized with (1-d), where d is
the damping factor). Then these two fields of all active internal vertices are iteratively
updated and all their neighbors’ delta values are updated accordingly (lines 5–10).
Finally, delta values of all the boundary vertices are reset (lines 11 and 12).
Inputs: G = (V,E): the input graph whose set of vertices is V and set of edges is E
MaxIteration: maximum number of iterations (set it to ∞ if not needed)
A: set of active internal vertices
B: set of boundary vertices
d: damping factor (typically, 0.85)
Data: v.pr: page rank value of vertex v
v.delta: current delta value of vertex v
v.incomingMessages: list of incoming messages of vertex v
v.numOutGoingEdges: number of outgoing edges incident upon v
v.neighbors: list of neighbors of v
1. if superstep > MaxIteration then exit
2. if superstep = 0 then
3. foreach v in V do v.pr = 0, v.delta = 0
Big Graph Analytics ◾ 111
5.2.3 Subgraph-Centric Frameworks
Although many graph problems can be solved by “thinking like a vertex,” many
problems cannot be solved via a TLAV framework. For example, computing local
clustering coefficient of a vertex requires the information of the whole subgraph
induced by that vertex as well as by its neighbors. To solve such problems, some
frameworks (such as NScale [40] and Arabesque [41]) propose to “think like a
subgraph”, i.e., they ask the programmers to write UDFs on subgraphs, instead of
vertices. We will discuss one such framework here, namely, NScale.
5.2.3.1 NScale
NScale [40] is a Hadoop-based big graph framework, which allows distributed pro-
cessing of subgraphs in the cloud. It provides three special features: (i) subgraph-
centric computation on a graph, (ii) computation on selected subgraphs, and (iii)
space-efficient packing of subgraphs into memory.
In this framework, the programmer needs to specify (i) the subgraphs in which
the user is interested and (ii) an UDF that will be executed on these subgraphs.
The UDF is written using a graph API called BluePrints. The NScale execution
engine ensures that the UDF has access to only its corresponding subgraph. It also
allows the users to select a subgraph formed by the k-hop neighborhood of a vertex
filtered by some given properties of the nodes and edges. Thus, the user can choose
any desired subgraph with special attributes (e.g., subgraphs formed by vertices
annotated with certain hashtags). Then the selected subgraph will entirely reside in
a single machine to be processed.
NScale has two major components:
1.
GEP (graph extraction and packing module): It extracts the relevant sub-
graphs, decides the optimum number of machines that will process these
subgraphs and distributes the workload in a balanced fashion.
2.
Distributed execution engine: It executes the UDFs on subgraphs of
interest.
112 ◾ Data Analytics
In the GEP module, Map Reduce jobs are executed depending on the size of the
graph and sizes of the machines. This module follows the following steps.
First step: In this map step, the GEP module reads the graph and finds out
the query vertices. It also filters the vertices and edges that do not match with
the purpose/interest of the user and evaluates the space necessary according to the
graph structure and weights. There are a few cases to consider here.
Case 1: If a single machine can contain the filtered graph, then all the vertices,
their weights, and their incident edges are sent to the reducer that constructs
the subgraph and applies the packing algorithm to map each vertex to a
partition.
Case 2: If a single machine cannot contain the filtered graph, then a distributed
process completes both subgraph extraction and packing. The steps of this
process depend on the radius of the subgraph of interest.
Second step: It is the packing stage where the extracted subgraphs are packed into
minimum possible number of partitions. A standard bin packing algorithm is used
to find the minimum number of partitions.
The distributed engine provides different execution modes: (i) the vector bitmap
mode allows parallel execution on subgraph-centric UDFs, (ii) the batch bitmap mode
decreases memory consumption but increases execution time, and (iii) the single-bit
bitmap mode decreases space requirement, but it allows only serial e xecution of UDFs.
NScale, like many other distributed graph frameworks, uses the bulk synchro-
nous protocol (BSP) to support iterative processes. However, unlike the original
BSP model, messages within a subgraph are transferred using shared memory,
which reduces communication overhead. Only messages between subgraphs are
transmitted via a barrier synchronization step to confirm the completion of message
exchange among the subgraphs.
5.2.4 Matrix-Based Frameworks
Some frameworks such as Hama [42], PEGUSUS [43], GBASE [44], and System
ML [45] represent the graph as a matrix (e.g., the adjacency matrix) and translate
different graph computations into matrix operations. Some of these frameworks
require the programmers to think of graphs as matrices, while others still rely upon
traditional programming models, thereby hiding the underlying matrix calcula-
tions. We call both these types of frameworks matrix-based frameworks. In this
section, we will discuss one such framework, namely, PEGUSUS, to give the reader
an idea about such frameworks.
5.2.4.1 PEGUSUS
PEGUSUS is a peta-scale graph framework that can be used to quickly process a
big static graph. It is implemented on the Hadoop platform. The main primitive
Big Graph Analytics ◾ 113
5.2.5 DBMS-Based Frameworks
Several frameworks, such as Pregelix [46], DG-SPARQL [47], and GraphX [20],
depend on DBMSs and their operations to store and analyze big graphs. These
frameworks attempt to bring the fruits of long research on the storage and query
optimization techniques in the area of DBMSs to the area of big graph a nalysis.
Typically, these types of frameworks use some kind of database as their back
end engines. For example, Pregelix uses traditional relational database, whereas
DG-SPARQL uses graph database in the back end. In the next two subsections, we
will discuss about these two frameworks.
5.2.5.1 Pregelix
Pregelix [46] is a big graph analytics system that uses an iterative dataflow approach.
It transformed the Pregel programming model into a set of database operations.
To reconstruct the execution of Pregel states, Pregelix uses a set of nested relations
shown in Table 5.3.
Here, each instance of Vertex relation is read as an input and constitutes a row
of this relation. Each vertex has three attributes: vertexID, halt, value, and neigh-
bors. The value and neighbors are user defined that represent its current state and
list of neighbors. Vertices whose halt value is False are considered to be active.
The Msg relation represents the messages sent to an instance of Vertex whose ID
is destinationID and whose received message is denoted by payload. GS relation
holds the global state of execution of an UDF. The aggregate attribute of this rela-
tion represents an aggregated value over all iterations and the superstep denotes the
current iteration count. If halt = True in this relation, then the program terminates.
Pregelix has been implemented on top of Hyracks platform, a data-parallel
engine capable of processing big data. Pregelix first loads the graph (i.e., the
Vertex relation) from a distributed file system to Hyracks, partitions it using the
vertexIDs, and then dispatches these partitions to Hyracks for processing them
in parallel. The partitioning can be done by a user defined partitioning function.
After finishing the full process (determined by the attributes of the GS relation), the
partitioned vertex relation is scanned and dumped back to the file system. During
execution, the message-passing behavior of Pregel is achieved via joining Vertex
and Msg relations. On the other hand, different UDFs of Pregel are written by
the programmers as database queries, which may in turn affect the relations of
Table 5.3. For example, the compute UDF may affect the relations in the following
way in a superstep: it may change some attributes of its corresponding Vertex tuple,
it may add/update the list of outgoing messages in Msg relation, it may change the
attributes of GS relation, or it may mutate the graph (via adding/deleting tuples
from Vertex relation).
5.2.5.2 DG-SPARQL
DG-SPARQL [47] is a distributed graph database management system. Graph data-
bases store the graph data as entities (vertex) and relationships (edges). Each vertex
in a graph database stores a list of relationship records that signify its relationships
to other vertices. These relationship records are organized by type and direction
and may hold additional attributes. Whenever a JOIN operation is executed, the
database just uses this list and directly accesses its neighbors, which eliminates the
need for an expensive search or match computation that are done in traditional
relational databases. As a result, graph databases often outperform relational data-
bases in terms of efficiency. Graph databases also allow the user to perform different
graph queries including subgraph matching queries, path algebras, regular path
queries, and reachability queries. Queries in a graph database are written using a
graph query language, such as OpenCypher, SPARQL, GRAPQL, or Gremlin.15
For example, DG-SPARQL uses a version of SPARQL as a query language.
DG-SPARQL is a distributed implementation of G-SPARQL [48], which is a
centralized graph DBMS. Following G-SPARQL, DG-SPARQL provides a hybrid
storage system in which the structure of a big attributed graph resides in the distrib-
uted memory and the graph data resides in a relational database whose replicas are
contained in each node. This hybrid approach allows DG-SPARQL to execute part
of the query using fast in-memory graph traversal algorithms. The rest of the query
is executed in RDBMS via SQL as usual. It also optimizes the query processing by
(i) using a divide and conquer algorithm that splits the query plan into some manage-
able chunks and (ii) cleverly selecting the appropriate number of database nodes based
on a cost model. It allows the user to search the graph based on both the structural
properties and the attributes of the vertices/edges of the graph. Specifically, the
following types of structural queries can be processed in DG-SPARQL: (i) reachability
query: check if any path exists between two vertices, (ii) shortest path query: find the
shortest path between two vertices, and (iii) pattern matching queries: search for the
incidences of subgraphs based on patterns in a big graph.
Figure 5.5 shows an attributed graph representing a student database which
contains information (age) about some students, courses, degrees, departments,
and the interrelations among them. Figure 5.6 shows a sample query on this graph
database, which searches for all pairs of students (A, B) such that A’s age is at least
22 years and A is senior to B. In this code, line 3 represents the requirement that
A must be senior to B, lines 4 and 5 represent the filtering condition that the age
116 ◾ Data Analytics
of A must be at least 22, and line 6 is used to filter out any possible paths between
vertices within a five-hop distance.
them an extra advantage over their distributed counterparts. Among these frame-
works, Ligra [49], GraphChi [50], X-Stream [37], FlashGraph [51], GraphMat [52],
and GRACE [53] are some of the well-known ones. Ligra uses a fast implemen-
tation of BFS [54] to encode traversal-based graph algorithms. Both GraphChi
and X-Stream use edge-centric GAS programming model and provide out-of-core
execution of graph algorithms by loading a graph into the memory, one partition
at a time. FlashGraph keeps vertex data in the memory and edge data in SSDs.
In order to decrease I/O cost, it only accesses those edge data that are required by
the algorithm and are not already available in the cache. Grace is an asynchronous
block-centric graph framework that can process graphs efficiently when they fit into
the main memory.
In this section, we will discuss about one single-machine framework, namely,
GraphMat, in greater details. GraphMat [52] is a single-machine multicore static
graph framework that achieves good multicore scalability with 1.1–1.7 times speed
up as compared to other notable distributed frameworks. It is implemented based
on a generalized sparse matrix vector multiplication (SPMV) scheme that is similar
to the GIM-V technique of PEGUSUS. However, GraphMat has two distinctions/
advantages over PEGUSUS: (i) unlike PEGUSUS, it does not suffer from network
communication overhead, and (ii) it does not expose the matrix calculations to the
user/programmer; rather its front end is based on the popular GAS programming
model of vertex-centric frameworks—which makes it easy for the programmers to
make a transition from vertex-centric frameworks to GraphMat.
In the back end, GraphMat uses SPMV that exploits the optimization tech-
niques developed over the last few decades for manipulating sparse matrices in order
to do efficient computations on matrix representations of graphs. Thus, GraphMat
combines the best features of “both worlds”: expressiveness of vertex-centric GAS
model and efficiency of optimized algorithms for sparse matrix computation.
Specifically, GraphMat is built on the idea that many vertex-centric iterative
processes can be transformed into some matrix vector operations. For example, for
an unweighted graph, multiplying the transpose of its adjacency matrix with a vector
of all ones yields a vector of vertex in-degrees (Figure 5.7). Similarly, vertex out-
degrees can be obtained by multiplying the adjacency matrix with an all one vector.
GraphMat programming model has four functions that the programmer/user
needs to define: send_message(), process_message(), reduce(), and apply(). The first
and last UDFs are analogous to the scatter() and apply() UDFs of the GAS model,
respectively. On the other hand, the second and third UDFs together mimic the
gather() UDF of GAS model. From the programmer’s point of view (front end),
these UDFs work in the following way. At first, each vertex is initialized and some
of them are activated. The send_message() function is called on each active vertex,
which produces a message based on the value of that vertex. The process_message()
function reads the incoming messages along with the information of the edge
through which it came and then processes them to compute a processed message.
The reduce() function combines all the processed messages received by a vertex to
118 ◾ Data Analytics
Figure 5.7 (a) A graph, (b) its adjacency matrix, and (c) in-degree calculation
via SPMV.
compute a single value. The apply() function uses this value to decide whether/how
the vertex data should be updated.
However, in the back end, these functions use SPMV to achieve the goal.
Specifically, send_message() UDF creates a sparse vector representing the active
vertices. The tasks of process_message() and reduce() UDFs are accomplished via
an operation (typically, multiplication) on this vector and the adjacency matrix
of the input graph. Finally, apply() does the same thing as the apply() UDF of
GAS model.
how PageRank scores (or some centrality measures) of vertices change with time,
how to classify vertices of dynamic networks [60], etc.
In general, two types of analyses can be done on dynamic graphs: (i) point-
in-time analysis—which analyzes a specific snapshot to answer some questions
about it (e.g., what are the highest degree vertices in the Facebook friendship graph
on Dec 30, 2017?) and (ii) time-range analysis—which analyzes graphs in a time
range to answer some questions related to their dynamic nature (e.g., which non-
hubs became hubs17 in Facebook friendship graph between Jan 1, 2010, and Jan 1,
2017?). While traditional big static graph frameworks can still be used for point-in-
time analysis, time-range analysis requires special purpose frameworks.
Note that although both temporal and streaming graphs raise similar research
questions, answering them often requires one to apply distinct techniques. In gen-
eral, graph frameworks should be designed specifically to store and analyze any of
these types of dynamic graphs. In the next two subsections we will discuss some
notable frameworks that can handle each of these types of dynamic graphs.
5.4.1.1 DeltaGraph
DeltaGraph is a distributed graph database system designed to support snapshot
queries in a big dynamic graph. Specifically, it supports two types of queries: (i)
single-point snapshot query—which is used to search for the snapshot/graph of a
given time point (e.g., “retrieve the subgraph induced by the vertices a, b, and c
on Jan 2, 2018”) and (ii) multipoint snapshot query—which is used to search for
all the snapshots between a range of time points (e.g. “retrieve all the subgraphs
induced by the vertices a, b, and c between Jan 2, 2018 and Jun 13, 2018”). To
process queries in an efficient manner, it maintains an in-memory index in the
form of a rooted directed acyclic graph (DAG). The root of this DAG is an empty
graph, the lowest level vertices of this DAG represent18 snapshots, and internal
nodes typically represent the intersection graph19 of its children. An edge between
a parent and a child in this DAG is associated with the information about the dif-
ference (called delta) between the graphs represented by the parent and child. Thus,
the graph corresponding to a node of this DAG can be formed by applying the
deltas across all the edges from the root to that node. A weight is assigned to each
such edge (parent–child relation) which captures the cost of reading its associated
delta and applying that on the parent graph to form the child graph. Thus, single
point snapshot queries can be efficiently answered by computing the shortest path
120 ◾ Data Analytics
from the root of this DAG to one of its leaves, which correspond to the given time
point. Similarly, multipoint snapshot queries can be answered by first finding the
leaves that fall under the given time range and then computing a Steiner tree that
connects those leaves to the root.
5.4.1.2 Chronos
Chronos has been designed to efficiently store and analyze big dynamic graphs.
It partitions the vertices of the input dynamic graph and then assigns each of
these partitions along with the entire history of vertex-states in that partition to
a worker machine. Han et al. [61] noticed that partitioning vertices in this fash-
ion yields much better performance than partitioning snapshots. Partitioning this
way allows Chronos to execute UDFs on each vertex across all the snapshots in a
batch. This strategy also allows Chronos to do incremental computations (doing a
computation on a snapshot utilizing the results of that computation done on a pre-
vious snapshot;e.g., computing the SSSP distances in the first snapshot, using these
results to update SSSP distances in the second snapshot, then using these results
to update SSSP distances in the second snapshot, and so on) efficiently. Chronos
also allows one to store and use deltas of consecutive snapshots to achieve greater
efficiency in incremental computation. It can be implemented both in a single pow-
erful machine and in a distributed system.
5.5.2 STINGER
STINGER (Spatio-Temporal Interaction Networks and Graphs Extensible
Representation) is a single-machine framework that supports storing and
122 ◾ Data Analytics
analyzing streaming graph updates that may come in a nonsteady rate and as
such may not have temporal resolution. STINGER maintains a single in-memory
representation of the most recent snapshot. For this purpose, it uses a modified
version of the CSR data structure to store graphs. This data structure combines
the ideas of both CSR and adjacency list. Specifically, it stores the record of the
neighbors of each vertex as a list of arrays. When a neighbor gets deleted, a nega-
tive value is placed in its cell—which creates a “hole” in that place. When a new
neighbor is added, its record is placed in a hole; if no hole is present, a new array
is created to place it and the new array is linked with the existing arrays of neigh-
bors. Thus, STINGER is able to update the graph (edge insertions/deletions)
efficiently. To achieve high parallelism, STINGER runs graph-reader processes
(called by the graph algorithm running in this framework) along with graph-
updater processes in parallel. The authors of STINGER reasoned that in a big
graph, the probability of conflict between graph readers and graph updaters will
be very low. However, this may cause inconsistencies since different passes of a
reader process may read different information. STINGER tolerates such incon-
sistencies because it assumes that the current graph is already an approximate
representation of the concerned real-world network, in which case, these incon-
sistencies will not affect the result significantly.
5.6 Conclusions
In this chapter, we have explored different categories of frameworks that are
capable of analyzing big graphs, discussed their features, and highlighted
their advantages/disadvantages. We also briefly discussed about some notable
frameworks. We hope that this discussion will help the reader to get an idea
about the evolution of research in this area. Specifically, the very first frame-
work for processing big static graphs was Pregel. It is a vertex-centric distributed
f ramework, which is conceptually easy to design and understand. However, it
may suffer from high idle time due to its strict adherence to the BSP model
even at the vertex level. To solve this issue, block-centric frameworks were pro-
posed. However, both these types of frameworks force the programmer to write
operations from a vertex-centric point of view, which makes it difficult to encode
graph algorithms that require the programmer to think like a subgraph (e.g.,
algorithms for computing dense clusters). Subgraph-centric, matrix-based, and
DBMS-based frameworks offer different solutions to this problem. While many
of these frameworks are designed as distributed systems, rapid decrease in costs
of high-configuration workstations encouraged many researchers to develop
single-machine frameworks of similar/superior capabilities. Also, currently
the focus of this line of research is shifting more and more towards processing
dynamic graphs since they better represent real-world networks than their static
counterparts.
Big Graph Analytics ◾ 123
Notes
1 Because big graph is really a type of big data.
2 We could not find any standard lower bound of graph size that has been used to
define big graphs. However, most papers on big graphs discuss about graphs with at
least millions, and most often, billions of vertices/edges.
3 We have found in the literature that whenever a graph has high volume it is consid-
ered to be a big graph, even if it does not have a high volume/velocity.
4 Vertex-centric frameworks are also known as vertex-oriented or TLAV frameworks.
5 In TLAV frameworks, this function is also called a vertex program kernel or simply, a
vertex program.
6 A vertex upon which the UDF is run is called an active vertex.
7 To keep our discussion simple, previously (in Section 5.2.1) we have discussed two
Pregel algorithms in such a way that it may seem to the reader that each vertex is
assigned to only one processor. However, in reality, a big graph contains billions of
nodes and as such multiple vertices may be assigned to the same worker.
8 Because hubs need to communicate with many more vertices than other vertices,
since it has too many neighbors as compared to others.
9 This approach is called vertex-cut partitioning because it “cuts/splits” the data of some
vertices into different machines that contain their incident edges.
10 Since these frameworks assign edges (instead of vertices) to workers and/or requires
the programmer to define some UDFs for processing edges, they are sometimes
called edge-centric frameworks [38]. However, we consider them to be vertex-
centric frameworks, just like Yan et al. [7] and Mccune et al. [10] did, because
these frameworks require the programmer to define some vertex-centric UDFs, i.e.,
f unctions for processing vertices (just like other vertex-centric frameworks).
11 This may happen when multiple vertices are assigned to the same worker and some
of those vertices want to send messages to the same recipient. For example, vertices a
and e in Figure 5.4(a) tries to send messages to vertex d.
12 The time required to finish communication between two machines in the network.
13 Γv is the set of adjacent vertices of v.
14 The authors of BLOGEL construct a web-graph by considering each webpage as a
vertex and put an edge between two webpages if one contains a link to another. They
also grouped all vertices in a block that have the same host name.
15 https://react-etc.net/entry/graph-query-languages-graphql-opencypher-gremlin-and-sparql.
16 Typically, time is considered to be a discrete variable in big graph analysis [7].
17 Vertices with very high (statistically significant) degrees are called hubs.
18 Each leaf of this DAG represents a snapshot; none of them actually contains a graph.
Graphs are partitioned into different workers, as usual.
19 Internal nodes can represent some other graph (other than intersection graph) formed
from the graphs of its children.
References
1. G. Malewicz, M. H. Austern, A. J. C. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G.
Czajkowski, “Pregel: a system for large-scale graph processing,” In Proceedings of the 2010
International Conference on Management of Data—SIGMOD ‘10, 2010, p. 135.
124 ◾ Data Analytics
2. A. Cuzzocrea and I.-Y. Song, “Big graph analytics: The State of the Art and Future
Research Agenda,” In Proceedings of the 17th International Workshop on Data Warehousing
and OLAP—DOLAP ‘14, 2014, pp. 99–101.
3. U. Kang and C. Faloutsos, Big graph mining: Algorithms and discoveries, ACM SIGKDD
Explor. Newsl., vol. 14, no. 2, ACM, p. 29, 30-Apr-2013.
4. A.R. Benson, D.F. Gleich, and J. Leskovec, Higher-order organization of complex
networks, Science, vol. 353, no. 6295, pp. 163–166, Jul. 2016.
5. P. Zikopoulos and C. Eaton, Understanding Big Data: Analytics for Enterprise Class Hadoop
and Streaming Data, New York: McGraw-Hill Osborne Media, 2011.
6. S. Ma, J. Li, C. Hu, X. Lin, and J. Huai, Big graph search: Challenges and techniques,
Front. Comput. Sci., vol. 10, no. 3, pp. 387–398, Jun. 2016.
7. D. Yan, Y. Bu, Y. Tian, and A. Deshpande, Big graph analytics platforms, Found. Trends
Databases, vol. 7, no. 1–2, pp. 1–195, 2017.
8. D. Yan, J. Cheng, K. Xing, Y. Lu, W. Ng, and Y. Bu, Pregel algorithms for graph
connectivity problems with performance guarantees, Proc. VLDB Endow., vol. 7, no. 14,
pp. 1821–1832, Oct. 2014.
9. Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J.M. Hellerstein, Distributed
GraphLab: A framework for machine learning and data mining in the cloud, Proc. VLDB
Endow., vol. 5, no. 8, pp. 716–727, Apr. 2012.
10. R.R. McCune, T. Weninger, and G. Madey, “Thinking Like a Vertex: a Survey of Vertex-
Centric Frameworks for Distributed Graph Processing,” vol. 48, no. 2, 2015.
11. G. Karypis and V. Kumar, A fast and high quality multilevel scheme for partitioning
irregular graphs, SIAM J. Sci. Comput., vol. 20, no. 1, pp. 359–392, Jan. 1998.
12. Z. Khayyat, K. Awara, A. Alonazi, H. Jamjoom, D. Williams, and P. Kalnis, “Mizan: a
system for dynamic load balancing in large-scale graph processing,” In Proceedings of the
8th ACM European Conference on Computer Systems—EuroSys ‘13, 2013, p. 169.
13. G.M. Slota, K. Madduri, and S. Rajamanickam, “PuLP: Scalable multi-objective multi-
constraint partitioning for small-world networks,” In 2014 IEEE International Conference
on Big Data (Big Data), 2014, pp. 481–490.
14. L. Wang, Y. Xiao, B. Shao, and H. Wang, “How to partition a billion-node graph,” In
2014 IEEE 30th International Conference on Data Engineering, 2014, pp. 568–579.
15. U.N. Raghavan, R. Albert, and S. Kumara, Near linear time algorithm to detect commu-
nity structures in large-scale networks, Phys. Rev. E, vol. 76, no. 3, p. 36106, Sep. 2007.
16. I. Stanton and G. Kliot, “Streaming graph partitioning for large distributed graphs,” In
Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery
and Data Mining—KDD ‘12, 2012, p. 1222.
17. D. Yan, J. Cheng, Y. Lu, and W. Ng, “Effective techniques for message reduction and
load balancing in distributed graph computation,” In Proceedings of the 24th International
Conference on World Wide Web—WWW ‘15, 2015, pp. 1307–1317.
18. J.E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin, “PowerGraph: distributed
graph-parallel computation on natural graphs,” In 10th USENIX conference on Operating
Systems Design and Implementation (OSDI), 2012, pp. 17–30.
19. R. Chen, J. Shi, Y. Chen, and H. Chen, “PowerLyra: differentiated graph computation
and partitioning on skewed graphs,” In Proceedings of the Tenth European Conference on
Computer Systems—EuroSys ‘15, 2015, pp. 1–15.
20. J.E. Gonzalez, R.S. Xin, A. Dave, D. Crankshaw, M.J. Franklin, and I. Stoica, “GraphX:
graph processing in a distributed dataflow framework,” In 11th USENIX Symposium on
Operating Systems Design and Implementation (OSDI), 2014, pp. 598–613.
Big Graph Analytics ◾ 125
21. S. Salihoglu and J. Widom, “GPS: a graph processing system,” In Proceedings of the 25th
International Conference on Scientific and Statistical Database Management—SSDBM,
2013, p. 1.
22. J.D. Cavanaugh, “Protocol Overhead in IP/ATM Networks,” 1994.
23. A. Lulli, “Distributed Graph Processing: Algorithms And Applications,” 2017.
24. E.W. Dijkstra, Hierarchical ordering of sequential processes, In The Origin of Concurrent
Programming, edited by P.B. Hansen, New York: Springer, 1971, pp. 198–227.
25. K.M. Chandy and J. Misra, The drinking philosophers problem, ACM Trans. Program.
Lang. Syst., vol. 6, no. 4, pp. 632–646, Oct. 1984.
26. L.G. Valiant and L.G., A bridging model for parallel computation, Commun. ACM, vol.
33, no. 8, pp. 103–111, Aug. 1990.
27. Y. Lu, J. Cheng, D. Yan, and H. Wu, Large-scale distributed graph computing systems,
Proc. VLDB Endow., vol. 8, no. 3, pp. 281–292, Nov. 2014.
28. C. Xie, R. Chen, H. Guan, B. Zang, and H. Chen, “SYNC or ASYNC: time to fuse
for distributed graph-parallel c omputation,” In Proceedings of the 20th ACM SIGPLAN
Symposium on Principles and Practice of Parallel Programming—PPoPP 2015, 2015, vol.
50, no. 8, pp. 194–204.
29. A. Ching, S. Edunov, M. Kabiljo, D. Logothetis, and S. Muthukrishnan, One trillion
edges: Graph processing at Facebook-scale, Proc. VLDB Endow., vol. 8, no. 12, pp.
1804–1815, Aug. 2015.
30. Y. Zhang, Q. Gao, L. Gao, and C. Wang, Maiter: An asynchronous graph processing
framework for delta-based accumulative iterative computation, IEEE Trans. Parallel
Distrib. Syst., vol. 25, no. 8, pp. 2091–2100, Aug. 2014.
31. M. Han and K. Daudjee, Giraph unchained, Proc. VLDB Endow., vol. 8, no. 9, pp.
950–961, May 2015.
32. C. Zhou, J. Gao, B. Sun, and J.X. Yu, MOCgraph: Scalable distributed graph processing
using message online computing, Proc. VLDB Endow., vol. 8, no. 4, pp. 377–388, Dec.
2014.
33. Y. Liu, C. Zhou, J. Gao, and Z. Fan, “GiraphAsync: supporting online and offline graph
processing via adaptive asynchronous message processing,” In Proceedings of the 25th
ACM International on Conference on Information and Knowledge Management—CIKM
‘16, 2016, pp. 479–488.
34. Q. Chen, S. Bai, Z. Li, Z. Gou, B. Suo, and W. Pan, “GraphHP: A Hybrid Platform for
Iterative Graph Processing,” 2017.
35. X. Ju, H. Jamjoom, and K.G. Shin, Hieroglyph: Locally-sufficient graph processing via
compute-sync-merge, Proc. ACM Meas. Anal. Comput. Syst., vol. 1, no. 1, p. 9, 2017.
36. S. Tasci and M. Demirbas, “Giraphx: parallel yet serializable large-scale graph processing,”
In Proceedings of the 19th international conference on Parallel Processing, Springer-Verlag,
2013, pp. 458–469.
37. A. Roy, I. Mihailovic, and W. Zwaenepoel, “X-Stream: Edge-centric graph processing
using streaming partitions,” In Proceedings of the Twenty-Fourth ACM Symposium on
Operating Systems Principles—SOSP ‘13, 2013, pp. 472–488.
38. D. Yan, J. Cheng, Y. Lu, and W. Ng, “Blogel: A block-centric framework for distributed com-
putation on real-world graphs,” Proc. VLDB Endow., vol. 7, no. 14, pp. 1981–1992, Oct. 2014.
39. Y. Tian, A. Balmin, S.A. Corsten, S. Tatikonda, and J. McPherson, From think like a
vertex to think like a graph, Proc. VLDB Endow., vol. 7, no. 3, pp. 193–204, 2013.
40. A. Quamar, A. Deshpande, and J. Lin, NScale: Neighborhood-centric large-scale graph
analytics in the cloud, VLDB J., vol. 25, no. 2, pp. 125–150, 2016.
126 ◾ Data Analytics
41. C.H.C. Teixeira, A.J. Fonseca, M. Serafini, G. Siganos, M.J. Zaki, and A. Aboulnaga,
“Arabesque: a system for distributed graph mining,” In Proceedings of the 25th Symposium
on Operating Systems Principles—SOSP ‘15, 2015, pp. 425–440.
42. S. Seo, E.J. Yoon, J. Kim, S. Jin, J.-S. Kim, and S. Maeng, “HAMA: an efficient matrix
computation with the mapreduce framework,” In 2010 IEEE Second International
Conference on Cloud Computing Technology and Science, 2010, pp. 721–726.
43. U. Kang, C.E. Tsourakakis, and C. Faloutsos, “PEGASUS: a peta-scale graph mining
system implementation and observations,” In 2009 Ninth IEEE International Conference
on Data Mining, 2009, pp. 229–238.
44. U. Kang, H. Tong, J. Sun, C.-Y. Lin, and C. Faloutsos, “GBASE: a scalable and general
graph management system,” In Proceedings of the 17th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining—KDD ‘11, 2011, p. 1091.
45. M. Boehm, M.W. Dusenberry, D. Eriksson, A.V. Evfimievski, F.M. Manshadi, N. Pansare,
B. Reinwald, F.R. Reiss, P. Sen, A.C. Surve, and S. Tatikonda, SystemML: Declarative
machine learning on spark, Proc. VLDB Endow., vol. 9, no. 13, pp. 1425–1436, Sep. 2016.
46. Y. Bu, V. Borkar, J. Jia, M.J. Carey, and T. Condie, Pregelix: Big(ger) graph analytics on
a dataflow engine, Proc. VLDB Endow., vol. 8, no. 2, 2015.
47. O. Batarfi, R. Elshawi, A. Fayoumi, A. Barnawi, and S. Sakr, A distributed query execu-
tion engine of big attributed graphs, Springerplus, vol. 5, no. 1, p. 665, 2016.
48. S. Sakr, S. Elnikety, and Y. He, “G-SPARQL: a hybrid engine for querying large attrib-
uted graphs,” In Proceedings of the 21st ACM International Conference on Information and
Knowledge Management—CIKM ‘12, 2012, p. 335.
49. J. Shun, G.E. Blelloch, J. Shun, and G.E. Blelloch, “Ligra: a lightweight graph pro-
cessing framework for shared memory,” In Proceedings of the 18th ACM SIGPLAN
Symposium on Principles and Practice of Parallel Programming—PPoPP ‘13, 2013, vol.
48, no. 8, p. 135.
50. A. Kyrola, G. Blelloch, and C. Guestrin, “GraphChi: large-scale graph computation on
just a PC,” In Proceedings of the 10th USENIX Conference on Operating Systems Design and
Implementation, 2012, pp. 31–46.
51. D. Zheng, D. Mhembere, R. Burns, J. Vogelstein, C.E. Priebe, and A.S. Szalay,
“FlashGraph: processing billion-node graphs on an array of commodity SSDs,” In 13th
USENIX Conference on File and Storage Technologies (FAST ‘15), 2015, pp. 45–58.
52. N. Sundaram, N. Satish, M. M. A. Patwary, S. R. Dulloor, M. J. Anderson, S. G. Vadlamudi,
D. Das and P. Dubey, GraphMat: High performance graph analytics made productive,
Proc. VLDB Endow., vol. 8, no. 11, pp. 1214–1225, Jul. 2015.
53. G. Wang, W. Xie, A. Demers, and J. Gehrke, “Asynchronous large-scale graph processing
made easy,” In 6 th Biennial Conference on Innovative Data Systems Research (CIDR’13), 2013.
54. J.K. Hollingsworth, K. ACM Digital Library., D. Sigarch., and S. I. G. on H. P. C.
ACM Special Interest Group on High Performance Computing, “Direction-optimizing
breadth-first search,” In Proceedings of the International Conference on High Performance
Computing, Networking, Storage and Analysis, 2012, p. 1161.
55. U. Khurana and A. Deshpande, “Efficient snapshot retrieval over historical graph data,” In
2013 IEEE 29th International Conference on Data Engineering (ICDE), 2013, pp. 997–1008.
56. D. Kempe, J. Kleinberg, and A. Kumar, “Connectivity and inference problems for
temporal networks,” In Proceedings of the thirty-second annual ACM symposium on Theory
of computing—STOC ‘00, 2000, pp. 504–513.
57. M. Karsai, N. Perra, and A. Vespignani, Time varying networks and the weakness of
strong ties, Sci. Rep., vol. 4, no. 1, p. 4001, May 2015.
Big Graph Analytics ◾ 127
58. H. Wu, J. Cheng, S. Huang, Y. Ke, Y. Lu, and Y. Xu, Path problems in temporal graphs,
Proc. VLDB Endow., vol. 7, no. 9, pp. 721–732, May 2014.
59. S.-Y. Chan, P. Hui, and K. Xu, “Community detection of time-varying mobile social
networks,” In International Conference on Complex Sciences, 2009, pp. 1154–1159.
60. C. Aggarwal and K. Subbian, Evolutionary network analysis, ACM Comput. Surv., vol.
47, no. 1, pp. 1–36, May 2014.
61. W. Hant, Y. Miao, K. Li, M. Wu, F. Yang, K. Zhou, V. Prabhakaran, W. Chen, and E. Chen,
“Chronos,” In Proceedings of the Ninth European Conference on Computer Systems—EuroSys
‘14, 2014, pp. 1–14.
62. A.P. Iyer, L.E. Li, T. Das, and I. Stoica, “Time-evolving graph processing at scale,”
In Proceedings of the Fourth International Workshop on Graph Data Management Experiences
and Systems—GRADES ‘16, 2016, pp. 1–6.
63. R. Cheng, J. Hong, A. Kyrola, Y. Miao, X. Weng, M. Wu, F. Yang, L. Zhou, F. Zhao, and
E. Chen, “Kineograph: taking the pulse of a fast-changing and connected world,” In Proceedings
of the 7th ACM European Conference on Computer Systems—EuroSys ‘12, 2012, p. 85.
64. Z. Cai, D. Logothetis, and G. Siganos, “Facilitating real-time graph mining,” In CloudDB,
2012.
65. W. Xie, Y. Tian, Y. Sismanis, A. Balmin, and P.J. Haas, “Dynamic interaction graphs with
probabilistic edge decay,” In 31st International Conference on Data Engineering (ICDE),
2015, pp. 1143–1154.
66. G. Feng, X. Meng, and K. Ammar, “DISTINGER: A distributed graph data structure for
massive dynamic graph processing,” In 2015 IEEE International Conference on Big Data
(Big Data), 2015, pp. 1814–1822.
67. S. Aridhi, A. Montresor, and Y. Velegrakis, “BLADYG: A novel block-centric framework
for the analysis of large dynamic graphs,” In Proceedings of the ACM Workshop on High
Performance Graph Processing—HPGP ‘16, 2016, pp. 39–42.
68. D. Ediger, J. Riedy, D.A. Bader, and H. Meyerhenke, Computational graph analytics for
massive streaming data, In Large Scale Network-Centric Distributed Systems, edited by
H. Sarbazi‐Azad , A.Y. Zomaya, Hoboken, NJ: John Wiley & Sons, Inc., 2013, pp. 619–648.
69. P. Macko and Peter, “LLAMA: A Persistent, Mutable Representation for Graphs,”
Doctoral dissertation, Harvard University, 2015.
DATA ANALYTICS II
TECHNIQUES
Chapter 6
Transition from
Relational Database to
Big Data and Analytics
Santoshi Kumari and C. Narendra Babu
M. S. Ramaiah University of Applied Sciences
Contents
6.1 Introduction..............................................................................................132
6.1.1 Background, Motivation, and Aim................................................133
6.1.2 C hapter Organization....................................................................133
6.2 Transition from Relational Database to Big Data......................................134
6.2.1 R elational Database.......................................................................134
6.2.2 I ntroduction to Big Data...............................................................135
6.2.3 R elational Data vs. Big Data..........................................................136
6.3 Evolution of Big Data................................................................................138
6.3.1 F acts and Predictions about the Data Generated............................139
6.3.2 A
pplications of Big Data................................................................139
6.3.3 F
undamental Principle and Properties of Big Data........................141
6.3.3.1 I ssues with Traditional Architecture for Big Data
Processing����������������������������������������������������������������������� 141
6.3.3.2 F undamental Principle for Scalable Database System......142
6.3.3.3 P roperties of Big Data System..........................................143
6.3.4 G eneralized Framework for Big Data Processing...........................144
6.3.4.1 S torage and Precomputation Layer..................................144
131
132 ◾ Data Analytics
6.1 Introduction
The term big data was invented to represent large data generated continuously
with the advancement in digital technology, smart devices, and cheap hardware
Relational Database and Big Data ◾ 133
6.1.2 Chapter Organization
This chapter is organized as follows: Section 6.2 provides a review on the t ransition
from relational database to big data and the difference between relational data and
big data. Section 6.3 elaborates on the evolution of big data, the basic principles and
properties of big data, and the generalized framework for big data systems. Section
6.4 describes big data analytics, the necessity of big data analytics, and the challenges
134 ◾ Data Analytics
in big data analytics. Lastly, tools and technologies for big data processing are dis-
cussed in Section 6.5. Conclusion and future work are presented in Section 6.6.
6.2.1 Relational Database
In 1970, E. F. Codd [14] introduced a new approach for calculating and manipulat-
ing data in relational database format. Since then, RDBMS has been implemented
and used for more than four decades, satisfying the business needs. It is a tradi-
tional method for managing structured data in which rows and columns are used
to store the data in the form of tables and each table has a unique primary key.
The key characteristics of relational database are to support ACID (atomicity,
consistency, isolation, and durability) property [15] that guarantees consistency in
handling all the transactions. RDBMS includes a relational database and a schema
to manage, store, query, and retrieve the dataset. The RDBMS maintains data
integrity through the following characteristics:
Data warehouses and data marts are the key methods for managing the structured
datasets. A data warehouse integrates data from multiple data sources that are used
for storing, analyzing, and reporting, whereas a data mart is used to access the
data stored in a data warehouse so that data is filtered and processed. Preprocessed
Relational Database and Big Data ◾ 135
data is given as an input for data mining and online analytical processing to find
new values to solve various business problems. The two ways to store data in a data
warehouse are as follows:
There are several challenges faced by today’s enterprises, organizations, and govern-
ment sectors due to some limitations of the traditional database system which are
as follows:
Hence, big data analytics, tools, and technologies were introduced to overcome the
limitations of traditional database systems for managing complex data generated
rapidly in various formats characterized by [16] volume, velocity, variability, and
variety.
◾◾ Variety: Data is generated in different formats such as audio, videos, log files,
and transaction history from various sources, which no longer fits traditional
structures.
◾◾ Veracity: Large amount of unorganized data is generated, such as tweets,
comments with hash tags, abbreviations, and conversational text and speeches.
In order to address the above characteristics of big data, several new technologies
such as Not only SQL (NoSQL), Hadoop, and Spark were developed.
(Continued)
138 ◾ Data Analytics
Table 6.1 (Continued) Difference between Relational Data and Big Data
SI. No RDBMS Big Data
1. Data generated in every 2 days is equal to data generated from the beginning
of time until 2003.
2. Over 90% of all the data in the world was produced in the past 2 years.
3. By 2020, the amount of digital information generated is expected to grow
from 3.2 ZB today to 40 ZB. Data generated by industries gets doubled in
every 1.2 years.
4. Every minute, 200,000 photos are uploaded on Facebook, 204 million emails
are sent, 1.8 million likes are shared on Facebook, and 278,000 tweets are
posted, and around 40,000 search queries are served by Google every second.
5. In YouTube, every minute, around 100 hours of video are uploaded and it
would take 15 years to watch all the videos uploaded in a single day.
6. Thirty billion pieces of information are exchanged between Facebook users
every day.
7. Every minute, around 570 new websites are hosted into existence.
8. Around 12 TB of tweets is analyzed every day to measure “sentiment.”
9. An 81% increase in data over the mobile network had been observed per
month from 2012 to 2014.
1.
Understanding and aiming customers: Understanding the necessities and
the requirements of customers is one of the important factors for many busi-
ness entities to improve their business. Big data applications play an important
role in understanding the customers, their behaviors, and their inclinations
by analyzing the behaviors and sentiments of the customers from previously
collected large data. In order to get a more comprehensive picture of their
customers, many companies are keen to increase their datasets with social
media data, browser logs, as well as text analytics and sensor data. Nowadays,
using big data, e-business giants such as Flipkart and Amazon are able to
predict what products are on demand and can be sold, and they can suggest
similar products to the customers. Similarly, telecom companies are now able
to better understand the customer expectations and can make better deci-
sions. Exit polls of elections are more predictive using big data analytics [22].
140 ◾ Data Analytics
2.
Understanding and improving business practices: Big data helps in boost-
ing the business processes [23]. Analytics on social media data, search histo-
ries on web sites, and weather forecasts help retailers in adjusting their stocks.
GPS systems and sensor-equipped vehicles are used to track the delivery of
goods, and analysis of live traffic data helps in finding the shortest path. Most
importantly, customer feedback on social media helps to improve the busi-
ness processes.
3.
Health care: A large amount of patient medical history help in understand-
ing the symptoms and in predicting possible diseases and solutions [24,25].
Better prediction on a disease pattern enables to provide better treatments.
For example, a success factor of in vitro fertilization treatment can be pre-
dicted by the analysis of multiple attributes of N number of patient records.
The risk factors in a patient’s treatment can be identified by analyzing con-
ditions such as blood pressure levels, asthma, diabetes, genetics, and previ-
ous records. Future judgments in medicine would not be limited to small
samples, instead it would include a huge set of records.
4.
Sports: IBM SlamTracker tool used for video analytics in tennis tournaments
works on big data analytics. The performance of players in football and base-
ball games can be tracked and predicted based on the analysis of historical
data and sensor technology in sports equipment. Big data analysis and visu-
alization can help players a lot to improve their performance, for example, a
cricket player can improve his/her bowling skills by understanding what kind
of shots are played by the opponent.
5.
Science and research: Current potentials of big data analytics are trans-
forming science and research technology. For example, CERN, the Swiss
nuclear physics lab data center, has 65,000 processors. It generates huge
amounts of data by experimenting on the discovery of universe coverts
and analyzes 30 PB of data on many distributed computers across 150 data
centers. This data is used in many other research areas to compute new
insights.
6.
Optimizing machine and device performance: A large data used to train
machines and devices with the help of artificial intelligence and machine
learning makes better and smarter devices without human involvement. The
more the training data, the more accurate and smarter the device. For exam-
ple, Google’s self-driving car uses data captured from sensors, cameras, and
GPS systems to analyze the traffic movement and for safe driving without
human intervention.
7.
Security and law enforcement: Big data analytics is used by many developed
countries such as the United States, the United Kingdom, Japan, and Singapore
to advance security and law enforcement. For example, the National Security
Agency in the United States and National Computer Systems in Singapore
use big data analytics to track and avoid terrorist activities.
Relational Database and Big Data ◾ 141
8.
Smart cities: Big data analytics [26] help optimize traffic flow by predicting
weather using real-time traffic data collected from sensors, GPS, and social
media. Analysis of data from tweets, comments, blogs and end-user feedback
helps in building better transportation systems and hence in making better
decisions for building smarter cities and a smarter planet.
9.
Financial operation: Big data analytics plays an important role in mak-
ing financial decisions and improving financial operations. Understanding
costumer requirements and inclinations helps to provide best services, such
as insurance, credit, and loan facilities. Further financial operation process
improvements can be made based on the analysis of feedbacks. Nowadays,
most of the financial trading such as investments in stock market and pur-
chase and sale of shares is dependent on big data analytics.
1.
Architecture was not completely fault tolerant: As the number of machines
increases, it is more likely that a machine would go down as it is not hori-
zontally scalable. Manual interventions are required such as managing queue
failures and setting replicas to keep the applications running.
2.
Distributed nature of data: Data is scattered in pieces on many clusters, and
the complexity is increased at the application layer to select the appropriate
data and process it. Applications must be aware of the data to be modified
or must inspect the scattered pieces over the clusters and process it and then
merge the result to present the final result.
3.
Insufficient backup and unavoidable mistakes in software: Complexities
are pushed to the application layer with the introduction of big data technol-
ogy. As the complexity of system increases, the possibility of making mistakes
142 ◾ Data Analytics
will also increase. Systems must be built robust enough to avoid or handle
human mistakes and limit damages. In addition, a database should be aware
of its distributed nature. It is more time consuming to manage distributed
processes.
The big data system scalability and complexity issues of traditional systems are
addressed and resolved in a systematic approach.
To implement the above arbitrary function on random dataset with small latency,
Lambda Architecture provides some general steps for developing scalable big data
systems. Hence, it becomes essential to understand the elementary properties of big
data systems to develop scalable systems.
Relational Database and Big Data ◾ 143
errors, and running efficiently when machines are scaled are the important
factors to be considered for developing systems.
8.
Easy to restore: Systems must be able to provide the basic necessary informa-
tion to restore the data when something goes wrong. It should have enough
information replicas saved on distributed nodes to easily compute and restore
the original data by utilizing saved replicas.
Query Query
Data
Reduction and
Processing Map-Reduce
View
MAP-REDUCE,
Knowledge
SPARK
Discovery Layer
Storage and
Precomputation
Data Storage
Layer
and CLOUD
HDFS
Management
to be addressed in present and future research work is real time data processing and
computation [29]. Instead of precomputing the data from the storage layer, real-
time views are updated with new data that gets generated, eventually achieving the
highest possible latency.
This layer provides solutions for datasets generated in real time to improve
latency, whereas the storage layer produces precomputed views on the entire data-
set. Datasets that are no longer essential in real-time processing are removed, the
results are temporarily saved, and the complexity is pushed to the application layer.
Hence, the real-time data processing layer is more complex than the storage and
serving layers.
Finally, valuable results are obtained by joining the results from the precom-
puted views and real-time views. Hence, future research work would be focused
on bringing together the batch and real-time views to produce new and valuable
insights and to make better decisions. It requires advanced machine learning and
analytical techniques to improve the computational speed with maximum accuracy
for continuously changing random datasets.
A generalized flexible architecture with distinct components focused on spe-
cific purposes leads to more acceptable performance. Applications must be robust
enough to precompute values, corrupted values, and results by eventually executing
the computation on whole datasets to relocate or fix problems.
6.4.1.1 Volume
A large amount of unstructured data is generated and archived compared to tradi-
tional data. This data is generated continuously from various sources, such as sys-
tem logs, sensor data, click streams, transaction-based data, email communications,
Relational Database and Big Data ◾ 147
housekeeping data, and social media. The amount of data is increasing [10] to a
level that the traditional database management and computation systems are inca-
pable of handling it. The solution based on data warehouse may not be capable of
analyzing and processing huge data due to the lack of a parallel processing design.
So, increasing volume is one of the biggest challenges as it requires a new level of
scalability for storage and analysis.
6.4.1.2 Velocity
The amount of data increased is exponential to the increase of IoTs, sensor-equipped
devices, e-business, and social media. The data generated continuously at high speed
makes it challenging to process and analyze it. It is essential to device an algorithm
to get quick results from streaming data. For example, online interactions and real-
time applications require high rate of analysis.
6.4.1.3 Variety
A variety of data is generated from various sources in different structures, such as
text, video, audio, images, and log files. A combination of structured, unstructured,
and semistructured data is not supported by the relational database system. Hence,
this requires modified storage and processing methods for manipulation of hetero-
geneous data.
Many large-scale organizations and enterprises require big data tools and tech-
nologies to analyze their past history and customer information to understand cus-
tomer needs, in order to improve their business and finally to make better decisions
and predictions to survive in the competitive era. They also need [16] robust big
data analytics tools and technologies to make new innovations, process improve-
ments, monitoring, security, and many other functions. The analytical methods
used to handle big data in different areas are listed in Sections 6.4.2.1–6.4.2.5.
6.4.2.1 Text Analytics
The process of mining valuable and meaningful information from text data is called
text analytics. Some of the examples of text data are mail threads, chat conversa-
tions, reviews, comments, tweets, feedbacks, financial statements, and log records.
Text analytics comprise natural language processing, statistical analysis [29,30],
machine learning [32], and computational linguistics. Meaningful abstracts are
extracted from large-scale text data by applying many text mining and analyti-
cal methods. Text analytics is basically used for question answering, information
extraction, sentiment analysis, and text summarization. Text analytics is very essen-
tial in analyzing social media data [31] such as tweets and Facebook comments to
understand peoples’ sentiments and events happening in real time.
6.4.2.2 Audio Analytics
Audio analytics refers to the extraction of valuable and meaningful information
from audio data. It is most commonly used in call centers and health care services
for improving skills and services provided by call centers and also for improving
patient treatments and health-care services. It is also essential in customer relation-
ship management to understand the customers and improve the quality of products
and facilities to satisfy their needs and to maintain their relationships.
6.4.2.3 Video Analytics
Video analytics refers to extracting valuable data by tracking and analyzing the
video streams. It is used in key application areas, such as marketing and operations
management. Analyzing video streams of sports such as tennis and cricket helps to
improve the performance of the sports person.
people interest for particular products, and so on. Many organizations are trying
to improve their markets by analyzing social media data, such as people behav-
ior, sentiments, and opinions. According to [33], social media are categorized as
follows:
6.4.2.5 Predictive Analytics
Predictive analytics is used to make predictions on future events based on the anal-
ysis of current and historical data. According to [33,30], statistical methods form
the base for predictive analysis.
Making use of in-memory data analytics, big data tools have improved the per-
formance of data query notably. Big data analytics is not just about making better
decisions but also about real-time processing that motivates businesses to derive
new values and improve performance and profit rates from insights gained.
Big data beats RDBMS in several ways including robust backups, recovery,
faster search algorithms, overall scalability, and low-latency reads and writes.
6.4.3.2 Data Management
The traditional methods of managing structured data includes two important
parts. One is a schema to store the dataset and another is a relational database for
data retrieval. Data warehouse and data marts are the two standard approaches for
managing large-scale structured datasets. SQL is used to perform operations on
relational structured data. Data warehouse is used to store, analyze, and report the
outcomes to users. Access and analysis of the data obtained from a warehouse is
enabled by a data mart.
6.4.3.3 Data Analysis
According to Moore’s law, to cope with increasing data size, researchers gave more
attention to speeding up the analysis algorithms. As the data size increases sig-
nificantly faster than the CPU speed, there is a remarkable change in processor
technology, even though processors are doubling the clock cycle frequency. It is
essential to develop on-line, sampling, and multi-resolution analysis means. On the
other hand, development of parallel computing is required with increasing num-
bers of cores in processors. Large clusters of processors, distributed computing,
and cloud computing are developed fast to aggregate several different workloads.
In real-time applications like navigation, social networks, finance, biomedi-
cine, astronomy, intelligent transport systems, and IoT, speed is the top priority.
It is still a big challenge to be addressed for stream processing by giving quick
Relational Database and Big Data ◾ 151
1.
Clustering algorithms: In data clustering, many challenges are emerging
in addressing the characteristics of big data. One of the important issues
that need to be addressed in big data clustering is how to reduce the data
complexity. Big data clustering is divided into two groups [36]: (i) single-
machine clustering using sampling and dimension reduction solutions and
(ii) multiple-machine clustering using parallel and Map Reduce solutions
[37]. Using sampling and dimension reduction methods, complexity and
memory space required for data analytical processes will be reduced.
Inappropriate data and dimensions are discarded before data analysis pro-
cess starts. To reduce the data size for data analysis processes, data sampling is
used, and for reducing the whole dataset, dimension reduction is used.
To perform the clustering process in parallel, CloudVista [38] uses cloud
computing. It is a common solution for clustering big data. To handle large-
scale data, CloudVista uses balanced iterative reducing and clustering using
hierarchies (BRICH) and sampling methods.
2.
Classification algorithms: Many researchers are working toward developing
new classification algorithms for big data mining and transforming traditional
classification algorithms for parallel computing. Classification [39] algorithms
are designed in such a way that they take input data from distributed data
sources and use various sets of learners to process them. Tekin et al. presented
“classify or send for classification” as a novel classification algorithm.
In the distributed data classification method, the input data should be
processed in two different ways by each learner. One performs classifica-
tion functions, whereas the other forwards the input data to another labeled
learner. Big data classification problem improves the accuracy using these
kinds of solutions.
For example, to perform big data classification, Rebentrost et al. [40]
defined a quantum-based support vector machine and showed that with
O(log NM) time complexity the proposed classification algorithm can be
152 ◾ Data Analytics
6.4.3.5 Visualization of Data
Information hidden in large and complex datasets can easily be conveyed in both
functionality and visual forms. The challenges in data visualization [46] are to
Relational Database and Big Data ◾ 153
represent facts more instinctively and effectively by using distinct patterns, graphs,
and visualization techniques. For valuable data analysis, information should be
abstracted in some schematic form from complex datasets, and it should include
variables or attributes for the units of information.
To extract and understand the hidden insights from the data, e-commerce com-
panies, such as eBay and Amazon, use big data visualization tools, such as Tableau
[47]. This tool helps to convert large complex datasets into interactive results and
intuitive pictures. For example, data about thousands of customers, goods sold,
feedback, and their inclinations. However, there are many challenges in the current
visualization tools, such as scalability, functionalities, and response time that can
be addressed in future work.
Message Protocol
Distributed Protocol
5. Visualization Tableau
3. Data Processing
Batch Stream High level language Graph
Map Spark S4 Strom
Reduce Hive/Pig Pregel/Giraph
Thri
Zookeeper
1. File System
stack of big data tools was built on Hadoop by Apache to support different applica-
tions as shown in Figure 6.3. The later version of Hadoop is called Apache YARN
in which a new layer, called resource management layer, is added for efficient utili-
zation of resources in clusters of big data.
There are five layers in big data systems (Figure 6.3). Distributed file stor-
age is the bottom layer for storing large distributed data, above which there is
a cluster resource management layer. The purpose of this layer is to manage
large clusters of hardware resources and to allow the upper layers to utilize the
resources efficiently. The data stored in distributed file systems are processed by
the data processing layer as batch, stream, or graph processing. Preprocessed
data is fed to the data analytic layer to analyze and extract more valuable infor-
mation. To represent valuable results, high-level abstractions are built in the
visualization layer.
6.5.1 Tools
6.5.1.1 Thrift
Thrift is a scalable cross-language services library and code generation tool set to
support scalable back-end services. Its major goal is to provide efficient and reli-
able communication across different programming languages by selecting portions
of each language that require the most modification into a common library and
finally implementing them in each language. Thrift [54] supports many languages
such as Haskell, Java, C++, Perl, C#, Ruby, Cocoa, Python, D, Delphi, Erlang,
OCaml, PHP, and Smalltalk.
Relational Database and Big Data ◾ 155
6.5.1.2 ZooKeeper
Yahoo developed a distributed coordination system called ZooKeeper [8], and later,
it was taken over by the Apache Software Foundation. To coordinate the distrib-
uted applications, it offers an integrated service. ZooKeeper provides the following
support for distributed coordination:
6.5.1.3 Hadoop DFS
HDFS is a hierarchical file system consisting [55] of directories and files similar to
a UNIX file system. Users can perform all the administrative and manipulation
operations such as create, delete, copy, save, and move files to the HDFS as in a
normal UNIX system.
HDFS architecture: HDFS architecture consists of two units: a single name
node and multiple data nodes. Name node is a local file system responsible for
managing the file system namespace and tracks. Edit log keeps track of the logs and
updates whenever changes are made to the file system. Additionally, it keeps track
of all the blocks in the file system assigned to data nodes and mapping of blocks to
data nodes.
Name nodes are replicated for fault tolerance. To manage a large number of
files, file systems are divided into a number of blocks and are saved on data nodes.
The operations on files and directories such as opening, closing, and renaming are
performed by the name node. Mapping of blocks to data nodes is tracked by the
name node. The client directly communicates to the data nodes by obtaining a list
of files from the name node to read a file.
Block report is used to manage the copies of files. A separate file is created for
each block stored in the local file system by the data node. It also creates directories
for dividing the files belonging to different applications.
Fault tolerance in HDFS: HDFS replicates the files into blocks and keeps
them in different data nodes for fault tolerance. The name node uses the replicated
copy of blocks to process the requested data chunk if any data node goes down.
156 ◾ Data Analytics
6.5.2 Resource Management
Clusters of commodity servers are cost-efficient solutions for intensive scientific com-
putations and are used for running large Internet services. The issues of traditional
resource management for Hadoop and Storm are described in two aspects: First, a
system that runs Hadoop or Storm should be commodity hardware. Second, Hadoop
requires a lot of configurations and scheduling to support fine-grained tasks.
To address these issues of traditional resource management system, new tools
such as YARN [56] and Mesos [57] are introduced.
6.5.3.1 Apache HBase
HBase is stimulated by the Google Bigtable application [4]. It is built on HDFS
and is a multidimensional, distributed, column-oriented data storage method. It
provides faster access to records and updates for the data tables. It uses HDFS to
store the data and ZooKeeper framework for distributed coordination. Row key,
column key, and a timestamp are used for distributed index mapping in the HBase
multidimensional data model. The row keys are used for organizing the mapping.
Each row has its unique key, and a set of columns and number of columns are
added dynamically to the column families.
6.5.3.2 Apache Cassandra
Apache Cassandra [7] was developed by Facebook and is based on a peer-to‐peer
distributed key-value store construct. Cassandra is a row-oriented data model where
all nodes are treated as equal. Cassandra is suitable for real-time applications where
a large data needs to be handled with a faster random access.
Three orthogonal properties called consistency, availability, and partition toler-
ance (CAP) [58] are considered to develop distributed applications. It is difficult
to satisfy all three properties together to have tolerable latency for the operations,
according to the CAP theorem. By loosening the value of strong consistency to sub-
sequent consistency, Cassandra satisfies two properties, high availability and parti-
tion acceptance. Cassandra uses partitioning and replication similar to the Amazon
Dynamo database model.
Architecture: Cassandra data model is based on the design of Google Bigtable.
The data model is partitioned into a number of rows, and each row contains a
number of columns. Similar to SQL, Cassandra offers a query language CQL
(Cassandra Query language). Tables are designed to hold duplicates of the data, but
CQL does not support join operations. For faster access to data values, Cassandra
present indexes on columns.
6.5.4 Data Processing
Basic data processing frameworks are divided into batch mode, stream mode, or
graph processing and interactive analysis mode based on their processing meth-
ods and speed. The resource managers, Mesos and YARN, manage these runtime
systems in clusters at the lower layers. Unstructured data from HDFS as well as
structured data from NoSQL are used as an input to these systems. Output of these
158 ◾ Data Analytics
structures are redirected to these storage layers or cached for performing analytics
and visualization on it.
6.5.4.1 Batch Processing
Batch processing is suitable for processing large amounts of data stored in batches.
Hadoop Map Reduce is a basic model introduced to process huge amounts of data
in batch. However, it is not suitable for all kinds of batch processing tasks such
as iterative processing. To overcome some of these disadvantages, new processing
models, Spark and Apache Hama, are presented.
1. Hadoop
Hadoop is a distributed processing framework to processes big datasets over
groups of computers [16] using simple programming models. It is planned
and designed for scale-up from single servers to thousands of machines. The
role of each computer is to provide storage and local computation, instead
of depending on hardware, to provide high-availability. Machine learning
libraries are designed and modified to handle failures at the application layer,
hence providing an extensive service on top of the clusters.
Hadoop framework was introduced and published by Google, illustrat-
ing its method to manage and process a large data. Afterward, Hadoop is the
typical structure for storing, processing, and analyzing terabytes to exabyte
of data. As Doug Cutting started developing Hadoop, the framework got its
name “Hadoop” from his son’s toy elephant.
Yahoo is a main contributor to Hadoop advancement. Using 10,000-core
Hadoop clusters, by 2008, Yahoo’s web search engine index was generated.
It was developed to run on local hardware, and without any system interven-
tion, it can scale up and down. Three important functions of Hadoop frame-
work are storage, resource management, and processing.
– Hadoop Map Reduce (prior to version 2.0): Hadoop Map Reduce
structure consists of three main components: HDFS, job tracker (master),
and task tracker (slave). HDFS is used to store and share the data among
Map Reduce jobs computational tasks. First, the job tracker reads the
input data from the HDFS and splits it into partitions to run map tasks
on each input partition, and the intermediate results are stored in the
local file system. Second, reduce tasks read the intermediate results from
map tasks and run the reduced code on it. The results of the reduce phase
are saved in HDFS.
– Hadoop Map Reduce Version 2.0: A new version of Map Reduce intro-
duces the resource allocation and scheduling tool, Apache YARN [59].
Task tracker is replaced by YARN node managers. To keep track of the
finished jobs, job history server is a new added feature to the architecture.
Relational Database and Big Data ◾ 159
Initially, clients request YARN for the resources required for their
jobs. The resource manager assigns a place to execute the master task on
jobs.
– Hadoop characteristics:
1. Fault tolerant: The Hadoop cluster is highly prone to failures as thou-
sands of nodes are running on the commodity hardware. However,
data redundancy and data replication are implied to achieve fault
tolerance.
2. Redundancy of data: Hadoop divides data into many blocks and
stores them across two or more data nodes to improve the redun-
dancy. The master node preserves information of these nodes and
data mapping.
3. To scale up and scale down: The distributed nature of Hadoop file
system allows Hadoop to scale up and scale down by adding or delet-
ing the number of nodes required in the cluster.
4. Computations moved to data: Queries are computed on data nodes
locally and the results are obtained by combining them in paral-
lel to avoid the overhead of bringing the data to the computational
environment.
2. Spark
Spark was built on top of HDFS as an open-source project [60] by the
University of Berkley to address the issues of Hadoop Map Reduce. The
objective of the system is to support iterative computation and increase
the speed of distributed parallel processing to overcome the limitations of
Hadoop Map Reduce.
In-memory fault-tolerant data structure, resilient distributed datasets
(RDDs), was introduced by [61,62] the Berkeley University for efficient data
sharing across parallel computations. It supports batch, iterative, interactive,
and streaming in the same runtime with significantly high performance. It
also allows applications to scaled up and scale down with efficient sharing of
data. Spark differs from Hadoop by supporting simple join and group-by basic
operations.
Spark with RDDs runs applications with 100 times faster in memory and
10 times faster on disk compared to Hadoop Map Reduce.
Spark overcomes some of the limitations of Hadoop as follows:
1. Iterative algorithms: Spark allows applications and users to explicitly
cache data by calling the cache () operation, so that subsequent queries
can use intermediate results stored at the cache and provide dramatic
improvements in time and memory utilization.
2. Streaming data: Spark offers an application programming interface to
process the streaming data. It also gives an opportunity to design meth-
ods to process real-time streaming data with minimum latency.
160 ◾ Data Analytics
1. Storm
Storm is an open-source distributed real-time computation framework, dedi-
cated to stream processing. It offers a fault-tolerant mechanism to execute
computation on an event as it runs into a system.
Using Apache Storm, it is easy to process real-time streaming [63] data.
It has many useful applications, such as real-time analytics, online machine
learning, continuous computation, distributed RPC (remote procedure call),
and ETL (extract, transform, and load). Storm is easy to set up and operate.
It is also scalable and fault tolerant. Storm typically does not run on top of
Hadoop clusters, and it uses Apache ZooKeeper and its own master worker
processes to manage topologies.
2. S4
S4 offers a modest programing model [51] for programmers and offers an ease
and efficient automated distributed execution, for example, automatic load
balancing. On the other hand, in Storm, the programmer should take care
of load balancing, adjusting size of the buffer, and the level of parallelism for
getting optimum performance.
6.5.4.3 Graph Processing
The earlier method of graph processing on top of Map Reduce was inefficient as
it took entire graphs as input and processed them and then wrote the complete
updated graph into the disk. Pregel [64], Giraph [65], and many other systems are
developed for efficient graph processing and to overcome the limitations of Map
Reduce.
Relational Database and Big Data ◾ 161
1.
Pregel
Pregel was constructed on the bulk synchronous parallel model (BSP) [64]
as a graph processing parallel system. In the BSP model, a set of processors
that are interconnected by a communication network follow a different set
of computation threads in which individual processors are armed with a fast
local memory. The platform based on the BSP model consists of the following
three important mechanisms:
– Trained system for processing local memory transactions (i.e., processors)
– Efficient networks for communication between these systems
– A hardware support for synchronization between the systems
2.
Apache Giraph
Giraph is developed on top of Apache Hadoop for graph processing. Giraph
is built on Pregel for distributed processing of a graph [65]. Pregel is built
on top of Apache Hadoop, and it runs into map tasks of Map Reduce. To
coordinate between its tasks, Pregel uses Apache ZooKeeper, and for inter-
node communications, it uses Netty. A set of vertices and edges are used to
represent a graph in Giraph. Vertices perform computational tasks and con-
nections are represented by edges. Compared to Giraph, Pregel is used for
very large-scale graph processing.
1.
PIG
It is a procedure-oriented high-level programming language developed by
Yahoo for scripting Hadoop Map Reduce jobs. The language is also called Pig
Latin. Data pipelines are represented in a more usual way as it is a procedure-
oriented language. Due to its procedural nature PIG is suitable for iterative
data processing, data pipeline applications, and transformation jobs, and is
best suited for processing unstructured data.
A major advantage of PIG [66] compared to a declarative language is
that one can have control and check the operations performed over the data.
In declarative languages, programmers must have a good knowledge of the
algorithms and data operations and should also have knowledge on query
optimizer to choose the appropriate query to process the data.
2.
Hive
Hive [67] is developed on Hadoop as a high-level language to perform
analytics on big data. It supports large amounts of structured data process-
ing. It offers a method to map a structure of the data stored in HDFS and
162 ◾ Data Analytics
1.
Mahout
Mahout is a set of machine learning algorithm library [4] built on Hadoop
Map Reduce to support various analytical tasks. It also aims to include vari-
ous algorithms for machine learning on different distributed systems. Mahout
library includes several algorithms for various tasks as shown in Table 6.2.
2.
MLlib
It is a machine learning library developed on Spark [9,68] that consists of a set
of machine learning algorithms, as shown in Table 6.3, for classification and
clustering, regression analysis, and collaborative filtering.
A summary of the chapter on relational data and big data analytical
methods and technologies is presented in Table 6.4 and Figure 6.4. An over-
view of characteristics and tools and technologies for managing structured
and unstructured data is represented in the form of a flowchart to easily
understand the overview of bog data tools and technologies. It also shows the
k-means Clustering
area of applications for which big data technology and relational database are
suitable.
Relational Atomicity, DBMS, data SQL On-line Data mining, Graph, chart Employee
data consistency, warehouse, transaction clustering, details
(structured isolation, and data mart processing classification management,
data) durability [14] (OLTP), hospital
On-line management,
analytical insurance
processing company
(OLAP)
Big data Volume, HDFS (DFS) NoSQL [19] Batch: Mathematic: Tableau Social media
(unstructured velocity, HBase [4] Hadoop Statistics, (Jason computing,
data) veracity, verity Cassandra Map Fundamental Brooks, health care,
[17] [7] reduce [2] Mathematics 2016) government,
Spark [60] R TOOL finance,
PYTHON business, and
enterprise
(Continued)
Table 6.4 (Continued) Relational and Big Data Tools and Technologies
Storage and Data
Data Characteristics Management Tools Processing Analytics Visualization Application
High-level
languages:
Pig
Hive
166 ◾ Data Analytics
NoSQL
VS
BIG DATA VS SQL
RELATIONAL
DATA
HDFS
NOSQL
COORDINATION:
BIG DATA ZooKeeper
HADOOP MAP-
REDUCE
BATCH SPARK
PREGEL
TOOLS AND S4
TECGNOLOGIES
STREAM
PROCESSING
STROM
GRAPH GIRAPH
HIVE
HIGH LEVEL
LANGUAGES
PIG
DATA MINING
ANALYTICS STATISTICS
MLlib
MACHINE
LEARNING
VISUALIZATION
MAHOUT
Figure 6.4 Flow chart of big data analytical tools and technologies.
6.6.2 Conclusion
Enormous amounts of data are generated with increasing speed in different formats
due to digitization around the world. Big data technology will definitely enter every
domain, enterprise, and organization. Traditional database management system
Relational Database and Big Data ◾ 167
fails to scale for growing data needs such as multiple partitioning and paralleliz-
ing abilities. It is also incapable of storing, managing, and analyzing unstructured
data generated from different sources such as sensors, smart applications, wearable
technologies, smartphones, and social networking websites. Evolution of big data
analytics and tools and technologies made it possible to efficiently handle huge
unstructured growing data. One of the most popular open source frameworks,
Hadoop, is a generally recognized system for large-scale data analytics. It is mainly
known and accepted for support in large-scale distributed parallel computing of
clusters, and is cost-effective, fault tolerant, reliable, and provides highly scalable
support for processing and managing terabyte to petabyte of data. However, it is
not suitable for real-time data analytics. To overcome the incapability of this earlier
version of Hadoop system for real-time analytics, a new framework was introduced,
known as Spark. To support real-time analytics, Spark with RDDs gives results in a
fraction of seconds. Several areas such as business, social media, government, health
care, and security are implementing big data technologies to gain knowledge from
previously unused data to make better decisions and predictions. In the future, it
will be motivating to overcome the drawbacks of the Spark and Hadoop systems
and work toward real-time analytics. The challenges in batch processing and stream
processing analytical systems also need to be addressed in the future work.
References
1. McKinsey & Company. “Big Data: The Next Frontier for Innovation, Competition, and
Productivity.” McKinsey Global Institute, p. 156, June 2011.
2. J. Dean and S. Ghemawat. “MapReduce.” Communications of the ACM, vol. 51, no. 1,
p. 107, January 2008.
3. “Apache SparkTM—Lightning-Fast Cluster Computing.” http://spark.apache.org/ (Accessed
January 25, 2017).
4. A. Duque Barrachina and A. O’Driscoll. “A Big Data Methodology for Categorising
Technical Support Requests using Hadoop and Mahout.” Journal of Big Data, vol. 1, p. 1,
2014.
5. F. Aronsson. “Large Scale Cluster Analysis with Hadoop and Mahout.” February 2015.
6. N. Marz and J. Warren. “Big Data—Principles and Best Practices of Scalable Realtime
Data Systems.” Harvard Business Review, vol. 37, pp. 1–303, 2013.
7. L. Avinash and P. Malik. “Cassandra: A Decentralized Structured Storage System.” ACM
SIGOPS Operating Systems Review, pp. 1–6, 2010.
8. F. Junqueira and B. Reed. ZooKeeper: Distributed Process Coordination. O’Reilly Media,
Inc., 2013.
9. X. Meng, J. Bradley, S. Street, S. Francisco, E. Sparks, U. C. Berkeley, S. Hall, S. Street, S.
Francisco, D. Xin, R. Xin, M. J. Franklin, U. C. Berkeley, and S. Hall. “MLlib: Machine
Learning in Apache Spark.” Journal of Machine Learning Research, vol. 17, pp. 1–7, 2016.
10. C. Lynch. “Big Data: How Do Your Data Grow?” Nature, 2008.
11. “Welcome to ApacheTM Hadoop®!” http://hadoop.apache.org/ (Accessed January 25, 2017).
12. T. White. Hadoop: The Definitive Guide. O’Reilly Media, Inc., 2012.
168 ◾ Data Analytics
13. K. Mayer-Schönberger, V. and Cukier. Big Data: A Revolution that Will Transform How
We Live, Work, and Think. Boston, MA: Houghton Mifflin Harcourt, 2013.
14. E. F. Codd. “A relational Model of Data for Large Shared Data Banks.” Communications
of the ACM, vol. 26, no. 6, pp. 64–69, 1983.
15. T. Haerder and A. Reuter. “Principles of Transaction-Oriented Database Recovery.” ACM
Computing Surveys, vol. 15, no. 4, pp. 287–317, 1983.
16. P. Zikopoulos and C. Eaton. Understanding Big Data: Analytics for Enterprise Class Hadoop
and Streaming Data. McGraw-Hill Osborne Media, 2011.
17. D. Laney. “META Delta.” Application Delivery Strategies, vol. 949, p. 4, 2001.
18. R. Cattell. “Scalable SQL and NoSQL Data Stores.” ACM SIGMOD Record, vol. 39, no.
4, p. 12, May 2011.
19. J. Han, E. Haihong, G. Le, and J. Du. “Survey on NoSQL Database.” In 2011 6th
International Conference on Pervasive Computing and Applications (ICPCA), Port
Elizabeth, South Africa, 2011.
20. D. Borthakur. “The Hadoop Distributed File System: Architecture and Design.” Hadoop
Project Website, 2007.
21. G. Bell, T. Hey, and A. Szalay. “Beyond the Data Deluge.” Science, vol. 323, pp. 1297–1298,
2009.
22. E. Al Nuaimi, H. Al Neyadi, N. Mohamed, and J. Al-jaroodi. “Applications of Big Data
to Smart Cities.” Journal of Internet Services and Applications, vol. 6, no. 1, p. 25, 2015.
23. H. Chen, R. Chiang, and V. Storey. “Business Intelligence and Analytics: From Big Data
to Big Impact.” MIS Quarterly, 2012.
24. T. Huang, L. Lan, X. Fang, P. An, J. Min, and F. Wang. “Promises and Challenges of Big
Data Computing in Health Sciences.” Big Data Research, vol. 2, no. 1, pp. 2–11, 2015.
25. W. Raghupathi and V. Raghupathi. “Big Data Analytics in Healthcare: Promise and
Potential.” Health Information Science and Systems, vol. 2, no. 1, p. 3, 2014.
26. S. Kumar and A. Prakash. “Role of Big Data and Analytics in Smart Cities.” International
Journal of Science and Research, vol. 5, no. 2, pp. 12–23, 2016.
27. L. Garber. “Using In-Memory Analytics to Quickly Crunch Big Data.” Computer, vol. 45,
no. 10, pp. 16–18, October 2012.
28. M. Özsu and P. Valduriez. Principles of Distributed Database Systems. Springer Science &
Business Media, 2011.
29. S. Shahrivari. “Beyond Batch Processing: Towards Real-Time and Streaming Big Data.”
Computers, vol. 3, no. 4, pp. 117–129, 2014.
30. J. Friedman, T. Hastie, and R. Tibshirani. The Elements of Statistical Learning. Vol. 1, New
York, NY: Springer series in statics, 2001.
31. R. Irfan, C.K. King, D. Grages, S. Ewen, S.U. Khan, S.A. Madani, J. Kolodziej, et al. “A
Survey on Text Mining in Social Networks.” The Knowledge Engineering Review, vol. 30, no. 2,
pp. 157–170, 2015.
32. R. Bekkerman, M. Bilenko, and J. Langford. Scaling Up Machine Learning: Parallel and
Distributed Approaches. Cambridge: Cambridge University Press, 2011.
33. A. Gandomi and M. Haider. “Beyond the Hype: Big Data Concepts, Methods, and Analytics.”
International Journal of Information Management, vol. 35, no. 2, pp. 137–144, April 2015.
34. J. Ahrens, B. Hendrickson, G. Long, and S. Miller. “Data-intensive Science in the US
DOE: Case Studies and Future Challenges.” Computing in Science & Engineering, vol. 13,
no. 6, pp. 14–24, 2011.
35. W. Fan and A. Bifet. “Mining Big Data : Current Status, and Forecast to the Future.”
ACM SIGKDD Explorations Newsletter, vol. 14, no. 2, pp. 1–5, 2013.
Relational Database and Big Data ◾ 169
56. V. Vavilapalli, A. Murthy, and C. Douglas. “Apache Hadoop Yarn: Yet Another Resource
Negotiator.” In Proceedings of the 4th annual Symposium on Cloud Computing, SoCC’13,
Santa Clara, CA, 2013.
57. B. Hindman, A. Konwinski, M. Zaharia, and A. Ghodsi. “Mesos: A Platform for Fine-
Grained Resource Sharing in the Data Center.” In NSDI’11 Proceedings of the 8th USENIX
Conference on Networked Systems Design and Implementation, Boston, MA, 2011.
58. S. Gilbert and N. Lynch. “Perspectives on the CAP Theorem.” Computer, vol. 45, no. 2,
pp. 30–36, 2012.
59. V. K. Vavilapalli, S. Seth, B. Saha, C. Curino, O. O’Malley, S. Radia, B. Reed, E.
Baldeschwieler, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J.
Lowe, and H. Shah. “Apache Hadoop YARN.” In Proceedings of the 4th Annual Symposium
on Cloud Computing—SOCC’13, pp. 1–16, New York, NY, 2013.
60. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. “Spark : Cluster
Computing with Working Sets.” In HotCloud’10 Proceedings of the 2nd USENIX
Conference on Hot Topics in Cloud Computing, p. 10, Boston, MA, 2010.
61. M. Zaharia, M. Chowdhury, T. Das, and A. Dave. “Resilient Distributed Datasets: A Fault-
tolerant Abstraction for In-memory Cluster Computing.” In NSDI’12 Proceedings of the 9th
USENIX Conference on Networked Systems Design and Implementation, p. 2, San Jose, CA,
2012.
62. M. Zaharia. “An Architecture for Fast and General Data Processing on Large Clusters.”
Berkeley Technical Report, p. 128, 2014.
63. T. da Silva Morais. “Survey on Frameworks for Distributed Computing: Hadoop, Spark
and Storm.” Proceedings of the 10th Doctoral Symposium in Informatics Engineering—
DSIE’15, vol. 15, pp. 95–105, 2015.
64. G. Malewicz, M. Austern, A. Bik, and J. Dehnert. “Pregel: A System for Large-scale Graph
Processing.” In Conference: SPAA 2009: Proceedings of the 21st Annual ACM Symposium
on Parallelism in Algorithms and Architectures, Calgary, Alberta, Canada, August 11–13,
2009.
65. O. Batarfi, R. El Shawi, A. G. Fayoumi, R. Nouri, S.-M.-R. Beheshti, A. Barnawi, and S.
Sakr. “Large Scale Graph Processing Systems: Survey and an Experimental Evaluation.”
Cluster Computing, vol. 18, no. 3, pp. 1189–1213, September 2015.
66. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. “Pig latin.” In Proceedings of
the 2008 ACM SIGMOD International Conference on Management of Data—SIGMOD’08,
Vancouver, p. 1099, 2008.
67. A. Thusoo, J. Sen Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff,
and R. Murthy. “Hive.” Proceedings of the VLDB Endowment, vol. 2, no. 2, pp. 1626–
1629, August 2009.
68. X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkataraman, D. Liu, J. Freeman, D. Tsai,
M. Amde, S. Owen, D. Xin, R. Xin, M. J. Franklin, R. Zadeh, M. Zaharia, and A.
Talwalkar. “MLlib: Machine Learning in Apache Spark.” Journal of Machine Learning
Research, vol. 17, pp. 1–7, 2015.
Chapter 7
Contents
7.1 Introduction..............................................................................................172
7.2 Graph + Big Data = Big Graph..................................................................175
7.2.1 The Scale of Big Graph: How Big Is Big Graph?............................175
7.2.2 V ’s of Big Graph............................................................................ 176
7.2.3 Graph Databases............................................................................ 176
7.3 Big Graph Analytics..................................................................................179
7.3.1 Definition......................................................................................179
7.3.2 Relationships: The Basics of Graph Analytics................................179
7.4 Big Graph Analytics Approaches...............................................................180
7.4.1 In-Memory Big Graph Analytics...................................................183
7.4.2 S SD-Based Big Graph Analytics....................................................184
7.4.3 D isk-Based Big Graph Analytics....................................................185
171
172 ◾ Data Analytics
7.1 Introduction
The attainment of Google has made the phrase “to Google” synonymous with
the search for anything. The success mantra behind Google’s dominance as
a search engine is its ability to exploit the direct and indirect connections
Big Graph Analytics ◾ 173
among the humongous amounts of data. Google has adopted the graph-centric
approach to represent and model the relationships between documents and has
used graph analysis techniques to understand their semantics and contexts and
find out relevant information. The more vast and diverse data it can explore, the
more relevant information can be extracted. With the explosion of the digital
world, the amount of digital data originated from varied sources is increasing
colossally. In today’s world, the devices, processes, people, and other entities
are becoming more connected than ever before. As the number of data sources
increases, it is vital that we define the underlying relationships between them.
Social media, sensors, surveillance intelligence, industrial control systems, and
connected devices have become visibly important in the data-procurement
scenario. With this explosive increase of global data, the term “big data” was
coined and mainly used to describe huge datasets that are generated from
numerous heterogeneous sources [1]. Mining and analyzing these data helps us
to get an in-depth knowledge of the hidden values that bring new opportuni-
ties for discovering new values. As the variety and volume of our data increase,
more sophisticated methods are required to unveil the valuable information
hidden in it.
Although big data generated from diverse sources might appear to be discrete
and isolated on the surface, they are all nodes of a bigger network that binds them
through complex relationships. The modern big data increasingly appears in the
form of large-scale complex graphs/networks. Some of the examples of massive
complex networks include the World Wide Web, the physical Internet, phone
networks, online social networks (OSNs), and biological networks. In addition
to their big sizes, these graphs/networks are transient, noisy, and dynamic. They
also conform to most of the popular V’s (e.g., volume, velocity, variety) that
define big data. Hence, these large-scale graph-structured data are referred to as
the “big graph.”
The term “graph analytics” refers exactly to the study and analysis of data that
can be transformed into big graph representations.
Graph analytics is a fast-growing field in both big data mining and visualiza-
tion community that is utilized for numerous multidisciplinary and high-impact
applications, such as network security, finance, and health care. Even though the
analysis of unstructured collections of multidimensional points has already been
addressed in the past by several methods, graph analytic technologies form a rather
recent trend and they pose many challenges, concerning not only the performance
of the data mining algorithms that promote knowledge discovery through algorith-
mic computation but also the production of effective graph visualizations in order
to enhance human perception.
Graph analytics is all about modeling, storing, retrieving, and performance
analysis of graph-structured data by applying a combination of statistical, graph
theoretic, and graph database techniques. These techniques enable us to understand
174 ◾ Data Analytics
the structure of a large network and how it changes in different conditions, identify
clusters/modules or closely interacting subgroups inside the graphs, find subgraphs
that are similar to a given pattern, or find paths between the pairs of entities that
satisfy different constraints.
The main purpose of graph analytics is to analyze graph-structured data to
find the answers to questions (e.g., Who are the most influential people in a com-
munity? What are the important infrastructure nodes in the Internet or urban
networks?) for better practical actions and decision-making.
Thus, graph analytics can be used to model all sorts of relationships and pro-
cesses in a wide range of applications. For instance, they can reveal the patterns
across varying datasets that signal at the beginning of cyberattacks.
Graph techniques can also identify the root (cause) of surrounding or bigger
events, i.e., they can help in finding the most influential people in social media.
Alternatively, graph analytics can help in the identification of communities that
revolve around a certain theme, i.e., it helps in detecting patterns in communica-
tion that might indicate a threat to national defense by identifying the groups of
people who have been communicating about terrorist events, something security
agencies and/or authorities might be interested in.
Throughout the history of computation, graph analytics has always been an
interesting topic for researchers, but due to the big data movement, which has seen
an increased use of the advanced analytics on huge volumes of semistructured or
unstructured data, the research on big graph processing has also attracted the atten-
tion of the computer science fraternity in recent years.
In the modern digital ecosystem, the need for differently advanced graph ana-
lytics to make out something of the big data becomes more than essential as the
sources, types, and the amount of data continue to expand. In this respect, the con-
textual impact of data and the impact of graph analytics technologies on organiza-
tions seeking to discover the cause, effect, interrelation, and the influence of events
on business outcomes need to be defined. In this way, big graph analytics are not
only able to identify the key insights in the big data but also able to visualize them.
Moreover, they can detect correlations of the findings and determine their nature
and their significance within the given environment. New tools and algorithms
need to be developed regularly to help the business stakeholders optimize their key
business processes and open new opportunities.
The rest of the chapter is organized as follows: Section 7.2 examines the bigness
of the graph and how graphs are related to big data. To store big graph data, we
need special databases. This section also mentions some popular graph databases.
Section 7.3 defines and explains big graph analytics. Section 7.4 mentions differ-
ent big graph analytics approaches while categorizing different frameworks for each
approach. Sections 7.5 and 7.6 discuss different techniques and algorithms used in
graph analytics. Section 7.7 mentions the issues and challenges involved in big graph
analytics. Section 7.8 identifies a number of such application areas for big graph ana-
lytics. Section 7.9 concludes the chapter.
Big Graph Analytics ◾ 175
LinkedIn network, very nearly 8 million users (nodes) and 60 million friendship
relationships (edges) exist. The latest estimate states that the user count reached 500
million [7]. And in the semantic web network, DBpedia ontology contains nearly 4
million objects (nodes) and almost 400 million facts (edges) [5].
Volume: The volume represents the size of the graph in terms of the number of
nodes and edges. As the volume of graph increases, the analysis gets difficult.
Data-to-analysis time is too high if the volume of the graph is big [8].
Velocity: The velocity is the rate at which the graphs increase in size and com-
plexity (in large graphs, the streaming edges). These streaming edges make
graph analytics very difficult. The continuous stream of edges does not fit in
the memory. And this makes it difficult to compute a metric, e.g., the shortest
distance between two nodes and counting of strongly connected groups [8].
Variety: There are different sources and types of data that are combined to form
the graphs. The graphs data are often created through integration, e.g., XML/
JSON, relational, document, and graph-structured data [8]. The different
types of large graphs may have different meanings that can make the things
more complex if they are combined. Some examples are citation networks,
social networks, protein–protein interaction networks, ontologies, and linked
data/semantic web.
Valence: Valence refers to the degree of connectedness or interdependence. If
the valence is high, then the data elements are strongly related. In graph
analytics, this relatedness is significantly exploited. In some cases, as the
time valence increases, the average distance between the arbitrary node pairs
decreases and the parts of massive graph become denser [8].
7.2.3 Graph Databases
Legacy relational database management systems (RDBMSs) have been successfully
utilized to store, manage, and retrieve data for a long time. But the emergence of
big data has exposed the weakness of RDBMSs in handling relationships between
a huge number of heterogeneous data points semantically, which is absolutely nec-
essary to generate unforeseen insight from various data sources. Their tabular data
models and rigid schemas make it difficult to add connections that are new and/or
diverse in nature [9]. As the relationships between data points are more important
than the isolated data points, to leverage those data relationships, different DBMS
approaches are needed, which prioritize storing the relationship between the data
Big Graph Analytics ◾ 177
rather than the data itself. Graph databases do exactly the same. A graph database
is a database that uses graph structures to represent and store data. The relation-
ships between data (nodes) are represented through the directed edges of the graph.
Storing big data using graph database is like having a mini web inside our big data
applications. To crawl through that web, one can set off from a logical starting
point and fan out through the edges until the goal is attained. Graph databases
not only store the relationships between the data points effectively, but they are
also flexible in adding new and diverse kinds of relationships or in adapting a data
model to new business requirements [9]. A graph database is more appropriate for
dealing with complex, densely connected, and semistructured data. Some of the
popular graph databases are discussed briefly as follows:
Neo4j (Neo Technology): Neo4j [10], the world’s leading graph database, is a
high-performance, scalable, and robust database that is appropriate for enter-
prise deployment. The features of Neo4j are high-speed querying through
traversals, scales to trillions of nodes and relationships, declarative graph
query language, and so on.
DEX: DEX [11] is a high-performance graph database querying system. For
temporary graphs, DEX includes best integrity model. DEX makes the graph
querying possible in various networks, e.g., pattern recognition, link analysis,
keyword search, and social network analysis.
InfoGrid: InfoGrid [12] is an open-source web graph database and is being
developed in Java. It includes many important software components, which
make the buildup of web applications on a graph very easy.
InfiniteGraph: InfiniteGraph [13] is a scalable, cross-platform, and distributed
graph database. It is able to manage very complex relationships which require
multiple hops and are designed to yield very high throughput. InfiniteGraph
provides very high performance in terms of the query and also allows graph-
wise indexes on multiple key fields.
HyperGraphDB: HyperGraphDB [14] is an open-source graph database based
on generalized hypergraphs. It is a transactional, embedded graph database
designed as a universal data model. HyperGraphDB is used in the model-
ing of the graph data for very complex, large-scale knowledge representation
applications, e.g., natural language processing, artificial intelligence, bioin-
formatics, and so on.
Titan: Titan [15] is a very highly scalable graph database for querying and stor-
ing graphs having billions of vertices and trillions of edges. In real time,
Titan supports a large number of concurrent users executing the complex
graph traversals. It gives very fast responses to very complex queries and also
provides various important features, such as support for the global graph data
analytics using Apache Giraph integration.
Neo4j Java Transactional Native graph storage and Does not support ACID-compliant
processing sharding transactional database
InfoGrid Java Web graph Data integration, Data Linkage Combine data
referencing description integration with
language knowledge discovery
process
Titan Java Highly scalable Batch graph processing Dependent on the Scaling feature
using Hadoop framework third-party
storage backend
Big Graph Analytics ◾ 179
7.3.1 Definition
The big graph analytics is essentially an important and effective tool for knowledge
discovery in big data. In big graph analytics, the underlying data can be modeled or
is natively structured as a set of graphs. Graph analytics leverage graph-structured
data to visualize and understand relationships that exist between devices or people
in a network. It is used to model the pairwise relationships between objects, people,
or nodes in a network. Big graph analytics can uncover insights about the strengths
and directions of the relationships hidden in the big data. Example of strengths of
the relationship—How frequently do individuals or nodes communicate with each
other? What other individuals or nodes tend to join that conversation? Example of
directions of the relationship—Who does typically start the conversation? Is it a two-
way conversation or does one always lead? In what situations and how often does the
conversation get forwarded to others?
Big graph analytics defines and strengthens these complex relationships by
using mathematical concepts of graph theory. Graph analytics can help to answer
the questions—What is the maximum number of friends an individual has? How
many friends do individuals have on average? How interconnected are the groups of
users? Are there isolated groups of individuals who are connected to each other, but not
to individuals that are not in their group?
are often interested to know—Who are the most social influencers? or Who has the
social power to influence the perspectives of others?
Consider the important question—How two or more persons might be con-
nected to social media? This question may seem simple, but when we look closely it
gets complicated. The simplest problem is in looking at how two or more persons
may be connected on Facebook. These connections might be friends or friends of
friends. Sometimes, two individuals may only get the connection by sharing a few
“Likes.” This information may be valuable to the business. So, we want to know
those specific persons out of the trillion users on Facebook so that we can target
those persons directly for online advertisements.
Pregel
GraphLab
GraphX
In-Memory
Trinity
PowerGraph
Ringo
GraphChi
TurboGraph
Disk-Based X-Stream
GridGraph
Chaos
Pregel C/C++ In-memory Vertex-centric Not suitable for Design for sparse graph
scalable computation dense graph
graph
processing
system
GraphLab C++ Graph-based Distributed data graph It does not support It preserves strong data
computation in which the program dynamically consistency guarantees
the state is stored as a evolving graphs
directed graph
GraphChi C++ Disk-based PSW method Limited parallelism, To minimize the number of
parallel graph separate steps for random accesses, it uses PSW
engine I/O processing and method
CPU processing
(Continued)
182 ◾ Data Analytics
Table 7.2 (Continued) Comparison of Different Big Graph Analytics Frameworks
Name Language Type Features Weakness Strength
TurboGraph C++ Disk-based Pin-and-slide model — Full parallelism and full overlap
parallel graph (implements column of CPU processing and I/O
engine view of matrix-vector processing
multiplication)
7.5.1.1 Degree Centrality
In the real-world interactions, we often consider people with more connections to be
more important. A degree centrality is a simple centrality measure which counts, e.g.,
How many neighbors does a node have? There are two types of degree centrality measures:
1.
In-degree: It is the number of predecessor nodes or the number of incoming
links.
2.
Out-degree: It is the number of successor nodes or the number of outgoing
links.
Big Graph Analytics ◾ 187
Since the in-links are given by other nodes in the network, typically we are inter-
ested in the in-degree, while the out-links are determined by the node itself.
7.5.1.2 Eigenvector Centrality
By incorporating the importance of the neighbors, eigenvector centrality tries to
generalize degree centrality. Eigenvector centrality is defined for both the directed
graphs as well as the undirected graphs. To keep track of neighbors, we can use the
adjacency matrix A of a graph. The eigenvector centrality is defined as
n
c e ( vi ) =
1
λ
∑A
j =1
c ( vi )
j , i e
where ce ( vi ) denote the eigenvector centrality of node vi, λ is some fixed constant,
and A denotes the adjacency matrix of the graph, to keep track of the neighbors,
assuming ce = ( ce ( vi ) , ce ( vi ) ,.., ce ( vi )) is the centrality vector for all the nodes.
T
7.5.1.3 Katz Centrality
A potential problem with eigenvector centrality is that it works well only if the graph is
(strongly) connected. In this case, by adding the bias term to the centrality value, the
problem can be rectified. For all the nodes, irrespective of the network topology, the
bias β is added to the centrality values. The Katz centrality measure [33] is formulated as
C Katz ( vi ) = α ∑A
j =1
j , i C Katz ( v j ) + β
where α is a constant. The bias term β avoids the zero centrality values.
7.5.1.4 PageRank Centrality
A major problem with Katz centrality arises when a node with high centrality is
connected to many nodes and then all these nodes get high centrality. In many
cases, however, this is not so significant if a node is only one among the many that
are linked. To solve this problem, one can divide the passed centrality value by the
number of outgoing links (out-degree) from that node such that each connected
neighbor from the source node gets a fraction of the source node’s centrality. The
PageRank centrality [34] of a node vi is defined as
C p (v j )
n
C p ( vi ) = α ∑A
j =1
j , i
d out
j
+ β
7.5.1.5 Closeness Centrality
Closeness centrality [35] is a centrality measure that measures the mean distance
from one vertex to the other vertices. The concept of closeness centrality is that
the more the central nodes , the more quickly they can reach the other nodes. The
closeness of a node v is defined as
1
closeness ( v ) =
∑ d vi
i ≠ v
7.5.1.6 Betweenness Centrality
It measures the number of shortest paths in which a vertex is in sequence of vertices
in the path. For a node vi, the betweenness centrality [36] is defined as the number
of shortest paths between other nodes that pass through node vi.
σ st ( vi )
C b ( vi ) = ∑
s ≠ t ≠ vi
σ st
where σ st denotes the number of shortest paths from node s to t, and σ st ( vi ) denotes
the number of shortest paths from s to t that pass through vi.
Betweenness centrality needs to be normalized to be comparable across
networks.
7.5.2 Path Analysis
Path analysis is used to identify and explore all the connections between a pair of
nodes. For example, in the graph, path analysis can be used to determine the short-
est distance between two nodes. Route optimization is an obvious use case that is
particularly applicable to supply, logistics, distribution chains, and traffic optimiza-
tion for smart cities.
7.5.3 Community Analysis
In general, a community is formed by individuals in such a way that the individuals
within the group interact with each other more often than with those outside the
group. The distance-based and density-based analyses are used to find the groups
Big Graph Analytics ◾ 189
7.5.4 Connectivity Analysis
In a graph, how nodes are connected with each other is defined by connectivity. To
determine weakness in utility networks, such as power grid, connectivity analysis
can be used. Another example where connectivity analysis is used is in the compari-
son of connectivity across the networks.
7.6.2 Connected Component
In a graph, the connected component is the maximal subgraph in which all the
vertices are connected with each other. A graph can be made of many connected
components. Consider a social network that is made up of two or more groups of
friends in which none of the individuals from one group knows any individuals
from the other groups and vice versa. Each group of friends represents different
components because the members of one group are not connected with the mem-
bers of the other groups.
7.6.4 Graph Search
A graph is basically a collection of nodes where each node might point to the other
nodes. The graphs can be directed like one-way streets or can be undirected like
190 ◾ Data Analytics
two-way streets in the city. Suppose we want to go through this graph, and specifi-
cally suppose we want to do something, like figure out if there is a path from one
node to another. There are two common ways of doing this:
1.
Breadth-first search (BFS): The BFS is a level-by-level graph traversal algo-
rithm that performs searching on unweighted, undirected graph from the
given start vertex and assigns the distance to each vertex.
2.
Depth-first search (DFS): The DFS is typically a recursive graph traversal algo-
rithm that explores as deeply as possible using one neighbor before backtracking
to other neighbors. DFS does not go through all the edges. The vertices and
edges, which depth first search has visited, form a tree (graph spanning tree).
7.6.5 Clustering
Clustering is a task of grouping the set of nodes that have denser relations among
each other in comparison with the rest of the network. Based on the characteristics
of objects, the bunch of objects is placed in one group that has high intra-cluster
similarity and low inter-cluster similarity. Clustering is used in many applications,
such as data or text mining, statistics, image processing, and machine learning.
7.7.1 High-Degree Vertex
Graphs having high-degree vertices are very common, e.g., in OSNs, a very popular
person (e.g., Prime Minister of India: Narendra Modi) is connected to many other
persons; in text analysis we have a popular word that appears in many documents; and
so on. These types of graphs are computationally challenging and difficult to partition.
7.7.2 Sparseness
In real-world graphs, most vertices have relatively few neighbors, whereas a few have
many neighbors. For example, in the Twitter follower graph, some people have more
than 50 million followers [40] (e.g., President of the United States: Donald J. Trump),
whereas a normal user just has a hundred followers or even lesser. More computation,
synchronization, and communication are required to split the sparse graphs.
Big Graph Analytics ◾ 191
7.7.3 Data-Driven Computations
The graph computation algorithms are in most cases completely data driven. In these
algorithms, if the structure of graph computations is not known a priori, then the
parallelism based on the partitioning of the computation can be difficult to express.
7.7.4 Unstructured Problems
In general, data in graph problems are unstructured and irregular in nature. While par-
allelizing the graph problems, the irregular structure of the graph problem data makes it
difficult to extract the parallelism by partitioning the graph problem data [16].
7.7.5 In-Memory Challenge
The size of natural graphs is massive in nature, and it does not fit in a single mem-
ory. To reduce the response time, the graph data should reside in the RAM, instead
of the hard disk drive (HHD) or SSD.
7.7.6 Communication Overhead
The high-degree vertices (like more popular people on Twitter) lead to communica-
tion overheads. At present, high-degree vertices are in the billions, but in the future,
it will be in the trillions and beyond.
7.7.7 Load Balancing
Many of the natural graphs have followed the power law of the degree distribution. It
means that most of the vertices are connected to a small number of vertices and only
those small number of vertices are connected to a larger number of other vertices. When
we analyze such graphs, there is a need to pay extra attention to the load balancing.
Basically, the predominant uses of graph analytics are in those areas that require
addressing relationship-based problems. Some of the prominent applications of big
graph analytics are briefly discussed in Sections 7.8.1–7.8.7.
7.8.2 Behavior Analytics
In social media, individuals exhibit different behaviors that can be categorized into
individual and collective behaviors. Individual behavior is the behavior that an
individual targets toward
Collective behavior is the behavior when a group of individuals having or not hav-
ing any coordination act in an aligned manner.
7.8.3 Biological Networks
Interactions arise naturally in biology. Genes produce proteins, proteins regu-
late the functions of other proteins, cells transmit signals to other cells, etc. All
Big Graph Analytics ◾ 193
these interactions can be assembled into the graphs where each node represents
the biological entity and each edge represents associations or interactions between
entities.
7.8.4 Recommendation Systems
Graph-based recommendation systems help many companies to personalize con-
tent, services, and products by leveraging the multitude of connections. One popu-
lar example of recommendations is the friend recommendations in the OSNs.
7.8.5 Smart Cities
Nowadays, cities have various interacting networks, such as mobile-to-mobile and
broadband Internet Protocol networks, power networks, and sewage and water net-
works. Graph analytics is used for planning the smart hubs, energy-optimal rout-
ing, estimating the congestion patterns, etc.
7.9 Conclusions
In this chapter, we have presented an overview of big graph and big graph analytics.
We have understood the importance of graph in big data analytics. As big data is
growing even bigger, traditional database techniques are turning out to be ineffec-
tive in generating innovative insights needed for the organizations to be competitive.
Storing big data graphically ensures prioritizing relationships between data over the
data that automatically reflects the semantics of the data connections. This has helped
data analysts to expose new knowledge, thus helping organizations to take innovative
business decisions. Big graph analytics may be used for a variety of analytical purposes,
such as identifying notable persons in OSNs, spotting fraudulent acts in financial net-
works, and analyzing consumers’ buying patterns in order to consider better market-
ing strategies. Familiar DBMSs are incapable of coping with the expectations from
big data. Graph databases are required to store big graph data. It gives responses to
complex queries in a few milliseconds. Many such graph databases are emerging in the
market, Neo4j being one of them. Different approaches for big graph analytics such
as in-memory, SSD-based, and disk-based are adopted by different big graph analyt-
ics frameworks. Several techniques (e.g., centrality analysis, path analysis, community
analysis, and connectivity analysis) and algorithms (e.g., PageRank, clustering, etc.) are
used in big graph analytics. Big graph analytics is extremely helpful in industries, such
as online business solution, health care, online media, financial sector, social network,
communication, and retail. Though big graph has a lot of promises, some hitches such
as sparseness, high-degree vertex, and load balancing are to be taken care of.
Big Graph Analytics ◾ 195
References
1. M. V. Rijmenam. Think Bigger: Developing a Successful Big data Strategy for Your Business.
Amacom, 2014.
2. H. Motoda. “What Can We Do with Graph-Structured Data?—A Data Mining Perspective.”
In 19th Australian Joint Conference on Artificial Intelligence, Hobart, Australia, 2006.
3. R. Valdes. “The Competitive Dynamics of the Consumer Web: Five Graphs Deliver a
Sustainable Advantage.” July 13, 2012. Available: www.gartner.com/doc/2081316/
competitive-dynamics-consumer-web-graphs (Accessed December 27, 2017).
4. R. Marsten. “Is Graph Theory Key to Understanding Big data?” March 2014. Available:
www.wired.com/insights/2014/03/graph-theory-key-understanding-big-data/ (Accessed
December 4, 2017).
5. S. Sakr. “Processing Large-scale Graph Data: A Guide to Current Technology.” June 10,
2013. Available: www.ibm.com/developerworks/library/os-giraph/ (Accessed December
30, 2017).
6. “Number of Monthly Active Facebook Users Worldwide as of 3rd Quarter 2017 (in
millions).” Statista, 2017. Available: www.statista.com/statistics/264810/number-of-
monthly-active-facebook-users-worldwide/ (Accessed December 30, 2017).
7. B. Darrow. “LinkedIn Claims Half a Billion Users.” Fortune, April 24, 2017. Available:
http://fortune.com/2017/04/24/linkedin-users/ (Accessed December 30, 2017).
8. R. Kumar. “Graph Analytics.” February 16, 2016. Available: www.ranjankumar.in/
graph-analytics/ (Accessed August 2, 2017).
9. J. Webber. “The Top 5 Use Cases of Graph Databases: Unlocking New Possibilities with
Connected Data.” Neo4j, 2015.
10. “Neo4j Graph Database.” Neo Technology. Available: https://neo4j.com.
11. N. Martínez-Bazan, V. Muntés-Mulero, S. Gómez-Villamor, J. Nin, M.-A. Sánchez-
Martínez and J.-L. Larriba-Pey. “Dex: High-performance Exploration on Large Graphs
for Information Retrieval.” In 6th ACM Conference on Information and Knowledge
Management (CIKM ‘07), Lisbon, Portugal, 2007.
12. N. Giannadakis, A. Rowe, M. Ghanem and Y.-k. Guo. “InfoGrid: Providing Information
Integration for Knowledge Discovery.” Information Sciences, vol. 155, no. 3–4, pp. 199–
226, 2003.
13. “InfiniteGraph,” Objectivity, Inc. Available: www.objectivity.com/products/infinitegraph/
(Accessed January 27, 2018).
14. B. Iordanov. “HyperGraphDB: A Generalized Graph Database.” In International
Conference on Web-age Information Management (WAIM’10), Jiuzhaigou Valley, China,
2010.
15. “Titan: Distributed Graph Database.” Available: http://titan.thinkaurelius.com/
(Accessed January 27, 2018).
16. D. K. Singh and R. Patgiri. “Big Graph: Tools, Techniques, Issues, Challenges and Future
Directions.” In Sixth International Conference on Advances in Computing and Information
Technology (ACITY 2016), Chennai, India, 2016.
17. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser and G.
Czajkowski. “Pregel: A System for Large-scale Graph Processing.” In ACM SIGMOD
International Conference on Management of Data (SIGMOD’10), New York, 2010.
18. L. G. Valiant. “A Bridging Model for Parallel Computation.” Communications of ACM,
vol. 33, no. 8, pp. 103–111, August 1990.
19. “Welcome to Apache Giraph!” The Apache Software Foundation, October 24, 2016.
Available: http://giraph.apache.org/ (Accessed January 27, 2018).
196 ◾ Data Analytics
37. L. Page, S. Brin, R. Motwani and T. Winograd. “The PageRank Citation Ranking:
Bringing Order to the Web.” In 7th International World Wide Web Conference, Brisbane,
Australia, 1998.
38. R. G. Gallager, P. A. Humblet and P. M. Spira. “A Distributed Algorithm for Minimum-
Weight Spanning Trees.” ACM Transactions on Programming Languages and Systems, vol.
5, no. 1, pp. 66–77, 1983.
39. A. Lumsdaine, D. Gregor, B. Hendrickson and J. Berry. “Challenges in Parallel Graph
Processing.” Parallel Processing Letters, vol. 17, no. 1, pp. 5–20, 2007.
40. “Twitter: Most Followers.” Available: http://friendorfollow.com/twitter/most-followers/
(Accessed August 16, 2017).
41. Sparsity Technologies. “Graph Database Use Case: Insurance Fraud Detection.” May 12,
2015. Available: http://sparsity-technologies.com/blog/graph-database-use-case-insurance-
fraud-detection/ (Accessed January 27, 2018).
Chapter 8
Application of Game
Theory for Big
Data Analytics
Mohammad Muhtady Muhaisin
Huawei Technologies Bangladesh Limited
Taseef Rahman
Axway Inc.
Contents
8.1 Introduction............................................................................................. 200
8.1.1 Chapter Roadmap.........................................................................201
8.2 Basics of Classical and Evolutionary Game Theory...................................202
8.2.1 C lassical Game Theory..................................................................202
8.2.2 E volutionary Game Theory............................................................202
8.2.3 Nash Equilibrium......................................................................... 204
8.2.4 P areto Efficiency........................................................................... 204
8.2.5 R epeated Game............................................................................ 204
8.2.6 Bayesian Game..............................................................................205
8.2.7 Chicken Game.............................................................................. 206
8.2.8 Tit-for-Tat Game............................................................................207
8.2.9 Stackelberg Game..........................................................................207
8.2.10 Potential Game............................................................................. 208
8.3 Game-Theoretic Application in Big Data Analytics.................................. 208
199
200 ◾ Data Analytics
8.1 Introduction
In today’s technology-driven world, data is being produced at an astronomical
rate. As a matter of fact, more than 85% of the data in the world today has been
created in the past 2 years. Also, experts suggest a 4,300% increase in annual
data production by 2020 [1]. This staggering amount of data generated every sec-
ond, which is to be processed and analyzed, has given rise to the domain of big
data. Expounded as the data that requires processing beyond the conventional
methods, big data size is an ever-varying factor. Added to that, with the fear of
being left behind, many business entities today are becoming data-rich, but with
very poor data insight. They are storing data that they have no idea what to do
with, without extracting meaningful information from it. Furthermore, these
stored data have a finite life span as after some time they become outdated. As
a result, novel tools are being deployed to extract valuable information from this
ocean of data collected and stored.
Rowly et al. [2] distinguished between knowledge and wisdom by formulating
data, information, knowledge, understanding, and wisdom (DIKUW) hierarchy
represented in Figure 8.1. Game theory is a branch of mathematics that helps gain
“wisdom” by finding an optimal solution for a given situation when “rational” deci-
sion makers are at play. Big data explosion has paved the path for implementing the
game theory features in this domain because the game theory system theoretically
becomes more accurate with each iteration as it collects data in every stage. To
make sense of the huge chunk of data at hand, game theory data science might turn
out to be an auxiliary concept that data analysts can deploy to forecast and devise
effective data-driven decisions under strategic circumstances. Game theory has the
potential to supersede the intuitive evaluation of big data analysis with quantifiable
data-driven decision-making.
When implemented, “rational” independent players participating in the game
always try to maximize their own profit. Whether a player gains a profit is decided
by the success of that particular step or action [3].
Modern game theory began with an idea regarding the existence of mixed-
strategy equilibria in two-person zero-sum games and its proof by John von
Neumann. The evolutionary game theory, on the other hand, comes into play when
more than two players are involved [4]. Then the best strategy to be chosen may
depend upon the strategies used by other players [5]. It also takes into account not
only the outcomes of various competing strategies but also the frequencies with
which those various strategies are found in the populace.
Recently, the application of game theory in various facets of big data analytics
has gained traction due to its ability to converge fast into optimality. To name a
few, game theory is used to set up pricing mechanisms in decentralized transac-
tions, in incentivizing cooperative behavior in crowdsourcing platforms, in intru-
sion detection in cybersecurity platforms, and in devising data exchange protocols
among Internet of Things (IoT) devices. Besides, game theory is emerging as the
perfect tool to address the V factors of big data analytics—velocity, volume, value,
variety, and veracity.
The major contributions of this chapter are as follows:
Thus, readers will be able to get a lucid idea about the basics of game theory, with
the scopes of application of game theory and evolutionary game theory in big data
analytics discussed in various sections of this chapter.
8.1.1 Chapter Roadmap
The chapter is organized as follows: Section 8.2 focuses on briefing the readers on
the basics of classical and evolutionary game theory and on some technical terms
frequently used in the chapter. After that, Section 8.3 aims at reviewing the litera-
ture on the application of game-theoretic approaches in multiple facets of big data,
followed by Section 8.4 that sheds light on the challenges to be addressed. Lastly,
Section 8.5 concludes the chapter.
202 ◾ Data Analytics
which represents all possible strategy pairs of these two players. Aij ,(i , j = 1, 2,3,...k )
denotes the payoff of strategy i while competing with strategy i against strategy j.
Let xi (i = 1, 2,...k ) denote the ratio or relative frequency of each strategy [6,7].
All xi sum up to 1, i.e.,
n
∑x = 1 (8.1)
i =1
i
fi = ∑x a (8.2)
j =1
j ij
ϕ= ∑ x f (8.3)
i =1
i i
where the superscript stands for time derivation. Equation 8.4 represents that the
number of players selecting strategy i increases with the relative difference between
the expected payoff of strategy i and the average payoff of all the strategies. This
equation only stands for an infinitely large and well-mixed population where every
node can equally play games with all the other nodes. Now, if we want to con-
sider the evolutionary game dynamics on a k-regular graph [8], the modified payoff
matrix would become A′ = [aij′ ]. This modified matrix is the sum of the original n ∗ n
payoff matrix, A = [aij ] and a n ∗ x modifier matrix M = [mij ] where mij describes
the local competition between strategies i and j. Now, the expected payoff g i for the
local competition of strategy i is defined as
n
gi = ∑x m (8.5)
j =1
j ij
It has to be noted that the average payoff of the local competition of strategy i sums
to zero.
n
∑x g = 0 (8.6)
i =1
i i
204 ◾ Data Analytics
Ψ = ∑x ( f + g ) = ∑x f (8.7)
i =1
i i i
i =1
i i
8.2.3 Nash Equilibrium
For an N finite set of networking nodes, Ti are all the possible strategies where
i represents the player and T = (t1 , t 2 ,...t 3 ) is the set of strategies for each player
and T−i is the strategy profile T without agent i’s strategy [9]. In relation to that,
Vi (T ) is the payoff for a node i if the strategy profile is T. Ti is the best response
to T−i if Vi (t i ,T−i > Vi (t i′,T−i )) for every strategy t i′ available to agent i where
(t i ,T−i = (t1 , , t i −1 , t i′, t i +1 , , t n )). This set of strategies t = (t1 , , t n ) can be termed
as nash equilibrium, when for every i, ti is the best response to Ti , which means no
node can do better by unilaterally changing its strategy. Every game with a finite set
of participants/players is bound to have at least one nash equilibrium. For the game
to reach an equilibrium, each node has to identify its optimal strategy in response
to the strategies of the other players and all the nodes have to play with their opti-
mal strategies simultaneously.
8.2.4 Pareto Efficiency
Pareto efficiency, another important criterion of game theory, is defined as a situa-
tion where there is no way for a node to choose a different strategy and get benefited
without making other nodes worse off [3]. A strategy profile T Pareto dominates
a set of strategies T ′ if no player ends up with a worse payoff with T than with T ′.
Mathematically put, Vi (T ) ≥ Vi (T ′ ) for all i and at least one player gets a better
payoff with T than with T ′, i.e., Vi (T ) ≥ Vi (T ′ ) for at least one i.
8.2.5 Repeated Game
In repeated games, a set of players will play the game repeatedly with the same strat-
egy, taking into consideration the history of the past behavior. For better under-
standing, let us consider a repeated prisoner’s dilemma game.
It is a strategic game in which a player, in this case a prisoner, is unaware of
the other player’s action and chooses its action simultaneously and independently.
The strategy chosen by a player and the corresponding payoff has been provided in
Table 8.1. Each of the players, i.e., prisoners, aims to reduce their sentences [10].
Application of Game Theory ◾ 205
Cooperate 2, 2 10, 0
Defect 0, 10 4, 4
If cooperation is present, then both of them will remain silent and each will be
sentenced to 1 year in prison. In case they defect, i.e., both confess, then each will
be sentenced for 3 years. If only one of them confesses, then she/he will be freed,
while the other will end up with 4 years’ incarceration. Although it will be ben-
eficial for both the prisoners to cooperate, i.e., not confessing to the crime, but as
each has an incentive to get freed, in isolation both players will defect leading to
a unique nash equilibrium (defect, defect) of the game in case of a non-repeated
game. But if the players repeatedly take into account the past outcomes, then the
mutual desirable outcome will be to cooperate, which will ensure a long-term gain
but a short-term loss.
8.2.6 Bayesian Game
In games of complete information, the players know not only their own prefer-
ences, but also the preferences of the other players as well. But in many situations,
however, there is likely to be considerable uncertainty on the part of each player
regarding the preferences of his or her opponents. To incorporate this possibility,
Bayesian games or games of incomplete information is introduced. In Bayesian
games, each player can be one of a certain number of types. Each player knows his/
her own type but cannot observe the type of the other players directly, which influ-
ences each player’s payoff.
In the Battle of Sexes, suppose there are two players—player 1 and player 2.
They have different preferences on whether to watch ballet or football. But player
1 does not know whether player 2 wishes to meet or wishes to avoid player 1.
Therefore, this situation is of “incomplete information.” We represent this by con-
sidering that player 2 has two different types, one type that wishes to meet player
1 and the other that wishes to avoid him. More explicitly, suppose that these two
types have probability 1/2 each (Tables 8.2).
Cooperate 5, 2 0, 0
Defect 0, 0 2, 5
206 ◾ Data Analytics
Then the game takes the form of one of the following two with probability 1/2.
Crucially, player 2 knows which game it is (she knows the state of the world), but
player 1 does not. Most importantly, from player 1’s point of view, player 2 has two
possible types (or equivalently, the world has two possible states each with prob-
ability 1/2 and only player 2 knows the actual state). Let us consider the following
strategy profile (C , (C , F )), which means that player 1 will play C, i.e., ballet, while
in state 1, player 2 will also play F (when she wants to meet player 1) and in state
2, player 2 will play F, i.e., football (when she wants to avoid player 1). Clearly,
given the play of B by player 1, the strategy of player 2 is the best response. Let us
now check if player 1 is also playing the best response. Since both states are equally
likely, the expected payoff of player 1 is
E [C ,(C , F )] = 1/ 2 ∗ 2 + 1/ 2 ∗ 0 = 1 (8.9)
E [ F ,(C , F )] = 1/ 2 ∗ 0 + 1/ 2 ∗ 1 = 2 (8.10)
E [ F ,( F ,C )] = 1/ 2 ∗ 1 + 1/ 2 ∗ 0 = 2 (8.11)
E [C ,( F ,C )] = 1/ 2 ∗ 0 + 1/ 2 ∗ 2 = 1 (8.12)
8.2.7 Chicken Game
Let us understand the chicken game with a quintessential scenario. Let us say that
there are two cars (players) heading toward each other. If the players continue on
the same path, they bump into each other; if one swerves out of the way and the
other does not, then that player “loses” and is labeled as the chicken, while the sec-
ond, implicitly braver player, wins.
The payoff matrix is mentioned in Table 8.3. In the table, (−2, 2) and (2, −2) are
two pure nash equilibria because if either of the players deviates from their current
strategies in these two cases, then they would end up with more negative payoffs.
But the same cannot be said for (−100, 100) and (0, 0) because deviating from these
Application of Game Theory ◾ 207
Swerve −2, 2 0, 0
strategies would yield a higher payoff for each individual. So, they can be ruled out
from being equilibria stages.
8.2.8 Tit-for-Tat Game
Tit for tat is a game theory with a payoff matrix similar to that of prisoner’s
dilemma. It is relevant to a problem called the iterated prisoner’s dilemma. This
strategy can be implemented in games involving repeated moves. A player using
this strategy will first cooperate, then replicate an opponent’s previous action. If
the opponent had been cooperative, the player would have been cooperative and
vice versa.
This strategy states that a person is more successful in game theory if he/she
is cooperative with another person. The idea is that if a person rewards another
with good behavior, according to this theory, then he/she reciprocates with good
behavior. This strategy is based on the concept of forgiveness (retaliation). When
faced with a dilemma, an individual cooperates only when the other person has an
immediate history of cooperating and does not cooperate when the counterparty
previously defaulted.
8.2.9 Stackelberg Game
The original Stackelberg game is an asymmetric duopoly. Player 1 takes on the
role of leader, while player 2 takes on the role of follower. It is a two-period model
where in the first period the leader chooses its strategy. This decision is irreversible
and cannot be changed in the second period. The leader might emerge in a game
because of historical precedence, size, reputation, innovation, information, and so
forth. In the second period, the follower chooses its strategy after observing the
strategy chosen by the leader (the strategy chosen by the follower must, therefore,
be along its reaction function).
There are some variations of this game. If both players are followers, then you
have a Cournot game. If both players believe themselves to be the leaders, then
neither will be successful at following the strategy above. This is a leader–leader
game, and it will lead to a Stackelberg disequilibrium. The leader–leader model
208 ◾ Data Analytics
will require both parties to start with incorrect assumptions about each other, and
it will lead to nonoptimal output in the market.
In a multistage game, as rational players, the players will revise their expecta-
tions. As a result, either one will take the role of follower or a player will exit the
market, providing with the remaining leader a monopoly.
8.2.10 Potential Game
A potential game is one where the incentives of all the players to change their strate-
gies can be expressed using one global function called potential function. Two types
of potential games are the ordinal and cardinal potential games. In cardinal games,
the difference in individual payoffs for each player obtained by individually changing
one’s strategy only must have the same value as the difference in values for the poten-
tial function. In ordinal games, only the signs of the differences have to be the same.
The potential function is a useful tool to analyze equilibrium properties of
games, since the incentives of all players are mapped into one function, and the set
of pure nash equilibria can be determined by finding the local optima of the poten-
tial function. Convergence and finite-time convergence of an iterated game toward
a nash equilibrium can also be understood by studying the potential function.
Liu et al. [12] model the interaction between the customers and the hotel indus-
try as a noncooperative two-stage bargain game. The customers around the world
look for the best hotel to stay and receive the highest cost performance. Here, first,
the necessity of the customers and hotel managements have been analyzed, fol-
lowed by sorting out the decision factors that affect the chosen policies by the cus-
tomers. Then analyzing the interactions between the clients and the hotel operators
based on big data, the modeled game has been simulated. The results supported by
Monte Carlo simulation infer that the proposed game with big data generates the
best output because the two-stage game provides the customer with an extra chance
to choose and the hotel with an extra chance to accept the price. The outcome infers
that this novel method can reduce the price for the customers to select a hotel at a
lower price with lesser failure rate, thus benefiting the hotel operators as well.
Lancelot et al. [13] address the problem of joint policy correlation (JPC) in the
multi-agent reinforcement learning mechanism. A new algorithm is put forward
that allows independent learners to measure the extent of correlation of policies,
which in turn helps in visualizing the acuteness of the overfitting problem. This
novel algorithm is capable of immensely lessening JPC while participating in the
partially observable coordination games. Furthermore, it generates a defense mech-
anism utilizing the decision-making approach of other participating players in a
competitive imperfect information game.
Map Reduce, a framework developed by Google, processes big data from dis-
tributed servers and runs several responsibilities in parallel to each other. Till today,
Map Reduce applications have been bottlenecked by various limitations, e.g., exe-
cution time and capacity allocation. Gianniti et al. [14] focus on establishing novel
capacity allocation techniques for ensuring more robust performance among pri-
vate clouds. Game-theoretic approach leading to a generalized nash equilibrium
solves the problem of joint admission control, which in turn helps improve the
capacity allocation of multi-class Hadoop clusters (Table 8.4).
The explosion of online services and the resultant big data containing personal
information of the users have resulted in assigning intrinsic economic values to
these personal data. Chessa et al. [15] put forward a game-theoretic mechanism
termed as public good game, which is capable of quantifying the worth of reveal-
ing personal data. Selecting the cooperative game appropriate for the model, the
authors quantify the value of personal data using core and Shapley values. Also,
while implementing network games, it is inferred that creating more links among
neighbors helps them increase the value of their personal data. This in turn provides
higher incentives that benefit all as they also ascertain a stable network.
Jiang et al. [16] proposed the application of game theory in detecting com-
munities in dynamic social networks. Every day people are getting more involved
with each other on social networks, which has led to the generation of big data in
this domain. Game theory, being the tool for extracting valuable information from
these complex relations in networks, has been used in this work in the form of a
noncooperative game. Here, each rational independent node is able to switch, join,
210 ◾ Data Analytics
Gianniti et al. To ensure the robust cloud performance with the novel
[14] capacity allocation technique.
Tao et al. [17] To analyze the behavior of protocols in big data domain
Zheng et al. [18] To maximizing the data transmission rate in wireless sensor
networks
Zheng et al. [28] To propose a novel incentive mechanism for data controllers
in WCN
(Continued)
Application of Game Theory ◾ 211
Wang et al. [30] To apply data mining on manufacturing data with rules
derived from game theory
game is chosen because in the designed hierarchy the agents (network operators)
of the network help the leader (ISP) in attaining its objective. Moreover, the fast
and smooth linear convergence is independent of the network size as the objective
functions have been designed as convex functions. This ensures the model’s proper
functioning in big data networks. Lastly, this work verifies the designed pricing
framework’s fast convergence in a virtualized wireless network where the agents
always aim to maximize the total data transmission rate by allocating the maxi-
mum resource possible.
Pillai et al. [19] in their work apply the uncertainty principle of game theory
to design a resource allocation mechanism among the virtual machines of a cloud
system. To cope with the surge in demand for infrastructure-as-a-service of cloud
platform, the distributed virtual machines need to form a coalition among them-
selves. This work shows that the coalition and uncertainty-oriented model is better
suited for optimal resource allocation and utilization that is capable of attaining a
lower task completion time in comparison with the existing systems.
In modeling security games, the efficiency of machine learning application
has been tested previously where a huge amount of historical data was fed to the
machine learning system. As a result, the system becomes sensitive to the attacks
which only has relevance to previous attacks as per the historical data. Chung
et al. [20] introduce rationality by using Q-learning approach which reinforces the
decision-making model by adapting to newer intrusion patterns by analyzing the
previous iterations. As a result, this novel approach is better suited in taking neces-
sary actions to maximize the payoff of the security system administration against
newer attacks.
Kaur et al. [21] propose an automated employee performance evaluation using
IoT-based monitoring devices. The authors implement a game-theoretic approach
while checking the performance appraisal to ensure unbiased evaluation. In the
designed model, employees get rewards for positive performance and vice versa.
Prior to applying game theory for making the decision, data is mined from the IoT
devices using colocation mining. Performance comparison carried out in this work
infers that implementation of game-theoretic approach reduces the execution time
for the system significantly in comparison to the other decision-making processes,
such as fuzzy systems or Bayesian model. The equilibrium obtained shows that the
strategy of rewarding the employees ensures better performance from them, which
helps in attaining a stable business environment.
Tosh et al. [22] propound the concept of establishing a structure called cyber
information exchange (CYBEX) with a view to promote mutual security infor-
mation sharing for improving security infrastructure of the funds. As the funds
will always have a tendency to dominate the market, the competition among them
will always be noncooperative. But the authors devise a mechanism based on the
distributed noncooperative game model to impose a self-enforcement mechanism,
which maximizes the participation from the firms in sharing their security infor-
mation and also maximizes the CYBEX incentive for its sustenance in the market.
Application of Game Theory ◾ 213
that controllers can ask agents to help in their respective tasks. Here, previous issues
in the case of applying game theory in wireless big data processing, namely, conver-
gence conditions, convergence speed, and scalability, are also discussed.
In real-life applications of privacy-preserving distributed data mining (PPDM),
where multiple parties are involved, every party does not always behave according
to the protocols. Kargupta et al. [29] intend to solve this problem by forming a
game where each party tries to optimize its benefit, assuming a nonideal behavior
from each party. It presents a case study in the form of equilibrium analysis of
secure sum computation as an example. Furthermore, to achieve an optimal solu-
tion, a distributed penalty function mechanism is discussed to help achieve equi-
librium without any collusion between the parties involved.
To address the difficulty of applying game theory in manufacturing engineer-
ing, Wang et al. [30] discuss a new approach that amalgamates game theory and
data mining to utilize real-life manufacturing datasets. It divides an iterative pro-
cess called game mining into six stages with an aim to provide a general framework.
Alongside discussing the game behavior and levels and addressing the drawbacks
of game theory, it utilizes mining algorithm based on the work of association and
dynamic mining to achieve the desired goal.
Owing to the need to preserve privacy, there have been various adversaries mod-
eled till now, namely, semi-honest, malicious, and covert. But game theory gives
us the scope to assume that each party acts in its own interest. So, this paper [31]
discusses a protocol involving two parties in light of game theory. Here, verifiable
random functions are used as cryptographic algorithms underlying the protocol.
It is also shown that it satisfies nash equilibrium and stability conditions.
Smartphone users can be used by crowdsourcers on the condition of receiving
enough rewards for participation. So, this paper [32] visualizes a noncooperative
dynamic game consisting of multiple mobile crowdsourcers and a crowd of smart-
phone users to study the interrelationship between them. The game is later evolved
to a repeated game to consider joint acts between the crowdsourcers. The problems
related to the amount of reward for smartphone users, interaction with the market,
and stability of the market are also discussed.
Luo et al. [33] model heterogeneous crowdsourcing with a view that each work-
er’s probabilistic knowledge is different. Here, an incentive mechanism is designed
in an incomplete information scenario using an auction-based framework. In the
process, they also discuss strategy autonomy, i.e., the phenomenon of workers
behaving the same in a heterogeneous setting as they would in a homogeneous set-
ting. Later, they obtain accurate solutions for achieving equilibrium despite assum-
ing an unorthodox setting.
Realizing the great potential of smartphone users as a medium of crowdsensing,
Wang et al. [34] discuss two types of incentive mechanisms: crowdsourcer-centric
model and user-centric model. In the crowdsourcer-centric model, crowdsourcer is
the leader and vice versa for the user-centric model. For the first case, the condi-
tions for Stackelberg equilibrium are considered. For the latter case, an auction-based
Application of Game Theory ◾ 215
Figure 8.2 Emerging fields of application of game theory in big data domain.
8.5 Conclusion
There is obviously a shift in the progress of decision making processes where previ-
ously time consuming, inefficient models were built for ensuring accurate estimates
and judgments. Reality was simulated using data-driven models accumulated from
big data pool. Strategic decision-making from a huge amount of data is a research
area with far-reaching implications/effects. In summary, the concept of game the-
ory is not a core part of traditional data science, rather a tool that can analyze a
huge amount of data with meaningful visualizations and quantifiable data-driven
decision-making. This chapter has synopsized game-theoretic applications in the
field of big data analytics. Application of game theory is already being used in
Application of Game Theory ◾ 217
References
1. Bernard Marr. “Big data overload: Why most companies can’t deal with the data explo-
sion.” Forbes, April 28, 2016. Available: www.forbes.com/sites/bernardmarr/2016/04/28/
big-data-overload-most-companies-cant-deal-with-the-data-explosion/#312d21ce6b0d.
(Accessed February 7, 2018).
2. Jennifer Rowley. “The wisdom hierarchy: Representations of the DIKW hierarchy.”
Journal of Information Science 33, no. 2 (2007): 163–180.
3. Drew Fudenberg and Jean Tirole. Game Theory. Mit Press: Cambridge, MA, 1991.
4. Peter D. Taylor and Leo B. Jonker. “Evolutionary stable strategies and game dynamics.”
Mathematical Biosciences 40 (1978): 97–98.
5. Erez Lieberman, Christoph Hauert, and Martin A. Nowak. “Evolutionary dynamics on
graphs.” Nature 433, no. 7023 (2005): 312–316.
6. Hisashi Ohtsuki and Martin A. Nowak. “The replicator equation on graphs.” Journal of
Theoretical Biology 243, no. 1 (2006): 86–97.
7. K. Habibul Kabir, Masahiro Sasabe, and Tetsuya Takine. “Evolutionary game theo-
retic approach to self-organized data aggregation in delay tolerant networks.” IEICE
Transactions on Communications 93, no. 3 (2010): 490–500.
8. K. HabibulKabir, Masahiro Sasabe, and Tetsuya Takine. “Design and analysis of self-
organized data aggregation using evolutionary Game Theory in delay tolerant networks.” In
The Third IEEE WoWMoM Workshop on Autonomic and Opportunistic Communications, 2010.
9. John Nash. “Non-cooperative games.” Annals of Mathematics 54, no. 2 (1951): 286–295.
10. Ramona Trestian, Olga Ormond, and Gabriel-Miro Muntean. “Theory-based network
selection: Solutions and challenges.” IEEE Communications Surveys and Tutorials 14, no. 4
(2012): 1212–1231.
11. Lori Mann Bruce. “Game theory applied to big data analytics in geosciences and remote
sensing.” In 2013 IEEE International on Geoscience and Remote Sensing Symposium
(IGARSS). IEEE, 2013.
12. Shasha Liu, et al. “Game theoretic approach of a novel decision policy for customers based
on big data.” Electronic Commerce Research 18, no. 2 (2018): 225–240.
13. Marc Lanctot, et al. “A unified game-theoretic approach to multiagent reinforcement
learning.” In Proceedings of the Conference on Advances in Neural Information Processing
Systems (NIPS), Long Beach, CA, 2017.
14. Eugenio Gianniti, et al. “A game-theoretic approach for runtime capacity allocation in
MapReduce.” In Proceedings of the 17th IEEE/ACM International Symposium on Cluster,
Cloud and Grid Computing. IEEE Press, 2017.
15. Loiseau Chessa, et al. “A cooperative game-theoretic approach to quantify the value of
personal data in networks.” (2017). Doi:10.1145/3106723.3106732.
16. Fei Jiang and Jin Xu. “Dynamic community detection based on game theory in social
networks.” In 2015 IEEE International Conference on Big Data (Big Data). IEEE, 2015.
17. Xiuting Tao, et al. “A game-theoretic model and analysis of data exchange protocols for
internet of things in clouds.” Future Generation Computer Systems 76 (2017): 582–589.
18. Zijie Zheng, Lingyang Song, and Zhu Han. “Bridging the gap between Big Data and
Game Theory: A general hierarchical pricing framework.” In 2017 IEEE International
Conference on Communications (ICC). IEEE, 2017.
218 ◾ Data Analytics
19. Parvathy S. Pillai and Shrisha Rao. “Resource allocation in cloud computing using the
uncertainty principle of game theory.” IEEE Systems Journal 10, no. 2 (2016): 637–648.
20. Keywhan Chung, et al. “Game theory with learning for cyber security monitoring.” In
IEEE 17th International Symposium on High Assurance Systems Engineering (HASE). IEEE,
2016.
21. Navroop Kaur and Sandeep K. Sood. “A game theoretic approach for an IoT-based auto-
mated employee performance evaluation.” IEEE Systems Journal 11, no. 3 (2017): 1385–1394.
22. Deepak Tosh, et al. “An evolutionary game-theoretic framework for cyber-threat informa-
tion sharing.” 2015 IEEE International Conference on Communications (ICC). IEEE, 2015.
23. Yu Zhang and Mihaela van der Schaar. “Reputation-based incentive protocols in crowd-
sourcing applications.” In INFOCOM, 2012 Proceedings IEEE. IEEE, 2012.
24. M. Feldman, C. Papadimitriou, J. Chuang, and I. Stoica. “Free-riding and Whitewashing
in Peer-to-Peer Systems.” In Workshop on Practice & Theory of Incentives in Networked
Systems, 2004.
25. Baik Hoh, et al. “Trucentive: A game-theoretic incentive platform for trustworthy mobile
crowdsourcing parking services.” In 2012 15th International IEEE Conference on Intelligent
Transportation Systems (ITSC). IEEE, 2012.
26. Yi Wang. “Integration of data mining with Game Theory.” In Knowledge Enterprise:
Intelligent Strategies in Product Design, Manufacturing, and Management, Proceedings of
PROLAMAT 2006, IFIP TC5, International Conference, Shanghai, China, June 15–17,
2006, 275–280.
27. Xiaotong Wu, et al. “Game theory based correlated privacy preserving analysis in big
data.” IEEE Transactions on Big Data (2017). Doi: 10.1109/TBDATA.2017.2701817.
28. Zijie Zheng, et al. “Game theoretic approaches to massive data processing in wireless
networks.” Distributed, Parallel, and Cluster Computing 25, no. 1 (2018): 98–104.
29. Hillol Kargupta, Kamalika Das, and Kun Liu. “Multi-party, privacy-preserving distrib-
uted data mining using a game theoretic framework.” In Practice of Knowledge Discovery
in Databases, Vol. 4702, 2007.
30. Yi Wang. “Combining data mining and game theory in manufacturing strategy analysis.”
Journal of Intelligent Manufacturing 18, no. 4 (2007): 505–511.
31. Atsuko Miyaji, and Mohammad Shahriar Rahman. “Privacy-preserving data mining:
A game-theoretic approach.” In IFIP Annual Conference on Data and Applications Security
and Privacy. Springer: Berlin, Heidelberg, 2011.
32. Jia Peng, et al. “Behavior dynamics of multiple crowdsourcers in mobile crowdsourcing
markets.” IEEE Network 30, no. 6 (2016): 92–96.
33. Tie Luo, et al. “Incentive mechanism design for heterogeneous crowdsourcing using all-
pay contests.” IEEE Transactions on Mobile Computing 15, no. 9 (2016): 2234–2246.
34. Dejun Yang, et al. “Incentive mechanisms for crowdsensing: Crowdsourcing with smart-
phones.” Biological Cybernetics 24, no. 3 (2016): 1732–1744.
35. Neeraj Kumar, et al. “Optimized clustering for data dissemination using stochastic coali-
tion game in vehicular cyber-physical systems.” The Journal of Supercomputing 71, no. 9
(2015): 3258–3287.
36. Maryam Pouryazdan, et al. “Game-theoretic recruitment of sensing service providers for
trustworthy cloud-centric Internet-of-Things (IoT) applications.” In 2016 IEEE Globecom
Workshops (GC Wkshps). IEEE, 2016.
Chapter 9
Mohiuddin Ahmed
Canberra Institute of Technology
Contents
9.1 Introduction.............................................................................................. 219
9.1.1 Chapter Roadmap........................................................................ 220
9.2 Big Data Projects.......................................................................................221
9.3 Project Management Body of Knowledge.................................................223
9.4 Projects in Controlled Environment 2.......................................................225
9.5 A gile......................................................................................................... 226
9.6 ISO 21500:2012....................................................................................... 228
9.7 Key Insights..............................................................................................230
9.8 Conclusion................................................................................................233
References..........................................................................................................233
9.1 Introduction
Over the years, project management (PM) has developed into a discipline in its
own right and has received increasing interest from researchers. PM comprises a
number of tools and techniques, and if applied effectively, it can lead to the delivery
of specified outcomes.
219
220 ◾ Data Analytics
Snyder et al. [1] stated that though stories about the evolution of PM go back to
the building of pyramids in Egypt however, modern PM started in the 1950s and the
first article on PM appeared in 1958. Snyder stated that in 1960 very few people knew
what the project manager was about, but now the project manager is a key member
of management teams. Johnson et al. [2] stated that PM did not get any mention
in management literature before World War II (WWII). He argues that during the
Cold War there were pressures to develop new defense systems swiftly and operational
research, systems engineering, and PM because of increasing realization by the scien-
tists, managers, and engineers that conventional methods of management and devel-
opment were inadequate. From its inception to the early years of evolution, PM was
technical and industry specific. Morris et al. [3] highlighted the journey of PM from
tools such as Gantt chart, program evaluation and review technique (PERT), and crit-
ical path method in the 1950s to work breakdown structure (WBS) and value analysis
in the 1960s; resources scheduling, concurrency, and people in the 1970s; technology,
procurement, risk, and fast tracking in the 1980s; and philosophy, quality, knowledge
management, and Project Management Office in the 1990s and 2000s.
Tracing the technical and institutional factors behind the emergence of PM,
Johnson et al. [4] stated that the Manhattan Project, to build a nuclear bomb,
was the single most important project of the WWII that was managed by Leslie
Groves of army corps of engineers. The Manhattan Project was a significant event
in the history of PM for developing a scientific product. From its origins in the
military, PM spread to other sectors in the United States. The emergence and devel-
opment of PM has its roots in the United States, except a little influence from
Germany because Von Braun’s team was involved in the development of rocket sys-
tems. Initially, PM was limited to the military projects that required secrecy; there
were only a few influences from other industries in the United States because these
industries did not face similar levels of technological challenges, uncertainties, and
pressures of rapid development as experienced by the defense projects. Therefore,
the management style that emerged in aerospace and computing industries has the
signatures of technical and organizational problems of the military [4]. Pollack et al.
[5] provided a comprehensive analysis of PM literature from 1962–2012, where it
is identified that PM has moved from a technical and industry specific focus to
interpersonal issues. The main themes of current research in the PM discipline are
environment issues, strategic planning, project knowledge management, business,
and innovation [5]. Since PM has played a key role in research and development of
new technologies, it will be interesting to explore how PM can assist data analytics
as an emerging discipline.
9.1.1 Chapter Roadmap
The rest of the chapter is organized as follows: Section 9.2 discusses the big data
projects. In Sections 9.3–9.6, different types of PM methodologies are described in
detail and the relationship with data analytics is established. Finally, in Section 9.7,
Project Management ◾ 221
the key insights gained from these methodologies are discussed from the perspec-
tive of data analytics. The chapter concludes in Section 9.8.
the basic principles of data management. If the very data that instigates big data
project’s proof of concepts is not reliable, then the project will not be able to solve
the business problem, which will ultimately lead to failure in fulfilling the expec-
tations of the top management and also failure int creating value for the business.
Due to these factors highlighted by Informatica, there are only 13% organizations
where big data projects are in full production; therefore, big data project managers
are facing a phenomenal task of increasing the success rate of big data projects by
addressing the various factors discussed earlier.
In order to improve the success rate of big data projects, we can seek guidance
from the mainstream PM. Before we discuss how PM can help big data projects to
achieve higher success rates, it would be pertinent to discuss how PM as a discipline
is contributing toward the economic activity and helping businesses achieve their
strategic objectives and generate value from projects [9].
PM plays an important role as an economic activity globally, and it has
been tremendously adopted by various industries, particularly since the 1980s.
Projects are important tools in the hands of organizations for achieving their
strategic objectives. Therefore, projects have been employed as management
tools by various industries. It is important to explore how far projects have been
effective as management tools for organizations. Projects have been employed to
achieve strategic objectives by various industries since the 1950s, starting with
the defense industry in the United States. The important role played by projects
as an effective tool can be gaged from the share of projects in the global economic
activity. Bredillet et al. [10] stated that globally more than 20% of the economic
activity occurs in the form of projects, and in the emerging economies this ratio
is more than 30%. Referring to the World Development Indicators, Bredillet
et al. stated that out of $48 trillion of the world’s gross domestic product, 22%
is entirely project based. These statistics indicate the role of PM in value creation
at the global level [10].
Since projects were adopted to achieve strategic objectives, the success of p
rojects
also became an important issue for practitioners and researchers. According to PM
literature, where the success debate started in the 1980s, a project is considered suc-
cessful if it is completed within the cost, time, and meeting scope specifications,
though the success debate has moved further to include stakeholder’s satisfaction as
another key variable There is an increasing emphasis on value generation from PM
and value is not just limited to economic and financial gains but encompasses social,
ethical, and ecological value as well. Since 2010 various authors have challenged the
classical success criteria of triple constraints (on cost, time, and scope) and argue for
assessing project success not on the basis of project outputs rather on the basis of
project outcomes. Project outputs are the artifacts or deliverables, whereas project
outcomes are goals for which projects are initiated and implemented. Project out-
comes lead to value for the organization, which is also known as project benefits.
PM literature makes a powerful case for PM as the most effective management tool
to implement organizational strategy, achieve strategic objectives, and obtain value
Project Management ◾ 223
for the organization. Big data analytics need to learn from current debates in PM
and evaluate its project’s success not only on the basis of triple constraints but also
on the basis of project outcomes and benefits. In order to ensure project success, the
mainstream PM discipline employs various PM frameworks and methodologies,
and according to Pearson et al. [11], following are the key PM methodologies:
◾◾ Scope management
◾◾ Time management
◾◾ Cost management
◾◾ Quality management
◾◾ Human resources management
◾◾ Communications management
◾◾ Risk management
◾◾ Procurement management
◾◾ Governance management
◾◾ Integration management
Figure 9.1 PMBOK project management life cycle process. (Adapted from [11].)
responsible for overall business assurance of the project. The senior user represents
the end users of the product, and therefore specifies the characteristics and features
of the project product. The senior supplier represents those who are involved in
designing, developing, facilitating, procuring, implementing and, possibly, oper-
ating and maintaining project product. The senior supplier role is based on the
assumption that project environment is based on the customer/supplier relation-
ship. PRINCE2 is best suited for public sector projects spread over a longer duration
with strong governance and reporting mechanisms, whereas big data projects, as
stated by Informatica, should not run for more than 6 weeks. Therefore, PRINCE2
does not provide a project framework that can resolve the PM challenges faced by
big data projects. Particularly, being software projects, big data projects cannot be
initiated with finalized product description that is required by PRINCE2 as the
product details may evolve with the project progress. However, PRINCE2 offers
a good mechanism for the involvement of key stakeholders in important decision-
making at various stages. Big data projects can also learn from PRINCE2 about
benefits management to win over the confidence of executives about the value of
investment in big data projects.
9.5 Agile
Larson et al. [12] stated that the traditional approaches to PM such as PMBOK are
based on comprehensive upfront planning of the entire project, which inherently
requires the availability of project product description and demands a high level of
predictability to be effective. However, in certain cases, particularly the software
development projects, the finite product information is not possible; therefore, the
need for a new project methodology was direly felt and Agile endeavored to pro-
vide the role of an effective PM methodology. Larson et al. [12] stated some of the
following key features of Agile:
Figure 9.4 shows the incremental and iterative Agile process of product development.
Larson et al. [12] argued that traditional PM approaches were designed to work
in a predictable environment where the project scope is well defined and the tech-
nology is stable. On the contrary, Agile is the response to project environment,
which are unpredictable and undefined; therefore, agile is more experiential and
adaptive of managing projects in unstable and unpredictable circumstances. Agile
approach to PM identifies three main roles that play key responsibilities in man-
aging project activities: product owner, self-organizing team, and scrum master.
Product owner helps identify the product description and initial features. The prod-
uct owner has the authority to change product features at the end of each sprint
(iteration). Agile project teams, unlike traditional PM teams, are self-managing and
the team members decide who is going to do what work, and the team is responsible
for the commitments made at the review meetings. The role of a scrum master is
quite similar to that of a project manager who facilitates scrum meetings. Scrum
meetings are a series of coordinated meetings to manage the development process;
in other words, these are project team meetings held on a daily basis.
Though Agile is best suited for software development projects because of its
iterative and flexible nature, it does not provide any guidance on the other aspects
of PM, such as cost and time management, risk management, quality management,
and governance. Larson et al. [12] stated that Agile approach does not address
management’s top concerns about possible costs and expected timeline for a given
project. In many cases, a maximum budget ceiling is identified which is not expected
to exceed a certain limit. Agile PM approach faces project integration challenges
228 ◾ Data Analytics
when various teams are working on different product features. Therefore, to ensure
effective integration, processes external to Agile would be necessary.
The Agile approach is also silent on post project implementation activities such
as project review, and most of all, it does not guide on benefits management such as
how to identify, implement, and realize benefits of software solutions delivered as
a project product. It does not seek to direct how an organization can get value by
realizing benefits from a big data solution. The fundamental characteristics of Agile
are that it facilitates working on project product with initial features, and its itera-
tive nature makes it a strong candidate project approach for big data projects.
9.6 ISO 21500:2012
The ISO 21500:2012 framework has been developed by ISO and it is more of a
guidance framework than a full-fledged methodology like the other frameworks
discussed previously. It provides a high-level description of various PM processes
that can be employed by public or private sector organizations irrespective of the
organization size, project size, and complexity. In other methodologies, projects
are conceived within the context of portfolios and programs, but ISO 21500:2012
framework sees projects as stand-alone entities; therefore, issues about general man-
agement have been addressed only in the context of PM rather than at higher levels,
such as programs and portfolios.
This framework considers organizational strategy as the starting point in the
PM process, where projects are initiated to exploit opportunities in the light of
organizational strategic objectives. Figure 9.5 provides an overview of the ISO
21500:2012 framework.
ISO 21500:2012 has great similarities to the PMBOK framework proffered by
PMI in project life cycle. The similarities between the two frameworks can be seen
in Table 9.1.
According to Varajaoa et al. [13], ISO 21500:2012 is closely aligned with
PMBOK and have almost similar project processes, as can be seen in Table 9.2.
Since there are lot of similarities between ISO 21500:2012 and PMBOK, it is
advisable to use PMBOK which is a globally recognized PM standard and which
situates projects within the context of programs and portfolios, thus offering
effective PM. ISO 21500:2012 framework can be tailored to implement big data
projects; however, like PMBOK and PRINCE2 methodologies, ISO 21500:2012
framework also requires scope definition and product description at the planning
phase, whereas big data proof of concepts being software projects are started with
the basic product features and are finalized as the product gets developed.
However, system development life cycle (SDLC) proposed by Kimball et al. [14]
offers a good model, which can be applied to big data projects. Kimball’s SDLC
model is fundamentally geared toward designing, developing, and deploying data
warehousing business intelligence systems. The Kimball model comprises 11 main
Project Management ◾ 229
Initiating Initiate
Planning Planning
Implementing Executing
Closing Closing
processes, and these are nested in PM context, which makes this model unique.
Most of the common SDLCs are predominantly designed for software develop-
ment, but these overlook the PM aspects in software development. It is possible that
such SDLCs either consider PM processes out of the scope of software development
or assume that PM processes can be addressed at the organization level. However,
the model demonstrates that SDLC can be more effective when implemented in
combination with PM processes and PM context.
As can be seen in Figure 9.6, Kimball’s model assumes that there exists a PM
environment in the organization, where the key PM aspects such as time and
cost management, quality and risk management, governance and stakeholders’
230 ◾ Data Analytics
Integration Integration
Scope Scope
Time Time
Cost Cost
Quality Quality
Risk Risk
Stakeholders Stakeholders
Procurement Procurement
Communication Communication
management are taken care of. Therefore, the Kimball model starts with business
requirements definition with the assumption that PM processes have already been
completed for a given project. Figure 9.6 shows Kimball’s model.
9.7 Key Insights
As mentioned at the start of this chapter, the success rate of IT projects is abysmally
low and the project failure rate in bigger organizations is even higher; therefore,
Project Management ◾ 231
big size organizations who are prime beneficiaries of big data projects would also
be facing similar low success rates for such projects. Project success is a major issue
even in mainstream PM. Although modern project management started in the
1950s, project success started appearing in literature from the 1980s [1,15]. During
the early years, project success remained focused on the project delivery within
the triple constraints of “on cost,” “on time,” and “within specifications.” As the
concept of project success matured gradually, stakeholders’ satisfaction was added
as another criterion for success. The efforts to identify project success factors con-
tinued, and various authors such as Pinto and Slevin [16] identified ten success fac-
tors for projects such as project mission, top management support, project schedule
plan, client consultation, personnel matters such as human resources and training,
technical tasks—adequate technology to support projects, client acceptance, moni-
toring and feedback, communication, and troubleshooting. Similarly, Freeman and
Beale [17] identified seven main elements of project success and stated that the first
five have been more frequently mentioned:
◾◾ Technical performance
◾◾ Efficiency of execution
◾◾ Managerial and organizational implications (mainly customer satisfaction)
◾◾ Personal growth
◾◾ Manufacturability and business performance
◾◾ Technical innovation
◾◾ Manufacturability and business performance
Dvir and Shanhar [18] investigated the relationship between project planning
and project success and argued that planning is central to modern-day PM and a
minimum level of planning is required for project success. The authors argue their
research found three factors such as “development of functional requirements,”
“development of technical requirements,” and “implementation of PM processes
and procedures” have critical roles in project success. The authors identified a posi-
tive relationship between the efforts made to determine project goals, functional
requirements, and technical specifications of the product and project success, par-
ticularly from the end users’ perspective. However, the current debate on project
success in the mainstream PM literature has extended beyond the execution of
the project and it now centers on project benefits realization, which values project
outcomes more than project outputs. Since organizations spend precious resources
on projects, in return, it is reasonable for them to expect that the intended benefits
will flow into the organization. There is a growing realization that benefits accrued
by the host organization should be a criterion of project success. Cook-Davies et al.
[19] reviewed the six bodies of knowledge in PM and concluded that expected ben-
efits have become the benchmark for stage-gate decision-making and continuous,
informal assessment in PM. Zwikael et al. [20] challenged the most commonly
used criteria of project success, which is based on project outputs rather than on
232 ◾ Data Analytics
current PM practices in various industries, big data projects need to give due impor-
tance to benefits management and realization, as in the end every project is initiated
and implemented for project outcomes so that benefits and value can flow into the
investing organization. For this to happen, big data projects need to adopt program
and portfolio management as this will facilitate benefits management throughout
the organization. By ensuring that benefits are realized, the m anagers of big data
projects can improve the project success rates and also enhance the confidence of top
management in the efficacy and value of big data projects.
9.8 Conclusion
The chapter establishes the relationship between PM and big data analytics. The
existing PM methods were developed without considering the impact of big data.
However, the increasing amount of data in any project will have to be treated well
for effective PM and outcomes. Therefore, in this chapter, we have highlighted the
importance of PM for optimized data analytics. We have critically analyzed the state-
of-the-art methodologies for PMs and have provided key insights based on the analy-
sis. This chapter will be particularly helpful for project managers who have to deal
with large amounts of data and, at the same time, have to look for successful projects.
References
1. Snyder, James R. (1987) www.pmi.org/learning/library/modern-project-management-
disciplines-direction-1810, accessed on December 29, 2016.
2. Johnson, Stephen B. (1997) Three Approaches to Big Technology: Operations Research,
Systems Engineering, and Project Management. Technology and Culture, vol. 38, no. 4,
pp. 891–919.
3. Morris, Peter W.G. (2010) Research and the future of project management. International
Journal of Managing Project in Business, vol. 3, no. 1, pp. 139–146.
4. Johnson, Stephen B. (2013) Technical and Institutional Factors in the Emergence of
Project Management. IJPM, vol. 31, pp. 670–681.
5. Pollack, Julien and Adler, Daniel (2015) Emerging Trends and Passing Fads in Project
Management Research: A Scientometric Analysis of Changes in the Field. IJPM, vol. 33,
pp. 236–248.
6. Standish Group (2017) CHAOS Report. www.projectsmart.co.uk/white-papers/chaos-
report.pdf, accessed on February 25, 2017.
7. Pearson, Neil, Erik W. Larson, and Clifford F. Gray (2013) Project Management in Practice.
McGraw Hill, Sydney, NSW.
8. Larson, Erik W., Honig, Beverly, Gray, Clifford F., Dantin, Ursala, and Baccarini, David
(2014) Prokect Management: The Managerial Process. McGrawHill Education, Sydney, NSW.
9. Varajaoa, Joao, Richardo Colomo-Palaciosb, and Helio Silva (2017) ISO 21500:2012 and
PMBoK 5 Processes in Information Systems Project Management. Computer Standards
and Interfaces, vol. 50. pp. 216–222.
234 ◾ Data Analytics
10. Kimball, Ralph, Margy Ross, Warren Thornthwaite, Joy Mundy, and Bob Becker (2008)
The Data Warehouse Lifecycle Toolkit (2nd ed.). Wiley Publishing, Indianapolis, IN.
11. Ballard, Asbjorn, Iris Tommelein, Per Morten Schiefloe, and Glenn Ballard (2014)
Understanding Project Success through Analysis of Project Management Approach.
International Journal of Projects in Business, vol. 7, no. 4, pp. 638–660.
12. Pinto, Jefrey K. and Dennis P. Slevin (1987) Critical Factors in Successful Projects
Implementation. IEEE Transactions on Engineering Management, vol. 34, no. 1, pp. 22–27.
13. Freeman, M. and P. Beal (1992) Measuring Project Success. Project Management Journal,
vol. 23, no. 1, pp. 8–17.
14. Dvir, Don and Aron Shenhar (2003) An Empirical Analysis of the Relationship between
Project Planning and Project Success. IJPM, vol. 21, no. 2, pp. 89–95.
15. Cooke-Davies, Terry (2000) Towards Improved Management Practice, PhD thesis, Lead
Metropolitan University. In Cook-Davies, T. (2002) The Real Success Factors on Projects.
IJPM, vol. 20, pp. 185–190.
16. Zwikael, Ofer and John Smyrk (2012) A General Framework for Gauging the Performance
of Initiatives to Enhanced Organizational Value. British Journal of Management, vol. 23,
no. 1, pp. 6–22.
17. Zwikael, Ofer (2016) Editorial for IJPM Special Issue on Project Benefit Management.
IJPM, 34.
18. Turner, Rodney, Frank Anbari, and Christophe Bredillet (2013) Perspectives on Research
in Project Management: The Nine Schools. Global Business Perspectives, vol. 1, no. 1,
pp. 3–28.
19. Informatica (ND). How to Run a Big Data POC in Six Weeks. https://govinsider.asia/
wp-content/uploads/docs/[Workbook]%20How%20to%20Run%20a%20Big%20
Data%20POC%20in%206%20weeks.pdf, accessed on February 27, 2018.
20. Cao, Longbing (2017) Data Science: Challenges and Directions. Communications of the
ACM, vol. 60, no. 8. pp. 59–68.
21. Datameer (ND). The Guide to Big Data Analytics. www.datameer.com/pdf/big-data-
analytics-ebook.pdf?mkt_tok, accessed on February 25, 2017.
Chapter 10
Mohiuddin Ahmed
Canberra Institute of Technology
Contents
10.1 Introduction..............................................................................................236
10.1.1 Chapter Roadmap.........................................................................237
10.2 Emergence of Industrial Revolutions.........................................................237
10.2.1 F ourth Industrial Revolution (Industry 4.0)..................................239
10.2.2 D efinition of Industry 4.0..............................................................239
10.2.3 C ore Components of Industry 4.0.................................................241
10.3 Blockchain and Cryptocurrency................................................................245
10.3.1 D efinition of Blockchain.............................................................. 246
10.3.2 C omponents of Blockchain............................................................247
10.3.3 W orking Procedure and Algorithm................................................250
10.3.4 C ryptocurrency.............................................................................252
10.4 Blockchain’s Impact on Industry 4.0.........................................................253
10.4.1 H
ow Blockchain Supports Industry 4.0.........................................253
10.4.2 A
pplication Domains of Blockchain in Industry 4.0......................257
10.4.3 A daptation Issues and Open Research Challenges........................ 260
10.4.4 C hallenges Associated with Law, Policy, and Standardization.......263
10.4.5 Recommendations for Adaptation................................................ 264
235
236 ◾ Data Analytics
10.1 Introduction
The fourth industrial revolution has begun and this is not going to be like other
revolutions. It is not about the steam that powered our factories in the first
revolution or the mass production model that dominated the second or even the
emergence of computer-driven systems from the third revolution that we are living
in today. The fourth industrial revolution or Industry 4.0 is about connectivity and
communication. The current digital economy is based on the trust formed between
groups of business-minded authorities. Meanwhile, the fourth industrial revolution
demands an automated business process with ubiquitous digitization, i.e., a tech-
nology that can narrow the gap between human and system. It is an opportunity
to radically change the way industry responds to the needs of society. The world is
waiting for another industrial revolution, which is capable of automatic machine-
to-machine (M2M) communication via a secure and decentralized cyber-physical
system (CPS). This exponential challenge must be mobilized by technologies such
as blockchain, shared ledger, and cryptocurrency.
The main theme of blockchain is a distributed ledger instead of a traditional
centralized system. A closer look at this technology shows the following key
characteristics that have opened immense opportunities for Industry 4.0:
Now, cryptocurrencies are for blockchain, what email was for the Internet in the
early 1990s. Digital currency or cryptocurrency is an Internet-based commodity
exchanging medium that acts like a bank check, note, and coin. The first usage
of blockchain technology, bitcoin (BTC), came in the year 2009. Since then quite
Blockchain in the Era of Industry 4.0 ◾ 237
a few have been generated. Some superior cryptocurrencies are BTC, ethereum,
ripple, BTC cash, litecoin, etc.
Initially, BTC was the only dominant application of blockchain technology,
but with further technological achievements, it can be used in other sectors too.
Major industries that can utilize the aforementioned properties of blockchain tech-
nology are finance and insurance, health and medicine, Internet of Things (IoT),
academia, copyright, vehicle sharing, supply chain management, energy manage-
ment, gambling, government, retail, human resource management, etc. Presently,
we have middlemen everywhere, for example, bank for currencies, postal service
for product insurance, election commission for election, contract lawyer, court and
registry office for land registration, etc. Blockchain can remove these middle par-
ties, secure communication, and build a digital reputation for a trustless secured
business. However, there are arguments in the context of technology, standardiza-
tion, policy, and social adaptability. Although standardization and synchronization
for an acceptable blockchain are ongoing, it is important to clear up the upcoming
fourth industrial revolution issues such as energy and power consumption, aberrant
cryptocurrency value, transaction time and cost, and privacy and regulations.
Blockchain technology has many things to offer, which will surely affect the
future industry structure in Industry 4.0. After this revolution, industries can
exploit the real digital enterprises, with physical products at the pivot, and decen-
tralized storage, augmented interfaces, and immutable crypto transaction at the
end. To underscore the effect of blockchain on Industry 4.0, this chapter lists
challenges, scopes, and current status of these technologies in detail. In addition
to that, their flaws, effects, and requirements are also pointed out with adequate
examples, which can be a good reference point for future study.
10.1.1 Chapter Roadmap
Rest of the chapter is organized as follows: Section 10.2 elaborates the fourth
industrial revolution and its components within the scope of Industry 4.0.
Section 10.3 contains discussion on blockchain and cryptocurrencies. It also pres-
ents the scope of blockchain technology and its correlation with Industry 4.0.
In Section 10.4, the impacts of blockchain on diverse industries are highlighted.
Section 10.5 highlights the changes that blockchain can bring in the industrial
architecture with a potential use case. Finally, Section 10.6 concludes the chapter
followed by necessary references.
Figure 10.1 Industrial evolution with key developments (Industry 1.0 to Industry
4.0).
revolution will be capable of planning ahead with a clear strategic vision and will
be focused on discovering how smart products and processes can be developed.
Industry 4.0 will take away isolated silo-driven development, which limits the
scopes and values of new projects. Instead, it will move the current industry toward
large-scale and proactive integration across their enterprise and among their cus-
tomer, supply base, and products to ensure that they are aware of every possible
opportunity a smart factory has to offer. Under the umbrella of Industry 4.0, a
trader understands the value chain from which products are designed, how they
are produced, who they are sold for, and how they are purchased. An organization
may become radically different in the future with a range of products, services, and
processes that we cannot currently predict. We must plan ahead and fight radical
changes with integrated radical actions to welcome the new era of the fourth indus-
trial revolution (Industry 4.0). Figure 10.1 shows a gradual change of the industrial
revolution from the year 1784 to 2017.
◾◾ Industry 1.0
From the beginning of the eighteenth century, for the greater benefit of workers,
water and steam-powered machines were invented. Due to industry 1.0, pro-
duction capabilities maximized, business grew from individual to organization
level, and from stakeholders to customers all came under the same platform.
◾◾ Industry 2.0
By the beginning of the twentieth century, industries got the taste of electric-
ity and division of labor. By the virtue of electricity, machines had their own
power sources, which empowered factories for mass production. In this revo-
lution, workers did only a part of the total job, which increased the overall
productivity and efficiency.
◾◾ Industry 3.0
The invention of electronic cheapest, integrated circuit, the programmable
logic controller (PLC) spawned the development of computerized software
Blockchain in the Era of Industry 4.0 ◾ 239
components of Industry 4.0 are mentioned in the text that follows (Figure 10.3),
though there are other ongoing technologies connected with Industry 4.0.
Manufacturing industries of the fourth industrial revolution will have the
abovementioned components (Figure 10.3). A secure CPS is used with cloud com-
puting and big data analysis to improve policies. IoT, Internet of systems (IoS), and
additive manufacturing technologies are used to increase the quality and quantity
of the product. Training and production systems become more secure and efficient
with smart sensors and augmented reality. Finally, blockchain and cryptocurren-
cies can provide a middleman less, secure, and decentralized transaction facilities.
A detailed description of Figure 10.3 is given as follows:
◾◾ Cyber-physical systems
Physical and computational elements can be controlled through the Internet
via a CPS. These interactions among the machines are dynamic, which change
the context to represent a complete system in a digital way. With intelligent
embedded sensors, machines will be able to communicate, thus resulting in a
smart network for a particular product throughout its supply chain. “Smart
anything” is its main concept, which includes everything surrounding us
(e.g., cars, buildings, homes, manufacturing, hospitals, factory, grids, and
appliances). It is a bridge between the computing and noncomputing devices
around us. Lee proposed two main components of CPS: (i) a real-time bidi-
rectional communication for physical and cyber world and (ii) an intelligent
data analysis for building an intelligent cyber space [15]. For example, the
Figure 10.3 Core components of the fourth industrial revolution (Industry 4.0).
Blockchain in the Era of Industry 4.0 ◾ 243
a common, transparent, and shared economy; finally, blockchain filled that space.
Companies are facing shared social economy, crowd funding, digital identities, and
entrepreneurship. Therefore, stakeholders must welcome blockchain-based transac-
tion for fitting their company to global payment settlement, accounting, or cus-
tomer loyalty. It was first started in the year 2008 and Satoshi Nakamoto explained
about the general idea of the blockchain [26]. The main idea of his BTC white
paper was about money transactions between peers without requiring the help of a
third party, such as banks or trusted middleman. Instead of trust what was needed
was a digital transaction system based on cryptography. It helped to solve the
double-spending problem through something called proof of work (POW) through
the use of digital wallet. Due to gradual development over time, this technology
has become the largest way of solving global economic barrier. As BTC becomes
popular worldwide, similar cryptocurrencies are also being developed. Figure 10.5
explains the gradual development of BTC and blockchain over time. It started
with the domain registration of bitcoin.org in 2008. The official publication of the
blockchain idea publicly happened in the same year. In 2017, BTC and blockchain
became mature enough to join as a key technology of Industry 4.0.
10.3.1 Definition of Blockchain
The blockchain is a decentralized ledger of all transactions across a peer-to-peer
(P2P) network. Without a central authority and middleman, this technology can
confirm the transaction with the approval of an already established and proved
Blockchain in the Era of Industry 4.0 ◾ 247
group. The blockchain is the base technology on which BTC and other cryptocur-
rencies are based. Swan mentioned in his book Blockchain: Blueprint for a New
Economy about two versions of blockchain technology based on its use: version 1.0
was all about currency and version 2.0 reflects smart contract with all other shared
ledger use [26]. There are different studies and research trying to introduce and
define blockchain and BTC technology from their own viewpoints. According
to Robert Hackett of the Fortune magazine, “This coding breakthrough—which
consists of concatenated blocks of transactions—allows competitors to share a
digital ledger across a network of computers without the need for a central author-
ity. No single party has the power to tamper with the records: the math keeps
everyone honest” [27]. The blockchain can also allow more advanced transactions
other than transferring money with smart contracts [28]. In 1997, Nick Szabo
defined backsmart contract as a cryptographic contract. It is a digital protocol
to satisfy common contractual conditions such as payment, license, and busi-
ness deal. A smart contract can reduce transaction costs by removing the trusted
intermediaries [29].
Zibin Zheng in 2017 has found four key characters of blockchain technologies:
decentralization, persistency, anonymity, and auditability [30]. The International
Organization for Standardization (ISO) has also started a technical committee on
blockchain in Australia to propose “Standardization of blockchain technologies
and distributed ledger technologies.”
10.3.2 Components of Blockchain
Major technologies constituting blockchain are shown in Figures 10.6–10.8). Hash
function, architecture of a block, and public and private keys in digital signature
are discussed as follows:
◾◾ Hash
A hash algorithm changes arbitrary data into definite length hash. Generally
written as hexadecimal, a slight change in original data results in a significant
difference in output hash value by the use of this hash function. It makes fal-
sification difficult that guarantees blockchain data security. Every new block
creation or transaction utilizes the calculation of hash value. The hash value
calculator that is mostly used is double SHA256 hashing algorithm and this
technology was mentioned in this study [31]. An example of hash function
is as follows: If an input data in BTC is “2190,” after going through a hash
function it becomes “e80G1.” For the next transaction, if the input data is
changed slightly to “2191,” after going through a hash function it will gener-
ate “2cc6L” as an output. The following scenario shows that in Figure 10.6,
2190 and 2191 are almost similar, but “e80G1” and “2cc6L” are difficult to
predict. Blockchain adopts this security mechanism to generate random hash
values for data storage and authentication.
248 ◾ Data Analytics
◾◾ Block
Block can be easily defined as a source of information storage for a transaction.
A block is thus a permanent store of records that, once written, cannot be
altered or removed. If we make a comparison with record kipping common
style it’s a page of a ledger, we only go to the next page of a ledger when it fin-
ishes. Similarly, a block indicates the current transaction, and when the next
block is created, it becomes the latest one. Each time a block is “completed,” it
gives way to the next block in the blockchain. There is no limit to the number
for blocks being created; it will continue to grow as long as people use them
for transactions. Its parts are header, transaction counter, and block content,
which are shown in Figure 10.7. The header contains the basic information
about a particular block such as block version, timestamp, nonce, and parent
block hash. The basic size of BTC is around 100 bytes per transaction, out of
which 80 bytes belongs to the header.
Block header contains the following components as shown in Figure 10.7:
Block Version: It includes the ID, the size of the block, and the version number
(the set of rules that a particular block conforms to is also mentioned).
ParentBlock Hash: 2x SHA256 hash of the previous block header, this is
how the blockchain is created by pointing to the previous block.
MerkleTree Root Hash: Encodes blockchain data efficiently and securely.
It uses double SHA-256. A double hashing is done to generate a tree
element.
Timestamp: Used to figure out the mining difficulty re-target. When a node
connects to another, it generates a timestamp where 1–2 h delay is normal.
Target Threshold: This is related to mining and how hard it is to successfully
mine the block.
Blockchain in the Era of Industry 4.0 ◾ 249
◾◾ Digital signature
This digital signature was first used by Johnson as the elliptic curve digital
signature algorithm (ECDSA) in the year 2000 [32]. A digital signature is a
procedure to prove the authenticity of the blockchain transmission. It mainly
contains public and private keys to provide a double-layer protection of trans-
action information. Signing phase and verification phase are the two steps
needed for this verification. A private key is reserved for personal transactions,
whereas a public key is sent globally for outside security. In the signing part,
one encrypts data with the private key and sends the original data-encrypted
result, both. In the verification part, the receiver can easily check the validity of
250 ◾ Data Analytics
the received data with the sender’s public key. The sender sends data and hash
value, and, on the other hand, the receiver can verify the information that was
actually generated from the hash and the private key of the sender. It can be
71–73 bytes long. The private and public keys for BTC are 32 and 33 bytes,
respectively. A more detailed elaboration is given in Figure 10.8.
executed the consensus algorithm and every node with the last block information
approved that block, it extends the chain base on this block.
In order for a node in the network to accept the verification of creation of blocks
or transactions by other nodes , they must agree with each other on how the system
works, so they must find a consensus on that. In the blockchain world, some of the
well-known consensus algorithms are (i) PBFT [33], (ii) POW [26], and (iii) POS
[34] which is the backbone of Bitshares [35]. The consensus algorithm that collec-
tively uses the trusted sub-networks of a larger network is called ripple [36]. Manu
Sporny, in 2017, compared different blockchain-related algorithms and identified
the feasibilities and risks in his study [37].
Because the POW requires a huge amount of energy, it has motivated the research-
ers to find more suitable alternative blockchain protocols. This would eliminate the
need of POW by changing the protocol which consumes lesser energy with the same
guarantee of work [38]. So, the POS has solved this enormous computation power
that is required by the POW. By changing the concept where security levels of the
network were not dependent on energy consumption, it provides an energy-saving
efficiency and more cost-competitive end-to-end cryptocurrency transactions. POS
also requires hashing by the nodes in the network, but it is needed for a limited search
space. It is also known as proof of ownership because the verification is done by the
one who owns the highest amount of cryptocurrency. There is a belief that the owner
having the highest amount will not corrupt the network [30], but there are also others
252 ◾ Data Analytics
who think otherwise because of the concept “rich gets richer.” Although POS helped
us to reduce the massive energy consumptions, it has some disadvantages as well,
for example, several potential security issues where the coin age might be abused by
some malicious nodes helping them to gain significant network weight that leads to
a successful double-spending. Another issue related to the coin age is that the honest
nodes can be seen abusing the system by staking only on a periodical basis, which can
be considered as an unsecured network. In the current system, all the existing com-
ponents of a POS are predictable enough to allow the precomputation of future POS,
wherein (BlackCoin’s Proof-of-Stake protocol v2) Vasin [39] illustrated these issues
and provided a solution for them. Improved POS also enables more transactions per
second and also it is a censorship resistance that prevents maximizing the profit of the
members at the expense of nonmembers, and through this improved version, it can
recover from netsplits as well as from mass crash failure.
Another group introduced Casper “the Friendly Ghost” for improving the POS
[40]. First, there is security-deposit-based security and authentication where the
node’s “bonded validators” must place a security deposit (bonding) for serving the
consensus by producing blocks. Second, gambling on consensus, Casper makes its
validators bet a big part of their security deposits on how the consensus process is
going to turn out. The consensus process “turns out” according to their bet and the
validators have to bet their deposit on how they expect the rest to be betting their
deposits as well.
10.3.4 Cryptocurrency
Cryptocurrency is just an application of blockchain technology in real life. It is
virtual money for P2P electronic transactions. It started with BTC, and currently,
there are more than 900 cryptocurrencies in the market, which will be discussed in
detail in later sections. “Announcing the first release of BTC, a new electronic cash
system that uses a peer-to-peer network to prevent double-spending. It’s completely
decentralized with no server or central authority.”—Satoshi Nakamoto announced
this on January 9, 2009 [26]. A comparison analysis was done by Manu Sporny
and Ashiq Anjum in their study stated below, which clearly compares the perfor-
mances of different crypto coins that vary with situations [37]. According to that
study, interplanetary file system (IPFS) and hashgraph performance are better in
the context of system availability, scalability, failure tolerance, and latency. On the
contrary, BTC, Ethereum, and Blockstack are better in terms of auditability, liveli-
ness, and denial of service, but their performance decreased with higher transac-
tions and verification times. It shows that BTC, Ethereum, and Blockstack spend
around 30–60 minutes for achieving consensus. IPFS and Steller are more scalable
than other cryptocurrencies. Over 900 cryptocurrencies have been listed up to now
[35]. A study by Vigna in 2016 discussed how cryptocurrencies are changing the
trends of the age-old economic system and presenting a transparent, shared, and
independent transaction system [41].
Blockchain in the Era of Industry 4.0 ◾ 253
Every technology evolved with problems and limitations, and with time they
got their actual shape for future commanding. Several reports identified the follow-
ing issues and challenges of blockchain and BTC [30,42]. Common transactions
will be impossible in actual business with the current blockchain problems, such
ase lack of data modification scope, long approval time, huge power consumption,
storage, and security issues. These were identified as flaws of the current blockchain
technology in a survey done by the Japan Ministry of Economy, Trade and Industry
[43]. They mentioned some security issues such as network cutoff (eclipse attack)
and 51% error. 51% error can happen in the current POS protocol where an indi-
vidual or a group of people getting more than 50% share can control the activity of
the chain. More issues like this will be discussed in Section 10.4.
Uncertainty will be removed from every kind of business and transparency will
take that place. Many companies are facing problems in managing different ven-
dors in the horizontal supply chain, where every vendor is having an individual
policy and company architecture, which creates a communication gap in business
development. Blockchain can remove this barrier easily. With the blockchain, each
company can independently monitor and validate other companies without a cen-
tral authority. From raw material to complete value development, the cycle is open
not only to stakeholders but also to the consumers. For example, a car manufactur-
ing company needs to manufacture gear lever, seat belt, steering wheel, windscreen,
fuel gauge, temperature gauge, etc. A car manufacturer needs to do business with
several other organizations for a successful product development. Now, other orga-
nizations may or may not have a similar infrastructure to support the supply chain
mechanism of that particular car company. This is where blockchain is offering
vertical networking for a smart production system for every organization involved
in that product development life cycle. Figure 10.10 clearly shows that blockchain
and Industry 4.0 fit each other in several aspects, which makes blockchain a real
contender for becoming a member of the fourth industrial revolution.
Blockchain provides a platform for digital transformation of the current
industry to adapt to Industry 4.0. This technology is offering the following items
to current industry architecture to make it more acceptable and dynamic. The
left side of Figure 10.10 shows the needs of Industry 4.0. Autonomy, CPS, P2P
Figure 10.10 Common features between Industry 4.0 and blockchain technology.
Blockchain in the Era of Industry 4.0 ◾ 255
communication, and other latest technologies are the requirements of the fourth
industrial revolution. On the other hand, the right side reflects the characteristics
of the blockchain such as scalability, transparency, decentralization, and secured
communication. The abovementioned characteristics of blockchain technology are
capable enough of fulfilling the requirements of Industry 4.0.
◾◾ Digital trust
From a very lower level of business to the top level, the term that makes a
successful completion of the deal is trust. Employees to customers, everyone
wants trust without dependencies. Currently, blockchain is there to generate
digital trust among the business parties. Juri Mattalia describes that digital
trust requires three factors in a system: security, identifiability, and trace-
ability [46]. When we choose to do transactions, we trust that the products
and services provided are secure against malware and data abuse. Again, trust
must be there regarding the identity of the involved parties. Finally, if, for any
circumstance, the contract is not executed, then the contractual right is well
preserved for both parties, which is also a kind of trust. Blockchain transac-
tion shelters every party involved in a business-to-business communication.
Therefore, in the fourth industrial revolution, digitalized trust will be offered
by blockchain technologies.
◾◾ Intelligent data management
Data mining, especially big data analysis, has already been deployed in all
sectors. In future, manufacturers will use artificial intelligence and neural
networking for big data mining in business intelligence development [47].
Business intelligence can transform data into information and that informa-
tion transfers as knowledge for future use. By virtue of blockchain technol-
ogy, data will be more secure and reliable. Data manipulation and privacy
have been big issues until now, but decentralized databases and a two-layer
protection mechanism of blockchain will encourage more data mining. From
medical sector to insurance sector, data will more be secure with stakeholders
(researchers, customer, public authorities, etc.) participating in the network
as blockchain “miners” [48]. So, it is obvious that every kind of industry can
deploy blockchain technology for business intelligence development by min-
ing data.
◾◾ Smart ecosystem
Companies and organizations along with their suppliers, lead producers,
competitors, and other stakeholders form the business ecosystem. Blockchain
has a lot to contribute in generating a smart ecosystem for business. P2P
and cyber-physical communication protected by blockchain can create a
new era of the smart ecosystem that can reduce cost, time, and system loss
for Industry 4.0. It brings the concept of “smart contract” [49] that helps
to exchange money or anything of value in a transparent and indisputable
way. This not only keeps the contract stable but also ensures that it happens
256 ◾ Data Analytics
successfully and, best of all, it ensures privacy and execution of the agreed
terms between the parties.
◾◾ Digital supply chain (DSC)
Kari Korpela described the DSC as an integrated business process where an
effective connection will be there between parties involved in service, delivery,
marketing, and selling [50]. In DSC, product will be trackable throughout the
supply chain. This addition of DSC brings transparency in the fourth industrial
revolution. In the current DSC transactions, organizations are executing the
transactions via a reliable third party, preferably banks, and here comes the need
for blockchain that helps in minimizing the need for an unnecessary third-
party “bank.” One of the blockchain advantages is that it includes a public
ledger of the transactions without the need for the transaction party identities.
This is because it uses a public key infrastructure to help in notifying the coun-
terparties. Every party involved in an agreement can verify the details of execu-
tion. Blockchain can check whether every aspect of the contract has been well
maintained, and if not, a party can legally cancel the contract or agreement.
◾◾ Cybersecurity
Blockchain saves data in hash functions with timestamps, so it is highly
unlikely to be tempered since data once written cannot be overwritten; thus,
there is no chance of manipulation. According to Pentagon, blockchain tech-
nology can be used as a cybersecurity shield. The Washington Times stated
that US military sees blockchain technology as a reliable way of maintaining
security. American military and security sector can reduce the mega hacking
of drone, satellite, and communication media with the help of blockchain
technology [51]. The bank’s information systems are considered to be one of
the most secure systems. Individual banks maintain their interbank com-
munication with highly encrypted data transfer. Transfer of money, settle-
ments, information by banks were trustworthy until recent cyberattacks on
the banking sector started growing rapidly. Day by day people are losing their
faith on these banks due to corruption and political, technical, and social
competition. Security systems like “SWIFT” have also been overcome by
cyber attackers. A distributed ledger or a shared database offered by block-
chain technology can also be a secure option for banking sector communica-
tion. Confidentiality, integrity, and availability of a blockchain service are
the three main security challenges for Industry 4.0 as identified in a study
by Chhetri [52]. Since blockchain can recode every detail of a transaction,
interbank communication improves in the context of data validation, time
validation, and identity validation. Since security related issues lose a lot of
money for a business and manufacturing industry, blockchain technology
can be a good option to secure a business. Hacking and fraudulence cost busi-
nesses $400 billion a year. The report also mentioned that global revenues for
companies that offer digital security solutions were a total of $2.7 billion in
2015 and are predicted to be more than $4.7 billion in 2020 [53].
Blockchain in the Era of Industry 4.0 ◾ 257
For this reason, research industries and software industries have also started
developing banking applications to hold this rising market. For e xample, R3,
a US-based software company, has invested $107 million for developing a
blockchain or shared ledger-based finance-related application. This company
released the second version of “CORDA” software after collaborating with
more than 100 banks, layers, and professionals to shape it as the best available
banking application [54,55]. According to McKinsey survey in May 2016, half
of the bank executives believe that within the next 3 years, there will a substan-
tial effect of blockchain and cryptocurrencies on the banking and financing
sector [56]. Another similar kind of survey reported that blockchain will be
implemented in 15% of the banks present all over the world by the end of 2018
[57]. IBM reported that in the next 4 years, blockchain will be implemented
commercially in 66% of the global banks [58]. Blockchain-enabled smart con-
tract applications can easily be extended in building a P2P or crowdfunding
insurance model. The new model is expected to focus on supply and demand
matching and risk calculation aspects, where insurers will build platforms for
potential customers to post their customized requirements for insurance plans.
Insurers would apply their risk assessment techniques to perform necessary
calculations. Once the calculations are finalized, insurers can place their pre-
mium estimations giving a chance for investors to subscribe to the plan.
◾◾ Medical and health care
In the health-care industry, blockchain can secure the valuable patient data.
Sometimes, the health sector uses patient information for improving the
intelligence of health-care systems without any prior permission from the
patient. This privacy issue can be solved by a decentralized and secured block-
chain technology [59]. Again, Deloitte has surveyed the health care oppor-
tunities in Blockchain: Opportunities for Health Care [60], which identified
blockchain as a promising new distributed framework to amplify and support
the integration of health care information. Blockchain-based systems have
the potential to reduce costs of current intermediaries between doctors and
patients. In the health–care industry, blockchain technology use cases can be
quality medicine, patient care and outcomes research, and global standard
health-care system. The application field can be clinical data sharing, public
health data storage, research and clinical trials, administrative and finan-
cial health information, and doctor and food identification and prioritiza-
tion. For example, wearables and IoT devices collect patient information, and
blockchain can secure this by replacing individual cloud to the shared ledger.
Another example can be when a patient with several health issues undergoes
a severe operation, blockchain consensus technique can be applied among
specialist doctors to finalize the medical decision.
◾◾ Energy and power grid
Blockchain decentralized and privacy-protected scope can authenticate and
manage the billing process in power grid billing systems. James Basden
Blockchain in the Era of Industry 4.0 ◾ 259
describes two use cases of blockchain that can be used in the future power
grid sector: (i) Participants of a microgrid project living in a city can sell
electricity produced by their solar panels to the neighbors. Information and
transactions of this application can be maintained by blockchain technology
where each house can be considered as a “node” or miner in reference to the
blockchain. (2) Selling of unused power during the break time of a company
to another company of the same grid [61]. Private companies are deploying
fossil fuel-based electricity generation companies. They are supplying power
to the national grid where governments are facing difficulties in dealing with
environmental pollutions, stealing, and corruptions. A shared database in
this sector can track the exact flaws. Even the payment, billing, and user
identity validation take longer times for this centralized power grid system.
Blockchain transparency, census, and reputation log can remove this extra
time taken for validation.
◾◾ Vehicle and transport
Business related to vehicle sharing and renting will be highly affected by
blockchain technology. Nowadays, from bicycle to car are deploying as a
shared business in every country. The decision, rating, and approval can be
done by a shared database like the blockchain technology. Scania truck sus-
pension, booking, and maintenance system discussed by Lindberg can be a
good example for other companies of similar type [62]. Sensing and smart
vehicle communication algorithm was discussed by Sean Rowan where the
vehicle-to-vehicle communication security was ensured by a similar a lgorithm
like the blockchain private–public key [63].
◾◾ Marketplace
The blockchain trustless business communication policy will have a huge
impact on online marketplaces of the fourth industrial revolution. Payment
system, client-to-seller relation, product guarantee, and review after business
all can be blockchain based with more security and reliability. Fraud buy-
ers and sellers will be easy to detect if it is a public blockchain. Buyers and
sellers will be able to connect themselves without any middlemen and the
payment system will be more secure too. The smart contract with a built-in
reputation system can open markets such as OpenBazzar [64]. In the real
estate business, blockchain can help in tracking, verifying, and transfer-
ring document and property, which can be done with the smart contract of
blockchain [65].
◾◾ Internet of Things
If we consider the success of Airbnb and Uber for the shared economy, we
can make an estimate of how the IoT can be occupied by the blockchain
technology. On the one side, we can share our devices, vehicles, buildings,
and even services by the means of embedded sensors and network connectiv-
ity, and on the other side, blockchain helps us to keep a record of all these in
a safer way [66]. IBM has started to use its shared cloud for product tracing.
260 ◾ Data Analytics
◾◾ Transaction time
For building a consensus (approval for transaction), there is a requirement
of 10–60 minutes or even longer to get the approval for a transaction. This
kind of situation is not acceptable for business in the current blockchain,
so it has to be changed in order to get the business attention to blockchain.
Long finality time (a transaction finalized time) is a big issue in making this
technology fit for the smart factories of the fourth industrial revolution [71].
Blockchain in the Era of Industry 4.0 ◾ 261
600 KB during the end of 2016 and increased rapidly to become 3.6 MB at
the end of 2017 [72]. This trend can demotivate stakeholders while deploy-
ing blockchain technologies into a real business. An architectural change is a
must to resolve this problem.
◾◾ Energy consumption
Resources (machine power) use high-configuration computers for informa-
tion processing and transactions. Electricity consumption and the main-
tenance cost of these computers are huge. People worldwide bring more
energy-hungry computers to mine the digital currency. BTCs estimated that
the annual electricity consumption is 34.34 TWh. Electricity consumed per
transaction is 240 kWh, which raises the question that whether industries
will invest this much electricity for communication, transaction, and security
[73]. The answer is probably yes since atomic energy is producing enormous
electricity that can fulfil the needs of blockchain technology. Reports also
suggested that the entire BTC network now consumes more energy than
some countries lsuch as Serbia, Denmark, and Belarus. Another issue pointed
out by the report was carbon footprint, which is 117.63 kg of CO2 per trans-
action. Blockchain architecture and principle must be evaluated again to
fit into the fourth industrial revolution. The biggest problem with POW is
energy wastage. BTC, in particular, is voracious in its appetite for energy.
Multichainblog.com revealed the energy consumption data for the month of
October–November, 2017 [73]. On November 17, 2017, energy consumption
for BTC was 28.5 TWh. The dangerous fact is that it was just 22 TWh on
October 20, 2017. This rapid growth in energy consumption can limit the
use of blockchain.
◾◾ Security issues
The blockchain is mainly a P2P network technology with a strong security
shield. Still, technical issues are arising, such as 51% error, eclipse attack, race
attack, finery attack, brute force attack, and block discarding [43,74,75]. A
51% attack is generated when fraudulent people consume more than 50%
of POS. Eclipse attack happens when the network is cut off. When a person
tries to spend two BTCs at the same time, it is called race attack. Private
mining to generate blockchain fork produces brute force attack. Studies also
mentioned about some social security issues such as bribery attack, wallet
stealing, and fake delay creation. When BTC was started, miners were hon-
est and worked without any influence, but since the market started going
high (1 BTC = US$9,000) powerful stakeholders started finding an alternate
way for corruption. In Industry 4.0, security threats will be seriously taken;
therefore, blockchain must fulfil the security needs of the future industry
[52]. Stakeholders are paying money to bring about corruption, delay specific
consensus, steal keys, etc. In the real business system, the third party involved
resolves the issues. Since there are no middlemen in this technology, correct-
ing issues are also difficult. Too many hackers are also getting involved for
Blockchain in the Era of Industry 4.0 ◾ 263
between users and standard developers is a must before developing the standards
and regulations. In June 2016, a workshop was conducted by the World Wide Web
Consortium (www.w3.org) to discuss web and blockchain technology [76]. A coor-
dination between ISO TC 307, ITU-T, and SWIFT took place in 2017 to stan-
dardize blockchain protocol-related issues. Federal Reserve System is also taking
initiatives to standardize their monetary transactions via blockchain technology
[77]. Unstable market values of cryptocurrencies will create problems in real-life
blockchain implementation in the industry. Currently, bank charges are fixed and
currency rates are also predictable. On the other hand, in cryptocurrency, fees are
hard to predict due to frequent changes in its value. Taxation procedures, profit-
making, price estimation, and cost analysis will be much more complicated.
Financial stability is very much needed; otherwise, stakeholders will move away
from blockchain due to higher business risks. Prices for the same product can vary
significantly from one day to the next, which is very unusual for any business.
According to blockchain.info, at the end of 2016, transaction fees cost around 136
(BTC, which was raised to 784 BTC at the end of 2017. This dissimilarity hap-
pened due to uneven changes in its market value. The same website stated that the
market value for 1 BTC was US$3,777 on December 17, 2016, which went up to
US$19,500 on December 17, 2017, without any legit variable. It is predicted that
popularity, energy and storage consumption, and the price of other cryptocurren-
cies are the main factors affecting BTC’s value [72].
from company level to country level, policy makers must synchronize the laws and
regulations. They should also think elaborately about new technologies to cope with
the current trends. The vital requirement for bringing about a successful fourth
industrial revolution (Industry 4.0) is the cooperation among aforementioned par-
ties (Figure 10.12). For example, if a bank wants to deploy a blockchain technology,
there must be collaboration among the bank’s customers, software developers, and
other banks. If a fruitful cooperation takes places among them, only then bank
owners can invest reliably.
possible solution. By involving the blockchain shared ledger system, they can trace
and reduce the product supply time. Next we will discuss a digital product delivery
system (DSC) as a blockchain use case.
◾◾ Information management
Blockchain can manage all the information related to the supply chain of
an industry. Receipt, delivery report, identification numbers, and guarantee
papers all can be protected by blockchain technology.
◾◾ Utilized function
Secured and shared data storage of blockchain technology has a lot to con-
tribute in supply management. The three important blockchain functions
used in supply chains [43] are “execution of applications using a dedicated
script (smart contracts),” “ensuring traceability of data and enabling transpar-
ent transactions (shared ledger and consensus),” and “maintaining the ecosys-
tem against any attacks by malicious users without a central authority (public
and private key security).”
◾◾ Impact on industrial structure
Order placement for raw material, quotation supplying, delivery, checking,
and payment methods are also traceable. If at any level of the chain, defec-
tive goods delivery reports are found against a particular supplier, then that
information will be open and visible to all.
◾◾ Other possibilities
Retailers (downstream), wholesalers (middle stream), and manufacturers
(upstream) communication procedure can be made independent by block-
chain where everyone can contact each other. Here, the reputation or trust
is not provided by a middleman and communication will happen from the
root level to the top level if blockchain is used for supply chain management.
10.5.2 Comparative Analysis
Table 10.1 shows the comparative analysis of the effects that blockchain will have
on the future industry.
each industry has its own database; therefore other companies can only have
information that is supplied by that particular company. This creates data
manipulation and system loss. Blockchain can replace a shared system where
intercompany communication will be automatic. Industry 4.0 will certainly
use this technique to make business interactions more transparent.
◾◾ Human resources and procurement management
Potential contractors and partner identification and recruitment will be easier
since both employer and employee can have a highly secured, distributed
database. A company structure in recruitment can be changed as shown in
Figure 10.14 where previous companies and academies can generate a reputa-
tion database and the next destination can be selected automatically from this
data. Similarly, product and services can be ranked, which will generate more
revenue for any reputed manufacturer in Industry 4.0 architecture.
◾◾ Trade and transaction management
Middlemen from transactions will die out and industry’s logistic and business
architecture will be different. In current transactions (Figure 10.15), exporters
and importers do business via banks. After blockchain-based shared storage
provides direct transaction facilities, product development will be cost and
time efficient.
10.6 Conclusion
The blockchain is not a magic, but it is an innovative, general-purpose technology,
offering a new way of value creation for almost all domains of industries. In this chap-
ter, we have discussed the influence of blockchain and cryptocurrencies on Industry
4.0. First, we precisely mentioned the prerequisite of fourth industrial revolution, and
consequently, we identified the key characteristics and components of blockchain
technology that blends with the ongoing industrial revolution. While reporting the
challenges and issues of blockchain technologies, we also mentioned that collabora-
tion is a must among stakeholders for bringing about a solid blockchain standard.
Finally, we have described the future scopes and applicable sectors for blockchain in
the fourth industrial revolution. In summary, this chapter is a sole attempt toward
highlighting the impact of blockchain on Industry 4.0. We believe that our discus-
sion will help readers to easily understand the strategic, organizational, economic,
informational, and technological aspects of blockchain and Industry 4.0.
References
1. Ganschar, O., Gerlach, S., Hämmerle, M., Krause, T., & Schlund, S. (2013).
Produktionsarbeit der Zukunft-Industrie 4.0 [Production work of the future industry]
(pp. 50–56). D. Spath (Ed.). FraunhoferVerlag, Stuttgart.
2. Kagermann, H., Lukas, W. D., & Wahlster, W. (2011). Industrie 4.0: Mitdem Internet
der Dinge auf demWegzur 4. industriellen Revolution [Industry 4.0: With the Internet of
Things on the way to the 4th industrial revolution]. VDI nachrichten, 13, 11.
3. The new High-Tech Strategy. (2014, December 03). Retrieved December 13, 2017, from
www.hightech-strategie.de/de/The-new-High-Tech-Strategy-390.php.
4. Kagermann, H., Helbig, J., Hellinger, A., & Wahlster, W. (2013). Recommendations
for implementing the strategic initiative INDUSTRIE 4.0: Securing the future of
German manufacturing industry; final report of the Industrie 4.0 Working Group.
Forschungsunion.
5. Bungart, S. (2014) Industrial Internet versus Industrie 4.0. Interview in: Produktion.
Technik und Wirtschaftfür die deutsche Industrie. Retrieved from www.produktion.de/
technik/automatisierung/industrial-internet-versusindustrie-4-0-338.html
6. Bauernhansl, T., Ten Hompel, M., & Vogel-Heuser, B. (Eds.). (2014). Industrie 4.0 in
Produktion, Automatisierung und Logistik: Anwendung, Technologien und Migration
[Industry 4.0 in production, automation and logistics: application, technologies and migra-
tion] (pp. 1–648). Springer Vieweg, Wiesbaden.
7. Dais, S. (2017). Industrie 4.0–Anstoß, vision, vorgehen. In B. Vogel-Heuser,
T. Bauernhansl, and M. ten Hompel (Eds.), Handbuch Industrie 4.0 Bd. 4 (pp. 261–277).
Springer Vieweg, Berlin.
8. MacDougall, W. (2014). Industrie 4.0: Smart manufacturing for the future. Germany
Trade & Invest.
9. C. Baur & D. W. (2015, June). Manufacturing’s next act. Retrieved December 14, 2017, from
www.mckinsey.com/business-functions/operations/our-insights/manufacturings-next-act
10. Davies, R. (2015). Industry 4.0.Digitalisation for productivity and growth. Briefing
from EPRS. European Parliamentary Research Service.
270 ◾ Data Analytics
11. L. G. (2015, May 18). What is industrie 4.0 and what should CIOs do about it? Retrieved
December 14, 2017, from www.gartner.com/newsroom/id/3054921
12. Hofmann, E., & Rüsch, M. (2017). Industry 4.0 and the current status as well as future
prospects on logistics. Computers in Industry, 89, 23–34.
13. Geissbauer, R., Vedso, J., & Schrauf, S. (2016). Industry 4.0: building the digital
enterprise: 2016 global industry 4.0 survey. PwC, Munich.
14. Deloitte, A. G. (2015). Industry 4.0 challenges and solutions for the digital transforma-
tion and use of exponential technologies. McKinsey Global Institute.
15. Lee, J., Bagheri, B., & Kao, H. A. (2015). A cyber-physical systems architecture for
industry 4.0-based manufacturing systems. Manufacturing Letters, 3, 18–23.
16. Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013). Internet of Things (IoT):
A vision, architectural elements, and future directions. Future Generation Computer
Systems, 29(7), 1645–1660.
17. Wollschlaeger, M., Sauter, T., & Jasperneite, J. (2017). The future of industrial communi-
cation: Automation networks in the era of the internet of things and industry 4.0. IEEE
Industrial Electronics Magazine, 11(1), 17–27.
18. Hossain, M. S., & Muhammad, G. (2016). Cloud-assisted industrial Internet of
Things (iiot)–enabled framework for health monitoring. Computer Networks, 101,
192–202.
19. Energie, B. F. (2016, September 14). AutonomikfürIndustrie 4.0. Retrieved December 14,
2017, from www.bmwi.de/Redaktion/DE/Publikationen/Digitale-Welt/autonomik-fuer-
industrie-4-0.html. Federal Ministry for Economic Affairs, Germany.
20. Wang, S., Wan, J., Li, D., & Zhang, C. (2016). Implementing smart factory of
industrie 4.0: an outlook. International Journal of Distributed Sensor Networks, 12(1),
3159805.
21. Abramovici, M., Göbel, J. C., & Savarino, P. (2017). Reconfiguration of smart prod-
ucts during their use phase based on virtual product twins. CIRP Annals-Manufacturing
Technology.
22. Lee, J., Kao, H. A., & Yang, S. (2014). Service innovation and smart analytics for industry
4.0 and big data environment. Procedia Cirp, 16, 3–8.
23. Pierdicca, R., Frontoni, E., Pollini, R., Trani, M., & Verdini, L. (2017) The use of augmented
reality glasses for the application in industry 4.0. In L. De Paolis, P. Bourdot, and A. Mongelli
(Eds.), Augmented Reality, Virtual Reality, and Computer Graphics. AVR 2017. Lecture Notes in
Computer Science, Vol. 10324. Springer, Cham.
24. Rehnberg, M., & Ponte, S. (2017). From smiling to smirking? 3D printing, upgrading
and the restructuring of global value chains. Global Networks.
25. Huber, A., & Weiss, A. (2017, March). Developing human-robot interaction for an
industry 4.0 robot: How industry workers helped to improve remote-HRI to physical-
HRI. In Proceedings of the Companion of the 2017 ACM/IEEE International Conference on
Human-Robot Interaction (pp. 137–138). ACM, Vienna, Austria.
26. Nakamoto, S. (2008, October). Bitcoin: A peer-to-peer electronic cash system.
27. Hackett, R. (2016, May 23). Wait, what is blockchain? Retrieved December 15, 2017,
from http://fortune.com/2016/05/23/blockchain-definition/
28. Kosba, A., Miller, A., Shi, E., Wen, Z., & Papamanthou, C. (2016, May). Hawk: The
blockchain model of cryptography and privacy-preserving smart contracts. In 2016
IEEE Symposium on Security and Privacy (SP) (pp. 839–858). IEEE, San Jose, CA.
29. Szabo, N. (1997). Formalizing and securing relationships on public networks. First
Monday, 2(9).
Blockchain in the Era of Industry 4.0 ◾ 271
30. Zheng, Z., Xie, S., Dai, H., Chen, X., & Wang, H. (2017, June). An overview of
blockchain technology: Architecture, consensus, and future trends. In G. Karypis
and J. Zhang (Eds.), 2017 IEEE International Congress on Big Data (BigData Congress)
(pp. 557–564). IEEE, Honolulu, HI.
31. Pilkington, M. (2015, September 18). Blockchain technology: Principles and applications
In F. Xavier Olleros and M. Zhegu (Eds.), Research Handbook on Digital Transformations.
Edward Elgar, 2016. Retrieved from SSRN, https://ssrn.com/abstract=2662660
32. Johnson, D., Menezes, A., & Vanstone, S. (2001). The elliptic curve digital signature
algorithm (ECDSA). International Journal of Information Security, 1(1), 36–63.
33. C. Miguel & L. Barbara. Practical byzantine fault tolerance. In Proceedings of the
Third Symposium on Operating Systems Design and Implementation, Vol. 99, New
Orleans, USA, 1999, pp. 173–186.
34. Kiayias, A., Russell, A., David, B., & Oliynykov, R. (2017, August). Ouroboros: A provably
secure proof-of-stake blockchain protocol. In J. Katz and H. Shacham (Eds.), Advances in
Cryptology – CRYPTO 2017. CRYPTO 2017. Lecture Notes in Computer Science, Vol. 10401.
Springer, Cham.
35. Chen, S., Chen, C. Y. H., Härdle, W. K., Lee, T. M., & Ong, B. (2017). Econometric
analysis of a cryptocurrency index for portfolio investment. In Handbook of Blockchain,
Digital Finance, and Inclusion (Vol. 1, pp. 175–206). Academic Press, San Diego, CA.
36. D. Schwartz, N. Youngs, and A. Britto.(2014). “The Ripple protocol consensus algo-
rithm”, [PDF file] Ripple Labs Inc White Paper. Retrieved from https://ripple.com/
files/ripple consensus whitepaper.pdf
37. Sporny, M. (2017, April). LD-DL’17 workshop keynote talk by many sporny: Building
better blockchains via linked data. In Proceedings of the 26th International Conference on
World Wide Web Companion (pp. 1429–1429). International World Wide Web Conferences
Steering Committee.
38. O’Dwyer, K.J., Malone, D. (2014). Bitcoin mining and its energy footprint. ISSC
2014/CIICT 2014, imerick, June 26–27.
39. Vasin, P. (2015). Blackcoin’s proof-of-stake protocol v2. 2014. Retrieved from http://
blackcoin.co/blackcoin-pos-protocol-v2-whitepaper.pdf
40. Zamfir, V. (2015). Introducing Casper “the friendly ghost”. Ethereum Blog. Retrieved
from https://blog. ethereum.org/2015/08/01/introducing-casper-friendly-ghost
41. Vigna, P., & Casey, M. J. (2016). The Age of Crypto Currency. Picador St Martin’s Press,
New York. Retrieved from https://theageofcryptocurrency.com/
42. Kiviat, T. I. (2015). Beyond bitcoin: Issues in regulating blockchain tranactions. Duke LJ,
65, 569.
43. The Results of a Survey on Blockchain Technologies and Related Services were compiled.
(2016). Retrieved December 17, 2017, from www.meti.go.jp/english/press/2016/0531_01.
html. Japan’s Ministry of Economy, Trade and Industry (METI).
44. Geissbauer, D., & Vedso, J. (2016, April). Industry 4.0. Retrieved December 16, 2017,
from www.pwc.com/gx/en/industries/industry-4.0.html
45. Challenges and solutions for the digital transformation and use of exponential | Deloitte
Taiwan | Manufacturing. (2015, September 2). Retrieved December 16, 2017, from
www2.deloitte.com/tw/en/pages/manufacturing/articles/industry4-0.html
46. Mattila, J., & Seppälä, T. (2016). Digital trust, platforms, and policy. ETLA Brief No., 42.
47. Mishra, D. P., Samantaray, S. R., & Joos, G. (2016). A combined wavelet and data-mining
based intelligent protection scheme for microgrid. IEEE Transactions on Smart Grid, 7(5),
2295–2304.
272 ◾ Data Analytics
48. Azaria, A., Ekblaw, A., Vieira, T., & Lippman, A. (2016, August). Medrec: Using block-
chain for medical data access and permission management. In International Conference on
Open and Big Data (OBD) (pp. 25–30). IEEE, Vienna, Austria.
49. Peters, G. W., & Panayi, E. (2016). Understanding modern banking ledgers through block-
chain technologies: Future of transaction processing and smart contracts on the internet of
money. In P. Tasca, T. Aste, L. Pelizzon, and N. Perony (Eds.), Banking Beyond Banks and
Money. New Economic Windows. Springer, Cham.
50. Korpela, K., Hallikas, J., and Dahlberg, T. 2017. Digital supply chain transformation
toward Blockchain integration. In 50th Hawaii International Conference on System Sciences
(HICSS 2017), Waikoloa, HI, pp. 4182–4191.
51. Boylan, D. (2017, August 17). Pentagon eyes bitcoin blockchain technology as cybersecu-
rity shield. Retrieved December 16, 2017, from www.washingtontimes.com/news/2017/
aug/17/pentagon-eyes-bitcoin-blockchain-technology-as-cyb/
52. Chhetri, S. R., Rashid, N., Faezi, S., & Faruque, M.A.A. (2017). Security trends and
advances in manufacturing systems in the era of industry 4.0. IEEE/ACM ICCAD,
pp. 1039–1046. doi:10.1109/ICCAD.2017.8203896.
53. Yakowicz, W. (2015, September 8). Companies Lose $400 Billion to Hackers Each
Year. Retrieved December 16, 2017, from www.inc.com/will-yakowicz/cyberattacks-
cost-c ompanies-400-billion-each-year.html
54. Rutter, K., & Rutter, C. (2017, December 11). Kevin Rutter. Retrieved December 16,
2017, from www.r3.com/research/
55. Guo, Y., & Liang, C. (2016). Blockchain application and outlook in the banking industry.
Financial Innovation, 2(1), 24.
56. How blockchains could change the world. (2016, May 15). Retrieved December 16, 2017,
from www.mckinsey.com/industries/high-tech/our-insights/how-blockchains-could-
change-the-world
57. Blockchain Could Start Making Some Real Waves the Banking Industry Next Year. (2016,
November 28). Retrieved December 16, 2017, from http://fortune.com/2016/09/28/
blockchain-banks-2017/
58. Jia, C. & Feng, H. (2016) Blockchain: From Digital Currencies to Credit Society [M].
CITIC Publishing Group, Beijing, China.
59. Yue, X., Wang, H., Jin, D., Li, M., & Jiang, W. (2016). Healthcare data gateways: found
healthcare intelligence on blockchain with novel privacy risk control. Journal of Medical
Systems, 40(10), 218.
60. Krawiec, R. J., Housman, D., White, M., Filipova, M., Quarre, F., Barr, D., …
Tsai, L. (2016, August). Blockchain: Opportunities for health care. In Proc. NIST
Workshop Blockchain Healthcare (pp. 1–16).
61. Basden, J., & Cottrell, M. (2017). How utilities are using blockchain to modern-
ize the grid. Harvard Business Review, Retrieved from https://hbr.org/2017/03/
how-utilities-are-using-blockchain-to-modernize-the-grid
62. Lindberg, J. (2017). Blockchain technology in Scania Services: An investigative study
of how blockchain technology can be utilized by Scania. Blockchain technology in
Scania Services : An investigative study of how blockchain technology can be utilized
by Scania (Dissertation). Retrieved from http://urn.kb.se/resolve?urn=urn:nbn:se:uu:
diva-327959
63. Rowan, S., Clear, M., Gerla, M., Huggard, M., & Goldrick, C. M. (2017). Securing
vehicle to vehicle communications using blockchain through visible light and acoustic
side-channels. CoRR, abs/1704.02553.
Blockchain in the Era of Industry 4.0 ◾ 273
64. Online Marketplace | Peer-to-Peer Ecommerce. (n.d.). Retrieved December 16, 2017,
from www.openbazaar.org/
65. Blouin, M. R., & Serrano, R. (2001). A decentralized market with common values
uncertainty: Non-steady states. The Review of Economic Studies, 68(2), 323–346.
66. Huckle, S., Bhattacharya, R., White, M., & Beloff, N. (2016). Internet of things, blockchain
and shared economy applications. Procedia Computer Science, 98, 461–466.
67. Christidis, K., & Devetsikiotis, M. (2016). Blockchains and smart contracts for the
internet of things. IEEE Access, 4, 2292–2303.
68. Bieler, D. (2017). Blockchain and the Internet of Things: the IoTblockchain picture. Retrieved
December 16, 2017, from www.i-scoop.eu/blockchain-distributed-ledger-technology/
blockchain-iot/
69. Atzori, M. (2015). Blockchain technology and decentralized governance: Is the state
still necessary? Retrieved from http://dx.doi.org/10.2139/ssrn.2709713
70. White, M., Killmeyer, J., & Chew, B. (2017, September 11). Will blockchain transform
the public sector? Retrieved December 17, 2017, from www2.deloitte.com/insights/us/en/
industry/public-sector/understanding-basics-of-blockchain-in-government.html
71. Peters, G. W., & Panayi, E. (2016). Understanding modern banking ledgers through
blockchain technologies: Future of transaction processing and smart contracts on the
internet of money. In P. Tasca, T. Aste, L. Pelizzon, and N. Perony (Eds.), Banking Beyond
Banks and Money. New Economic Windows. Springer, Cham.
72. Bitcoin Charts & Graphs—Blockchain. (2017, December 17). Retrieved December 17,
2017, from https://blockchain.info/charts
73. The Blockchain Immutability Myth. (2017, May 4). Retrieved December 17, 2017, from
www.multichain.com/blog/2017/05/blockchain-immutability-myth/
74. Usborne, S. (2017, May 15). Digital gold: why hackers love Bitcoin. Retrieved
December 17, 2017, from www.theguardian.com/technology/2017/may/15/digital-gold-
why-hackers-love-bitcoin-ransomware
75. Conti, M., Lal, C., & Ruj, S. (2017). A survey on security and privacy issues of
Bitcoin. IEEE Communications Surveys & Tutorials.
76. MIT Media Lab. (2016, June 30). Retrieved December 17, 2017, from www.
w3.org/2016/04/blockchain-workshop/
77. Mills, D. C., Wang, K., Malone, B., Ravi, A., Marquardt, J. C., Badev, A. I., … Ellithorpe,
M. (2016). Distributed ledger technology in payments, clearing, and settlement.Finance
and Economics Discussion Series 2016–095. Washington: Board of Governors of the
Federal Reserve System. Retrieved from https://doi.org/10.17016/FEDS.2016.095
78. Four blockchain use cases transforming business. (2017, May 25). Retrieved December 17,
2017, from www.ibm.com/blogs/internet-of-things/iot-blockchain-use-cases/
Chapter 11
Contents
11.1 Introduction..............................................................................................276
11.1.1 Chapter Roadmap.........................................................................277
11.2 Origin of Dark Data.................................................................................277
11.3 R isks of Dark Data...................................................................................278
11.4 Dark Data Analytics: An Untapped Opportunity.....................................279
11.4.1 I mplication of Dark Data in the Health Sector............................. 280
11.4.2 D ark Data for Gaining Market Advantage....................................281
11.4.3 D ark Data for Social Media Insights.............................................282
11.4.4 Retailers Providing Personalization with the Help of Dark Data.....282
11.5 Different Ways to Eliminate Dark Data....................................................283
11.5.1 T ools and Technique for Collecting and Analyzing Dark Data.....283
11.5.2 A Brief Introduction to DeepDive................................................ 284
11.5.3 S ix Steps to Identify and Manage Dark Data.................................285
11.6 Dark Data Solution Provided by Companies............................................ 286
11.6.1 A I Foundry’s Agile Solutions for Transformation of Dark Data......286
11.6.2 D ark Data Fracking by Datumize.................................................287
11.6.3 N uix Information Governance Solution........................................287
11.6.4 Deloitte: Insight’s Way to Start Extracting Value from Dark Data..... 288
11.7 International Data Corporation’s Research on Organization’s Ability
to Derive Value from Dark Data...............................................................289
11.8 Recommendations on Managing Dark Data.............................................290
11.9 Conclusion................................................................................................291
References..........................................................................................................291
275
276 ◾ Data Analytics
11.1 Introduction
The whole world is now more linked and shared than ever before. As the Internet
becomes even more indispensably attached to our everyday lives, we start to tread
on the path that leads to a technology-driven future. We are also creating huge
amounts of data in the process, both willingly and unwillingly. Humongous
amounts of data are being generated that most of us are not even aware of because
most of them are unstructured data hiding in plain sight. A c ompany must increase
productivity, decrease cost, be up to date with market opportunity, and influ-
ence potential customers to keep up with the evolving environment. All of these
goals can be achieved if we can tap into the realm of dark data, i.e., unstructured,
unused, and unexplored big data. Dark data analytics is still new and untapped. It
is a unique opportunity to go beyond the traditional routine spreadsheet analytics.
There are both risks and opportunities in dark data. Let us look at a simple but
insightful example about how an employee of Target used its unstructured, unused,
and unexplored customer data to gain a business advantage.
Target, a discount store retailer, found out about the pregnancy of a girl even
before her father did only by analyzing the buying patterns of pregnant women [1].
Target assigns a unique ID number to its customers that is linked with their email IDs,
names, and credit card numbers, and it remains in their database. When this event
happened, competitions were fierce among the retail stores for acquiring pregnant
customers so that they would purchase everything related to their babies in the future
and persuade them to become loyal and long-lasting customers. So, the store that
reached a pregnant woman before anyone else hit the jackpot and gained a profitable
buyer. Target statistician, Andrew Pole, was given the responsibility to find a way so
that they could reach out to pregnant women before any other store could. He was
given the job of predicting customer pregnancy. So, he observed the buying behaviors
among the ladies who signed up for their baby registries in the past. And surprisingly,
there were significant differences between a regular and a pregnant woman’s buying
patterns. Crawling through the database, Pole identified about 25 products with dif-
ferentiable characteristics. All of them were analyzed together by Pole. After that he
was able to assign each shopper a “pregnancy prediction” value based on their buying
pattern. Now, let us go back to the story where the girl was sent an email with coupons
for baby products and her father got angry at Target for sending this email as his high-
school-going daughter could not be pregnant. A few days later, Target called him to
apologize, but instead of the Target authority, the father was apologizing as he did not
know about his daughter’s pregnancy beforehand. Now, this may seem interesting and
very much profitable for Target, but this event also posed privacy questions. Should a
retail store know about someone’s pregnancy even before the family does? Many ques-
tions were raised which made them change their strategy to mix pregnancy-related
advertising with day-to-day ads so that people did not feel that their privacy was being
violated [1]. Even though this event raised controversy, we can see that the data was
available all the time, but it was hidden, and all it needed was some attention.
Dark Data for Analytics ◾ 277
Dark data offers the possibilities to shape the business in the future. From this
example, it is clear that how much opportunity lies in the untapped world of dark
data. But dark data analytics is not as easy as it sounds. There are different tools
and techniques out there to derive value from dark data. Hadoop and DeepDive
are noteworthy solutions to the problem of dark data. Corporate solutions are also
provided by prominent companies to transform and extract value from dark data.
Solutions also vary according to the source and type of data. Data is going to be the
driving force in the near future and to be in the driving seat. We need to ensure that
this huge amount of dark data is handled effectively and efficiently.
11.1.1 Chapter Roadmap
Rest of the chapter is organized as follows: Section 11.2 explains how data goes
dark in the first place. Section 11.3 contains a discussion on risks that dark data
possesses, followed by the various opportunities of dark data discussed in Section
11.4. Section 11.5 describes the tools and techniques for collecting and analyzing
dark data and portrays ways of eliminating dark data. Section 11.6 discusses dif-
ferent dark data solutions provided by different companies. Section 11.7 deals with
International Data Corporation’s research on an organization’s ability to derive
value from dark data. Section 11.8 contains recommendations on managing dark
data, and Section 11.9 concludes the chapter.
lastly the spam emails that alert us not to trust every source indicate veracity or the
accuracy of the source. Besides, these data can be irregular and inconsistent with
seasonality. All of these make it really difficult to search, process, store, and ana-
lyze. Due to the lack of awareness, proper knowledge, accessibility measures, and
extraction methods, big data turns into its evil twin, i.e., dark data (Figure 11.1).
Dark data includes emails, documents, presentation files, including both inter-
nal and external archived sensor data from Internet of Things (IoT), spreadsheets,
backups of operation database, images, partial documents, audio files, CCTV sur-
veillance footage of a superstore, geographic data, partial codes, zip files, log files,
and everything that is unused, underutilized, neglected, or forgotten in the system.
Networking devices generate a huge amount of information. If this huge amount
of data is not analyzed, it will simply go dark. Due to our inability to handle data
efficiently and inconsistency in the format of the data that analytics do not support,
we cannot turn these into actionable information [5].
To some extent, data buried in the deep web can be considered as dark data,
too. Deep web is not indexed by our typical search engines and includes data from
academia, government agencies, legal documents, scientific studies, subscription
only information, financial records, user communities, and many more. And dark
web is something inside the deep web that is completely untraceable and hidden,
and it cannot be accessed with the standard web browsers [6].
inactive, and unmanaged. The data remains dark when it is forgotten. Unmonitored
and unsecured data always poses threat because there are different compliance rea-
sons, and there is also the risk of sensitive information falling into the wrong hands.
Hackers can easily hack into the system when the location and even the content is
unknown to the authority, and it can be deadly in terms of reputation and business
profits, and may result in complete shutdown. Dark data poses serious security risks
in this ever-connected world where one can easily hack into a system and cause
havoc. A company can be unaware of the existence of a data just to find out about
it only after it is exposed to the public. How can one protect or hide something
that they do not even know about? Also, when a company is asked to provide some
data, it really hurts their reputation when they are unable to do that. Apart from
the obvious risks of losing business opportunities derived from dark data, we also
have to be aware of the other risks associated with it [7].
even emojis that can tell a lot about social patterns. One can analyze the associated
metadata, translate audio to text, use photo recognition, and categorize emojis into
different intensities of feelings expressed to gain more insight and understand the
content. Now, it requires a lot of effort than for the textual ones and may not be
worth it in many cases, but it does not need to turn into dark data. At the very out-
set, we can filter out and sort the data from the most important to the least, which
will definitely result in the most time-efficient outcome. And in a world of sharing
nontextual information, which is only going to increase in the future, it is very
unlikely that effort to mitigate dark data will go in vain. So better late than never;
it is high time that we took nontextual data seriously and built a self-sustainable
and long-lasting system around it [5].
user permission, we can eliminate the legal issues and then develop a model that
will predict future sickness. It can be done by monitoring people’s consumption
patterns and also by observing historical data on how a certain type of consump-
tion leads to different outcomes. We can observe Mr A’s health compared to Mr B’s,
based on their food habits over a year and deduce the reason for different or similar
outcomes. Now, think about millions of user data accumulated in a single system
and analyze it to find a relationship between food habits and health. We will be able
to predict health issues successfully and help prevent diseases, instead of curing it
after it has already happened.
Indiana University Health (IU Health) wants to personalize individual health
care and improve overall health outcomes. To achieve this goal, they are exploring
the world of unstructured and non-traditional data. The relationship between the
hospital and patient has been based on individual visits and treatments. A holis-
tic approach is missing in the current system. IU Health wants to build loyalty
among patients and provide efficient but affordable health care solutions with the
help of unexplored data. For that they need to have a 360° understanding about
their patient. Both verbal and written free-form notes are given to patients by their
consultants and all of that remains unmanaged. With the help of voice recogni-
tion, text analysis, and deep learning those unmanaged data can be utilized for the
patient’s betterment in the future. These historical medical records will enrich the
file of a patient by adding relevant information. All of this will allow IU Health to
develop an in-depth understanding about patient needs and formulate a model on
how patients can use the health-care services more efficiently. They are also trying
to find patterns of sickness, access to health care, and historical mapping of a local
area’s health outcomes. Cognitive computing and both external data and patient
data can find out how socioeconomic factors may affect someone’s relationship with
health-care providers. In the end, they want to use this scattered and unused dark
data and unveil its potential to improve individual health care as well as the overall
health of the population [13].
images to understand their sense of style. They begin the process with a detailed
questionnaire about their choice and preference in clothing. Sixty data scientists
work together to analyze all of it and produce a detailed understanding about their
customers. Kroger Co., a grocery supermarket chain, is using its customer’s pur-
chasing history to give on-time suggestions about purchasing a product on a 4-inch
display mounted in the aisles [13].
Have you wondered how ads are shown on Facebook related to a product, right
after you have searched it on Google? How are they even doing that? No wonder
that your particular interest is channeled through the mediums and is used in mar-
keting different products. You are visiting a website and a few minutes later their ad
is shown on Facebook. They are exploiting every possible source of customer infor-
mation to get to potential buyers. They are not giving any chance for the creation of
dark data, so others also should not sit idle and watch valuable data turn into dark
data. Data is now the competitive currency in today’s digital transformation [6].
Consumer behavior has changed dramatically over the years. Digital marketing
has taken over traditional marketing and every bit of customer information is now
critical for company success.
became 100 MB/second. Now, it takes 2.5 hours to read all of it and this is really
a long time. We can read data simultaneously from multiple sources to reduce the
time when drives work in parallel and provide shared access to each other. But
there are two problems when we connect multiple disks: One is the high chance
of hardware failure and another is to combine data from different disks during the
analysis. And, this is where Hadoop steps in [17]. One of the main reasons behind
the popularity of Hadoop is that Google, Facebook, Amazon, and Yahoo use it on
the largest sets of data in the world. Hadoop also addresses the two main problems
stated above with its two main components. Hadoop Distributed File System helps
to manage data by splitting and putting it on different nodes, then replicating and
managing it. Map Reduce processes and calculates the data on each node [18].
We need to have a way to transform data into different forms according to the
needs of different analytics platforms. Syncsort’s Big Data solutions suite enables
one to access data, to translate, and integrate it with the help of DMX-h to move it
easily into Hadoop. It allows integration with traditionally dark environments such
as mainframes [5].
Hadoop clusters and NoSQL databases have the ability to process this huge
amount of data, which makes it possible to integrate dark data into the application
and unlock business value. A Hadoop-based data warehouse in Edmunds.com Inc.
has speeded up the old time-consuming process that made the data invalid by the
time it was analyzed. It has opened up new dimensions of data and helped in reduc-
ing the operating cost. People who work with keyword acquisition for paid searches
and online presence can now probe real-time data to analyze the change in buying
patterns, which will eventually affect the marketing tactics. And, according to the
blog post of Philip Potloff, chief information officer at Edmunds, the newly estab-
lished system saved more than $1.7 million between February and mid-June. Merv
Adrian, a Gartner analyst, said that increasing number of companies are going to
audit archives of data to point out the dark ones and streamline them to possible
business requirements. We know that the transactions are occurring, but we do not
know what is happening around them and that is something we need to figure out
so that we can turn the dark data into a potential business gold mine [19].
DeepDive to fight human trafficking. The assumption was that trafficked individu-
als offer sexual services at a lower cost than others. Ads are given online in plain
sight in different websites. In this case, DeepDive is used for searching all over the
Internet for low-cost sexual services that is related to trafficking. All of this resulted
in the real perpetrators getting arrested [22]. While training data is used by most
of the machine learning techniques, DeepDive learns from distant supervision that
is less costly than the traditional method [23]. DeepDive uses machine learning
techniques, but it is different from traditional systems in many ways. DeepDive
asks users for features rather than algorithms and can extract data from millions
of sources. It allows developers to write simple rules and adapt by learning from
the user feedbacks. It can achieve higher quality than humans. PaleoDeepDive has
achieved a winning performance in extracting complex knowledge by assembling
synthetic paleontological databases in entity relation extraction competitions [21].
Let us see how DeepDive functions following the four steps: The first step is data
processing that includes adding raw input data, giving the raw data a structure, and
then feature representation. The second step is distant supervision of the dataset
to enable machine learning. It is also known as data programming for extracting
features, training classifiers, and predicting relations. After that, the third step is
to specify the model on which DeepDive will perform learning and inferencing.
We need to specify the connections and features of the variables that the system
will predict for us and DeepDive will perform statistical inference over the model.
Lastly, it will determine the probability of each variable being true. In the last step,
we will have to measure its accuracy, i.e., accuracy of the model—estimating the
precision with Mindtagger, browsing the data with Mindbender, and monitoring
the statistics with dashboard and calibration plots [24].
1. We have to identify the source of the data and define it. System-generated
data is more likely to be helpful compared to employee-generated data. We
need to collect in a conducive form that is analyzable. But there is always a
chance that these efforts may go in vain.
2. The next step is doing a cost–benefit analysis to understand the usefulness
of the data in hand. The data should serve a definite purpose, and we should
avoid the temptation to preserve the data after the purpose is met.
3. Now, we have to select what data we will keep and what we will delete. We
have to find out which ones are useful and can be useful in the future, and,
on the other hand, we have to delete data with no apparent value. It is wise to
think that some of it might come in handy in the future, but storing it may
create chaos and confusion when it gets mixed up with important data.
286 ◾ Data Analytics
4. We also have to justify why we deleted some data by mapping it to the reten-
tion and disposition schedule. The data that is not yet addressed is addressed
by creating new categories on the retention schedule. This is important
because some policy may or may not address issues that may arise.
5. We have to execute our defensible disposition plan and make sure it is actu-
ally done. But it is important to keep the documents, mentioning the “why”
and the “when” of deletion.
6. Every year, we need to look back and calculate whether the repurposed data
is adding any value to the organization or not. It not, then there is no reason
to use it again and it should be buried in a proper, defensible manner.
Dark data may seem like a buzzword, but it is going to be a part of every organiza-
tion in recent future as the volume of electronically stored information is only going
to increase. The best way to respond to that problem is to implement an information
governance program. It will properly and efficiently manage all of the organization’s
data produced throughout its life cycle. It will also recognize the possibility of any dark
data occurrence and implement measures to make it usable or to properly delete it [25].
We can also get the value out of dark data in three steps. We have to find,
review, and determine the value. Now, the first step is the hardest one. Different
costs such as architectural cost are associated with it, and there are legal issues;
this three-step data analysis process needs to be done by disrupting the ongoing
workflow of an organization. All of this makes the executives reluctant in finding
dark data. Now, to do all this with minimum disruption, we need to search across
every file and folder, then categorize it attribute wise, provide administrative access
through all the storage platforms, and lastly report according to the usage and
purpose so that the value can be determined. One of the most important things
is to ensure smooth integration of data from all over the platform. Dark data have
unfulfilled value that can provide actionable insights which can lead to profits [7].
The four key steps in information governance are identifying the sources of
data, analyzing key categories and issues, managing accordingly—like delet-
ing unnecessary data, analyzing data with potential value, and securing sensitive
data—and lastly monitoring the whole process with relevant metrices and evalu-
ating the whole process. After completing these four steps, we have to utilize the
feedback to improve and evolve [26].
1. First, we need to shed light on the dark data by understanding its nature and pri-
oritizing it. How much do we have, in which formats, where is it, how old it is, and
what does it contain? We need technological tools to index and normalize it. After
that, with automated tools and targeted searches, we have to classify everything
into three categories: redundant data, risky data, and data with business value. By
keeping an inventory, it will be easy to prioritize and set objectives for the next
phase. It is not possible to address all the pain points at once. First, we need to
address projects with higher returns on investment and then process the rest.
2. The first phase will allow the organization to make informed decisions. We
need to analyze and review different data sources that warrant deeper inspec-
tion with powerful searching and advanced analytical tools. After that, they
can generate a list of targeted sources to act on.
3. After the first two steps, we should know what to do with different sources
of dark data. This is where analysis is turned into actions. Risky and sensitive
data must be reviewed by HR, legal, risk, or compliance departments, alerting
respective personnel. Organizations should dispose or delete all the redundant,
obsolete, or trivial data with no legal or business value. All other data should
contain some sort of business value that can be used by connecting it through
an enterprise search tool, moving or copying it to modern archive, quarantin-
ing some with a suitable security profile, or by migrating it into a manage-
ment software based on the requirement. In simple words, this step is all about
channeling the data smoothly throughout the organization according to the
specific business needs of a particular department.
4. Next, we need to implement sustainable policies after the generation of a com-
plete index of unstructured data throughout the previous stages. With regular
updates and iterations of its proven processes, an organization can be “ever-
green” and can maintain a “living index” of real-time user-generated commu-
nications. It will be able to search instantly for eDiscovery and investigations.
All of this will allow an organization to reduce data volume, which will decrease
the time and cost of initiatives such as audits, eDiscovery, or investigations even by
50%. It will also minimize the risk from dark data and maintain compliance and
allow us to adhere to policies. Lastly, but most importantly, it will help us to extract
the values from dark, unstructured data that can bring unimaginable benefits to
an organization. The overall aim of information governance is to take control over
all the data that goes around in an organization and develop a self-sustaining pro-
cess for the future to reduce the amount of dark data as much as possible because
volume of data is only going to increase exponentially in the coming days [29].
Now, it is not possible to overhaul the whole system and be data driven overnight,
rather it is a continuous process. Small steps toward the big goal can lead to data-
driven insights for now and game changing opportunities for the future. The first
step is to ask the appropriate questions rather than rummaging through everything
that one has in his/her disposal. Now, we need to identify the potential sources of
dark data that can answer the questions, for example, if a marketing team wants
to increase sales in a certain region, then its analytics team can look for relevant
data on that target region only. The data can include video footage of customer
traffic or product placement pictures. Next, we need to look outside for available
demographic or local data outside the organization. For example, a doctor can
give advice to his/her asthma patients by reviewing the weather data and assist the
patient through a flare-up during the pollen season. Employers can find out the
relationship between commute time to workplace and job satisfaction by using
geospatial tools, employee turnover, and traffic patterns. A company also needs to
have expert data scientists in the team to handle all these steps for turning dark
data into operational data. But the insights generated by the data scientist as a
printout of advanced Bayesian statistics will not be comprehendible by most of
the business heads, so we need to represent these results in an easier form. Most
people inside an organization need to understand the complex insights before they
can turn insight into action, and visualization tools can help in this aspect in the
form of an infographic or a dashboard. After developing new strategies, it is wise
to extend it throughout the value chain, from suppliers to customers, for holistic
upgradation [13].
being produced so that it isn’t overlooked or forgotten later. All the data will pass
through a system and then will be divided according to needs and deleted after a
particular time period when it no longer has any potential. Sometimes, we do not
know whether we will need some information or not. In that case, it is wise to ask
the right questions and understand the nature and value of the data. If not done
properly, we can need up wasting time on useless information. So, asking the right
question by identifying business needs is really important. After that, categorizing
the data according to its purpose is crucial. There are technologies that can turn
unstructured data into analyzable data, and after that, the existing big data tools
can do the rest. But before that, we have to feed the system the right data and ask
the right questions to make brilliant use of it. We will get rid of dark data from our
organization only when we build a self-sustainable system to stop data from going
dark in the first place. Then it will only be big data, not dark data.
11.9 Conclusion
In this chapter, we discussed how dark data has become a pressing issue due to its
sheer volume and nature. Whether they are aware of it or not, every organization
has a huge amount of unused, unexplored, and unstructured information at its
disposal. Proper knowledge about one’s business and about how to use this data
in improving business is really necessary. Ignorance can be costly in this case, in
terms of both risks and returns. Opportunity in data analytics is endless, and there
are also different ways and systems to manage dark data. Hadoop and DeepDive
are the potential solutions to this dark data problem. Information governance also
warns us about managing the total information chain properly to ensure that data
does not go dark and is not left unchecked. Dark data analytics is still an untapped
opportunity for us, but if we can manage all our data sources properly, then we can
mitigate the risks and use this wonderful opportunity.
References
1. K. Hill. How Target Figured Out a Teen Girl Was Pregnant Before Her Father Did.
16 February 2012. Available: www.forbes.com/sites/kashmirhill/2012/02/16/how-target-
figured-out-a-teen-girl-was-pregnant-before-her-father-did/#20d6cae36668. [Accessed 16
December 2017].
2. J. Koetsier. Data Deluge: What People Do on the Internet, Every Minute of Every Day.
25 July 2017. Available: www.inc.com/john-koetsier/every-minute-on-the-internet-
2017-new-numbers-to-b.html. [Accessed 13 December 2017].
3. R. Jacobson. 2.5 Quintillion Bytes of Data Created Every Day. How Does CPG & Retail
Manage It?—IBM Consumer Products Industry Blog. 24 April 2013. Available: www.
ibm.com/blogs/insights-on-business/consumer-products/2-5-quintillion-bytes-of-data-
created-every-day-how-does-cpg-retail-manage-it/. [Accessed 13 December 2017].
292 ◾ Data Analytics
20. C. Zhang, J. Shin, C. Ré, M. Cafarella and F. Niu. Extracting Databases from Dark Data
with DeepDive. In Proceedings of the 2016 International Conference on Management of
Data—SIGMOD’16, pp. 847–859, 2016.
21. DeepDive. DeepDive. 2017. Available: http://deepdive.stanford.edu/. [Accessed 19
December 2017].
22. C. Ré. DeepDive: A Dark Data-System. 28 Decemebr 2016. Available: www.youtube.
com/watch?v=NjdT34L0-uU. [Accessed 19 December 2017].
23. DeepDive. Distant Supervision. 2017. Available: http://deepdive.stanford.edu/distant_
supervision. [Accessed 19 December 2017].
24. DeepDive. DeepDive. 2017. Available: http://deepdive.stanford.edu/example-spouse.
[Accessed 19 December 2017].
25. F. A. Pulzello. Six Steps to ‘Dark Data’ Information Governance. September 2014.
Available: http://searchcompliance.techtarget.com/tip/Six-steps-to-dark-data-information-
governance. [Accessed 19 December 2017].
26. S. Ryan. Illuminating Dark Data. 2013. Available: https://conferences.heanet.ie/2013/
files/65/Lightning%20Talk%20-%20Shane%20Ryan%20-%20Illuminating%20
Dark%20Data.pdf. [Accessed 20 December 2017].
27. Aifoundry. Dark Data Transformation—AI Foundry. 1 April 2014. Available: http://
aifoundry.com/agile-solutions/dark-data-transformation/. [Accessed 16 December 2017].
28. Datumize. Data Fracking. Improve your Access to Dark Data. 2017. Available: https://
datumize.com/wp-content/uploads/2017/08/Datumize-DataFracking.pdf. [Accessed 19
December 2017].
29. NUIX. INFORMATION GOVERNANCE: Building Business Value from Dark Data.
2015. Available: www.techworld.com.au/whitepaper/372690/information-governance-
building-business-value-from-dark-data/?type=other&arg=0&location=featured_list.
[Accessed 19 December 2017].
30. D. Schubmehl. Unlocking the Hidden Value of Information—IDC Community. 15 July
2014. Available: https://idc-community.com/groups/it_agenda/bigdataanalytics/unlocking_
the_hidden_value_of_information [Accessed 06 August 2018].
31. T. Srivastava. What is Hadoop? | Big Data Hadoop Technology. 21 May 2014. Available:
www.analyticsvidhya.com/blog/2014/05/hadoop-simplified/. [Accessed 15 December 2017].
DATA ANALYTICS III
APPLICATIONS
Chapter 12
Adam A. Alli
Islamic University in Uganda
Contents
12.1 Introduction..............................................................................................298
12.1.1 W hat Is Big Data?.........................................................................298
12.1.2 Chapter Roadmap........................................................................ 300
12.2 Big Data Technologies...............................................................................301
12.2.1 B ig Data Architecture Framework.................................................302
12.2.2 Big Data Learning Experience Cycle............................................. 304
12.2.3 Benefits of Big Data.......................................................................309
12.2.3.1 Enabling Personalized Learning......................................309
12.2.3.2 Proper Decision-Making................................................. 310
12.2.3.3 Measure Return on Investment....................................... 311
297
298 ◾ Data Analytics
12.1 Introduction
Many big data algorithms, applications, and techniques constitute the buzzword
“big data,” which find its way beyond computer science into the essential tools
for business, government, and very recently in education. The word big data was
coined in 2001 by researchers to describe voluminous and complex datasets that
cannot be processed by traditional means. Since then, organizations and, later on,
academic institutions have created avenues to build tools that assist in defining (i)
goals around their big data, (ii) milestones in the maturity and use of big data appli-
cations, (iii) capacity of use, and (iv) areas that require the use of big data. Given
that big data analytics is growing rapidly, we are not at liberty to understand what
big data is all about and whether it is useful, but the main concern is to understand
how we can yield its use in different areas of society that prove to be useful in
the development of society. Considering this developmental issue, the Technical
Vocational Education and Training (TVET) sectors are very much interested in big
data applications to get benefits from it.
TVET, which is gaining acceptance in many countries as a part of tertiary edu-
cation, is dedicated to converting skilled manpower for the economic development
of any country. The TVET sector constantly needs to be evaluated and transformed
so that it can follow the fast pace of changing trends in different sectors of the mar-
ket, which in turn can create a variety of demands in the workplace [1,2]. A recent
survey shows that 80% of the data created in the world is unstructured and this
assertion does not isolate TVET institutions. One challenge associated with this
massive generated data is how to structure them and make use of them as an asset
in boosting educational process in the TVET industry. The second challenge is
how we can store it. Though a number of technologies have been developed to sup-
port the analytics and storage function of big data, which include but not limited to
Apache Hadoop, Map Reduce, Pig, Hive, HBase, Sqoop, Spark, Oozie, etc., they
are still expensive for small institutions, more specifically in the TVET institutions.
enable easy storage of information and accumulation of data repositories with time.
When this information is of varying sizes, very extensive, and of highly complex
magnitude, it becomes big data [3,4]. According to Michalik et al. [5], big data
is defined by its size; it comprises a large, complex, and independent collection
of datasets, each with the potential to interact. In addition, an important aspect
of big data is the fact that it cannot be handled with standard data management
techniques due to inconsistency and unpredictability of the possible combinations.
In other words, big data can be defined as data that is growing too large and fast,
different, complex, important, and valuable for providing interesting insights and
patterns. Big data characteristics are summarized as volume (size), variety (sources,
formats, and types), and velocity (speed and frequency), and add complexity to the
data, which is, in fact, another attribute in concern [6]. Big data is context-specific
having different data formats, such as structured, semistructured, and unstruc-
tured data that ranges from sector to sector. The common challenge is that all these
sectors must be able to make use of the data by processing and enabling data-driven
improvements through high-level big data analytics [7,8].
In order to improve the current practices, TVET institutions will increasingly
need to capture and analyze data from diverse sources so as to respond to diverse
data flows, such as data coming from learning management systems (LMSs), smart
devices (phones, wearable devices etc.), call center voice and access point messages,
image and video feeds picked up from online classes, emails, transaction data and
social networks, and other learning platforms/sources where students and teachers
are continually interacting and engaging in learning and teaching activities [9].
Besides, from a technical point of view, the key challenge in the education industry
is to adapt and learn from big data. In this case, two things need to be considered:
data coming from different sources and vendors and the current platforms that were
not designed for this data. From a practical point of view, staff and institutions
need to learn the new data management and analysis tools and new skills that are
required to visualize and analyze data from the analytics platforms.
With a number of big data technologies becoming available for institutions,
TVET sectors can now leverage data to (i) understand how long it took for a stu-
dent to access content after starting the course, (ii) visualize the time of day they
interacted with the data or with each other, (iii) find out how long they spent look-
ing at specific content, (iv) seek whether they collaborated with each other or logged
on to the same forums, (v) find out whether they asked questions and what type of
questions were asked, and (vi) understand which format did they engage in during
their course. Using the information obtained from big data platforms, users (teach-
ers and administrators) can process data quickly and establish insights into the
interesting patterns about their students. These sorts of pattern interactions produce
huge data that have significance in improving the quality of technical education. As
more data gets analyzed, the decision makers in TVET sectors particularly need to
find out ways to interpret the data in a meaningful way so that they can predict the
future direction in a learner’s learning behavior.
300 ◾ Data Analytics
Graduates from TVET institutions are deployed straight in the field of work.
These graduates are a source of data for TVET institutions about how and what
should be included in the curriculum to match the current market demands. Using
big data analytics, it is easy to collect data from the market and other TVET insti-
tutions, and thereafter analyze them in a meaningful way that has a direct impact
on the future labor market prediction and market demands.
In many cases, the TVET curriculum of developing countries follows
teacher-centered teaching and uses a conventional teaching method, i.e., the
same teaching content, the same teaching methods, and the same time for deliv-
ery are used [10]. Therefore, it places less emphasis on students’ understand-
ing of taught subjects. Previous literature reports that such curriculum often
leaves gaps in the development of a learner. Using big data analytics on how
students interact during learning can change the way learning and development
is achieved by students and this process can predict the course of action between
teachers and students. This, in turn, insinuates in finding ways to provide tai-
lored instructions to individual students who have difficulties in learning. In the
past, many students have failed to reach anywhere near their potential because
of the inability to correctly find the deficiencies in learning potential. Big data
presents tools necessary for teachers and students to better understand each
other in the learning environment, which in turn gives students better opportu-
nities for success in learning.
In brief, many technologies are making way in our society, among them
Internet of Things (IoT), social media, virtual reality, and 3D printing, which
will be useful to TVET sectors. These technologies will use and create avenues
to produce a myriad of data that contain huge information, attributes, and fea-
tures that can be used in TVET sectors when proper analytics and strategy are
applied.
12.1.2 Chapter Roadmap
In this chapter, we present the prospects and applications of big data in the
TVET sector. Big data technologies present a myriad of opportunities and prac-
tices in teaching, learning, and administration of TVET institutions. Using big
data in management and teaching in the TVET sector does not only provide
the ability to make informed decisions but also enable the prediction of learn-
ers’ behavior, thus leading to customized learning among others. The rest of the
chapter has been o rganized as follows: Section 12.1 provides an overview of big
data. Section 12.2 presents big data technologies that are commonly available
in the industry to activate big data functionalities such as data input, cleaning,
performing analytics through modeling, the framework, the learning cycle, pos-
sibilities, and benefits of big data technologies in TVET. Section 12.3 presents
tools and analytical p latforms for educational purposes. Section 12.4 provides
recommendation and conclusion.
Big Data: Prospects and Applications in the TVET Sector ◾ 301
(Continued)
302 ◾ Data Analytics
and to track their progress. There are many other benefits that may not reflect in
this section. Therefore, Section 12.2.1 provides literature that enables the reader to
understand and measure the value that is added to TVET by big data technology.
◾◾ Input data
◾◾ Big data management
◾◾ Big data analytics and tools
◾◾ Big data infrastructure (BDI)
◾◾ Big data security and privacy
The first stage describes the input data that is going to be processed on the big
data platform. Different forms of data are inserted at this stage. It can be structured
Big Data: Prospects and Applications in the TVET Sector ◾ 303
Clean
data
o Computers
o Cloud Results of o Application
o Networks analytics o Methods and tools
o Operations o Target use
o Support
o etc
or unstructured with varying data formats like text, audio, video, etc. In the sec-
ond stage, the focus goes on managing the raw data that is processed in the first
stage. It utilizes the big data life cycle management system to process the data in the
flow. The third stage contains big data tools and configurations needed to process the
large datasets into useful information. Some tools are commonly accessible, such as
Hortonworks (https://hortonworks.com) or Cloudera (www.cloudera.com), to run the
analytics algorithms on the large input files. The fourth stage performs multiple tasks,
for instance, BDI contains different forms of storing devices, infrastructure, services
delivery, and operational support for the processed data. The fifth stage is responsible
for the security and privacy of results of the analytics process. It is necessary to secure
the results analyzed by the BDI. So, security mechanisms such as Kerberos security or
other encryptions and crypto-algorithms are used to secure big data systems.
Technology is moving at a fast pace creating a big gap or a big disruption.
Technical education also faces the same challenge in training technicians using these
technologies at places of work. This gap creates a constant change in the way TVET is
delivered. The main goal of TVET is to prepare the skilled manpower for the future
world of work. The current and future work force would face a number of dynam-
ics that make TVET institutions fail in finding out what exact skills are required
by its graduates in the future for a particular profession. Especially, TVET stake-
holders of many developing countries may not provide straight answers to questions
such as: What future skills will be required in a particular discipline? What will be the
304 ◾ Data Analytics
Summarizes insights
Predictions’ signals
Training trends
Very results patterns
Test classifications
Optimize comparison
correlations etc
reasons, students’ learning styles and preferences in TVET sectors are changing,
and therefore, students’ learning experiences are connected with diverse learn-
ing theories that were not seen before. The main learning theories are briefly
described in Table 12.2 to link them with the learning experience of the existing
students.
These theories provide a systematic process and applications of big data tech-
nologies to facilitate the acquisition of skills and knowledge in various learning
environments in the TVET sectors.
(Continued)
306 ◾ Data Analytics
(Continued)
Big Data: Prospects and Applications in the TVET Sector ◾ 307
11. Hortonworks The Hortonworks is used for big data analysis in the
ultimate cost-effective and open-source architecture
for all types of data. This tool also provides Hadoop
services such as HDFS, Ambari Cluster, MapReduce,
Oozie, and Spark.
Web: https://hortonworks.com
12. FlockDB It is used for graph databases. This tool is useful for
graphical data designing and processing in TVET sectors.
Web: https://github.com/twitter-archive/flockdb
The steps (Figure 12.2) involved in designing learning experiences are described
as follows:
required not only in learning but also in educational management and col-
laboration [16]. A key challenge that teachers face is adapting their teaching
techniques to the needs of different students, each of whom learns differently
and at his/her own pace. Big data tools can be used for adaptive learning,
i.e., it can be used for helping teachers adjust their pace of instructions to an
individual student’s interests, abilities, and prior knowledge.
ii. Relating experiences to learning theories: Teaching in face-to-face, blended,
and online modes should be designed on the basis of learning theories for mean-
ingful outcomes. The above stated theories (Table 12.1) have to be linked with
both teaching and learning processes while considering big data applications.
Therefore, a systematic process and the application of big data technologies
facilitate the acquisition of skills and knowledge in various learning environ-
ments using different teaching and learning techniques and experiences.
iii. Design teaching and learning experience: Different nations have different
learning styles, curriculums, and qualification framework. When designing
online learning management systems to facilitate international learning, we
have to consider or put in mind that different people interact differently on
the system user interface [17–19]. In this case, big data developers have to
customize data extraction and algorithms in relation to the environment and
learners’ interaction experiences.
iv. Data collection: Data collection is the systematic approach to gathering and
measuring information from a variety of sources to achieve accurate results.
Data collection enables an individual or an organization to answer relevant
questions, evaluate outcomes, and make predictions about future probabil-
ities and trends in TVET sectors. Data currently exist in different forms,
both structured and unstructured. In fact, data is increasingly available to the
extent that it no longer has to be collected, rather it just has to be connected
using the available attributes, data behavior, and outcomes [20]. For example,
to measure students’ motivation toward a technical subject, individual stu-
dent’s attitudes (attribute) will be required. The researchers and educators
need to connect it with the behavior and thereafter generate the outcomes.
Note: Data collected for any one at any point in time will limit the analysis
that can be made. Data collected over time in which trends can be identified will
enhance the abilities of instructors, administrators, and curriculum planners to
make the right decision for strengthening current educational programs [21].
v. Analysis of the data: In the establishment of big data analytics, it is impor-
tant to keep in mind that analyzing TVET data tends to be sophisticated and
requires the establishment of standards [21]. TVET is associated with indus-
trial tool engagement skills (hands-on) and theoretical knowledge. It is worth
mentioning that different techniques are involved in analyzing data, such
as predictive analysis, network analysis, and association and cluster analy-
sis, which can be utilized in finding educational questions, decision-making,
forecasting, and improving teaching practices [22].
Big Data: Prospects and Applications in the TVET Sector ◾ 309
vi. Model development: The data modeling is composed of the following main
aspects:
a. Conceptual modeling: get involved in the sector
b. Logical modeling: define attribute
c. Information modeling: identify the attribute required to describe a thing
or entity
d. Physical modeling: combine and flatten the data
e. Deployment modeling
The structural institutional data can be known, such as core business concepts
(I know where my business comes from, both theoretically and practically),
relationship (can be many to many), institutional drivers (agents), and attribute
(student ID, demography), and the like. On the other hand, semistructured
or unstructured data means some attributes are unknown. Therefore, there
is need to take the known key and separate it with the context that describes
it. This is observed in the ensemble data modeling [23,24]. In this case, if
the attribute is important to someone, then he/she keeps it. Developers dur-
ing designing models use different methods, like try and error (trying many
possibilities or variables available using linear and non-linear relationships).
This is done through model transformation and generalization. For example,
different models will work on different clusters of students such as urban tech-
nical students and village technical students. They both have different user
interactivities and accessibilities in this case.
vii. Redesigning the learning experiences: In this phase, big data technolo-
gists have to validate the model, assess the model, and find the relationship
[25,26]. For example, this is done by collecting research data such as students’
attitudes and survey data and then correlating it with models of students’
behaviors to learn the dominant characters of students that can predict their
behavior. When redesigning the learning experience, we have to consider
the sequence available, the association rules, and causal data to achieve the
desired learning outcomes.
sorts of adaptive and personalized learning processes, which are computer soft-
ware generated. The main goal of the education software available in adaptive
and personalized learning processes is to accomplish adaptability. Most technical
schools in developed nations apply some form of this in reading classes or stan-
dardized testing to place students in the appropriate reading or remedial read-
ing classes as needed. These processes are used to identify the specific needs or
support services so that students in technical education have a better chance at
graduating and retaining knowledge. Personalized learning also helps facilitators
to know which students need teachers’ assistance the most. In many cases, stu-
dents do not willingly talk about their difficulties or are not responsive in class.
TVET institutions will able to turn this potential for early intervention through
personalized learning experiences [28]. The importance of personalized learning
is further emphasized on:
It cannot deny that personalized learning is setting the trend for
future development of education. Big data in education makes the
process of learning become possible to meet every student’s need
through a personalized recommendation system which could recom-
mend adapted learning resources and proper path, as well as give
academic advice [15].
12.2.3.2 Proper Decision-Making
At the moment, big data and data-driven decisions in TVET are primarily used in
developed nations for predictive modeling, identification of the optimal patterns
of learning, tracking individual information, and research [30]. Earlier, decision
makers and educators were trying to assess teaching and learning by answering
questions like the following:
These questions were very useful but not sufficient. In the TVET sector, we also
have to ask the following:
All the above aspects lead toward the success of any course in TVET. Big data
in the TVET sector shows potential in providing learning support and facilitates
learning across learning management systems. When trying to assess the success
of any course, it is important to look at learners’ misconceptions and the goals of
learning. Decision makers intervene to help learning take place. Intervention from
teachers, researchers, parents, and counselors provides feedback about: Which stu-
dents finish their classes? And who learns? This provides improvements to the TVET
curriculum. Literature shows that different individuals learn in different ways. Let
us consider three different learning styles as an example. Auditory learners are more
likely to learn effectively by hearing, e.g., reading loudly to enhance recitation;
visual learners learn best by seeing, e.g., watching graphics and demonstrations,
and reading; and kinesthetic learners obtain knowledge best through “hands-on”
experiences, e.g., attending activities and writing notes.
It has been presented that students’ collaborative work on the same assignment
that follows different interaction patterns affects the performance and assessment
of the group work in collaborative learning. Continuous focus, self-reflection, live
collaboration, and fairly even distribution of workload are naturally more likely to
lead to more refined and coherent assignments and, consequently, lead to better
learning outcomes. But again there are some regulations that are widely accepted
in improving learning effectiveness, e.g., regular reviewing, more practice, and
active collaboration and discussion, and so forth. In relation to TVET apprentices,
learning behavior, knowledge composition, and knowledge acquisition are differ-
ent from case to case. It is thus vital to identify the optimal learning patterns for
individual students or groups to maximize learning performance and to identify
key impact factors that would allow the generalization and formulation of the best
practice guidelines in this space [17].
periods of time. Analysis of this data shows or predicts future patterns of learning
and teaching and related problems. On the meso level, performances can be mea-
sured by collecting data about the improvement in graduate rates. Another example
would be the measurement of students’ overall performances in specific teachers’
classes over a period of time.
Additionally, since large institutions should prepare reports for investors and
stakeholders of TVET, it is required to focus on a broad macro view of institu-
tional performance. Therefore, TVET institutions need to address key areas such
as teaching and learning that have to be reported to the stakeholders. Benefit is seen
in the development of models that can assess an institution’s current state of per-
formance while providing accurate predictions for future performance outcomes.
In relation to the existing model, descriptive models enable TVET facilitators
to detect student trends in admission, rate of improvement, and graduation rates.
A predictive model, the second type, is that whose purpose is to discover previ-
ously undetected relationships in data and to predict future outcomes. For example,
schools might be able to ascertain at-risk students during their academic programs
at primary and secondary levels. These prescriptive models provide the foundation
for action plans constructed upon knowledge acquired from both types of models;
thereby, allowing institutions to accurately appraise their current situations and
make data-driven choices about other possibilities. All this facilitates a great return
on investment in TVET sectors when properly utilized.
12.2.3.4 Performance Prediction
We predict the future in order to improve the present context, otherwise why
should we predict in the first place? Performance prediction can be done by sending
reports to TVET facilitators/tutors or parents and by gaining an actual picture of
class engagement and motivation in the teaching and learning situation. This also
provides information about answering questions such as: (i) How are the students
prepared for future learning? (ii) Which student will succeed now or later as the class
continues? These types of information could be achieved through assessment of top-
ics, knowing which students ask for help often in the LMS, and which ways and
how frequently each student gets involved in learning via LMS.
Synthesizing and applying knowledge to solve unseen problems enhance the
quality of student learning. Literature shows that TVET sectors are mostly less pre-
ferred [8]. By using big data in an appropriate way, TVET institutions can improve
student performance. For instance, it controls directly and indirectly the dropout
rates and therefore increases the graduation rates. These could be possible by estab-
lishing student relationships within the TVET learning environment and ensuring
interactions with peers and instructors via LMS. This will further provide a way of
gathering information about the performance of particular populations within the
student groups. In brief, identifying learners’ performance in a particular domain
of TVET sector will assist the overall improvement of this domain [5].
Big Data: Prospects and Applications in the TVET Sector ◾ 313
1. Reinforcement of learning.
2. Improve and control undesirable behavior following principles of classical or
operand conditioning and laws of learning.
3. Eliminate all incorrect responses and fix only the right responses.
Interactions
Learner experience (social
media, LMS etc.)
Web experience
Logs and feeds
particular students are doing provides an informal cheat code to the educators
(administrators) such as the following: (i) What do students find interesting on the
Internet? (ii) What games do they like playing? (iii) What do they watch and what social
network do they associate with in the teaching and learning contexts?
These sorts of acts and information assist the facilitators to know more about
the level of student behavior and their level of performance, hence providing a
platform where the student feels recognized and wanted.
the learners’ behavior, and consequently providing directions for the instructors
to tailor the way they teach. Considering learning analytics in TVET sector will
enable the administrators to visualize the actual pattern of TVET graduates, and
eventually be able to properly document future success factors of TVET. Lastly, we
provide useful information for readers, educators, curriculum developers, policy
makers, and other TVET stakeholders on implementing big data technologies to
ensure the quality TVET education.
Besides, big data technology facilitates the creation of new knowledge and
enhancement of competences in TVET sectors.
References
1. E. Clift, V. Liptak, and D. Rosen. Educational ideas and the future of higher educa-
tion: The quest for a new business model. In Von der Kutsche zur Cloud–globale Bildung
sucht neue Wege, edited by Manfred Schönebeck and Ada Pellert. Springer, Wiesbaden,
Germany, 2016, pp. 7–37.
2. D. E. C. B. Ms.Deepa. A. Big data analytics for accreditation in the higher education
sector. International Journal of Computer Science and Information Technologies(IJCSIT),
vol. 8, pp. 357–360, 2017.
3. D. C. Gibson and D. Ifenthaler. Preparing the next generation of education researchers
for big data in higher education. In Big Data and Learning Analytics in Higher Education,
edited by Ben Kei Daniel. Springer, Switzerland, 2017, pp. 29–42.
4. L. M.-s. Ni, J. Xiao, and H. Tan. The golden age for popularizing big data “平民化”——
大数据技术发展的新目标. Science China Information Sciences, vol. 59, p. 108101, 2016.
5. P. Michalik, J. Stofa, and I. Zolotova. Concept definition for Big Data architecture in
the education system. In 2014 IEEE 12th International Symposium on Applied Machine
Intelligence and Informatics (SAMI), 2014, pp. 331–334.
6. R. Kitchin and G. McArdle. What makes big data, big data? Exploring the ontological
characteristics of 26 datasets. Big Data & Society, vol. 3, p. 2053951716631130, 2016.
7. Y. Wang, L. Kung, W. Y. C. Wang, and C. G. Cegielski. An integrated big data analytics-
enabled transformation model: Application to health care. Information & Management,
vol. 55, pp. 64–79, 2018.
8. M. Nambobi. Big Data: Information is the new Oil mine in Africa. 2017. Available: https://
nambobi.com/big_data.
9. G. Bello-Orgaz, J. J. Jung, and D. Camacho. Social big data: Recent achievements and
new challenges. Information Fusion, vol. 28, pp. 45–59, 2016.
10. A. Alli and M. Nambobi. Adopting ICT and Green Skills in Teaching and Learning: BTVET
in Uganda. CreateSpace Independent Publishing Platform, North Charleston, SC, 2017.
11. S. Mangal. Essentials of Educational Technology. PHI Learning Pvt. Ltd., New Delhi,
2009.
12. Y. Song. The research on individual adaptive English studying of network education platform
based big data technology. In 2015 14th International Symposium on Distributed Computing
and Applications for Business Engineering and Science (DCABES), 2015, pp. 260–263.
13. W. Vollenbroek, K. Jagersberg, S. Vries, and E. Constantinides. Learning Education: An
‘Educational Big Data’approach for monitoring, steering and assessment of the process of
continuous improvement of education. 2014.
316 ◾ Data Analytics
Sports Analytics:
Visualizing
Basketball Records
in Graphical Form
Muye Jiang
University of Ottawa
Contents
13.1 Introduction.............................................................................................. 318
13.1.1 Chapter Roadmap.........................................................................321
13.2 Background and Related Work.................................................................321
13.3 Design Details..........................................................................................324
13.3.1 Converting Text Format Data........................................................324
13.3.2 D rawing Charts and Graphs..........................................................325
13.3.2.1 Line Chart.......................................................................325
13.3.2.2 Track Lines......................................................................326
13.3.2.3 Dynamic Elements..........................................................327
13.3.3 T echnologies..................................................................................327
13.3.4 S ystem Usage.................................................................................328
13.3.4.1 General Usage..................................................................328
13.3.4.2 Specific Usage..................................................................329
317
318 ◾ Data Analytics
13.1 Introduction
Statistics has long been an inherent component in professional sports. Sports
visualization is growing, yet most games tend to leave the statistics in numerical
format and use very few in-game visual representations. Basketball is a game that
still leaves the statistics in numerical and text format. An enormous amount of data
is generated in one basketball game. Live basketball play-by-play records in the
National Basketball Association (NBA), for example (Figure 13.1), are organized in a
tabular format listing events that occur during a game [1]. To search for a particular
event, a user must scroll up and down a long list of records and calculate the amount
of time at 12-min intervals per quarter. This makes it very difficult for users such
as coaches, players, audiences, sports reporters, and analysts to search for particular
events. We believe that live basketball records could be represented in a better format.
To our knowledge, no designer has attempted to redesign the format for show-
ing live basketball records. There is one visualization for showing the performance
of basketball players for an entire season [2] (Figure 13.2) and one design scheme
for showing three different game levels (season, game, and session) [3], as well as
some research regarding the potential applications of information visualization for
representing sports statistics [4,5], but no designer has yet focused on a design for
Sports Analytics ◾ 319
query. Our results show that the graph we designed and tested is clear, usable, and
visually appealing. We found evidence that it is both easier and faster to find critical
information about the game using the graph as opposed to the traditional text chart.
We have empirical evidence to support the assertion that the graph yields better user
experience than a traditional text chart, particularly for experts than for novices.
This graph displays basketball records clearly, and during a game, users will likely
find it quick and useful for finding information they wish to know.
The main goal of this project was to design a visualization that will better aid
visual search and query and test its effectiveness. Due to the large amount of scroll-
ing and computation1 required for the use of a text chart, it was hypothesized that
the new graph chart (Figure 13.3) will be more usable than a conventional text
chart (Figure 13.1) in many ways. For example, no scrolling is required while using
the graph chart and, thus, users will be able to detect an event almost instanta-
neously. Also, no computation of time is required because a continuous track line
is conveniently located at the top of the graph chart. In addition, to no need for
scrolling and computation, visualization elements such as colors, symbols, patterns,
and hover queries have been applied to the graph to aid visual search and query.
13.1.1 Chapter Roadmap
The subsequent sections of this chapter are organized as follows: Section 13.2
provides the background information about basketball and related literature.
Section 13.3 outlines the design approach along with the design principles, tech-
nology, and user tasks. Section 13.4 provides details regarding the user study, while
Section 13.5 explains the method that was employed for comparing a traditional
text chart vs. graph. Section 13.6 presents a report of the results of the study, and
Section 13.7 offers interpretation of the results, limitations, and suggestions for
future research.
in the data because the use of good visualization principles, such as colors and hover
queries, could increase visual salience and support rapid search and query [7].
We focused our sports data visualization research on basketball games because
basketball has a fixed game time for each quarter and its scores changes dramati-
cally, usually three or four times each minute, all the time. These basketball char-
acteristics make it a perfect choice for visualizing game data. However, very little
research has explored the area of sports using data visualization, and topics of these
research projects focus on other specific games, such as baseball [8]. In contrast,
game data visualization is very popular in Esports [9]. High-level professionalized
Esports, such as Starcraft II [10] (Figure 13.4) or Dota 2 [11] (Figure 13.5), are also
popular in game data visualization and are very detailed and well rounded. The
visualization (Figure 13.3) introduced in this chapter was inspired by one of these
professionalized Esports called League of Legends [12] (Figure 13.6).
Figure 13.6 Team gold advantage visualization in League of Legends [12] match
history record.
what is happening on the basketball court. Sometimes, they will use the teams’ or
players’ previous game data and compare it to the current game data to predict the
final result of the current game and evaluate players’ performance. The last type of
user is audiences. They simply focus on the performance of their favorite team(s)
or player(s) because they experience enjoyment when their favorite team wins or if
their favorite player scores a lot of points.
Little research has explored the merits of using visualizations for representing
sport statistics. Legg et al. [15] designed an algorithm for the layout of metaphoric
glyphs at different spatial scales called “MatchPad” and examined its effective-
ness for supporting rapid information seeking in a Rugby context. The researchers
found that MatchPad was useful for coaches and analysts in making important
match decisions in a short time period and was successful at helping examine
events in detail while maintaining a clear overview of the game situation. They
concluded that glyph-based visualization offers an efficient and effective way for
conveying large amounts of event records. In a different study, Dietrich et al. [8]
developed a new visual analytics tool called “Baseball4D” for analyzing, visual-
izing, and tracking the performance of baseball players. Although the design
was not empirically tested, the authors believe that their design is valuable for
teams and coaches to study gameplay and player performance. Losada et al. [16]
created an interactive visual prototype called “BKViz” for analyzing basketball
games which focuses on the processing and presentation of play-by-play data. To
validate the tool, the researchers consulted with experts and received much posi-
tive feedback with respect to the usefulness of their design, such as the capability
of comparing game situations where multiple players are or are not on the court
together, filtering of events to reveal when various plays occur, and linking of
multiple interactive visualizations to show the potential of predicting a variety
of conclusions. Although informative, there are a lot of basketball performance
visualizations, but they are all created after the games rather than during the
games (live).
13.3 Design Details
For basketball game data visualization, it should complete two major missions. The
first mission is converting the text format data into unified data structure and the
second mission is using the data structure to create charts and graphs. We separated
our program into two parts according to the two missions.
Figure 13.7 The event surrounded by a yellow box shows the time that an event
occurred at 1:10 (each quarter for lasts 12 min and 1:10 = 12:00–10:50), the type
of event is “scoring,” and the team that created this event is Boston Celtics. The
event also contains some detailed information, such as the name of the player and
the player’s scoring method [1].
13.3.2.1 Line Chart
The most important component of our visualization is the line chart. Time appears
on the x-axis because the two basketball teams (home and guest) that are playing this
game use the same timescale, and overtimes are rare in basketball games. The most
interesting feature of this chart lies on the y-axis. Usually, chart designers will plot
each team’s current scores on the y-axis. But, each team’s scores mean very little to
all the users (players, coaches, analysts, reporters, and audiences); instead, the score
difference between the two teams (or we can also call it score advantage) is the criti-
cal information that users need. As a result, we decided to show score advantage on
326 ◾ Data Analytics
Figure 13.8 Pre-attentive features applied to the design of BasketBallVis. (a) The
blue circles represent the home team has the score advantage, while the green
circles represent the visiting team has the score advantage. (b) Green circles that
are farther from the horizon (pink line) means that the visiting team has a larger
score advantage at that time.
the y-axis, and the points above the x-axis represent that the guest team has the score
advantage, whereas the point below the x-axis represents that the home team has
score advantage. To make the points in the line graph more meaningful, we added
more features to these points to contain more information. Since players and coaches
need quick and accurate access to this information, we selected two pre-attentive
features suggested by Ware [7] (Figure 13.8). The first feature is color to show which
team has score advantage (Figure 13.8a). The second feature is distance to the hori-
zon to show how large the score advantage is (Figure 13.8b).
Like the points, the score line is also a major part of a line chart, and we utilized
the line’s orientation to represent the game’s tendency. If the line is getting closer
to the top of the chart during a specific period, it means the guest team is gaining
an upper hand during this period (Figure 13.3: from time 30:00 to time 48:00).
Otherwise, it means the home team is gaining an upper hand (Figure 13.3: from
time 10:00 to time 20:00). In addition to color and size, we also found a way to
utilize the chart’s background. Each basketball game consists of four quarters, and
each quarter is exactly 12 minutes long. Thus, different background colors were
used to represent each quarter. Low-saturated light colors were used here because
this could make crucial small subsets, such as points, and the line more visible and
distinct from the background [7].
13.3.2.2 Track Lines
Track lines are used here to record the critical events of each team. There are four
kinds of critical events: injuries, substitutions, fouls, and timeouts. Although visual
appeal is not important for players and coaches, it is so for the audiences. A well-
designed symbol set can be appealing (meets audience’ requirements) and readily
found [7] (meets players’ and coaches’ requirements); thus, symbols were integrated
into the graph (Figure 13.9a). According to Ware [7], symbols should be distinct
Sports Analytics ◾ 327
Figure 13.9 (a) The four blue track lines to events belonging to the Boston team
and the four green track lines to events belonging to the Houston team. (b) A
red cross is commonly associated with first-aid and medical services, so it was
used to denote player injury. (c) Two rotating arrows forming a circle is usually a
symbol for swapping and recycling. We used it to represent players’ substitutions.
(d) Basketball referees will blow a whistle when players commit an offense or a
foul, so we used whistles to represent players’ fouls. The colors of the whistles
represent the number of fouls the players have at that time. (e) A clock is used for
showing time and we used it to represent timeout.
from the background and other symbols. Four symbols were created for denoting
each critical event by following this principle (Figure 13.9b–e).
13.3.2.3 Dynamic Elements
The events mentioned before usually contain more detailed information than
the event’s type and the time when the event happened. But displaying all these
detailed information at the same time would make our graph too complex and
unfocused. Thus, we chose to introduce interactions and dynamics into our graph.
Since coaches and players need to access information fast, “Hover Queries” [7],
which supports one of the quickest epistemic actions [17] in our graph, was applied.
When a user places the mouse cursor over a symbol on the track line or over a point
on the line graph, a pop-up message appears to reveal player and score information
(Figure 13.10a and b).
13.3.3 Technologies
Our program is based on HTML, CSS, and JavaScript programming language that
are the core technologies for web development. Recall that our program consists of
two parts. For the first part, since the text game data provided by the NBA’s official
328 ◾ Data Analytics
Figure 13.10 (a) This small window contains more information regarding a par-
ticular substitution event. “Lee” and “Zeller” are the two players’ names. (b) This
small window contains more information about a scoring event. “Ariza” is the
player’s name and “Driving Layup shot” is the scoring method.
website [1] utilizes a unified format, we used JavaScript language’s original string
methods to convert the data into our designed structure. For the second part, we
used SVG (Scalable Vector Graphics) to draw our visualization because SVG speci-
fication is an open standard developed by W3C (World Wide Web Consortium)
and it has many good characteristics. For example, SVG is based on vector images
and is very flexible. The visualization’s dynamic parts were implemented by the
D3.js JavaScript library [18]. D3’s core idea is based on data-driven documents
[18] so it is convenient to manipulate the document’s elements. For some of our
program’s functions, such as filtering a specific player’s game record on the visual-
ization, we filtered the game’s data, and D3 helped us remove the unbound game
record (other players’ record) from the visualization.
13.3.4 System Usage
The visualization is for supporting visual search and query during a live basketball
game. There are two kinds of usage for the visualization. The first kind of usage
is general usage that is designed for all kinds of users. The second usage is specific
usage that is designed for specific kinds of users.
13.3.4.1 General Usage
General usage is focused on the score advantage. When users want to know the
score advantage at a specific time, they will start with the x-axis (time-axis). Users
will find the specific location on the x-axis first and use it as a reference to find
the closest scoring event circle in the line chart. Then, users will try to identify
the circle’s color and the distance between the circles with respect to the horizon.
Sports Analytics ◾ 329
Finally, users will get the result which team has the advantage by the color and the
result is how much the score advantage is by the distance.
13.3.4.2 Specific Usage
Players, since they usually focus on their own performance, will use the visualiza-
tion’s filtering function to filter the events which are related to them. They can also
know the number of fouls they have accumulated by looking at the color of the
whistles located on the foul’s track line.
Coaches will use the slope of the line charts to evaluate his decisions and strat-
egies. For example, if the coaches called a player substitution at 5 minutes of the
game, he should look at the line’s slope during the 5 and 10 minutes of the game. If
the slope is inclined toward his team direction, this means the substitution is a good
decision. On the other hand, if the slope is inclined toward the opposite team’s
direction, this means the substitution is a bad decision. Aside from the evaluation
usage, the visualization can also help the coaches make decisions. For example, if
he finds his most valuable player’s record contains an orange or red whistle at the
first or second quarter of the game, he should call a substitution to prevent the most
valuable player from getting six fouls and get fouled out too early in the game.
Basketball analysts and reporters can compare the shape of the current game’s
line charts of the two teams with the shape of the previous game’s line charts to
predict which team will win the current game. Furthermore, they can also use the
line chart’s slope to explain which team is having the current advantage.
13.4 User Study
The dependent variables of this study are search efficiency, clarity, usability, and visual
appeal, while the independent variables are two types of charts: text and graph. The
research question of this study is: Can basketball records and events be better repre-
sented using a graph and symbols rather than text format? Following are six hypotheses:
H1: It will be easier to search for information on a graph than on a text chart.
H2: It will be easier to identify information on a graph than on a text chart.
H3: It will take less time to find information on a graph than on a text chart.
H4: A graph will be more usable than a text chart.
H5: A graph will be more visually appealing than a text chart.
H6: A graph will yield better user experience than a text chart.
via convenience and opportunistic sampling. Posters were posted around the uni-
versity campus and email notices were sent to colleagues in our class asking for their
participation. Twelve participants (ten males, two females; six experts, six novices)
ranging in age from 23 to 29 years old (M = 25.08 years, SD = 2.07 years) volun-
teered to participate in the study. Six user representatives in each user category were
sufficient for the purposes of this evaluation because of eventual data saturation
[19]. All participants were graduate students studying in the fields of engineering,
psychology, and computer science.
13.4.2 Measures
A combination of subjective and objective measures was gathered for analysis.
Subjective measures included search efficiency, clarity, usability, learnability, and
visual appeal. These were integrated into two post-record questionnaires: one for
evaluation of the conventional text chart (Figure 13.1) and the other for evaluation
of the new graph (Figure 13.3). The post-record questionnaire also included one
question that evaluated overall experience and one final question to provide sugges-
tions for improving the chart. Time was recorded as an objective measure of search
efficiency.
I could tell right away when team A had the largest score
advantage.
It was unclear which team was taking the lead at a specific time
(e.g., 20 minutes).
Sports Analytics ◾ 331
13.4.2.3 Visual Appeal
Lindgaard et al. [22] developed a 9-point scale (1 = very unappealing to 9 = very
appealing) for measuring characteristics (e.g., interesting—boring; good design—bad
design) of visual appeal, (R 2 = 0.94, p < 0.001). These characteristics were integrated
as statements (e.g., the chart was interesting) into the post-record questionnaires
consisting of three positively worded items and two negatively worded items. To
maintain consistency with other statements in the questionnaire, participants were
asked to indicate their responses on a 5-point Likert scale with responses ranging
from 1 = strongly disagree to 5 = strongly agree.
13.4.4 Procedures
Ethical approval for the current research was obtained from the university’s
Research Ethics Committee. The experiment was approximately 30 minutes long
332 ◾ Data Analytics
and was divided into two parts: an exploration session followed by an experimen-
tal session. In a closed laboratory setting, six participants explored and evaluated
the graph chart followed by the text chart, while this order was reversed for the
reminder of participants. For participants who were unfamiliar with basketball,
the experimenter briefly explained the mechanics of the game and the scoring sys-
tem. After initializing the informed consent, participants scrolled through and
explored a chart for about 1 minute. Next, participants were asked and timed on a
series of eight questions relating to the chart they explored and completed a post-
record questionnaire. The same procedure was repeated with a different chart type.
Finally, participants completed a demographics questionnaire, were debriefed, and
thanked for their participation.
13.5 Results
Data were analyzed using IBM SPSS Version 22.0 and figures were generated using
Microsoft Excel 2010. Prior to analysis, data were screened for missing values, out-
liers, and out of range values. Subjective results (Table 13.2) were drawn from the
analysis of the post-record questionnaires, whereas objective results were obtained
from the analysis of time. Nonparametric tests were selected to perform the analysis
because the use of an ordinal scale in the post-record questionnaires and the shape
of the distributions were skewed. A p-value threshold of 0.05 was set for all statisti-
cal tests4.
Search It was hard to find the information I 1.83 1.19 1.00 0.96 2.21
was looking for.
Clarity The chart was hard to understand. 2.00 1.13 2.00 0.76 2.13
It was easy to identify the trend of 4.50 0.52 4.50 0.00 0.92
the game.
I could tell right away when team A 4.67 0.65 5.00 −1.61 4.15
had the largest score advantage.
Search It was hard to find the information I was 2.33 3.42 1.75 2.50 5.94 0.11
looking for.
Clarity The chart was hard to understand. 2.24 3.25 1.83 2.50 4.21 0.24
It was easy to identify the trend of the 2.00 1.67 3.67 2.67 9.00 0.03*
game.
I could tell right away when team A had 2.25 1.25 3.42 3.08 12.89 0.01**
the largest score advantage.
It was unclear which team was taking the 2.25 2.92 2.50 2.33 1.08 0.79
lead at a specific time (e.g., 20 minutes).
5.00
4.00
Mean Ranks
3.00
2.50 2.50
Expert
2.00
Novice
1.00
0.00 0.00
0.00
Text Graph
Chart Type
Figure 13.11 Mean rank results of a Wilcoxon’s signed-rank test for the statement:
“It was easy to identify the trend of the game.”
5.00
4.00
3.00
Mean Ranks
3.00
2.50
Expert
2.00
Novice
1.00
0.00 0.00
0.00
Text Graph
Chart Type
Figure 13.12 Mean rank results of a Wilcoxon’s signed-rank test for the statement:
“I could tell right away that team A had the largest score advantage.”
25.00
22.35
20.00 17.89
Mean Ranks
15.00
11.25
Expert
10.00
Novice
4.50
5.00
0.00
Text Graph
Chart Type
Figure 13.13 Mean rank results of a Wilcoxon’s signed-rank test for search
efficiency. Lower mean rank scores indicate less search time.
336 ◾ Data Analytics
experts (Z = –4.71, p = 0.00) and novices (Z = –5.53, p = 0.00) suggest that the
graph is more efficient for performing visual search than the text chart.
13.5.3 Visual Appeal
Before conducting the analyses, the scores were averaged for the positively worded
items, where higher scores on the measure corresponded to higher levels of visual
appeal. Since the original 9-point scale was replaced with a 5-point scale, a reliabil-
ity analysis was conducted to test the internal consistency of the data. A Cronbach’s
alpha of 0.895 indicates high internal consistency. To compare which chart type
(text or graph) was more visually appealing with respect to the user type (expert or
novice), two separate Wilcoxon’s signed-rank tests were conducted on an averaged
visual appeal score. Marginally significant results (Figure 13.15) shown for both
experts (Z = –1.89, p = 0.06) and novices (Z = –1.84, p = 0.07) suggest that the
graph chart is more visually appealing than the text chart.
5.00
4.00 3.80
3.50
Mean Ranks
3.00
2.00 Expert
2.00
Novice
1.00
1.00
0.00
Text Graph
Chart Type
Figure 13.14 Mean rank results of a Wilcoxon’s signed-rank test for level of
usability and learnability.
Sports Analytics ◾ 337
5.00
Figure 13.15 Mean rank results of a Wilcoxon’s signed-rank test for visual appeal.
5.00
3.00
Expert
2.00
Novice
1.00
0.00
0.00
Text Graph
Chart Type
Figure 13.16 Mean rank results of a Wilcoxon’s signed-rank test for overall
experience.
13.5.4 Overall Experience
To compare which chart type (graph or text) elicited better overall experience
with respect to the user type (expert or novice), two separate Wilcoxon’s signed-
rank tests were conducted on the question: “On a scale of 1 (bad) to 10 (good),
please rate your overall experience with this chart” in the post-record question-
naire. Statistically significant results (Figure 13.16) shown for experts (Z = –2.23,
p = 0.03) suggest that the graph yields better experience compared to the text chart,
and nonsignificant results shown for novices (Z = –1.59, p = 0.11) suggest that both
the graph and the text chart yield the same experience.
338 ◾ Data Analytics
13.5.5 Subjective Report
Results in this section are comments that were gathered from the open-ended
question in the post-record questionnaire. They are all related to improving the
graph. One participant noted that additional information about the game, such as
steal, block, and missed shot, is needed. Another participant recommended that
the colors on the graph should match the official colors of the team. Furthermore,
several participants reported a slight confusion with the score advantage due to the
negative signs below the midline on the y-axis. Participants also suggested adding
more interactive elements on the graph, such as providing a link to a video clip of an
event or providing a legend for indicating the meaning of colored whistles.
13.6 Discussion
The purpose of this study was to explore the possibility of representing live
basketball records in graphical format. A graphical representation was designed
leveraging a variety of visualization principles, such as colors, patterns, symbols,
and hover queries [7]. The design was tested against a traditional text chart on
measures of clarity, search efficiency, usability, visual appeal, and user experience
with respect to two user types—experts and novices. Because of the application of
good design principles for supporting visual search and query, it was hypothesized
that the graph would be more efficient, usable, visually appealing, and yield better
user experience than the text chart. In general, results showed that the graph was
rated better than the text chart on all measures. Recall the six hypotheses of this
study that are as follows:
H1: It will be easier to search for information on the graph than on a text chart.
H2: It will be easier to identify information on a graph than on a text chart.
H3: It will take less time to find information on a graph than on a text chart.
H4: A graph will be more usable than a text chart.
H5: A graph will be more visually appealing than a text chart.
H6: A graph will yield better user experience than a text chart.
The finding that participants rated their ability to detect information such as the
trend of the game and score advantage to be faster while using the graph than a
text chart supports H1, H2, and H3. Additionally, the finding that irrespective of
the user type, expert or novice, it took less time to search and report details on the
graph than on the text chart also supports H1, H2, and H3. These findings suggest
that the graph is more efficient for supporting visual search and query than a tra-
ditional text chart. Likewise, H4 and H5 were supported in that both experts and
novices rated the graph to be more usable and visually appealing compared to the
text chart. Finally, H6 was partially supported in that findings differed depending
Sports Analytics ◾ 339
on user type. Experts rated their overall experience with the graph to be better than
with the text chart, whereas novices rated their overall experience of both chart
types to be about the same.
The findings of this study have implications for information visualization and
design. One major finding was that the graph was more efficient for performing
visual search and queries than a traditional text chart. This suggests that the graph
was well designed. Good use of visualization principles (colors, symbols, patterns, and
hover queries) enabled participants to easily find what they were asked to search for.
Thus, the traditional text chart could potentially be replaced by a graph. Basketball
is a quick-paced game and coaches need to make important decisions within a lim-
ited timeframe. Making injudicious decisions could result in lost of a game. A more
usable, search efficient, visually appealing, and clearer way of displaying records,
could reduce the chances of making unwise decisions and could increase the chances
of winning a game. With respect to the finding that novices did not have a preference
for graph or text chart with respect to overall experience, it is likely that novices have
never been exposed to or used a graph or text for looking up basketball records, and,
thus, chart type did not make a difference. Perhaps there needs to be some understat-
ing and enthusiasm for the game before one can appreciate the merits of the graph.
From a human–computer interaction perspective, the results of this study
provide some interesting insights for designers and users. The main goal of interac-
tion design is to help users quickly become experts by making the user-interface
as easy to use as possible. Although marginally significant, the graph was rated
more usable than the text chart. Particularly for experts, the graph elicited better
use experience than the text chart. It is possible that until now no designer has
attempted a way to improve the display of live basketball records. Results of this
study provide statistical evidence that users can quickly become familiar with this
new graph and, possibly, use it more effectively than a traditional text chart. As
for the subjective reports, although most comments were associated with minor
aesthetic issues, recall that one novice participant suggested adding more interac-
tive elements on the graph, such as providing a link to a video clip of an event. This
finding is interesting because simply by looking at and interacting with basketball
records evoked an interest in the game. From a design perspective, the interactive
elements in the graph could have encouraged further exploration.
user types. Another limitation is that the experiment was conducted in a closed
laboratory setting and, thus, results may not be valid in the field. To increase the
ecological validity, a future study could examine the usefulness of the graph during
a live basketball game. One more limitation of this study is the reliance on subjec-
tive measures with the exception of time as an indication of search efficiency. To
further test the efficiency and effectiveness of the graph for supporting visual search
and query, additional objective measures such as the level of accuracy or the num-
ber of errors made during search could be gathered. Moreover, time was recorded
rather loosely: the experimenter simply asked the question, and when a response
was heard, a timestamp was recorded. Recording time this way allows for much
variability with respect to individual differences. A more precise method for record-
ing time could be managed by a computer. For example, present each question on
a full screen and provide a “start” button to allow the user initiate time. Once the
user has an answer, provide a “stop” button to end time.
More generally, the graph can be adapted for other sports and application areas.
Baseball, for example, is another sport that generates a great deal of data during a
single game and, perhaps, the same graph designed in this research project could be
modified to show live baseball records. As for other application contexts, recall that
the graph was inspired by a graphical visualization from a multiplayer online game:
League of Legends [12]. What was learned in this research could in turn inform the
design of video or online game visualizations, such as the progress and performance
of teams and players.
13.7 Conclusions
We sought to improve the format of how live basketball records are represented.
After comparing both the conventional text chart and a newly designed graph, we
found that a graph is clearer, more usable, and visually appealing than a conven-
tional text chart. Not only does a graph better represent live records, but it also
invites interaction and further exploration. Due to the limited amount of research
in the area of sports visualizations, we would like to encourage other designers and
researchers to explore other possibilities of showing sports statistics using different
visualization methods in other sports contexts. In closing, this is a preliminary
attempt at improving the format of live basketball records and it provided some
new insights into the advantages of using visual methods for showing in-game
sports statistics, as well as the application of information visualization principles
for designing graphical charts.
Acknowledgments
We thank all the participants who kindly volunteered to participate in this research.
Sports Analytics ◾ 341
Funding
The authors received no financial support for the research and/or authorship of this
chapter.
Notes
1 Time begins at 12 minutes—countdown from 12:00 to 00:00 per quarter (4 quarters
in total, total time = 48 minutes). To know when an event occurs, a user must add and
subtract time. For example, to determine which team has the largest score advantage
at 30 minutes into the game, a user must add 12 + 12 + 12 to reach the third quarter
and subtract 6 from 36.
2 Novice = with very limited to no knowledge of basketball.
3 Expert = basketball amateurs; play and watch basketball frequently and uses text
chart while watching NBA game.
4 *p < 0.05; **p < 0.01
5 Example formula in Microsoft Excel spreadsheet: (((J5)–1)+(5-(K5))+((L5)–1)+
(5–(M5))+((N5)–1)+(5–(O5))+((P5)–1)+(5–(Q5))+((R5)–1)+(5– (S5)))*2.5
References
1. “NBA Media Ventures, LCC.” Retrieved from www.nba.com/games/20151023/
MEMORL/gameinfo.html?ls=iref:nba:scoreboard#nbaGIPlay.
2. F. Li. “NBAVis.”
3. W. Chen et al. “GameFlow: Narrative Visualization of NBA Basketball Games.”
IEEE Trans. Multimed., vol. 18, no. 11, pp. 2247–2256, 2016.
4. A. Cox and J. Stasko. “SportVis : Discovering Meaning in Sports Statistics through
Information Visualization.” Hum. Factors, pp. 10–11, 2006.
5. R. Therón and L. Casares. “Visual Analysis of Time-motion in Basketball Games.”
Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes
Bioinformatics), vol. 6133 LNCS, pp. 196–207, 2010.
6. K. Card, D. Mackinley, and B. Shneiderman. “Readings in Information Visualization:
Using Vision to Think.” p. 716, January 1999.
7. C. Ware. Information Visualisation: Perception for Design. 2012.
8. C. Dietrich, D. Koop, H. T. Vo, and C. T. Silva. “Baseball4D: A Tool for Baseball
Game Reconstruction & Visualization.” In 2014 IEEE Conf. Vis. Anal. Sci. Technol.
VAST 2014- Proc., pp. 23–32, 2015.
342 ◾ Data Analytics
Analysis of Traffic
Offenses in
Transportation:
Application of Big
Data Analysis
Charitha Subhashi Jayasekara, Malka N. Halgamuge,
Asma Noor, and Ather Saeed
Charles Sturt University
Contents
14.1 Introduction............................................................................................. 344
14.1.1 Chapter Roadmap........................................................................ 346
14.2 Material and Methods.............................................................................. 346
14.2.1 Data Inclusion Criteria................................................................. 346
14.2.2 Data Preprocessing........................................................................347
14.2.3 Data Analysis................................................................................ 348
14.2.3.1 Linear Regression............................................................349
14.2.3.2 Nonlinear (Polynomial) Regression.................................349
14.2.4 Data Analysis Algorithms..............................................................349
14.2.5 Statistical Analysis......................................................................... 351
343
344 ◾ Data Analytics
14.1 Introduction
Due to the rapid advancement in technology, the amount of data and the speed of
data collection are swiftly increasing. Especially in the traffic and transportation
field, multiple techniques are used to collect data on a daily basis. For instance,
surveillance cameras are operating 24 hours and speed cameras are recording
vehicle speeding offenses. Furthermore, offense details and traffic violations can
be automatically recorded or logged manually by the police. Therefore, at the
end of each day, extremely large datasets known as big data are created in differ-
ent formats with structured, semistructured, and unstructured data. However,
with the use of different data analysis techniques, this data can be analyzed to
identify patterns and trends. Also, this analysis is useful to draw attention to any
anomaly that can be noticed in the dataset. The biggest benefit is that processing
a large collection of data provides better accurate results, delivering more precise
outcomes when appropriately analyzed. Those identifications are helpful for vari-
ous purposes such as preventing future issues, making predictions, understand-
ing needs, improving safety, and making laws.
With the purpose of improving the road safety, this chapter has used descrip-
tive statistical analysis methods and regression modeling to analyze a dataset
Analysis of Traffic Offenses in Transportation ◾ 345
14.1.1 Chapter Roadmap
The chapter is organized as follows: Section 14.1 introduces the aim of this chap-
ter. It shows the importance of big data analysis and how it can be used in the traf-
fic and road safety area to get improved decision-making. Section 14.2 explains
the material and methods used. Once the data was collected it was necessary to
understand the data types, which methods were most appropriate, and how the
expected outcomes could be generated. Furthermore, the software tools that were
used are described. Section 14.3 includes the statistical tests and results. It shows
the regression models created and the generated expressions. Section 14.4 is the
discussion section that explains the trends found and decisions made using the
results. Section 14.5 is the conclusion that provides final thoughts on this chapter.
It briefly summarizes the rest of the chapter and also explains how the tests can be
improved in the future.
14.2.2 Data Preprocessing
The raw data was preprocessed before being used for the data analysis. Since there
are 916 types of traffic offenses they were categorized depending on the type of
offenses into 301 groups. Table 14.2 shows some of the offenses added under the
“Exceed speed limit” category.
348 ◾ Data Analytics
14.2.3 Data Analysis
The main aim of this analysis was to understand the seasonal traffic violation pat-
terns that explain the relationship between the offenses and the time of occurrence.
Additionally, the analysis also aimed to measure the frequency of offenses that
occurred during different times of the year.
Analysis of Traffic Offenses in Transportation ◾ 349
The IBM SPSS (version 23) software was chosen for the data analysis since it has
a number of features for statistical analysis, including predictive modeling suitable
for big data analytics.
There are several data analyzing methods and it is important to select the correct
method depending on the dataset and expected outcome. Considering the dataset
used in this chapter, all the data used was categorical and the counts of observations
were of different categories. Therefore, regression analysis was selected as the data
analysis method. The two major types of regression analysis used in this analysis
will be discussed next.
14.2.3.1 Linear Regression
Linear regression can be used to determine the relationship between one dependent
variable (denoted by y) and one or more independent variables (denoted by x). Its
formula generates a straight line. In this research, it was used to show the relation-
ship between weekdays and offense frequency on each weekday. As an example,
y = mx + b represents a linear regression model.
Algorithm 2 shows how the regression model was generated to analyze the weekly
offense frequency.
Algorithm 3 shows how the regression model was generated to analyze the offense
frequency during different time slots.
14.2.5 Statistical Analysis
The hypothesis test is important when selecting a dataset and the test was done for
this analysis using the SPSS tool. For the current research, the selected null hypoth-
esis (H0) and the alternative hypothesis (H1) were:
H0: There are no differences among the months/ day of week/ time and offense
frequency
H1: There is a difference between the months/ day of week/ time and offense
frequency
Table 14.3 shows the p-values of each group used to generate results. All the
p-values are less than 0.05. Therefore, the null hypothesis can be rejected.
14.3 Results
The analysis of the frequency of traffic offenses and the time periods in which they
occurred was the main focus of the study. A descriptive statistical analysis was used
to measure the frequencies and regression models created to analyze the data.
Table 14.3 p-Values among Groups Used for the Test
Type P-Value of Offense Frequency
Month 0.018
Time 0.016
352 ◾ Data Analytics
Figure 14.2 First 15 traffic offenses with the highest frequencies of occurrence.
354,372
104,409
According to Figure 14.3, 77% of offenses were directly related to the time at
which they occurred. This gives a strong indication that time is an important factor
to be considered when discussing traffic offenses.
R
Equation Square F df1 df2 Sig. a b c
14.3.4.3 Obtained Results
This study found that there was a significant difference in offense frequencies in
different months of the year. However, June shows the least number of offenses,
while January and December show the highest. Therefore, from the beginning of
the year, the number of violations decreases, and it increases when the year ends. The
difference between the highest and the lowest number of offenses was around 8,000.
Considering the top 15 traffic offenses, offenses such as alcohol- and bicycle-
related offenses showed the lowest number of occurrences during the middle of the
year, whereas offenses such as failing to keep a distance from the vehicle in front
and driving without effective lights were higher in the same period.
14.3.5.3 Obtained Results
Thursday and Friday showed the highest number of incidences, whereas Monday
was the lowest, although there is no significant fluctuation among the 7-day counts.
However, the graph shows that there is a very clear increase in the number of
offenses during the week (Monday to Friday). In other words, each day number of
offenses increases by almost the same amount. The difference between the Monday
and Friday counts was approximately 15,000.
However, all seven days of the week were considered to analyze the top 15
offenses. According to the graphs, offenses such as alcohol- and drug-related viola-
tions are higher on weekends, but most other offenses do not show a significant
increase. Also, offenses involving handheld devices, failing to wear the seatbelt,
and failing to keep a safe distance from the vehicle in front are outstandingly lower
during weekends.
1 0:00–3.00
2 3:30–6:00
3 6:30–9:00
4 9:30–12:00
5 12:30–15:00
6 15:30–18:00
7 18:30–21:00
8 21:30–23:30
For further analysis, a cubic graph was created through a regression curve
estimation as illustrated in Figure 14.13 to understand the variation better.
Table 14.8 shows the summary and parameter estimates.
The R-squared value is 0.905, which indicated that there was a strong connection
between the time slots and the number of expiations in each time slot. Furthermore,
the p-value of 0.016 showed that the relationship was significant. Using the resulted
parameter, the regression equation shown below was created.
364 ◾ Data Analytics
R
Equation Square F df1 df2 Sig. a B c d
14.3.6.3 Obtained Results
Compared to month and day of the week, time shows a highly significant
fluctuation. The number of offenses increases slowly from the morning, peak-
ing at 17:00, then dropping suddenly by midnight. That is, according to
Figure 14.12, the number of offenses is highest from 15:30 to 18:00 and lowest
between 03:30 and 06:00.
Although most of the offenses occurred during the daytime, bicycle-, alcohol-,
drug-, and smoking-related traffic violations are outstandingly higher during midnight.
Furthermore, the number of offenses of failing to wear a seatbelt, give way offenses,
and failing to keep a safe distance from the vehicle in front shows fluctuations during
the night time. To find the time periods of each offense, a cross-tabulation was used as
shown in Table 14.9.
Table 14.9 shows the maximum number of offenses by type that occurred on
Thursdays and Fridays. Offenses such as traffic light- and speeding-related inci-
dences that occur due to carelessness and impatience are highest from 15:30 to
18.00, whereas alcohol- and drug-related offenses show a variance. Therefore, in
a separate graph, alcohol- and drug-related offenses during each time slot are pre-
sented in Figure 14.15.
According to Figure 14.15, alcohol-related offenses are higher in the evening up
until midnight, whereas smoking- and drug-related offenses begin to rise from noon.
14.4 Discussion
The aim of this research was to identify the frequency of offenses during different
seasons of the year. This will help to identify possible reasons for the variation in
366 ◾ Data Analytics
2000
1500
Offense Count
1000
500
0
0:00-3.00 3:30-6:00 6:30-9:00 9:30-12:00 12:30-15:00 15:30:18:00 18:30-21:00 21:30-23:30
Time slot
Figure 14.15 Alcohol-, smoking-, and drug-related offenses against time slot.
Analysis of Traffic Offenses in Transportation ◾ 369
Table 14.10 Summary of the Proposed Regression Models
Coefficient Coefficient Coefficient Coefficient
Item Figure No. Best-Fit Model (a) (b) (c) (d) R2 Value
more frequently during the same days of the week. In addition, both pedestri-
ans and drivers are exposed to road dangers due to irresponsible alcohol intake,
especially during the festive season. The use of handheld mobile phone offenses
dramatically drops by approximately 70% on the weekends compared to weekdays.
This is because during the business hours and in the busy weekdays people use their
mobile phones more even while driving. This distraction can cause drivers to violate
traffic light laws, exceed speed limits, fail to keep a safe distance from the vehicle
in front, and not wear a seatbelt. Moreover, driving at night in hazardous weather
without effective lights is highest from Thursday to Saturday. This could be caused
by people tending to stay longer at parties during this period rather than the other
days and it leads them to drive at night, which could make the drivers sleepy and
lead to traffic accidents [21]. Next, an analysis of the time periods in which traffic
violations occur will be discussed.
Considering the overall frequencies, the time from 09:30 to 18:00 showed the
highest number of traffic violations. There was an 83% increase from the period of
03:30 to 06:00, with the highest number of recorded offenses occurring between
1530 and 1800. Exceeding the speed limit, traffic light-related offenses, the use of
handheld devices, failure to give way, and failure to keep a safe distance from the
vehicle in front are also higher from 15:30 to 18:00. This is the peak traffic time
when people return home from work violating more laws and increasing road dan-
ger. For instance, getting stuck in a long queue of vehicles at a traffic signal gives
drivers an urge to pass the lights as soon as possible, which can result in traffic-
related offenses, such as exceeding speed limits and failing to give way. The alcohol-
related offenses are highest between 18:30 and 03:00 as people consume alcohol
after regular business hours and this explains the reason for most of these offenses.
However, the number of smoking-, cannabis-, and other drug-related offenses were
highest most of the time, except during 03:30 to 09:00. Unlike alcohol, cannabis
and certain drugs stay in the body for a longer period. Therefore, officers can detect
the drugs even when they were taken hours ago. For instance, on an average, can-
nabinoid level is detectable for up to 9.8 days [22] in the body of a frequent cannabis
user. Furthermore, failure to wear a seat belt is highest from 09:30 to 12:00; as the
regular offices start around 9:00, the drivers might forget to wear seat belts in their
hurry to reach office on time.
With the discussion of traffic offenses during different time periods of a year
it is proved that application of big data analysis is important and can be used to
obtain thoughtful and reasonable decisions. Data science is a broad subject starting
from a simple descriptive analysis to deep learning neural networks. Depending
on the factors such as data types, necessary results, and available infrastructures,
different techniques and methods can be used for the analysis [23–27].
This research will help scholars understand the correlation between times and
incidences that lead to time specific traffic violations so that further actions can be
taken by creating feasible policies to reduce them.
372 ◾ Data Analytics
14.5 Conclusion
This study showed that there was a significant fluctuation in the number of traffic
offenses depending on the time and day of occurrence. Using 521,811 traffic offense
data instances, this study clearly illustrated that the traffic offense count increases
during festival times, weekends, and during the evening. Moreover, further analy-
sis shows variations in the different types of violations. For instance, it distinctly
showed that speeding offenses are higher during the evenings and drugs-related
traffic offenses are higher from noon to midnight. The analysis will be helpful for
police officers, traffic authorities, psychologists, and other researchers interested in
traffic-related offenses since it intends to deliver improved solutions to reduce road
danger and enhance road safety. Also, sustainable transport policies can be created
with active transport strategies to optimize danger reduction. However, based on
this study, further research can be carried out to find out the psychological aspects
that lead to offenses during different times.
In addition to the insights on traffic violations, this research also proved that
big data analysis is very useful for making important decisions. Unlike assump-
tions, analyzing real data is helpful in coming to more practical conclusions and
in predicting future trends. Therefore, the theories that were applied to traffic and
transportation in this chapter can be similarly used to a vast range of domains to
scrutinize real-world problems in detail.
References
1.
Corporate Statistics, Victoria Police. (2017). Vehicle Impoundment Statistics 2007–2016.
2.
Australian Bureau of Statistics. (2016). 9309.0- Motor Vehicle Census, Australia.
3.
Bucchi, A., Sangiorgi, C., & Vignali, V. (2012). Traffic psychology and driver behav-
ior. Procedia—Social and Behavioral Sciences, 53, 972–979. http://dx.doi.org/10.1016/j.
sbspro.2012.09.946.
4.
Precht, L., Keinath, A., & Krems, J. (2017). Effects of driving anger on driver behavior:
Results from naturalistic driving data. Transportation Research Part F: Traffic Psychology
and Behaviour, 45, 75–92. http://dx.doi.org/10.1016/j.trf.2016.10.019.
5.
Papadimitriou, E., Yannis, G., & Golias, J. (2009). A critical assessment of pedestrian
behaviour models. Transportation Research Part F: Traffic Psychology and Behaviour, 12(3),
242–255. http://dx.doi.org/10.1016/j.trf.2008.12.004.
6.
Hu, T., Xie, X., & Li, J. (2013). Negative or positive? The effect of emotion and mood on
risky driving. Transportation Research Part F: Traffic Psychology and Behaviour, 16, 29–40.
http://dx.doi.org/10.1016/j.trf.2012.08.009.
7.
Babson, K., & Feldner, M. (2015). Sleep and Affect (1st ed.).
8.
Hammar, L. M., Emami, A., Engström, G., & Götell, E. (2010). Reactions of persons with
dementia to caregivers singing in morning care situations. The Open Nursing Journal, 4, 35.
9.
Li, X., Er, M., Lim, B., Zhou, J., Gan, O., & Rutkowski, L. (2010). Fuzzy regression mod-
eling for tool performance prediction and degradation detection. International Journal of
Neural Systems, 20(05), 405–419. http://dx.doi.org/10.1142/s0129065710002498.
Analysis of Traffic Offenses in Transportation ◾ 373
10. Ahmadi, H., Pham, N., Ganti, R., Abdelzaher, T., Nath, S., & Han, J. (2010).
Privacy-aware regression modeling of participatory sensing data. Proceedings of the 8th
ACM Conference on Embedded Networked Sensor Systems—Sensys ‘10. http://dx.doi.
org/10.1145/1869983.1869994.
11. Duwe, G., & Freske, P. (2012). Using logistic regression modeling to predict sexual
recidivism. Sexual Abuse: A Journal of Research and Treatment, 24(4), 350–377. http://
dx.doi.org/10.1177/1079063211429470.
12. Wellings, K., Macdowall, W., Catchpole, M., & Goodrich, J. (1999). Seasonal variations
in sexual activity and their implications for sexual health promotion. Journal of the Royal
Society of Medicine, 92(2), 60–64. http://dx.doi.org/10.1177/014107689909200204.
13. Stamps, D., & Carr, M. (2012). Holiday season for a healthy heart. Critical Care Nursing
Clinics of North America, 24(4), 519–525. http://dx.doi.org/10.1016/j.ccell.2012.07.007.
14. Dai, J., Teng, J., Bai, X., Shen, Z., & Xuan, D. (2010). Mobile phone based drunk driving
detection. Proceedings of the 4th International ICST Conference on Pervasive Computing
Technologies for Healthcare. http://dx.doi.org/10.4108/icst.pervasivehealth2010.8901.
15. Pöysti, L., Rajalin, S., & Summala, H. (2005). Factors influencing the use of cellular
(mobile) phone during driving and hazards while using it. Accident Analysis & Prevention,
37(1), 47–51. http://dx.doi.org/10.1016/j.aap.2004.06.003.
16. Wellings, K., Macdowall, W., Catchpole, M., & Goodrich, J. (1999). Seasonal variations
in sexual activity and their implications for sexual health promotion. Journal of the Royal
Society of Medicine, 92(2), 60–64. http://dx.doi.org/10.1177/014107689909200204.
17. Gjerde, H., Normann, P., Christophersen, A., Samuelsen, S., & Mørland, J. (2011). Alcohol,
psychoactive drugs and fatal road traffic accidents in Norway: A case–control study. Accident
Analysis & Prevention, 43(3), 1197–1203. http://dx.doi.org/10.1016/j.aap.2010.12.034.
18. Doherty, E., & Haddou, J. (2016). Green light for dope-smoking picnics. Heraldsun.com.
au. Retrieved August 6, 2017, from www.heraldsun.com.au/news/law-order/police-allow-
dopesmoking-melbourne-picnics-dubbing-them-freedom-of-expression/news-story/7875
28d80a7e92926cdf467c12eadaa5.
19. Liu, S., Wang, M., Zhan, Y., & Shi, J. (2009). Daily work stress and alcohol use: Testing
the cross-level moderation effects of neuroticism and job involvement. Personnel Psychology,
62(3), 575–597. http://dx.doi.org/10.1111/j.1744-6570.2009.01149.x.
20. Henderson, A. (2010). Push to reinstate “Thirsty thursday” grog ban. ABC News.
Retrieved August 6, 2017, from www.abc.net.au/news/2010-10-22/push-to-reinstate-
thirsty-thursday-grog-ban/2308276.
21. Åkerstedt, T., Hallvig, D., Anund, A., Fors, C., Schwarz, J., & Kecklund, G. (2013).
Having to stop driving at night because of dangerous sleepiness—awareness, physiol-
ogy and behaviour. Journal of Sleep Research, 22(4), 380–388. http://dx.doi.org/10.1111/
jsr.12042.
22. Johansson, E., & Halldin, M. (1989). Urinary excretion half-life of 1-Tetrahydrocannabinol-
7-oic acid in heavy marijuana users after smoking. Journal of Analytical Toxicology, 13(4),
218–223. http://dx.doi.org/10.1093/jat/13.4.218.
23. Singh, A., Halgamuge, M. N., & Lakshmiganthan, R. (December 2017). Impact of dif-
ferent data types on classifier performance of random forest, Naïve Bayes, and k-nearest
neighbors algorithms. International Journal of Advanced Computer Science and Applications
(IJACSA), 8(12), 1–10.
24. Halgamuge, M. N. (November 2017). Machine learning for bioelectromagnetics: Prediction
model for weak radiofrequency radiation effect from mobile phone on plants. International
Journal of Advanced Computer Science and Applications (IJACSA), 8(11), 223–235.
374 ◾ Data Analytics
25. Wanigasooriya, C., Halgamuge, M. N., & Mohamad, A. (September 2017). The analyzes
of anticancer drug sensitivity of lung cancer cell lines by using machine learning clustering
techniques. International Journal of Advanced Computer Science and Applications (IJACSA),
8(9).
26. Gupta, A., Mohammad, A., Syed, A., & Halgamuge, M. N. (August 2016). A compara-
tive study of classification algorithms using data mining: Crime and accidents in den-
ver city the USA. International Journal of Advanced Computer Science and Applications
(IJACSA), 7(7), 374–381.
27. Bashyal, C., Halgamuge, M. N., & Mohammad, A. (January 2018). Review on Analysis
of the Application Areas and Algorithms used in Data Wrangling in Big Data. Lecture
Notes on Data Engineering and Communications Technologies Cognitive Computing for
Big Data Systems Over IoT, Frameworks, Tools and Applications, Springer, 14, Chapter
14, ISBN: 978-3-319–70687-0.
Chapter 15
Intrusion Detection
for Big Data
Biozid Bostami
Islamic University of Technology
Mohiuddin Ahmed
Canberra Institute of Technology
Contents
15.1 Big Data and Intrusion Detection System.................................................377
15.1.1 Chapter Roadmap.........................................................................377
15.2 W hat is Big Data?.....................................................................................378
15.3 Security Issues with Big Data....................................................................378
15.4 Intrusion Detection System.......................................................................379
15.5 Classification of Intrusion Detection Systems...........................................379
15.5.1 L ocation-Based Classification........................................................380
15.5.2 E valuation Criteria-Based Classification........................................380
15.6 Collaborative Intrusion Detection and Big Data.......................................381
15.6.1 W hy Is It Necessary?.....................................................................383
15.7 A rchitecture of CIDS................................................................................383
15.7.1 Centralized Architecture................................................................383
15.7.2 Hierarchical Architecture..............................................................384
15.7.3 Distributed Architecture................................................................385
15.8 Building Blocks of CIDS..........................................................................386
15.8.1 Local Monitoring...........................................................................386
15.8.2 Membership Management.............................................................387
15.8.3 Correlation and Aggregation.........................................................387
375
376 ◾ Data Analytics
Big data, a technology which stores the data in distributed cloud infrastructures,
is changing the business logic and application models. The valuable information
that big data contains is driving the decisions businesses are making toward
data-driven processes. However, when it comes to security, it is the concern of
the owners and users. Security and privacy can be increased by securing the
computing environment. In network security scheme, intrusion detection sys-
tems (IDSs) play a vital role. This chapter explores the collaborative intrusion
detection systems that enhance the security of the big data and also handle
the cloud vulnerabilities. Cyberattack sources are heterogeneous. Correlating
cyberattack incidents generated from different sources and time can provide a
concrete view and an increased awareness about the vulnerabilities of cyberat-
tacks. Conventional intrusion detection systems are not suitable for distributed
environments and they cannot detect cooperative attacks. For defending against
the correlated attacks collaborative intrusion systems are introduced that ana-
lyze the suspicious behaviors collected from other IDSs to increase the efficiency
of intrusion detection.
Coordinated attacks are also a common scenario in big data. Such attacks
like large-scale stealthy scans, worm outbreaks, and distributed denial-of-service
(DDoS) attacks occur simultaneously over multiple networks and this makes it
nearly impossible for a standalone intrusion detection system (IDS) to detect such
attacks as it can only monitor certain portions of the network. In this chapter, we
have presented a brief summary of the research done on coordinated attacks with
collaborative intrusion detection systems (CIDSs).
Finally, this chapter will be helpful for both the security and data analytics
professionals, academics, and graduate level students.
Intrusion Detection for Big Data ◾ 377
15.1.1 Chapter Roadmap
In this chapter, we give an overview of the intrusion detection systems for the big
data architecture. In this chapter, the key aspects of big data analytics are pre-
sented in detail. Section 15.2 starts with the definition of big data. Section 15.3
reflects the challenges associated with big data. Section 15.4 provides introduction
to intrusion detection systems. Section 15.5 discusses about the classification of
intrusion systems. In Section 15.6, we introduce collaborative intrusion systems
for big data. Section 15.7 focuses on the architecture of the collaborative intrusion
systems. Section 15.8 presents the building blocks, Section 15.9 describes different
types of attacks of CIDS, and Section 15.10 gives a brief idea about the importance
of CIDS in a cloud environment. In Section 15.11, we discuss about some of the
well-known coordinated attacks and in Section 15.12, we present the research done
378 ◾ Data Analytics
by different researchers and explore their limitations and strong points. Finally, in
Section 15.13, we conclude the chapter by discussing about the opportunities.
15.5.1 Location-Based Classification
Depending on the location of deployment in a network, IDSs can be classified into
two categories. They are host-based intrusion detection system and network-based
intrusion detection system [14].
Host-based intrusion detection system is responsible for monitoring individual
devices or hosts. It monitors all the incoming and outgoing traffic to Internet and
vice versa of the attached host as well as monitors the internal system calls. But it
is not aware of the overall network situation. That is the limitation of host-based
intrusion detection systems.
Network-based IDSs (NIDS) situated at the edge points of the network are
responsible for monitoring the traffic at those points. NIDs use the packet sniffers
and analyze the data to correlate the intrusion. Whenever suspicious packet or traf-
fic is detected the alarm is triggered. But, NIDs are unaware of the internal network
activities.
15.6.1 Why Is It Necessary?
Traditional standalone IDSs are likely to be invaded by the cooperative attacks,
so their performance is poor in the distributed environment cloud computing or
big data. By correlating suspicious evidence between different IDSs, collaborative
intrusion detection systems (CIDSs) improve the efficiency of intrusion detection.
Unlike conventional standalone IDSs, IDSs located at a local network’s entry points
share the information among themselves. In practice, the IDS is organized within
the CIDS over a large network following a decentralized or hierarchical pattern.
Communication takes place between each IDS directly or via a central coordinator
through the applied mode. By aggregating the network information received from
other IDSs in the CIDS, a complete attack diagram of the network is generated. The
central coordinator, which analyzes the aggregated information, generates a com-
plete attack diagram of the network. Collaborative intrusion and malware detection
offers an anonymous communication scheme. Participating IDSs can share infor-
mation with the interested partners based on the intrusion ontology. CIDS also pro-
vides a decentralized group formation algorithm and a robust collaborative model.
In the following section, we will be describing the architecture of collaborative
intrusion detection systems and their classification.
15.7 Architecture of CIDS
Early IDSs were designed to monitor a single host or port over a network by carry-
ing out local analysis. There was no scope for communication. Clearly, those design
approaches are not suitable for distributed networks. They were not designed to estab-
lish connection between events occurring in different places simultaneously. Thus,
collaborative IDS (CIDS) was proposed for the protection of large networks and large
IT ecosystems. CIDS is a collection of several monitors acting as sensors and collect-
ing data. They usually contain several analysis units that are responsible for intru-
sion detection from the collected data coming from sensors. Complex task done by
the network administrator is significantly reduced by the CIDS because it provides
high accuracy in intrusion detection [8]. Based on the specification, monitoring and
the analysis unit can be co-located. In this chapter, we will assume that a monitor is
built with both the monitoring unit and the analysis unit. By ensuring cooperation
between the monitors, CIDS becomes more scalable than traditional IDS. Following
their communication architecture, CIDS can be classified into three different catego-
ries: centralized, decentralized, and distributed CIDS.
15.7.1 Centralized Architecture
In this architecture, several IDSs are attached to their respective hosts for monitoring
the traffic and behavior (as shown in Figure 15.1). Monitored data is then shared with
384 ◾ Data Analytics
the central unit for analysis. The data sent to the central monitor can be either local
traffic or they can be alerts produced by intrusion detection. Hence, alert correlation
algorithm is applied on the alerts sent by other monitors or the detection algorithm
is applied by the central unit.
But this architecture has some limitations. As a standalone IDS, with the
increasing size of the system, centralized CIDS cannot be scaled easily. This archi-
tecture works for small-scale enterprise networks. It has two major drawbacks.
If the central unit fails then complete alert correlation will fail as well since it is
dependent on the central unit. It is commonly known as the single point of failure
(SPoF). The second drawback is that all the edge monitors send the monitored data
to the central unit. If the data flow exceeds the capacity of the central unit then
some data might get loss and the performance might decrease.
15.7.2 Hierarchical Architecture
For overcoming the drawbacks of a centralized design, the hierarchical architecture
was proposed. In this architectural design, the CIDSs are placed in a hierarchical
model that is based on multiple small communication groups. This group formation
is done by some of these features: (1) administrative controlling, (2) geographical
position, (3) similarity of the platform, and (4) intrusion type. Every communica-
tion group acts as a subset of the overall hierarchy. Every group has a parent analysis
node that is responsible for correlating all the alerts and traffic data belonging to
the respective group. After data from edge node is collected by the parent node of
a group then it forwards the processed data to the higher order node in the hier-
archy for analysis. Usually, IDS situated at the base level of the hierarchy acts as
the detecting unit. IDSs situated at the higher levels act as detection units as well
as correlation units. Higher-level nodes do all the analysis on the data sent by the
lower level nodes. After correlation and analysis of the alerts and the traffic data,
the result is sent to the higher level and this continues until it reaches the maximum
level from where the decision is distributed to the lower levels.
The hierarchical architecture (as in Figure 15.2) works better than the central-
ized design, but it still has some drawbacks. The whole system can be hampered
Intrusion Detection for Big Data ◾ 385
if the higher-level nodes stop their functionality. In that case, the whole hierarchy
will fail. Moreover, at the top level, it is difficult to scale, keeping pace with the
increased network. Again, due to input data abstraction at the top-level nodes,
there is limited detection coverage.
15.7.3 Distributed Architecture
In the distributed architecture, the task of analysis is distributed among all the
monitors in the network, making sure that each monitor has its own analysis unit
and is not directly dependent on other nodes for analysis. The distributed architec-
ture is similar to the peer-to-peer (P2P) architecture in which the data correlation,
data aggregation, and data analysis are done in a distributed manner with the par-
ticipation of all the monitors.
The fully distributed architecture focuses on the scalability property of CIDS.
But there are still many challenges and open issues related to accuracy of detection,
scale, and load balance.
15.8.1 Local Monitoring
Local monitoring can be of two types: host level monitoring and network level
monitoring. In case of host level monitoring, only the associated hosts behaviors are
considered. On the other hand, network level monitoring aims to cover the whole
network from attacks by deploying CIDS at selected points on the edge that are closer
to the ingress or egress routers of the network. A combination of both host level and
network level deploy can further increase the detection accuracy.
Monitoring can be divided into active monitoring and passive monitoring.
Passive monitoring comprises host level monitoring or networking level moni-
toring. Honeypots are used for passive monitoring for detecting the presence of
Intrusion Detection for Big Data ◾ 387
promising attack targets. Although honeypot’s false positive is zero, the false nega-
tive rate is quite high.
In local monitoring, the analysis unit uses a detection mechanism from the data
collected by the sensors. The detection mechanism can be signature based or can
be anomaly based. Even combination is also possible, which was proposed as Bro
IDS [48].
Local monitoring act as a vital part in designing an optimal CIDS because the
data collected by the local monitors is utilized by the other blocks.
15.8.2 Membership Management
The task of membership management unit is to establish overall connectivity
between all the neighboring monitors. Membership management can be of static
type or it can be dynamic by nature. In its static nature, whenever a new com-
ponent is added the network administrator needs to alter the overlay of the net-
work. Connections are predetermined in static membership. On the other hand,
in dynamic membership component a new component is added with the help of
a central server that provides a global view or follows the membership protocol.
This protocol is run locally at each CIDS. Since the membership controls the
connectivity among the neighboring CIDSs the architecture of the network is
also dependent on this unit. Based on the connectivity, the network architecture
can be centralized, hierarchical, or completely distributed as described in the
previous section.
There are four different approaches for alert data correlation. Here we will give
a brief description of them.
15.8.4 Data Dissemination
CIDSs that focus on correlating alerts and aggregating information need to be effi-
ciently distributed. Data dissemination is highly dependent upon the architectural
design of the membership unit. In centralized design, the data flow is always the
same. There the alert information always flows to the central analysis unit. On the
other hand, the decentralized design monitors and maintains a hierarchy between
them and sends the alert information in a bottom-up fashion. Distributed design
provides high level of freedom for data sharing on a flat overlay. In a distributed
approach, information can be shared in both directions for monitors having interest
in similar information. The information among the groups can be shared via gos-
siping approach [44] or by flooding selectively/partially using random walks [45] or
Intrusion Detection for Big Data ◾ 389
15.8.5 Global Monitoring
The capabilities of a particular CIDS are based on global monitoring. Global moni-
toring aims to detect the distributed attacks. Based on the detection scope, global
monitoring can be of two types: (1) generic type and (2) specific type. The generic
type aims to detect multiple types of attacks. Specific types are used for specialized
attacks. In a nutshell, global monitoring aims to detect malicious behavior over the
CIDSs that are connected with each other representing the collaborative detection
abilities.
15.9.1 External Attacks
The origin of the external attack is outside the network that is being monitored by
the CIDSs. The external attacks aim to explore the presence of the CIDS monitor-
ing points to plan future attacks to bypass the CIDS or to exploit the vulnerable
services of CIDS to degrade the quality of service. External attacks can be classified
into two types: (1) disclosure attack and (2) evasion attack.
15.9.1.1 Disclosure Attack
The disclosure attack aims at gathering the location information of the CIDSs in
the target network. A disclosure attacker expects a feedback from the CIDS. These
feedbacks can be of different types like the attack results publicly available in some
website over the Internet. Yoichi et al. [50] in their work proposed a passive approach
that takes the feedback from public websites. In their work, they gain information
about the CIDS location as well as other information by analyzing the feedback
graph that is updated periodically. An active approach was proposed by Shmatikov
et al. where disclosure attack gathers information from public sites by the method
named probe-response attacks [51]. In disclosure attacks, the attacker uses a known
attack signature to identify the attack feedbacks. Evasion attacks are followed by
the disclosure attacks. We will be discussing the evasion attacks in the next section.
390 ◾ Data Analytics
15.9.2 Evasion Attack
The evasion attack aims to avoid the IDS so that no alert is initiated. There are
many methods to achieve this goal. From the point of intrusion detection tech-
niques, the evasion attack is divided into two types: (1) signature-based evasion and
(2) anomaly-based evasion.
algorithm only analyzes a particular area of the traffic. So, the attacker first
creates a normal traffic profile and then blends the attack with it. Fogla
et al. [37] in his work showed methods using PAYL, a byte frequency-
based anomaly IDS, as a case study and demonstrates that polymorphic
blending attacks are indeed feasible.
15.9.2.1 Internal Attack
The origin of the internal attack is located inside the monitored network by an
insider. An insider can belong to a compromised CIDS monitoring unit or it can
be a compromised host in the network. A compromised host within the protected
network sets up a secret channel with an attacker outside the network. All the traf-
fic between the compromised host and the attacker is hidden from the monitors
[53,54]. The channel used by the compromised host is not usually used for com-
munication, for example, unutilized bits of the packet header or the timing infor-
mation. Internal attacks can become more harmful if a component of the CIDS is
compromised. A compromised/malicious CIDS monitor can expose the location of
other monitoring agents and it can compromise other monitoring agents by taking
advantage of the vulnerability of the CIDS protocol. It can lead to a total takeover
of the whole system at worst.
A generic countermeasure against malicious insiders in the distributed systems
is the adoption of reputation systems [55,56], such as EigenTrust [57]. In this case,
each of the monitors creates a level of trust depending on detection behavior or
other defined properties. If the trust level goes below a certain threshold, the moni-
tor is considered less trustworthy and preventive steps can be taken, for example,
blacklisting of the malicious monitor. Several proposals have been given for trust
management to avoid internal attacks [58–61]. Another problem with the reputa-
tion system occurs when the highly trusted monitors are compromised, which is
known as the betrayal attack [58]. In another attack known as the sleeper attack,
the compromised CIDS acts very gently until they gain a certain level of trust and
then the target attack is carried out [62]. Internal attacks are even more harmful
when combined with external attacks. Figure 15.4 summarizes the scenario.
In the following section, we will focus on the contribution of CIDS in the cloud
infrastructure followed by a short introduction of the cloud.
resources (e.g., networks, servers, storage, applications, and services) that can be
rapidly provisioned and released with minimal management effort or service pro-
vider interaction.” Cloud computing servers are basically operating systems (OS)
where the users are clustered into separate groups by using virtual machines. Since
the virtual machines reside in an operating system that is why threats of VM-to-VM
attacks increases. With the presence of a collaborative cloud computing model, a
large amount of data transfer occurs either between the local VMs or between the
different clouds. These are highly attracted to the attacker and are primary targets of
MITM or probing attacks. This information leakage can be followed by attacks like
DDoS, worm spread, etc. which can affect not only the compromised clouds but
also the connected cloud regions. Even the attacker can take advantage of other ser-
vices of cloud computing to attack their victims. A collaborative intrusion detection
system takes into consideration the suspicious activities and attack signatures from
multiple sources of IDS to protect the cloud systems. The CIDS provides:
In the following section, we will be looking into the coordinated attacks. We will
only present a short introduction to them.
15.11 Coordinated Attacks
Coordinated attacks are carried out by the attacker with the aim of monetary gain
[36]. Some of the known coordinated attacks are large-scale stealthy scans, worms,
Intrusion Detection for Big Data ◾ 393
and DDoS attacks. With the help of automation, it is easy to exploit different vulner-
able services simultaneously [52]. The highly distributed nature makes these attacks
more difficult to detect. In the following section, we review three main coordinated
attack methods by exploring their topology, purpose, target services, etc.
15.11.2 Worm Outbreaks
A worm is a special type of program that has the ability to replicate itself and without
any user interaction it can send copies to other computers over the network. The worm
outbreaks aim to gain two main goals: (1) degrading the system performance of the
targeted network and (2) creating compromised hosts for launching future attacks.
There are many known worms and some new worms are being created. It is difficult
to detect the worms because the newer worms may have completely unknown struc-
tures. However, these worms attack in two phases: in the first phase the worm infects
the target and during second phase it propagate to new hosts by self- replicating.
Some common worms are: SQL-Slammer, Code Red 2, W32/Sasser, etc.
creation of mobile agent can become costly if multiple nodes are attacked simul-
taneously and it may create communication disruption. Again, the central unit
is a SPoF. Again, if any host is compromised then the mobile agents will become
affected too.
The graph-based intrusion detection system (GrIDS) proposed a tree-like
infrastructure where networks are clustered into many zones called depart-
ments [21]. GrIDSs maintain a hierarchical model. Each department detects
and analyzes the intrusion. Each department has a parent node associated with
it. Locally detected attacks are sent to the parent nodes for future analysis.
GrIDS uses policy rules to detect alerts. That is why it may not detect attacks
of unknown policies. Moreover, the system performance can degrade for insider
attacks as well as for the DDoS attacks. However, GrIDS has a privacy protec-
tion mechanism that restricts access to the departments, and also the system is
very scalable.
Another hierarchical CIDS that can be used as a framework for different attack
detection engines is known as the autonomous agents for intrusion detection
(AAFID), it was proposed in Refs. [28,29]. The system has several transceivers and
monitors. The lower-level monitors monitor the network and send the feedback
to the transceivers for alert aggregation and correlation. The transceivers control
the local monitors as well as configure them. After aggregation, the report is sent
to the higher-level monitors. The higher level monitor is responsible for auditing
transceivers. The higher-level monitors can suffer from a single point of failure.
Again, the authors did not give any details about the aggregation, correlation, and
dissemination methods in their prototype.
Hierarchical intrusion detection (HIDE) was proposed in Ref. [27], which is
based on anomaly-based detection. The detection uses the statical preprocessing
and neural network classification. The monitors use the statistical approach to
detect the alert with reference to the model provided and after that the neural net-
work is used to determine whether an attack is normal or not. Lastly, final reports
are generated and provided to the user via the user interface. HIDE generates high
level of false positives. Moreover, the accuracy of the system depends on the design
of the algorithm. Zheng Zhang et al. [27] also points out that system cannot detect
the low volume attack traffic efficiently, which is a vital limitation.
Janakiraman et al. proposed a distributed collaborative intrusion detection sys-
tem, named intrusion detection and rapid action (INDRA) [25]. It follows the peer-
to-peer overlay and can detect generic attacks. INDRA uses the public-subscriber
method to exchange data between the monitors. The monitors in INDRA have
monitoring units to detect attacks and analysis unit, which generate alerts and take
action against the attacks. It disseminates the alert information with the help of
scribe [23]. Scribe creates a separate node for each class of attack and the monitors
subscribe to those nodes. INDRA has a feature for adding plugins for new classes
of attacks, but these need a manual approach of administration. Moreover, compro-
mised monitors can degrade the accuracy of the system. The authors suggested that
396 ◾ Data Analytics
keeping a blacklist consisting of suspicious peers can maintain the overall system
performance.
Zhou et al. introduced the large-scale intrusion detection (LarSID) in his
work [22,26]. It is a peer-to-peer-based CIDS and it uses the publish-subscribe
method on top of the distributed hash table. The local monitors monitor the
network and maintain a list of malicious IP addresses. The local list is shared if
a certain number of malicious IP addresses is added to the list. All the monitors
communicate over SSL. Moreover, they introduce the public key infrastructure
(PKI) for authentication. Even it can be scaled according to network size. The
main drawback is that it has no global monitoring ability, it can only detect
attacks with common sources.
A hybrid architectural-based CIDS named distributed overlay for monitoring
Internet outbreaks (DOMINO) was proposed in Ref. [24]. It consists of three
components: (1) axis overlay, (2) satellite communities, and (3) terrestrial con-
tributors. Axis overlay is considered as the central component and it consists
of trusted peers, which only use PKI authentication for communication. These
monitors can do the analysis of the traffic. These nodes introduce threshold fil-
tering to deal with the insider attacks. Satellite communities are localized ver-
sions of DOMINO protocols. These are dependent on the axis nodes. Terrestrial
contributors include all the nodes, even their non-trusty peers. DOMINO uses
signature-based detection and also uses honeypots. The alert data information is
shared in XML format periodically. There may arise a communication overhead
if the interval between the broadcasting is too short. Again, the system has no
policies for insider attacks.
Locasto et al. proposed a peer-to-peer-based intrusion system named wormi-
nator [15,20]. In worminator, each peer has a monitoring unit and every host is a
network-based IDS. Distributed correlation scheduling algorithm selects peers for
information sharing and information sharing is done by the bloom filter. Bloom
filters are used to exchange compressed alert data and to ensure security. However,
the scheduling algorithm was not given in detail by the authors.
CIDS for detecting distributed attack patterns, which is known as quick-
sand, was introduced in Ref. [11]. It uses a hybrid approach for attack detec-
tion at the global level, but at the local level each IDS is signature-based or
anomaly-based. The system has a central unit for attack signature storage and
update. Authors proposed the attack specification language (ASL) for describing
the attack signatures of the distributed attacks. With the help of ASL, the rela-
tionship between the attack and the host is transformed into an acyclic graph.
The central unit then disseminates the pattern graphs to the monitors. The local
monitors detect the attacks based on the updated pattern. The accuracy of the
system depends on the graphs patterns, so false positives might degrade the level
of accuracy.
In the following section, we present the summary for different collaborative
intrusion detection systems in Table 15.3.
Table 15.3 CIDS
Membership Aggregation and Global
Name Local Monitoring Management Correlation Data Dissemination Monitoring
(Continued)
398 ◾ Data Analytics
Table 15.3 (Continued) CIDS
Membership Aggregation and Global
Name Local Monitoring Management Correlation Data Dissemination Monitoring
References
1. Xiufeng Liu, Nadeem Iftikhar, and Xike Xie. 2014. Survey of real-time processing systems
for Big data. In Proceedings of the 18th International Database Engineering & Applications
Symposium (IDEAS’14), Ana Maria Almeida, Jorge Bernardino, and Elsa Ferreira Gomes
(Eds.). ACM, New York, pp. 356–361.
2. Mustafa Amir Faisal, Zeyar Aung, John R. Williams, and Abel Sanchez. 2012. Securing
advanced metering infrastructure using intrusion detection system with data stream mining.
In Intelligence and Security Informatics. PAISI 2012. Lecture Notes in Computer Science, Vol.
7299, Michael Chau, G. Alan Wang, Wei Thoo Yue, Hsinchun Chen (Eds.). Springer, Berlin.
3. Bazara I. A. Barry and H. Anthony Chan. 2010. Intrusion detection systems. In Handbook
of Information and Communication Security. Springer, Berlin, pp. 193–205.
4. Daniel A. Reed and Jack Dongarra. 2015. Exascale computing and big data.
Communications of ACM 58(7), 56–68
5. Big Data Information. www.nist.gov/el/cyberphysical-systems/Big-data-pwg
6. What is Big Data. www.ibm.com/Big-data/us/en/
7. Understanding Microsoft Big data solutions. https://msdn.microsoft.com/en-us/library/
dn749804.aspx
8. John R. Goodall, Wayne G. Lutters, and Anita Komlodi. 2004. I know my network:
collaboration and expertise in intrusion detection. In Proceedings of the 2004 ACM
Conference on Computer Supported Cooperative Work (CSCW ‘04). ACM, New York,
NY, pp. 342–345.
9. Mohiuddin Ahmed, Adnan Anwar, Abdun Mahmood, Zubair Shah, Michael J. Maher.
2015. An investigation of performance analysis of anomaly detection techniques for big
data in scada systems. EAI Endorsed Transactions on Industrial Networks and Intelligent
Systems 15(3), 1–16.
10. Subramanian Neelakantan and Shrisha Rao. 2008. A threat-aware signature based
intrusion-detection approach for obtaining network-specific useful alarms. In 2008
the Third International Conference on Internet Monitoring and Protection. Bucharest,
pp. 80–85.
400 ◾ Data Analytics
11. Christopher Krügel, Thomas Toth, and Clemens Kerer. 2002. Decentralized event cor-
relation for intrusion detection. In Information Security and Cryptology (ICISC). Springer,
Berlin and Heidelberg, vol. 2288, pp. 114–131.
12. R. Zuech, T.M. Khoshgoftaar, R. Wald. 2015. Intrusion detection and big heterogeneous
data: A survey. Journal of Big Data 2(1), 3.
13. Samuel Marchal, Xiuyan Jiang, Radu State, Thomas Engel. 2014. A big data architec-
ture for large scale security monitoring. In 2014 IEEE International Congress on Big data
(BigData Congress). IEEE, pp. 56–63.
14. Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A
survey. ACM Computing Surveys 41(3), 1–58.
15. Michael E. Locasto, Janak J. Parekh, Angelos D. Keromytis, and Salvatore J. Stolfo.
2005. Towards collaborative security and P2P intrusion detection. In IEEE Workshop on
Information Assurance and Security. IEEE, pp. 333–339.
16. J.F. Nieves and Y.C. Jia. 2009. Data clustering for anomaly detection in network intrusion
detection. Research Alliance in Math and Science, 1–12.
17. Like Zhang and Gregory B. White. 2007. An approach to detect executable content
for anomaly based network intrusion detection. In Parallel and Distributed Processing
Symposium, 2007. IPDPS 2007. IEEE International, pp. 1–8.
18. Ahmed Youssef and Ahmed Emam (2011). Network intrusion detection using data
mining and network behaviour analysis. International Journal of Computer Science &
Information Technology 3(6), 87–98.
19. Ahmed Patel, Mona Taghavi, Kaveh Bakhtiyari, Joaquim Celestino Júnior. (2013). An
intrusion detection and prevention system in cloud computing: A systematic review.
Journal of Network and Computer Applications 36(1), 25–41.
20. Michael E. Locasto, Janak J. Parekh, Salvatore Stolfo, and Vishal Misra. 2004.
Collaborative distributed intrusion detection. Technical report, Columbia University.
21. Steven Cheung, Rick Crawford, Mark Dilger, Jeremy Frank, Jim Hoagland, Karl Levitt,
Jeff Rowe, Stuart Staniford-Chen, Raymond Yip, and Dan Zerkle. 1999. The design
of GrIDS: A graph based intrusion detection system. Technical report, Department of
Computer Science, University of California at Davis.
22. Chenfeng Vincent Zhou, Shanika Karunasekera, and Christopher Leckie. 2005. A peer-to-
peer collaborative intrusion detection system. In International Conference on Networks. IEEE.
23. I.T. Antony, Miguel Castro, Peter Druschel, A.-M. Kermarrec, A.I.T. Rowstron. 2002.
Scribe: A large-scale and decentralized application level multicast infrastructure. IEEE
Journal on Selected Areas in Communications 20(8):1489–1499.
24. Vinod Yegneswaran, Paul Barford, and Somesh Jha. 2004. Global intrusion detection
in the DOMINO overlay system. In Proceedings of the Network and Distributed System
Security Symposium, NDSS 2004.
25. Ramaprabhu Janakiraman, Marcel Waldvogel, and Qi Zhang. 2003. Indra: A peer-to-
peer approach to network intrusion detection and prevention. In WETICE’03: Proceedings
of the Twelfth International Workshop on Enabling Technologies. IEEE Computer Society.
26. Chenfeng Vincent Zhou, Shanika Karunasekera, and Christopher Leckie. 2007. Evaluation of
a decentralized architecture for large scale collaborative intrusion detection. In The Tenth IFIP/
IEEE International Symposium on Integrated Network Management (IM 2007), pp. 80–89.
27. Zheng Zhang, Jun Li, C.N. Manikopoulos, Jay Jorgenson, and Jose Ucles. 2001. HIDE:
A hierarchical network intrusion detection system using statistical preprocessing and neu-
ral network classification. In IEEE Workshop on Information Assurance and Security. IEEE,
pp. 85–90. ISBN 0780398149.
Intrusion Detection for Big Data ◾ 401
28. Eugene H. Spafford and Diego Zamboni. 2000. Intrusion detection using autonomous
agents. Computer Networks 34(4), 547–570
29. Jai Sundar Balasubramaniyan, Jose Omar Garcia-fernandez, David Isacoff, Eugene
Spafford, and Diego Zamboni Ý. 1998. An architecture for intrusion detection using
autonomous agents. In IEEE Computer Security Applications Conference, pp. 13–24.
30. Pradeep Kannadiga and Mohammad Zulkernine. 2005. DIDMA: A distributed intrusion
detection system using mobile agents. In International Conference on Software Engineering,
Artificial Intelligence, Networking and Parallel/Distributed Computing. IEEE, pp. 238–245.
31. http://ids.surfnet.nl
32. Chenfeng Vincent Zhou, Shanika Karunasekera, and Christopher Leckie. 2007.
Evaluation of a decentralized architecture for large scale collaborative intrusion detection.
In The Tenth IFIP/IEEE International Symposium on Integrated Network Management (IM
2007), pp. 80–89.
33. Steven Snapp, James Brentano, Gihan Dias, Terrance Goan, Todd Heberlein, Che-Lin Ho,
Karl Levitt, Biswanath Mukherjee, Stephen Smaha, Tim Grance, Daniel Teal, and Doug
Mansur. 1991. DIDS (Distributed intrusion detection system)—Motivation, architecture,
and an early prototype. In Fourteenth National Computer Security Conference, pp. 167–176.
34. Vern Paxson. 2001. An analysis of using reflectors for distributed denial-ofservice attacks.
ACM SIGCOMM Computer Communication Review 31(3), 38–47.
35. Stuart Staniford, James A. Hoagland, and Joseph M. McAlerney. 2002. Practical auto-
mated detection of stealthy portscans. Journal of Computer Security 10(1–2), 105–136.
36. CERT Coordination center (CERT/CC). 2003. Module 4-types of intruder attacks.
37. Prahlad Fogla, Monirul I. Sharif, Roberto Perdisci, Oleg M. Kolesnikov, and Wenke Lee.
2006. Polymorphic blending attacks. In USENIX Security Symposium, pp. 241–256.
38. Kymie M.C. Tan, Kevin S. Killourhy, and Roy A. Maxion. 2002. Undermining an
anomaly-based intrusion detection system using common exploits. In Recent Advances in
Intrusion Detection. Springer, Berlin and Heidelberg, vol. 2516, pp. 54–73.
39. Oliver Dain and Robert K. Cunningham. 2001. Fusing a heterogeneous alert stream
into scenarios. In Proceedings of the 2001 ACM Workshop on Data Mining for Security
Applications, pp. 1–13.
40. Joaquin Garcia, Fabien Autrel, Joan Borrell, Sergio Castillo, Frederic Cuppens, and
Guillermo Navarro. 2004. Decentralized publish-subscribe system to prevent coordi-
nated attacks via alert correlation. In Sixth International Conference on Information and
Communications Security, pp. 223–35.
41. Steven T. Eckmann. 2002. Statl: An attack language for state-based intrusion detection.
Journal of Computer Security 10(1), 71–103.
42. David Wagner and Paolo Soto. 2002. Mimicry attacks on host-based intrusion detection
systems. In ACM Conference on Computer and Communications Security—CCS’02. ACM
Press, New York, pp. 255–264.
43. Tsung-Huan Cheng, Ying-Dar Lin, Yuan-Cheng Lai, and Po-Ching Lin. 2012. Evasion tech-
niques: Sneaking through your intrusion detection/prevention systems. IEEE Communications
Surveys & Tutorials 14(4), 1011–1020.
44. Ayalvadi J. Ganesh, A.-M. Kermarrec, and Laurent Massoulié. 2003. Peer-to-peer member-
ship management for gossip-based protocols. IEEE Transactions on Computers 52(2), 139–149.
45. Vivek Vishnumurthy and Paul Francis. 2006. On heterogeneous overlay construc-
tion and random node selection in unstructured P2P networks. In Proceedings IEEE
INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.
Barcelona, pp. 1–12.
402 ◾ Data Analytics
46. Phillip A. Porras, Martin W. Fong, and Alfonso Valdes. 2002. A mission-impact-based
approach to INFOSEC alarm correlation. In Recent Advances in Intrusion Detection. RAID
2002. Lecture Notes in Computer Science, Vol. 2516, Andreas Wespi, Giovanni Vigna, and
Luca Deri (Eds.). Springer, Berlin.
47. Zhi-Hua Zhou. 2011. When semi-supervised learning meets ensemble learning. Frontiers
of Electrical and Electronic Engineering in China 6(1), 6–16.
48. Vern Paxson. 1999. Bro: A system for detecting network intruders in real-time. Computer
Networks 31(23–24), 2435–2463.
49. Richard Zuech, Taghi M. Khoshgoftaar and Randall Wald. 2015. Intrusion detection
and big heterogeneous data: A survey. Journal of Big Data 2(1), 3.
50. Yoichi Shinoda, Ko Ikai, and Motomu Itoh. 2005. Vulnerabilities of passive internet
threat monitors. In Proceedings of the 14th USENIX Security Symposium, pp. 209–224.
51. Vitaly Shmatikov and Ming-Hsiu Wang. 2007. Security against probe-response attacks
in collaborative intrusion detection. In Proceedings of the Workshop on Large Scale Attack
Defense (LSAD’07). ACM, New York, pp. 129–136.
52. Stefan Savage, Geoff Voelker, Michael Vrable, and Nick Weaver. 2005. Internet
Outbreaks: Epidemiology and Defenses.
53. Butler W. Lampson. 1973. A note on the confinement problem. Communications of the
ACM 16(10), 613–615.
54. Sebastian Zander, Grenville J. Armitage, and Philip Branch. 2007. A survey of covert
channels and counter-measures in computer network protocols. IEEE Communications
Surveys 9, 44–57.
56. Claudiu Duma, Martin Karresand, Nahid Shahmehri, and Germano Caronni. 2006. A
trust-aware, P2P-based overlay for intrusion detection. In Proceedings of the International
Conference on Database and Expert Systems Applications (DEXA’06). IEEE, pp. 692–697.
57. Sepandar D. Kamvar, Mario T. Schlosser, and Hector Garcia-Molina. 2003. The eigen-
trust algorithm for reputation management in P2P networks. In Proceedings of the 12th
International Conference on World Wide Web (WWW’03), p. 640.
58. Carol Fung, Olga Baysal, Jie Zhang, Issam Aib, and Raouf Boutaba. 2008. Trust man-
agement for host-based collaborative intrusion detection. Managing Large-Scale Service
Deployment 5273(2008), 109–122.
59. Claudiu Duma, Martin Karresand, Nahid Shahmehri, and Germano Caronni. 2006. A
trust-aware, P2P-based overlay for intrusion detection. In Proceedings of the International
Conference on Database and Expert Systems Applications (DEXA’06). IEEE, pp. 692–697.
60. Poly Sen, Nabendu Chaki, and Rituparna Chaki. 2008. HIDS: Honesty-rate based
collaborative intrusion detection system for mobile ad-hoc networks. In Proceedings of
the 7th Computer Information Systems and Industrial Management Applications. IEEE,
pp 121–126.
61. Michael Brinkmeier, Mathias Fischer, Sascha Grau, and Guenter Schaefer. 2009.
Towards the design of unexploitable construction mechanisms for multiple-tree based
P2P streaming systems. In Kommunikation in Verteilten Systemen (KiVS). Springer, Berlin,
pp. 193–204.
62. Manuel Gil Pérez, Félix Gómez Mármol, Gregorio Martínez Pérez, and Antonio
F. Skarmeta Gómez. 2013. RepCIDN: A reputation-based collaborative intrusion detec-
tion network to lessen the impact of malicious alarms. Journal of Network and Systems
Management 21(1), 128–167.
Chapter 16
Contents
16.1 Introduction............................................................................................. 404
16.1.1 Chapter Roadmap.........................................................................405
16.2 Health Care in the Era of Industry 4.0......................................................405
16.3 Taxonomy of Cyberattacks in The Health Care Domain......................... 406
16.3.1 A ttacks on Medical Devices.......................................................... 406
16.3.1.1 Magnetic Resonance Imaging (MRI)..............................407
16.3.1.2 Robotic Surgical Machine...............................................407
16.3.1.3 Active Patient Monitoring Devices..................................407
16.3.2 Cyber-Physical Attacks..................................................................407
16.3.2.1 Attacks on Building Controls System............................. 408
16.3.3 I nsider Threat............................................................................... 408
16.4 Hacker’s Entry......................................................................................... 408
16.4.1 Reconnaissance............................................................................. 408
16.4.1.1 Footprinting................................................................... 409
16.4.1.2 Network Mapping.......................................................... 409
16.4.1.3 Scanning..........................................................................410
16.4.2 Hacker’s Access Hospital Network.................................................410
16.4.2.1 Phishing Attack...............................................................410
16.4.2.2 R ansomware.................................................................... 411
403
404 ◾ Data Analytics
16.1 Introduction
The health care sector is now embracing a plethora of newer technologies and, even-
tually, producing a huge amount of data. Managing these data for effective health
care systems is a challenge [1]. Health care data analytics is a multidisciplinary
field where scientific investigation is carried out to study social factors, financing
systems, health technologies, and personal behaviors. The objective of health care
analytics is to make health care effective, affordable, safe, accessible, and patient-
oriented [2,3]. The motivation behind the health care analytics is to benefit both
the patients and the health care providers. Health care brings together experts from
a wide range of disciplines such as medicine, social science, statistics, engineering,
policy, governance, etc. However, one of the challenges faced by health care sector
is in detecting and identifying cyberattacks and reducing cyber incidents [4]. Due
to the emerging malicious Internet users, i.e., cyber criminals, health care is being
affected. Some of the alarming cyber incidents are listed below:
◾◾ In 2015, cyber criminals took control of the personal information for 80 mil-
lion patients and stole tens of millions of records [4].
◾◾ According to 2017 Verizon Data Breach report [4], ransomware was ranked
22 for being the most common type of malware in 2014. However, in 2017,
it became fifth.
◾◾ Using phishing techniques, hackers compromised nearly 90,000 patients’
personal information at the University of Washington in 2013 [4].
◾◾ A new type of hijacking in health care is known as the medical device hijack
(called MEDJACK) [4]. A recent victim of MEDJACK is UCLA Health,
where personal data of 4.5 million patients were exposed.
Consequently, in the era of IoT, health care technology is changing in major ways
that creates more space for the cyber criminals to evade. It is estimated that the
number of health care IoT devices will increase from 95 million in 2015 to 646
million in 2020, which is almost 7-times in 5 years. Therefore, these devices will
lead to more accessible data or, in other words, to “big data.” It is important in the
health care sector that the big data gets utilized properly to predict epidemics, cure
diseases, and avoid preventable diseases [2,3].
Health Care Security Analytics ◾ 405
The aforementioned facts about IoT and big data emphasize the development
of robust cybersecurity strategies than ever before, especially for health care, as the
impact is most adverse when the data and devices are controlled by cyber criminals
[7–14]. To the best of our knowledge, this chapter is the first of its kind that cov-
ers cybersecurity in the health care domain extensively. Although there are a few
reports published by different organizations, such as Verizon, KPMG, Symantec,
etc., these reports [5] are not comprehensive enough to become helpful for the
research community.
16.1.1 Chapter Roadmap
The rest of the chapter is organized as follows. Section 15.2 covers the digitalization
of health care, i.e., industry 4.0. Section 15.3 discusses the cyberattacks in the health
care sector. Section 15.4 contains the ways hackers get access to a hospital network,
followed by counter measures in Section 15.5. Section 15.6 concludes the chapter.
In this scenario, hackers are far more interested as the incorporation of industry
4.0 opens a lot of opportunities for them [7–14]. Financial gain by hacking health
care systems is far more than any other systems as the information is sensitive
and the compromised organization can do nothing but comply with the demands
of the hackers. A cyberattack on a medical device can lead to misguided analysis
and wrong treatment of the patients. An attack on the cyber-physical systems of
a health care facility may jeopardize regular operations and affect critical utilities
such as electricity, water, heating, air conditioning, etc. Both medical devices and
cyber-physical systems faced a number of cyberattacks recently, and even after the
adoption of newer defense strategies, these attacks have not stopped.
16.3 Taxonomy of Cyberattacks in
The Health Care Domain
Cyberattacks in the health care domain can be classified into three groups in gen-
eral. The first type of attack is on medical devices where the devices connected to
the hospital network are compromised by cyber criminals who then exploits the
victims. The second type of attack is on cyber-physical systems, e.g., the smart
infrastructure of the health care facility. The third type of attack is alarming and
it’s called insider threat. The hospital employees are sometimes found exploiting
their access to the network and other devices, which leads to severe damages by
the cyber criminals [1]. Figure 16.1 below depicts the simple taxonomy of attacks
in health care. In the next subsections, we briefly discuss the three major types of
cyberattacks.
16.3.2 Cyber-Physical Attacks
Apart from the medical devices, the hackers are also interested in compromising
the cyber-physical devices. For example, the hospital building can be targeted [18].
There has been a lot of research on cyber-physical systems. However, in the scope of
health care facilities, there is no significant research. These attacks have the poten-
tial to do significant physical damage to hospital equipment and to create a con-
straint for regular patient treatments.
408 ◾ Data Analytics
16.3.3 Insider Threat
Insider threats are emerging types of attacks as it has been found in the last few
years that the legitimate users of the health care facilities are being exploited by
the cyber criminals. Therefore, these insiders have the credibility to cause physi-
cal damages to medical equipment and hospital facilities [19]. Insider threats to
critical infrastructure are more serious than outsider threats because an outside
attacker is less likely to know the hospital network vulnerabilities and weaknesses
than an insider would. Health care personnel with complete access to the hospital
network can easily perform reconnaissance on the network to identify vulner-
abilities and may inform the hackers. In summary, an insider attack is a malicious
attack on a network or computer system by a person with authorized system access.
There is an increasing number of incidents due to these types of attacks. However,
the current strategies are not enough to identify and detect these attacks.
16.4 Hacker’s Entry
In this section, we cover two important aspects by which the hacker gains information
to access the health care network. Figure 16.2 displays a simple taxonomy and major
ways to collect critical information about the network and to access the network.
16.4.1 Reconnaissance
Network reconnaissance is an effective way to test potential vulnerabilities in a
computer network [20–23]. Although it is a legitimate activity for the network
operator who is seeking to protect it, the hackers take advantage of it. The hackers
are interested to know more about the health care networks before launching the
attacks to devise strategies to bypass the protective shields. Next, we briefly discuss
Health Care Security Analytics ◾ 409
some popular techniques used for reconnaissance purpose and about how hackers
get to know more about the health care network.
16.4.1.1 Footprinting
One of the easiest ways to gather information about any network is to just use “who
is” [20]. Using “who is” it is possible to know about the organization’s employee direc-
tory that is the source of information for hackers to conduct social engineering. Also,
other important information obtained may include knowledge about the Internet
technologies being used, the operating system and hardware being used, IP addresses,
e-mail addresses and phone numbers, and policies and procedures. The main objec-
tive of footprinting is to learn as much as you can about a system, its remote access
capabilities, ports, services, and the security arrangements for the organization.
16.4.1.2 Network Mapping
There is an open source utility for network discovery and security auditing, called
Nmap (Network Mapper) [21]. Apart from systems and network administrators,
cyber criminals also find it useful for:
◾◾ Network inventory
◾◾ Management of service upgrade schedules
◾◾ Monitoring host or service
410 ◾ Data Analytics
This tool provides a number of functions for probing computer networks, host
discovery, service and operating system detection. The Nmap users are continuing
to develop and refine the tool. According to [21], Nmap is designed to rapidly scan
large networks. However, it also works well with single hosts. Nmap can be used
in almost every operating system and their binary packages are available for Linux,
Windows, and Mac OS X. Besides the command line-based Nmap, it also includes
the following:
16.4.1.3 Scanning
Port scanning is a popular technique to extract critical information, such as what
are the open ports and services available on a network host [22]. The security audi-
tors also use this technique to audit computers for vulnerabilities; however, it is
also used by hackers to target victims. For reconnaissance purpose, port scanning
is used to send requests to connect to the targeted computers and to track the
open ports or which respond to the request. Based on the port scanning informa-
tion, hackers know where to attack, especially the denial of service attack can
be launched very easily. Cyber criminals utilize port scanning since it is one of
the easiest ways to discover services on the target that they want to break into.
Due to the lack of security countermeasures in hospital networks, it is one of the
most convenient ways for people with malicious intents to launch cyberattacks in
the health care networks.
16.4.2.1 Phishing Attack
Phishing is a method to trick the victims. For example, the hackers send
legitimate-looking emails in an attempt to gather classified information from the
victims. Usually, the messages are framed in such a way that it seems to come from
well-known addresses. Websites that are frequently spoofed by phishers include
popular websites [24]. The ignorant employees of a health care facility will be easily
Health Care Security Analytics ◾ 411
compromised by these types of attacks. For example, spear phishing targets certain
hospital personnel and lures them to open emails that appear to come from a col-
league. When such emails are opened and some links are clicked, and the hospital
computers are compromised [24].
16.4.2.2 Ransomware
Ransomware is considered as extortion in cyber space; it is also known as crypto
viral extortion that restricts access to the infected computer system and it gives the
hackers power to demand a ransom. These types of attacks are extremely danger-
ous as it may so happen that the affected computer can be a server holding patient
files, which are being used as active monitoring devices. The hackers use malware to
infect the computers [25]. They demand a ransom of $17,000 to restore their opera-
tion. After investigation it is found that an employee opened an infected email or
downloaded the malware from a pop-up ad, which facilitated the virus to be planted
in the network. Currently, there are two different types of ransomware. The first is
through a program named Locky. Locky exploits spam email campaigns where an
email is sent across the system that contains infected MS Word documents. Once
the victim opens the document, macros will install on the host computer and begin
infecting the network. The second ransomware is via the Samas program, which
attacks web servers directly.
16.4.2.3 USB Stick
USB sticks are considered as gateways to plant malwares in any computing envi-
ronment. Cyber criminals often try to hand the infected USB sticks to hospital
employees via social engineering. Once the employee uses the USB stick in any of
the hospital’s networked computer, the malware is planted and then they can gain
as much information as they want via remote access. Infecting the USB sticks with
virus is another way of affecting the networked computers and getting access to
them. Consequently, the cyber criminals can identify the active medical devices.
Then, they are able to execute a wide range of malicious attacks, such as shutting
down the active medical devices or threatening to cause serious damage to the
devices unless a ransom is paid [26].
16.4.2.4 Password Cracker
A common approach to access any network is to crack the passwords of different access
points [27]. Cyber criminals use two major types of methods to guess passwords:
(i) brute-force and (ii) dictionary searches. Brute force is a widely used approach by
hackers to try different combinations until the right password is found. In diction-
ary search, the hacker looks at each word in the dictionary for the correct password.
Password dictionaries contain a wide range of subjects, such as policy, politics, games,
412 ◾ Data Analytics
movies, music, etc. There are other methods also, such as salting, rainbow tables, and
guessing, which are effective in cracking the passwords for any system. In a health
care network, the employees often use naive passwords that are easy to crack due to
the ignorance about the emerging attacks and social engineering.
16.5 Countermeasures
Countermeasures for health care cybersecurity is not much different from that for
any other domain. However, as reiterated time and again throughout the chapter,
the consequences of cyberattack are far more dangerous as it involves human lives.
There have been many studies and research for securing health care facilities from
cyber criminals; however, there is always room for improvement. We can think of
the countermeasures in two categories: detection and prevention.
Detection of cyberattacks in health care has recently got attention and newer
attacks are emerging, such as false data injection attacks. A recent research
Health Care Security Analytics ◾ 413
showcased the impact of false data injection attacks in health care [4]. In terms of
detection of cyberattacks, there are numerous tools and techniques in the literature
[7–14]; however, the challenge is to devise strategies to detect the zero-day attacks.
A few notable tools are listed below that are embraced by security analysts and
researchers across the globe:
The list is not comprehensive; however, it is a good starting point for the prac-
titioners. Now, when we delve into “prevention”, we need to be aware of the
proverb that “prevention is better than cure.” There are many IPS (intrusion
prevention systems) that are capable of detecting and preventing the system.
Apart from the regular systems which are available, it is also notable that there
are hardly any specific IPS for health care networks. Based on the discussion
above, it is clear that there is a lack of IDS and IPS designed for health care
systems.
Apart from the specific IDS and IPS, it is also important to establish a strong
security culture among the people working in the health care sector. As a part of the
security countermeasures, it is imperative to educate and train everyone involved
in the health care sector. It is often said that the most vulnerable point of any net-
worked/digital system is its users. The researchers who worked with the psychology
of information technology users have repeatedly showcased that the attitude of the
users is one of the main reasons for cyber incidents. Since the mass user cannot
predict the repercussions of cyber incidents, it is a challenge to raise the awareness.
Following steps can be executed to enhance security:
16.6 Conclusions
Since cybersecurity is an important application domain of data analytics, in this
chapter, we have summarized our investigation on health care. The chapter show-
cased the current status of cyber incidents in the health care sector, followed by
state-of-the-art health care systems. The taxonomy of attacks in the health care
sector provides a better understanding for the health care professionals. The discus-
sion on how hackers gain access to the hospital networks also provides meaningful
insights for the readers. To detect and identify the customized cyberattacks in the
health care sector, a discussion on countermeasures has been included that will help
the researchers in this area to devise newer strategies and robust intrusion detection
systems.
References
1. L. Ayala. Cybersecurity for Hospitals and Healthcare Facilities: A Guide to Detection and
Prevention. Berkely, CA: Apress, 2016.
2. J. Archenaa and E. M. Anita. “A survey of big data analytics in healthcare and gov-
ernment.” Procedia Computer Science, vol. 50, pp. 408–413, 2015. big Data, Cloud and
Computing Challenges.
3. S. M. R. Islam, D. Kwak, M. H. Kabir, M. Hossain, and K. S. Kwak. “The internet of
things for health care: A comprehensive survey.” IEEE Access, vol. 3, pp. 678–708, 2015.
4. M. Ahmed and A.S.S.M. Barkat Ullah. “False data injection attacks in healthcare.” In
15th Australasian Data Mining Conference, AusDM, 2017.
5. Connected Health. Available at www2.deloitte.com/content/dam/Deloitte/uk/Documents/
life-sciences-health-care/deloitte-uk-connected-health.pdf, accessed: February 10, 2018.
6. A. Hari and T. V. Lakshman. “The internet blockchain: A distributed, tamper-resistant
transaction framework for the internet.” In Proceedings of the 15th ACM Workshop on Hot
Topics in Networks, ser. HotNets’16. New York: ACM, 2016, pp. 204–210.
7. M. Ahmed. “Thwarting dos attacks: A framework for detection based on collective anom-
alies and clustering.” Computer, vol. 50, no. 9, pp. 76–82, 2017.
8. M. Ahmed. “Collective anomaly detection techniques for network traffic analysis.”
Annals of Data Science, January 2018.
9. M. Ahmed, A. Mahmood, and J. Hu. “A survey of network anomaly detection tech-
niques.” Journal of Network and Computer Applications, vol. 60, pp. 19–31, 2015.
10. M. Ahmed and A. Mahmood. “Network traffic analysis based on collective anomaly
detection.” In 9th IEEE International Conference on Industrial Electronics and Applications.
IEEE, 2014, pp. 1141–1146.
Health Care Security Analytics ◾ 415
11. M. Ahmed and A. Mahmood. “Network traffic pattern analysis using improved infor-
mation theoretic co-clustering based collective anomaly detection.” In International
Conference on Security and Privacy in Communication Networks. Springer International
Publishing, 2015, vol. 153, pp. 204–219.
12. M. Ahmed, A. Anwar, A. N. Mahmood, Z. Shah, and M. J. Maher. “An investigation
of performance analysis of anomaly detection techniques for big data in scada systems.”
EAI Endorsed Transactions on Industrial Networks and Intelligent Systems, vol. 15, no. 3, pp.
1–16, May 2015.
13. M. Ahmed, A. N. Mahmood, and J. Hu. “Chapter 1: Outlier detection.” In The State of the
Art in Intrusion Prevention and Detection. New York: CRC Press, January 2014, pp. 3–21.
14. M. Ahmed, A. N. Mahmood, and M. R. Islam. “A survey of anomaly detection techniques
in financial domain.” Future Generation Computer Systems, vol. 55, pp. 278–288, 2016.
15. A. Ray and R. Cleaveland. “An analysis method for medical device security.” In Proceedings
of the 2014 Symposium and Bootcamp on the Science of Security, ser. HotSoS ‘14. New York:
ACM, 2014, pp. 16:1–16:2.
16. J. Siegmund, C. K¨astner, S. Apel, C. Parnin, A. Bethmann, T. Leich, G. Saake, and
A. Brechmann. “Understanding understanding source code with functional magnetic
resonance imaging.” In Proceedings of the 36th International Conference on Software
Engineering, ser. ICSE 2014. New York: ACM, 2014, pp. 378–389.
17. T. Zhou, J. S. Cha, G. T. Gonzalez, J. P. Wachs, C. Sundaram, and D. Yu. “Joint surgeon
attributes estimation in robot-assisted surgery.” In Companion of the 2018 ACM/IEEE
International Conference on Human-Robot Interaction, ser. HRI ‘18. New York: ACM,
2018, pp. 285–286.
18. S. Wendzel, T. Rist, E. Andr´e, and M. Masoodian. “A secure interoperable architecture
for building-automation applications.” In Proceedings of the 4th International Symposium
on Applied Sciences in Biomedical and Communication Technologies, ser. ISABEL ‘11. New
York: ACM, 2011, pp. 8:1–8:5.
19. M. D. Carroll. “Information security: Examining and managing the insider threat.” In
Proceedings of the 3rd Annual Conference on Information Security Curriculum Development,
ser. InfoSecCD ‘06. New York: ACM, 2006, pp. 156–158.
20. A. Sridharan and T. Ye. “Tracking port scanners on the ip backbone.” In Proceedings of the
2007 Workshop on Large Scale Attack Defense, ser. LSAD ‘07. New York: ACM, 2007, pp.
137–144.
21. K. Rankin. “Hack and/: Dynamic config files with nmap.” Linux J., vol. 2010, no. 194,
June 2010.
22. J. Ye and L. Akoglu. “Discovering opinion spammer groups by network footprints.” In
Proceedings of the 2015 ACM on Conference on Online Social Networks, ser. COSN’15. New
York: ACM, 2015, pp. 97–97.
23. S. Standard, R. Greenlaw, A. Phillips, D. Stahl, and J. Schultz. “Network reconnaissance,
attack, and defense laboratories for an introductory cyber-security course.” ACM Inroads,
vol. 4, no. 3, pp. 52–64, September 2013.
24. Q. Cui, G.-V. Jourdan, G. V. Bochmann, R. Couturier, and I.-V. Onut. “Tracking phish-
ing attacks over time.” In Proceedings of the 26th International Conference on World Wide
Web, ser. WWW ‘17. Republic and Canton of Geneva, Switzerland: International World
Wide Web Conferences Steering Committee, 2017, pp. 667–676.
25. Y. L. Dion, A. A. Joshua, and S. N. Brohi. “Negation of ransomware via gamification and
enforcement of standards.” In Proceedings of the 2017 International Conference on Computer
Science and Artificial Intelligence, ser. CSAI 2017. New York: ACM, 2017, pp. 203–208.
416 ◾ Data Analytics
26. D. J. Tian, A. Bates, and K. Butler. “Defending against malicious usb firmware with
goodusb.” In Proceedings of the 31st Annual Computer Security Applications Conference, ser.
ACSAC 2015. New York: ACM, 2015, pp. 261–270.
27. J. Blocki, M. Blum, and A. Datta. “Gotcha password hackers!” In Proceedings of the 2013
ACM Workshop on Artificial Intelligence and Security, ser. AISec ‘13. New York: ACM,
2013, pp. 25–34.
28. M. Shobana, R. Saranyadevi, and S. Karthik. “Geographic routing used in manet for black
hole detection.” In Proceedings of the Second International Conference on Computational
Science, Engineering and Information Technology, ser. CCSEIT ‘12. New York: ACM, 2012,
pp. 201–204.
29. G. XIE, T. He, and G. Zhang. “Rogue access point detection using segmental tcp jitter.”
In Proceedings of the 17th International Conference on World Wide Web, ser. WWW ‘08.
New York: ACM, 2008, pp. 1249–1250.
30. V. Y. Hnatyshin and A. F. Lobo. “Undergraduate data communications and networking
projects using opnet and wireshark software.” In Proceedings of the 39th SIGCSE Technical
Symposium on Computer Science Education, ser. SIGCSE ‘08. New York: ACM, 2008, pp.
241–245.
31. L. L. Reynolds, Jr, R. W. Tibbs, and E. J. Derrick. “A gui for intrusion detection and
related experiences.” In Proceedings of the 43rd Annual Southeast Regional Conference -
Volume 2, ser. ACM-SE 43. New York: ACM, 2005, pp. 191–192.
32. R. Udd, M. Asplund, S. Nadjm-Tehrani, M. Kazemtabrizi, and M. Ekstedt. “Exploiting
bro for intrusion detection in a scada system.” In Proceedings of the 2Nd ACM International
Workshop on Cyber-Physical System Security, ser. CPSS ‘16. New York: ACM, 2016, pp.
44–51.
33. A. Hay, D. Cid, and R. Bray. OSSEC Host-Based Intrusion Detection Guide. Rockland,
MA: Syngress Publishing, 2008.
34. P. Szor. The Art of Computer Virus Research and Defense. New York: Addison-Wesley
Professional, 2005.
35. G. Post and A. Kagan. “The use and effectiveness of anti-virus software.” Computers &
Security, vol. 17, no. 7, pp. 589–599, 1998.
Index
417
418 ◾ Index
U W
Unsupervised learning, 19 Wilcoxon’s signed-rank test, 332, 335–337
User-defined function (UDF), 100–101, Wireless communication networks (WCNs),
106–107, 111, 118 213
Word Mover’s Distance (WMD), 79–80
V Word2vec, 83–85
Worm outbreaks, 393
Value management (VM), 232
VCPS, see Vehicular Cyber-Physical Systems X
(VCPS)
Vector algebra, 44 X-Stream, 108, 117, 185
Vector space model (VSM)
distance metrics, 77–80 Y
document-term matrix, 75–77
term-frequency approach, 80–81 YARN tool, 156
Vehicular Cyber-Physical Systems (VCPS), 215
Vertex-centric frameworks, 100–108 Z
classification of, 102–108
overview of, 100–102 ZooKeeper tool, 155
Taylor & Francis eBooks
www.taylorfrancis.com
Improved
A streamlined A single point search and
experience for of discovery discovery of
our library for all of our content at both
customers eBook content book and
chapter level