Big Data Analytics Notes
Big Data Analytics Notes
The term Big Data refers to all the data that is being
1.1 INTRoDUCTION generated across the globe at an unprecedented rate.
Now We are living in Big Data Era.
This data could be either structured or unstructured.
Few years ago, Systems or Organizations or
Today's business enterprises owe a huge part of their
Applications were using all Structured Data only success to an economy that is firmly knowledge-
Structured Data means In the form of Rows and
oriented.
Columns). It was very easy to use Relational Data Bases
Data drives the modern organizations of the world and
(RDBMS) and old Tools to store, manage, process and
hence making sense of this data and unraveling the
report this Data.
various patterns and revealing unseen connections
However recently, Nature of Data is changed. And within the vast sea of data becomes critical and a
Systems or Organizations or Applications are endeavor indeed.
hugely rewarding
generating huge amount of Data in variety of formats
There is a need to convert Big Data into Business
at very fast rate.
Intelligence that enterprises can readily deploy. Better
That means Data is not simple Structured Data(Not in data leads to better decision making and an improved
the form of simple Rows and Columns). It does not
way to strategize for organizations regardless of their
have any proper format, just Raw Data without any size, geography, market share, customer seg mentation
format. It is "very difficult or not possible" to use Old and such other categorizations. Hadoop is the platform
Technologies, Traditional Relational Databases and of choice for working with extremely large volumes of
Tools to store, manage, process and report this Data.
data.
Traditional Databases cannot Store, Process and
Big Data is a blanket term for the non-traditional
Analysis this kind of Data.
strategies and technologies needed to gather,
Then how to solve this problem? Here Big Data
organize, process, and gather insights from large
Solutions come into picture. datasets. While the problem of working with data that
Big Data Solutions solve all these problems very easily. exceeds the computing power or storage of a single
Let us start with understanding What is Big Data and computer is not new, the pervasiveness, scale, and
How important it is in our life. value of this type of computing has greatly expanded
We don't have a straightforward definition to Big Data. in recent years.
A t the right speed analyzing big data you can get a bet
understanding of current market conditions Eo
At the right time
Business Value
example, by analyzing customers purchasinn
To get the required behaviors, a company can find out the producte
this Big Data 4Vs Paradigm, we will get lot
By following that are sold the most and produce products
of benefits as shown below: By this, it can get aheadnf
according to this trend.
10)
The technology that SpaceCurve is developing can JBASO12017: Starting process 'Server: myserver
discover underlying patterns in multidimensional
17:01:58,833 INFO [orgjboss.as.process]
geodata.
(Shutdown thread) JBASO12016: Shutting down
Big Data: Data Formats process controller
Volume means "How much Data is generated". Now-a- Big Data refers to 3V (VVV) Paradigm:
days, Organizations or Human Beings or Systems are
generating or getting very vast amount of Data say
TB(Tera Bytes) to PB(Peta Bytes) to Exa Byte(EB) and
more.
Volume Velocity Variety
The name Big Data itself is related to a size which is
enormous. Size of data plays a very crucial role in
determining value out of data. Also, whether a
particular data can actually be considered as a Big Data
or not, is dependent upon the volume of data. Hence,
Fig. 1.2:BigData-3Vs
Volume' is one characteristic which needs to be
Three "Vs" Paradigm (Volume, Velocity, Variety) of Bi9
considered while dealing with Big Data.
Data was defined by "Doug Laney" in 2001.
vOLUME= Very large amount of data
If our Organization's Data is in
this 3Vs Paradigm, that
2. Velocity:
means we are in
Big Data Problems. So should use
Velocity means "How fast produce Data". Now-a-days, we
Big Data 4Vs In Simple Terminology: The reality is that there is a lack of skills available in the
market for big data technologies. The typical expert
VVolume) : The Amount of Data
has also
gained experience through tool
VVariety): The number of Type of Data
implementation and its use as a programming model,
VVelocity) : The Speed of Data Processing
apart from the big data management aspects.
VVeracity): The Correctness of Data 3. Getting Data into Big Data Structure:
1.4 CHALLENGES AND APPLICATIONS OF BIG It might be obvious that the intent of a big data
DATA management involves analyzing and processing a large
1.4.1 amount of data. There are many people who have
Challenges of Big Data
raised expectations considering analyzing huge data
The handling of big data is very complex. Some
sets for a big data platform. They also may not be
challenges faced during its integration include
aware of the complexity behind the transmission,
uncertainty of data Management, big data talent gap,
access, and delivery of data and information from a
getting data into a big data structure, syncing across
wide range of resources and then loading these data
data sources, getting useful information out of the big
into a big data platform.
data, volume, skill availability, solution cost etc.
The intricate aspects of data transmission, access and
1. The Uncertainty of Data Management:
loading are only part of the challenge. The requirement
One disruptive facet of big data management is the to navigate transformation and extraction is not limited
use of a wide range of innovative data management to conventional relational data sets.
tools and frameworks whose designs are dedicated to
Syncing Across Data Sources:
and analytical processing. The
supporting operational Once you import data into big data platforms you may
NoSQL (not only SQL) frameworks are used that
also realize that data copies migrated from a wide
differentiate it from traditional relational database
range of sources on different rates and schedules can
management systems and are also largely designed to
fulfill performance demands of big data applications rapidly get of the synchronization with the
out
such as manag ing a large amount of data and quick originating system. This implies that the data coming
response times. from one source is not out of date as compared to the
data coming from another source. It also means the
There are a variety of NosQL approaches such as
hierarchical object representation (such as JSON, XML commonality of data definitions, concepts, metadata
and BSON) and the concept of a key-value storage. and the Iike.
BIG DATA ANALYTICS (COMP., DBATU) INTRODUCTION TOBIG DATA
(1.8)
The traditional data IDC that the a
estimates
management and data Digital Universe report, amount
warehouses, the world's syste
IT
extraction and migrations
sequence of data transformation, of information stored in the systems is
By 202 the total
all arise the situation about every two years.
in doubling
which there are risks for be enough to
fill a stack of tablete
data to become amount will that
unsynchronized. reaches from the
earth to the moon 6.6 timec
And
or liability for abo
Extracting Information from the Data in
Big Data enterprises have responsibility
Integration: percent of that information.
able to connect different big data platforms and also "Managing unstructured
data is growing as a challeno
- rising from 31 percent in 2015 to 45 percent in 2016
provide transparency of the data consumers to
eliminate the requirement of custom coding. In order to deal with
data growth, organizations are
At the same time, if the number of data consumers turning to a number of different technologies. When it
grow, then one can provide a need to support an comes to storage, converged and hyper converged
increasing collection of many simultaneous user infrastructure and software-defined storage can make
accesses. This increment of demand may also spike at it easier for companies to scale their hardware. And
any time in reaction to different aspects of business technologies like compression, deduplication and
process cycles. It also becomes a challenge in big data tiering can reduce the amount of space and the costs
integration to ensure the right-time data availability to associated with big data storage.
the data consumers. On the management and analysis side, enterprises are
6. Miscellaneous Challenges: using tools like NosQL databases, Hadoop, Spark, big
Other challenges may occur while integrating big data. data analytics software, business intelligence
Some of the challenges include integration of data, applications, artificial intelligence and machine learning
skill availability, solution cost, the volume of data, the to help them comb through their big data stores to
rate of transformation of data, veracity and validity of find the insights their companies need.
data. 8. Generating Insights in a Timely Manner
The ability to merge data that is not similar in source Of course, organizations don't just want to store their
or structure and to do so at a reasonable cost and in big data they want to use that big data to achieve
time. It is also a challenge to process a large amount of business goals. According to the New Vantage Partners
data at a reasonable speed so that information is survey, the most common goals associated with big
available for data consumers when they need it. The data projects included the following:
validation of data set is also fulfilled while transferring Decreasing expenses through operational cost
data from one source to another or to consumers as
efficiencies
well. Establishing a data-driven culture
This is all about the big data integration and some
challenges that one can face during the
Creatingnew avenues for innovation and
disruption
implementation. These points must be considered and
should be taken care of if you are going to manage
Accelerating the speed with which new capabilne
and services
any big data platform.
are
deployed
7. Dealing with Data Growth
Launching new product and service offerings
All of those re
big data is goals can help organizations become fmo
The most obvious challenge associated with
competitive but only if they extract insights tro
simply storing and analyzing all that information. In its can
BIG DATA ANALYTICS (COMP., DBATU) (1.9) INTRODUCTION TO BIG DATA
their big data and then act on those insights quickly. In response, many enterprises are turning to neW
PWC's Global Data and Analytics Survey 2016 found technology solutions. In the IDG report, 89 percent of
"Everyone wants decision-making to be faster, those surveyed said that their
companies planned to
especially in banking, insurance, and healthcare" invest in new big data tools in the next 12 to 18
months. When asked which kind of tools
To achieve that speed, some organizations are looking they were
to a new generation of ETL and analytics tools that planning to purchase, integration technology was
second on the list, behind data analytics software.
dramatically reduce the time it takes to generate
11. Validating Data
reports. They are investing in software with real-time
Closely related to the idea of data integration is the
analytics capabilities that allows them to respond to
idea of data validation. Often organizations are
developments in the marketplace immediately. getting
similar pieces of data from different systems, and the
9. Recruiting and Retaining Big Data Talent data in those different systems doesn't always agree.
But in order to develop, manage and run those For example, the ecommerce system may show daily
applications that generate insights, organizations need sales at a certain level while the Enterprise Resource
professionals with big data skills. That has driven up Planning (ERP) system has a slightly different number.
demand for big data experts and big data salaries have Or a hospital's Electronic Health Record
(EHR) system
increased dramatically as a result. may have one address for a patient, while a partner
The 2017 Robert Half Technology Salary Guide pharmacy has a different address on record.
reported that big data engineers were earning The process of getting those records to agree, as well
between $135,000 and $196,000 on average, while as making sure the records are accurate, usable and
data scientist salaries ranged from $116,000 to $163, secure, is called data governance. And in the At Scale
500. Even business intelligence analysts were very well 2016 Big Data Maturity Survey, the fastest-growing
paid, making $118,000 to $138,750 per year area of concern cited by respondents was data
In order to deal with talent shortages, governance.
organizations
have a couple of options. First, many are increasing Solving data governance challenges is very complex
their budgets and their recruitment and retention and is usually requires a combination of policy changes
and technology. Organizations often set up a group of
efforts. Second, they are offering more training people to oversee data governance and write a set of
opportunities to their current staff members in an
policies and procedures. They may also invest in data
attempt to develop the talent they need from within.
management solutions designed to simplify data
Third, many organizations are looking to technology.
governance and help ensure the accuracy of big data
They are buying analytics solutions with self-service stores and the insights derived from them.
and/or machine learning capabilities. Designed to be 12. Securing Big Data
used by professionals without a data science degree, Security is also a big concern for organizations with big
these tools may help organizations achieve their big data stores. After all, some big data stores can be
data goals even if they do not have a lot of big data attractive targets for hackers or Advanced Persistent
experts on staff. Threats (APTs).
10. Integrating Disparate Data Sources However, most organizations seem to believe that their
The variety associated with big data leads to existing data security methods are sufficient for their
big data needs as well. In the IDG survey, less than half
challenges in data integration. Big data comes from a
of those surveyed (39 percent) said that they were
lot of different places enterprise applications, social
using additional security measure for their big data
media streams, email systems, employee-created
repositories or analyses. Among those who do use
documents, etc. Combining all that data and
additional measures, the most popular include identity
reconciling it so that it can be used to create reports
and access control (59 percent), data encryption (52
can be incredibly difficult. Vendors offer a variety of
percent) and data segregation (42 percent.
ETL and data integration tools designed to make the
13. Organizational Resistance
process easier, but many enterprises say that they have
It is not only the technological aspects of big data that
not solved the data integration problem yet.
can be challenging people can be an issue too.
BG
DATA
have to do some things differently. And that sort of Social Media Analytics trans
are now beginning to analyze big data to obtain Knowledge discovery Stream analytics
advances in the industry tools
insights. Recent technologic
their ability to work with such data,
have improved
files and often have
even though the are enormous
characteristics.
1.5 ENABLING TECHNOLOGIES FOR BIG
DATA
is a combination of Data virtualization
The big data analytics technology Data integration
methods. What
several techniques and processing
makes them effective
enterprises to obtain
management and implementation.
is their
relevant
collective
results
use by
for strategic 88
Data Preprocessing Data quality
enthusiasm, and ambition to
In spite of the investment
data to transform the enterprise, Fig. 1.4
leverage the power of
BIGDATA ANALYTICS (COMP., DBATU)
(1.13) INTRODUCTION TO BIG DATA
1. Predictive Analytics
7. Data Virtualization
One of theprime tools for businesses to avoid risks in I t enables
decision making, predictive analytics can
help
applications to retrieve data without
businesses. Predictive analytics hardware and software implementing technical restrictions such as data
formats, the physical location of
solutions can be utilized for discovery, evaluation and data, etc. Used by
Apache Hadoop and other distributed data stores for
deployment of predictive scenarios by
processing big real-time or near real-time access to
data. Such data can help companies to be data stored on
prepared for various platforms, data virtualization
is one of the most
what is to come and help solve
problems by analyzing used big data technologies.
and understanding them.
Data Integration
2. NoSQL Databases
A key operational challenge for most
These databases utilized for reliable and efficient
are organizations
handling big data is to process terabytes (or peta
data management across a scalable number of
storage bytes) of data in a way that can be useful for customer
odes. NoSQL databases store data as relational deliverables. Data integration tools allow businesses
to
database tables, JSON docs or key-value streamline data across a number of
pairings big data solutions
3. Knowledge Discovery Tools such as Amazon EMR,
Apache Hive, Apache Pig,
These are tools that allow businesses to mine Apache Spark, Hadoop, MapReduce, Mongo DB and
big data Couchbase.
(structured and unstructured) which is stored on
. Data Preprocessing
multiple sources. These sources can be different file
These software solutions are used for
systems, APIs, DBMS or similar platforms. With search manipulation of
data into a format that is consistent and can be used
and knowledge discovery tools, businesses can isolate for further analysis. The data
and utilize the information to their benefit. preparation tools
accelerate the data sharing process by formatting and
4. Stream Analytics
cleansing unstructured data sets. A limitation of data
Sometimes the data anorganization needs to process preprocessing is that all its tasks cannot be automated
can be stored on multiple platforms and in multiple and require
human oversight, which can be tedious
formats. Stream analytics software is highly useful for and time-consuming.
filtering, aggregation, and analysis of such big data. 10. Data Quality
Stream analytics also allows connection to external An important parameter for big data processing is the
data sources and their integration into the data quality. The data quality software can conduct
application
flow. cleansing and enrichment of large data sets by utilizing
5. In-Memory Data Fabric
parallel processing. This software is widely used for
getting consistent and reliable outputs from big data
This technology helps in distribution of large quantities
processing.
of data across system resources such as Dynamic RAM,
In conclusion, Big Data is already being used to
Flash Storage or Solid State Storage Drives which in
improve operational efficiency, and the ability to make
turn enables low latency access and processing of big informed decisions based on the very latest up-to-the-
data on the connected nodes. momentinformation is rapidly becoming the
6. Distributed Storage mainstream norm.
Away to counter independent node failures and loss1.6BIG DATA STACK
or corruption of big data sources, distributed file stores The Data Layer
contain replicated data. Sometimes the data is also At the bottom of the stack are technologies that store
replicated for low latency quick access on large masses of raw data, which comes from traditional
computer networks. These are generally non-relational sources like OLTP databases, and newer, less
databases. structured sources like log files, sensors, web analytics,
BIG DATA ANALYTICs (COMP DBATU) INTRODUCTION TO BIGDA
(1,14) DATA
document and media archives. Increasingly, Tools
Big Data Ingestion
happens in the cloud or on virtualized local storage
resources. A few examples
Organizations are moving away from legacy storage, ETL (Extract, Transforn
form, Load
towards commoditized hardware, and Stitch:A lightweight
managed services like Amazon $3.
more recently to tool which pulls data from multiple pre-integrato
rated data
cleans it as necessan
transforms and
Chartio: Cloud BI service allowing you to connect on-site. These are generally products from web service
data providers, such as Amazon or Microsoft, running either
sources, explore data, build SQL queries and transform
the data as needed, and create live Hadoop distributions from other, platform-focused
auto-refreshing vendors such
dashboards. as Hortonworks or MapR, or their own
distributions.
Looker: Cloud-based BI platform that lets you query
Beyond that, all of the top distributions have subtle
and analyze large data sets via SQL define metrics once
differences which could make them more or less
set up visualizations that tell a story with your data.
suitable for your business. Here's a non-exhaustive
17BIG DATADISTRIBUTION PACKAGE guide to some of the most popular on the market
Hadoop is the open source software framework at the today.
heart of much of the Big Data and analytics revolution. Cloudera
It provides solutions for enterprise data storage and Cloudera was the first vendor to offer Hadoop as a
analytics with almost unlimited scalability. Since its package and continues to be a leader in the industry.
release in 2011 it has rapidly grown in popularity and a Its Cloudera CDH distribution, which contains all the
strong ecosystem of distributors, vendors and
open source components, is the most popular Hadoop
consultants has emerged to support its use across distribution. Cloudera is known for acting quickly to
industry. innovate with additions to the core framework - it was
At its core, Hadoop is an Open Source system, which, the first to offer SQL-for-Hadoop with its Impala query
among other considerations, means it is essentially free engine. Other additions include user interface, security
for anyone to use. However, the requirement for it to and interfaces for integration with third party
be aligned to the needs of individual organizations has applications. It offers support for the whole of the
BIG DATA ANALYTICS (COMP., DBATUu)
distribution through (1.16) INTRODUCTION TO BIG DATA
its
subscription service. Cloudera Enterprise for storage and AWS loT to collect data from Internes
net
Hortonworks of Things-enabled devices.