Professional Documents
Culture Documents
Fast Data Enterprise Data Architecture
Fast Data Enterprise Data Architecture
strataconf.com
Presented by OReilly and Cloudera,
Strata + Hadoop World is where
cutting-edge data science and new
business fundamentals intersect
and merge.
n
Job # 15420
Scott Jarr
First release
ISBN: 978-1-491-91393-2
[LSI]
Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
1. Whats Shaping the Environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Data Is Everywhere
Data Is Fast Before Its Big
1
2
5
6
8
10
11
12
12
15
15
16
16
17
iii
20
20
20
21
26
26
27
29
30
31
31
33
33
34
34
9. Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
iv
| Table of Contents
Preface
vi
| Preface
CHAPTER 1
Data Is Everywhere
The digitization of the world has fueled unprecedented growth in data,
much of it driven by the global explosion of mobile data sources and
the Internet of Things (IoT). Each day, more devicesfrom smart
phones to cars to electric gridsare being connected and intercon
nected. It is safe to predict that within the next 1015 years, anything
powered by electricity will be connected to the Internet.
According to the 2014 EMC/IDC Digital Universe report, data is dou
bling in size every two years. In 2013, more than 4.4 zetabyes of data
had been created; by 2020, the report predicts that number will explode
by a factor of 10 to 44 zetabytes44 trillion gigabytes. The report also
notes that peopleconsumers and workerscreated some two-thirds
of 2013s data; in the next decade, more data will be created by things
sensors and embedded devices. In the report, IDC estimates that the
IoT had nearly 200 billion connected devices in 2013 and predicts that
number will grow 50% by 2020 as more devices are connected to the
Internetsmartphones, cars, sensor networks, sports tracking mon
itors, and more.
Data from these connected devices is fueling a data economy, creating
huge implications for future business opportunity. Additionally, the
rate of growth of new data is creating a structural change in the ways
enterprises, which are responsible for more than 80% of the worlds
data, manage and interact with that data.
As the data economy evolves, an important distinction between the
major ways in which businesses interact with data is emerging. Com
1
panies have begun to interact with data that is bigdata that has vol
ume and variety. Additionally, as companies embark on ever-more
extensive big data initiatives, they have also realized the importance
of interacting with data that is fast. The ability to process data imme
diatelya requirement driven by IoT macro-trendscreates new op
portunity to realize value via disruptive business models.
To illustrate this point, consider the devices generating all this data.
Some are relatively dumb sensors that generate a one-way flow of in
formationfor example, network sensors that push data to a process
ing hub but that cannot communicate with one another. More im
portant are two-way sensors embedded in smart devicesfor ex
ample, automotive in-vehicle infotainment and navigation systems
and smart meters used in smart power grids. These two-way sensors
not only collect data but also enable organizations to analyze and make
decisions on that data in real time, pushing results (more data) back
to the device. These smart sensors create huge streams of fast, smart
data; they can act autonomously on your inputs as well as act col
lectively on the groups inputs.
The EMC/IDC report states that embedded systemsthe sensors and
systems that monitor the physical universealready account for 2%
of the digital universe. By 2020 that will rise to 10%. Clearly, two-way
sensors that generate fast and big data require different modes of in
teraction if the data is to have any business value. These different
modes of interaction require the new capabilities of the enterprise data
architecture.
in motion has equal or greater value than historical data (data at rest)
new opportunities to extract value require that enterprises adopt
new approaches to data management. Many traditional database ar
chitectures and systems are incapable of dealing with fast datas chal
lenges.
As a result, the data management industry has been enveloped in con
fusion, much of it driven by hype surrounding the major forces of big
data, cloud, and mobility. Fortunately, many of the available technol
ogies are falling into categories based on problems they address, bring
ing the picture into better focus. This is good news for application
developers, as advances in cloud computing and in-memory database
architectures mean familiar tools can be used to tackle fast data.
Data Is Everywhere
CHAPTER 2
Introduction
The enterprise data architecture is a break from the traditional siloed
data application, where data is disconnected from the analytics and
other applications and data. The enterprise data architecture supports
fast data created in a multitude of new end points, operationalizes the
use of that data in applications, and moves data to a data lake where
services are available for the deep, long-term storage and analytics
needs of the enterprise. The enterprise data architecture can be rep
resented as a data pipeline that unifies applications, analytics, and ap
plication interaction across multiple functions, products, and disci
plines (see Figure 2-1).
Figure 2-1. Fast data represents the velocity aspect of big data.
Figure 2-2. Data has the greatest value as it enters the pipeline, where
realtime interactions can power business decisions, e.g., customer in
teraction, security and fraud prevention, and optimization of resource
utilization.
The actions companies take with data are increasingly correlated to
the datas age. Figure 2-2 represents time as the horizontal axis. To the
far left is the point at which data is created. Immediately after data is
created, it is highly interactive and for each event, of greatest value.
This is where the opportunity exists to perform high-velocity opera
tions on new or incoming datafor example, to place a trade,
make a recommendation, serve an ad, or inspect a record. This is the
beginning of a data management pipeline.
Shortly after data enters the pipeline, it can be examined relative to
other data that has also arrived recently, e.g., by examining network
traffic trends, composite risk by trading desk, or the state of an online
game leader board.
Queries on fresh data in motion are commonly referred to as realtime
analytics.
Architecture Matters
Interacting with fast data is a fundamentally different process than
interacting with big data that is at rest, requiring systems that are ar
chitected differently. With the correct assembly of components that
reflect the reality that application and analytics are merging, an en
terprise data architecture can be built that achieves the needs of both
data in motion (fast) and data at rest (big).
Building high-performance applications that can take advantage of
fast data is a new challenge. Combining these capabilities with big data
analytics into an enterprise data architecture is increasingly becoming
table stakes. But not everyone is prepared to play.
CHAPTER 3
Figure 3-1. Note the tight coupling of fast and big, which must be sep
arate systems at scale.
The first thing to notice is the tight coupling of fast and big, although
they are separate systems; they have to be, at least at scale. The database
system designed to work with millions of event decisions per second
is wholly different from the system designed to hold petabytes of data
and generate extensive historical reports.
10
SQL on Hadoop
Much innovation is happening in this space. The goal of many of
these products is to displace the data warehouse. Advances have
been made with the likes of Hawq and Impala. Nevertheless, these
systems have a long way to go to get near the speed and efficiency
of data warehouses, especially those with columnar designs. SQLon-Hadoop systems exist for a couple of important reasons:
a. SQL is still the best way to query data
b. Processing can occur without moving big chunks of data
around
Exploratory analytics
This is the realm of the data scientist. These tools offer the ability
to find things in data: patterns, obscure relationships, statistical
rules, etc. Mahout and R are popular tools in this category.
Job scheduling
This is a loosely named group of job scheduling and management
tasks that often occur in Hadoop. Many Hadoop use cases today
involve pre-processing or cleaning data prior to the use of the
analytics tools described above. These tools and interfaces allow
that to happen.
The big data side of the enterprise data architecture has, to date, gained
the lions share of attention. Few would debate the fact that Hadoop
has sparked the imagination of whats possible when data is fully uti
lized. However, the reality of how this data will be leveraged is still
largely unknown.
11
legacy data into the data lake where everything is stored. These appli
cations will migrate to full-fledged fast + big data applications in time
(this is discussed in detail in Chapter 7).
12
Furthermore, data events dont exist in isolation from other data that
may be static or coming from other sensors in the system. To continue
the precious metal mine example above, monitoring the location of
an expensive piece of equipment might raise a warning as it moves
outside an authorized zone. However, that piece of location data re
quires additional context from another data source. The movement
might be acceptable, for instance, if that machinery is on a list of work
orders showing this piece of equipment is on its way to the repair
depot. This is the concept of data fusion, the ability to make contex
tual decisions on streaming and static data.
Data is also valuable when it is counted, aggregated, trended, and so
forthi.e., realtime analytics. There are two ways in which data is
analyzed in real time:
1. A human wants to see a realtime representation of the mine, via
a dashboarde.g., how many sensors are active, how many are
outside of their zone, what is the utilization efficiency, etc.
2. Realtime analytics are used in the automated decision-making
process. For example, if a reading from a sensor on a human shows
low oxygen for an instant, it is possible the sensor had an anom
alous reading. But if the system detects a rapid drop in ambient
oxygen over the past five minutes for six workers in the same area,
its likely an emergency requiring immediate attention.
Physical asset management in a mine is a real-world use case to illus
trate what is needed from all the systems that manage fast data. But it
is representative. The same pattern exists for Distributed Denial of
Service (DDoS) detection, log file management, authorization of fi
nancial transactions, optimizing ad placement, online gaming, and
more.
Once data is no longer interactive and fast moving, it will move to the
big data systems, whose responsibility it is to provide reliable, scalable
storage and a framework for supporting tools to query this historical
data in the future. To illustrate the specifics of what is to be expected
from the big data side of the architecture, return to the mining exam
ple.
13
Assume the sensors in the mine are generating one million events per
second, which, even at a small message size, quickly add up to large
volumes of stored data. But, as experience has shown, that data cannot
be deleted or filtered down if it is to deliver its inherent value. There
fore, historical sensor data must move to a very cost-effective and re
liable storage platform that will make the data accessible for explora
tion, data science, and historical reporting.
Mine operators also need the ability to run reports that show historical
trends associated with seasonality or geological conditions. Thus, data
that has been captured and stored must be accessible to myriad data
management toolsfrom data warehouses to statistical modeling
to extract the analytics value of the data.
This historical asset management use case is representative of thou
sands of use cases that involve data-heavy applications.
14
CHAPTER 4
15
quo is beginning in earnest. Fast data is where many of the truly rev
olutionary advances will be made.
17
CHAPTER 5
Many solutions are emerging in the fast data market from players like
Amazon and Google, a testament to the fact that a data problem is
looming. Unfortunately, by focusing mainly on stream processing,
these solutions miss a huge part of the value organizations can gain
from fast data. Fast data is a new frontier. It is an inevitable step or
ganizations will take when they begin to deeply integrate analytics into
the data management architecture. Committing to a path that does
not address all these capabilities ensures an organization will be rewriting its systems far sooner than desired.
As data has become more immediately valuable, application develop
ers have realized applications now need to interact with fast streams
of data and analytics to take advantage of the data available to them.
This recognition surfaces the requirements of fast data.
19
21
Fast data systems must seamlessly integrate into systems designed to store
big data
One size does not fit all when it comes to database technology in the
21st century. So, while a fast in-memory operational database is the
correct tool for the job of managing fast data, other tools are optimized
for the storage and deep analytic processing of big data. Moving data
between these systems is an absolute requirement.
However, much more than just data movement is required. In addition
to the pure movement of data, the integration between big data and
fast data must allow:
Dealing with the impedance mismatch between the big systems
import capabilities and the fast data arrival rate.
Reliable transfer between systems, including persistence and
buffering.
Pre-processing of data, so when it hits the data lake it is ready to
be used.
In the smart grid example, fast data coming from smart meters across
an entire country accumulates quickly. This historical data has obvious
value in showing seasonal trends and year-over-year grid efficiencies.
Moving this data to the data lake is critical. But validations, security
checks, and data cleansing can be done prior to the data arriving in
the data lake. The more this integration is baked into data management
products, the less code the application architect needs to figure out,
e.g., how to persist data if one system fails and where to overflow data
if the data lake cant keep up with ingestion rates.
Fast data systems must have the ability to serve analytic results and
knowledge from big data systems quickly to users and applications, closing
the data loop
The deep, insightful analytics generated by BI reports and analyzed by
data scientists need to be operationalized, i.e., able to use realtime data.
This can be achieved in two ways:
Make the BI reports consumable by more people/devices than the
analytics system can currently support
Take the intelligence from the analytics and move it into the op
erational system
22
23
CHAPTER 6
At this point, it is natural to ask questions: What are these dataintensive applications? Where do they exist? While this book has pre
sented a number of detailed use case examples, there is no shortage of
places where fast data applications are producing new value for users
and businesses.
Few industries are immune from the pressures and opportunities that
vast amounts of data represent. However, as famously stated by Wil
liam Gibson, The future is already hereits just not very evenly dis
tributed.1
The early-to-mid market adoption of data-intensive applications can
be segmented into three broad categories, based on the industrys pro
gression to an evolved data-driven strategy.
1. William Gibson on NPRs Fresh Air, August 1, 1993. Also in The Science in Science
Fiction on Talk of the Nation, NPR (30 November 1999, Timecode 11:55).
25
26
Example: Online advertising has chased the same elusive goal all ad
vertising has soughtdelivering the right ad, to the right audience, at
the right time. But early entrants into the digital advertising world were
unable to get closer to that ideal than the print or broadcast advertisers
they were attempting to replace. A generic ad on an automotive web
site was no more targeted than a generic ad in an automotive magazine.
Now, with the addition of a data-driven architecture, the ad can be
targeted based on demographic trends (historic), current user profile
(static), and the previous clicks and current performance of the various
advertising exchange options (real time).
27
could argue that the nature of the traditional taxi industry does not
lend itself to a broad sharing and analysis of data, it is clear that markets
can be createdand destroyedby the ability (or failure) to recognize
fast data opportunities.
28
CHAPTER 7
Fast data is already streaming into the enterprise, and more is coming
on a daily basis. However, in many cases, enterprises are pushing this
fast data directly into the data lake, missing the opportunity to extract
valuable realtime insights from data streams using in-memory tech
nology. Realizing the benefits of this fast data requires a new enterprise
data architecture. Therefore, the way in which systems are designed
and built to leverage streams of data will define how quickly and per
vasively fast data applications will be rolled out within an organization.
To understand how enterprise adoption of fast data technologies will
occur, one needs to examine both the data sources and the applications
that utilize those data sources. Four broad usage environments will
drive enterprise adoption of fast data. The first three are combinations
of a specific application and the data source(s) that encompass that
application. The fourth category will be defined by corporations that
truly understand the value that exists in being data-driven, and are
prepared to implement an enterprise data architecture designed to
unify all data interaction within the enterprise.
Existing Applications
This category of usage exists when applications that manage data begin
to experience increasing volumes of data, exerting pressure on existing
applications. Given the normal architecture of these systems, the load
on the traditional database component will no longer meet the needs
of the application; a change in the application will be required.
29
30
Chapter 7: How Fast and Big Applications Will Enter the Enterprise
31
CHAPTER 8
33
34
Chapter 8: Getting There: Making the Right Fast Data Technology Choices
Some
Yes
Yes
No
No
Yes
Realtime analytics
Yes
Through add-on
Yes
No
Yes
Yes
No
Yes
35
CHAPTER 9
Conclusion
38
Chapter 9: Conclusion
Colophon
The text font is Adobe Minion Pro; the heading font is Adobe Myriad
Condensed; and the code font is Dalton Maags Ubuntu Mono.