Professional Documents
Culture Documents
Tamr EB Getting DataOps Right Full 05-23-19
Tamr EB Getting DataOps Right Full 05-23-19
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Getting DataOps
Right, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the authors, and do not represent the
publisher’s views. While the publisher and the authors have used good faith efforts
to ensure that the information and instructions contained in this work are accurate,
the publisher and the authors disclaim all responsibility for errors or omissions,
including without limitation responsibility for damages resulting from the use of or
reliance on this work. Use of the information and instructions contained in this
work is at your own risk. If any code samples or other technology this work contains
or describes is subject to open source licenses or the intellectual property rights of
others, it is your responsibility to ensure that your use thereof complies with such
licenses and/or rights.
This work is part of a collaboration between O’Reilly and Tamr. See our statement of
editorial independence.
978-1-492-03173-4
[LSI]
Table of Contents
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
DevOps and DataOps 2
The Catalyst for DataOps: “Data Debt” 2
Paying Down the Data Debt 3
From Data Debt to Data Asset 4
DataOps to Drive Repeatability and Value 4
Organizing by Logical Entity 5
3. DataOps as a Discipline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
DataOps: Building Upon Agile 14
Agile Operations for Data and Software 17
DataOps Challenges 22
The Agile Data Organization 25
iii
5. Key Components of a DataOps Ecosystem. . . . . . . . . . . . . . . . . . . . . . 37
Catalog/Registry 38
Movement/ETL 38
Alignment/Unification 39
Storage 39
Publishing 41
Feedback 41
Governance 41
iv | Table of Contents
CHAPTER 1
Introduction
Andy Palmer
1
DevOps and DataOps
Much like DevOps in the enterprise, the emergence of enterprise
DataOps mimics the practices of modern data management at large
internet companies over the past 10 years. Employees of large inter‐
net companies use their company’s data as a company asset, and
leaders in traditional companies have recently developed this same
appetite to take advantage of data to compete. But most large enter‐
prises are unprepared, often because of behavioral norms (like terri‐
torial data hoarding) and because they lag in their technical
capabilities (often stuck with cumbersome extract, transform, and
load [ETL] and master data management [MDM] systems). The
necessity of DataOps has emerged as individuals in large traditional
enterprises realize that they should be using all the data generated in
their company as a strategic asset to make better decisions every day.
Ultimately, DataOps is as much about changing people’s relationship
to data as it is about technology infrastructure and process.
The engineering framework that DevOps created is great prepara‐
tion for DataOps. For most enterprises, many of whom have adop‐
ted some form of DevOps for their IT teams, the delivery of high-
quality, comprehensive, and trusted analytics using data across
many data silos will allow them to move quickly to compete over the
next 20 years or more. Just like the internet companies needed
DevOps to provide a high-quality, consistent framework for feature
development, enterprises need a high-quality, consistent framework
for rapid data engineering and analytic development.
2 | Chapter 1: Introduction
• The competitive pressure of digital-native companies in tradi‐
tional industries.
• The opportunity presented by the “democratization of analytics”
driven by new products and companies that enabled broad use
of analytic/visualization tools such as Spotfire, Tableau, and
BusinessObjects.
4 | Chapter 1: Introduction
Next-level leaders who recognize the threat of digital natives are
looking to use data aggressively and iteratively to create new value
every day as new data becomes available. The biggest challenge
faced in enterprise data is repeatability and scale—being able to find,
shape, and deploy data reliably with confidence. Also—much like
unstructured content on the web—structured data changes over
time. The right implementation of DataOps enables your analytics
to adapt and change as more data becomes available and existing
data is enhanced.
• Customers
• Suppliers
• Products
• Research
• Facilities
• Employees
• Parts
To ensure clean, unified data for these core entities, a key compo‐
nent of DataOps infrastructure is to create a system of reference that
maps a company’s data to core logical entities. This unified system of
reference should consist of unified attributes constructed from the
raw physical attributes across source systems. Managing the path‐
ways between raw, physical attributes, changes to the underlying
data, and common operations on that data to shape it into
production-readiness for the authoritative system of reference are
the core capabilities of DataOps technologies and processes.
This book gets into much more detail on DataOps and the practical
steps enterprises have and should take to pay down their own data
debt—including behavioral, process, and technology changes. It
traces the development of DataOps and its roots in DevOps, best
practices in building a DataOps ecosystems, and real-world exam‐
ples. I’m excited to be a part of this generational change, one that I
truly believe will be a key to success for enterprises over the next
decade as they strive to compete with their new digital-native
competitors.
The challenge for the large enterprise with DataOps is that if it
doesn’t adopt this new capability quickly, it runs the risk of being left
in the proverbial competitive dust.
6 | Chapter 1: Introduction
CHAPTER 2
Moving Toward Scalable Data
Unification
Michael Stonebraker
7
required to transform the data into a common format for the desti‐
nation repository—it was necessary to write “merge rules” to com‐
bine the data sources, and additional rules to decide on the true
value for each attribute in each entity. Although fine for small opera‐
tions, like understanding sales and production data at a handful of
retail stores or factories, ETL systems failed to scale to large num‐
bers of data stores and/or large numbers of records per store.
The next generation of ETL tools offered increased functionality,
such as data cleaning capabilities and adaptors for particular data
sources. Like the first generation, these ETL tools were designed for
use by computer programmers, who had specialized knowledge.
Hence, they did not solve the fundamental scalability bottleneck: the
time of a skilled software professional. These ETL tools form the
bulk of the unification market today; however, most large enterpri‐
ses still struggle to curate data from more than a couple dozen sour‐
ces for any given data unification project. The present state of affairs
is an increasing number of data sources that enterprises want to
unify and a collection of traditional ETL tools that do not scale. The
rest of this book discusses scalability issues in more detail.
Unifying Data
The benefits to unifying data sources are obvious. If a category man‐
ager at Airbus wants to get the best terms for a part that their line of
business (LoB) is buying, that manager will typically have access
only to purchasing data from their own LoB. The ability to see what
other LoBs are paying for a given part can help that category man‐
ager optimize their spend. Added up across all of the parts and sup‐
pliers across all Airbus LoBs, these insights represent significant
savings. However, that requires integrating the LoB supplier data‐
bases for each LoB. For example, GE has 75 of them, and many large
enterprises have several to many of them because every acquisition
comes with its own legacy purchasing system. Hence, data unifica‐
tion must be performed at scale, and ETL systems are not up to the
challenge.
The best approach to integrating two data sources of 20 records each
is probably a whiteboard or paper and pencil. The best approach for
integrating 20 data sources of 20,000 records each might very well
be an ETL system and rules-based integration approach. However, if
GE wants to unify 75 data sources with 10 million total records, nei‐
Unifying Data | 9
Source Attribute name Record format
CRM-1 Tel. (xxx) xxx-xxxx
CRM-2 Phone_Number Xxxxxxxxx
DataLake Phone_Number xxx-xxx-xxxx
To consolidate the two CRM sources into the DataLake schema, you
need to write one mapping: Phone_Number equals Telephone. To
standardize the format of the number, you need to transform two
different formats to a third standard one.
Now let’s do this for six data sources:
We now have five different names for the same attributes, and one of
these attributes (Cell) might require some expertise to correctly map
it to Phone_Number. We also have four formats for phone numbers,
requiring four different transformations into the DataLake format.
In this simple example, we’ve gone from three rules to unify three
data sources to eight when doubling the amount of attributes.
Hence, the complexity of the problem is increasing much faster than
the number of data sources. Rules are problematic at scale because:
Unifying Data | 11
tional wisdom is to cluster records into a multidimensional space
formed by the records’ attributes, with a heuristically specified dis‐
tance function. Records that are close together in this space are
probably the same entity. This runs into the classic N**2 clustering
problem, and the computational resource required to do operations
with complexity N**2, where N is the number of records, is often
too great. Scalable unification systems must scale out to multiple
cores and processors (tenet 5) and must have a parallel algorithm
with lower complexity than N**2 (tenet 6).
Given the realities of the enterprise data ecosystem, scalable unifica‐
tion systems need to accommodate data sources that change regu‐
larly. Even though running the entire workflow on all of the data to
incorporate changes to a data source can satisfy some business use
cases, applications with tighter latency requirements demand a scal‐
able unification system to examine the changed records themselves
and perform incremental unification (tenet 7).
Scalable data unification must be the goal of any enterprise, and that
will not be accomplished using traditional ETL systems. It is obvi‐
ously the foundational task for enterprises looking to gain “business
intelligence gold” from across the enormous troughs of enterprise
data.
Nik Bates-Haus
13
DataOps: Building Upon Agile
DataOps is a methodology that spans people, processes, tools, and
services to enable enterprises to rapidly, repeatedly, and reliably
deliver production data from a vast array of enterprise data sources
to a vast array of enterprise data consumers.
DataOps builds on the many decades of accumulated wisdom
received from Agile processes. It is worth taking a moment to high‐
light some key goals and tenets of Agile, how they have been applied
to software, and how they can be applied to data. Agile software
development arose from the observation that software projects that
were run using traditional processes were plagued by the following:
These are the same frustrations that plague so many data delivery
projects today.
Agile Practices
The preceding has described the goal and tenets of Agile, but not
what to actually do. There are many variations of the Agile process,
but they share several core recommendations:
Deliver working software frequently
In days or weeks, not months or years, adding functionality
incrementally until a release is completed.
Get daily feedback from customers (or customer representatives)
Gather feedback on what has been done so far.
Accept changing requirements
Be prepared to do so even late in development.
Work in small teams
Work in teams of three to seven people who are motivated, trus‐
ted, and empowered individuals, with all the skills required for
delivery present on each team.
Keep teams independent
This means each team’s responsibilities span all domains,
including planning, analysis, design, coding, unit testing,
acceptance testing, releasing, and building and maintaining
tools and infrastructure.
Continually invest in automation
You should aim to automate everything.
DataOps Tenets
DataOps is an emerging field, whereas DevOps has been put into
practice for many years now. We can use our depth of experience
with DevOps to provide a guide for the developing practice of Data‐
Ops. There are many variations in DevOps, but they share a collec‐
tion of core tenets:
Let’s review these briefly, how they affect service availability, and the
expected impact on data availability.
DataOps Challenges
DataOps teams, particularly those working with Big Data, encounter
some challenges that other operations teams do not.
Query Interface
Data is not usable without a query interface. A query interface is a
type of API, so data consumers can be written and verified against
an abstract interface and then run against any provider of that API.
Unfortunately, most query interfaces are vendor or vendor/version
specific, and the vendors provide only one implementation of the
query interface, so much of the benefit of writing to an API is lost.
SQL is an attempt to create a standard data query API, but there is
enough variation between vendor implementations that only the
simplest of queries are compatible across vendors, and attaining
DataOps Challenges | 23
good performance always requires use of vendor-specific language
extensions.
Thus, even though we want to focus on DaaS, independent of any
particular vendor platform, the current reality is that the vendor and
version of most query interfaces must be transparent to end users,
and becomes part of the published interface of the data infrastruc‐
ture. This impedes upgrades, and makes it nearly impossible to
change vendors.
This problem is compounded by the fact that different data consum‐
ers require different kinds of query interface to meet their needs.
There are three very different modes of interacting with data, and
the DataOps team needs to provide interfaces for all of them:
Resource Intensive
Even moderate-scale data places significant demands on infrastruc‐
ture, so provisioning is another DataOps challenge. DataOps needs
to consider data storage, movement, query processing, provenance,
and logging. Storage must be provisioned for multiple releases of
data as well as for different environments. Compute must be provi‐
sioned intelligently, to keep data transfers within acceptable limits.
Network must be provisioned to support the data transfers that can‐
not be avoided. Although provisioning to support resource-
intensive loads is not unique to DataOps, the nature of data is such
that DataOps teams will have very little runway relative to other
Schema Change
Vendors change data with every release. Analysts require data
changes for every new analytic or visualization. These modifications
put schemas, and therefore ADIs, in a state of perpetual change.
Each change might require adjustment to the entire depth of the
associated data pipelines and applications. Managing the entire
DataOps ecosystem as versioned, tested code, with clear separation
between development and production environments, makes it pos‐
sible to respond quickly to these changes, with confidence that
problems will be caught quickly. Unfortunately, many tools still
assume that schemas change slowly or not at all, and the DataOps
team must implement responsiveness to schema change outside
these tools. Good factoring of code to centralize schema definition is
the only way to keep up with this rapid pace of change.
Governance
Regulations from both government and industry cover data access,
retention, traceability, accountability, and more. DataOps must sup‐
port these regulations and provide alerting, logging, provenance,
and so on throughout the data-processing infrastructure. Data gov‐
ernance tools are rapidly maturing, but interoperability between
governance tools and other data infrastructure is still a significant
challenge. The DataOps team will need to bridge the gaps between
these toolchains to provide the coverage required by regulation.
1 For more on this, see “DataOps: Building a Next Generation Data Engineering Organi‐
zation” by Andy Palmer and Liam Cleary.
Andy Palmer
• Highly automated
• Open
• Take advantage of best of breed tools
• Use Table(s) In/Table(s) Out protocols
• Have layered interfaces
• Track data lineage
29
• Feature deterministic, probabilistic, and humanistic data inte‐
gration
• Combine both aggregated and federated methods of storage and
access
• Process data in both batch and streaming modes
Highly Automated
The scale and scope of data in the enterprise has surpassed the abil‐
ity of bespoke human effort to catalog, move, and organize the data.
Automating your data infrastructure and using the principles of
highly engineered systems—design for operations, repeatability,
automated testing, and release of data—is critical to keep up with
the dramatic pace of change in enterprise data. The principles at
work in automating the flow of data from sources to consumption
are very similar to those that drove the automation of the software
build, test, and release process in DevOps over the past 20 years.
This is one of the key reasons we call the overall approach
“DataOps.”
Open
The best way to describe this is to talk about what it is not. The pri‐
mary characteristic of a modern DataOps ecosystem is that it is not a
single proprietary software artifact or even a small collection of arti‐
facts from a single vendor. For decades, B2B software companies
have been in the business of trying to get their customers “hooked”
on a software artifact by building it into their infrastructure—only
to find that the software artifact is inadequate in certain areas. But
because the software was built with proprietary assumptions, it’s
impossible to augment or replace it with other software artifacts that
represent “best of breed” for that function.
In the next phase of data management in the enterprise, it would be
a waste of time for customers to “sell their data souls” to single ven‐
dors that promote proprietary platforms. The ecosystem in DataOps
should resemble DevOps ecosystems in which there are many best
of breed free and open source software (FOSS) and proprietary tools
that are expected to interoperate via APIs. An open ecosystem
Best of Breed
Closely related to having an open ecosystem is embracing technolo‐
gies and tools that are best of breed—meaning that each key compo‐
nent of the system is built for purpose, providing a function that is
the best available at a reasonable cost. As the tools and technology
that the large internet companies built to manage their data goes
mainstream, the enterprise has been flooded with a set of tools that
are powerful, liberating (from the traditional proprietary enterprise
data tools), and intimidating.
Selecting the right tools for your workload is difficult because of the
massive heterogeneity of data in the enterprise and also because of
the dysfunction of software sales and marketing organizations who
all overpromote their own capabilities, i.e., extreme data software
vendor hubris.
It all sounds the same on the surface, so the only way to really figure
out what systems are capable of is to try them (or by taking the word
of a proxy—a real customer who has worked with the vendor to
deliver value from a production system). This is why people such as
Mark Ramsey, previously of GSK, are a powerful example. Mark’s
attempt to build an ecosystem for more than 12 best-of-breed ven‐
dors and combine their solutions to manage data as an asset at scale
is truly unique and a good reference as to what works and what does
not.
Best of Breed | 31
many different methods described under Data Services. A great ref‐
erence for these table-oriented methods is the popularity of Resilient
Distributed Datasets (RDDs) and DataFrames in the Spark ecosys‐
tem. Using service-oriented methods for these interfaces is critical,
and the thoughtful design of these services is a core component of a
functional DataOps ecosystem. Overall, we see a pattern of three key
types of interfaces that are required or desired inside of these sys‐
tems.
• Data access services that are “View” abstractions over the data
and are essentially SQL or SQL-like interfaces. This is the
power-user level that data scientists prefer.
• Messaging services that provide the foundation for stateful data
interchange, event processing, and data interchange
orchestration.
• REST services built on or wrapped around APIs providing the
ultimate flexible direct access to and interchange of data.
Conclusion
Everything presented in this chapter is obviously high level and has
an infinite number of technical caveats. After doing hundreds of
implementations at large and small companies, I believe that it’s
actually possible to do all of it within an enterprise, but not without
embracing an open and best-of-breed approach. At Tamr, we’re in
the middle of exercising all of these principles with our customers
every day, across a diverse and compelling set of data engineering
projects.
Conclusion | 35
CHAPTER 5
Key Components of a DataOps
Ecosystem
Andy Palmer
• Catalog/Registry
• Movement/ETL
• Alignment/Unification
• Storage
• Publishing
• Feedback
37
Catalog/Registry
Over the past 5 to 10 years, a key function has emerged as a critical
starting point in the development of a functional DataOps ecosys‐
tem—the data catalog/registry. There are a number of FOSS and
commercial projects that are attempts to provide tools that enable
large enterprises to answer the simple question, “What data do you
have?” Apache Atlas, Alation, and Waterline are the projects that I
see the most in my work at Tamr and discussions with my chief data
officer friends. I’ve always believed that the best data catalog/registry
is a “vendor neutral” system that crawls all tabular data and registers
all tabular datasets (files and inside of database management systems
[DBMS]) using a combination of automation and human guidance.
The problem with vendor-specific systems is inevitably their imple‐
mentation devolves into wars between vendor-oriented camps
within your organization and you end up not with one catalog/
registry, but multiple—which sort of defeats the purpose. If the ven‐
dor providing the catalog system is more open and interoperable
(per Chapter 4 on the principles of a DataOps ecosystem), it’s likely
OK. This is not an impossible thing to build yourself—I’ve seen two
to three built in the past few years that function well—but it is prob‐
ably just a matter of time before something like Apache Atlas
evolves to provide the basics in vendor-neutral systems that are
highly open and interoperable.
Movement/ETL
There are so many options for data movement that it’s a bit mind
numbing. They range from the existing ETL/extract, load, and
transform (ELT) vendors (Informatica, Talend, Oracle, IBM, Micro‐
soft) to the new breed of movement vendors (my favorites are
StreamSets, Data Kitchen, and KNIME) and the cloud data platform
vendors such as Google GCP/DataFlow, Microsoft Azure, Amazon
Web Services (AWS), Databricks, Snowflake, and the dozens of new
ones that every VC in the Bay Area is funding by the month.
The most interesting dynamic here, in my opinion, is that most
large enterprises use Python for most of their data pipelining, and
the idea of broadly mandating a single pipelining tool for their users
seems like a bit of a reach. Python is so easy for people to learn, and
if you use an orchestration framework such as Apache AirFlow, you
Alignment/Unification
The tools required to create consistency in data need to be strongly
rooted in the use of three key methods: rules, models, and human
feedback. These three key methods, implemented with an eye
toward an Agile process, are essential to tame the large challenge of
variety in enterprise data. Traditional tools that depend on a single
data architect to a priori define static schema, controlled vocabulary,
taxonomy, ontologies, and relationships are inadequate to solve the
challenges of data variety in the modern enterprise. The thoughtful
engineering of data pipelines and the integration of rules, active
learning–based probabilistic models of how data fits together, and
the deliberate human feedback to provide subject matter expertise
and handle corner cases is essential to success in the long-term
alignment and molding of data broadly.
Storage
The biggest change in data systems over the past 10 years has been
the evolution of data storage platforms—both cloud and on-
premises. When my partner Mike and I started Vertica back in 2004,
the database industry had plateaued in terms of new design patterns.
At the time, everyone in the enterprise was using traditional “row
Alignment/Unification | 39
stores” for every type of work load—regardless of the fundamental
fit. This was the gist of Mike and Ugur Cetintemel’s “‘One Size Fits
All’: An Idea Whose Time Has Come and Gone” paper and my Use‐
nix talk in 2010. Overall, the key next-generation platforms that are
top of mind now include the following:
Feedback
This is perhaps the most impactful and least appreciated component
of a next-generation DataOps ecosystem. Currently, data in most
enterprises flows unidirectionally from sources through determinis‐
tic and idiosyncratic pipelines toward data warehouses, marts, and
spreadsheets—and that’s where the story stops. There is an incredi‐
ble lack of systematic feedback mechanisms to enable data to flow
from where data is consumed back up into the pipelines and all the
way back to the sources so that the data can be improved systemati‐
cally over time. Most large organizations lack any “queue” of data
problems identified by data consumers. At Tamr we’ve created
“Steward” to help address this problem, providing a vendor-neutral
queue of data consumer–reported problems with data for the
enterprise.
Governance
Governance is a key component of a modern ecosystem—the evolu‐
tion of data privacy has raised governance up to the top of the prior‐
ity list—driven by a need to comply to many important new
regulations. My belief is that the best governance infrastructure
focuses on the automation of information access policy and the
prosecution of that policy across users in the context of key roles
that are aligned with policy. Focusing on governance in the context
of information use helps avoid boiling the infinite proverbial ocean
Publishing | 41
of data source complexity. Having a codified information access pol‐
icy as well as methods for running that policy in real time as users
are consuming data should be the core goal of any governance infra‐
structure initiative.
Liam Cleary
Interoperability
Interoperability is perhaps the biggest departure in the DataOps
stack from the data integration tools or platforms of the past.
Although a good ETL platform certainly appeared to afford intero‐
perability as a core principle, with many out-of-the-box connectors
supporting hundreds of data formats and protocols (from Java Data‐
base Connectivity [JDBC] to Simple Object Access Protocol
[SOAP]), this, in fact, was a reaction to the complete lack of intero‐
perability in the set of traditional data repositories and tools.
In practice, if your ETL platform of choice did not support a native
connector to a particular proprietary database, enterprise resource
planning (ERP), customer relationship management (CRM), or
business suite, the data was simply forfeited from any integration
project. Conversely, data repositories that did not support an open
data exchange format forced users to work within the confines of
that data repository, often subsisting with rigid, suboptimal, worst-
of-breed, tack-on solutions. The dream of being able to develop a
center of excellence around a single data integration vendor quickly
43
evaporated as the rigid constraints of the single vendor tightened
with increasing data variety and velocity.
Results Import
In practice, however, even DataOps tools can, and often must, over‐
lap in capability. The answer to this apparent redundancy reveals
one of the unique hallmarks of interoperability in the DataOps
stack, namely the tool’s ability to both export and import its results
for common overlapping capabilities. In composing your DataOps
pipeline, this allows your team to decide where best the artifacts and
results of a particular common ability should be natively generated
or imported.
Consider that a data extract tool, a data unification tool, and a data
catalog are each capable of generating a basic dataset profile (record
count, attribute percentage null, etc.):
Metadata Exchange
The ability of the DataOps team to rapidly compose, build, and
deploy data unification, cataloging, and analytics pipelines, with
plug-and-play components, seems to present a formidable systems
integration challenge. In DataOps however, the focus of interopera‐
bility is not the need for closer systems integration—for example,
with each tool possessing native support for each other tool—but
rather the need for open interoperable metadata exchange formats.
This objective reveals the second unique hallmark of interoperabil‐
ity in the DataOps stack, namely the ready exchange and enrich‐
ment of metadata.
To illustrate, consider a data element richly decorated with the fol‐
lowing metadata:
Data type: numeric
Description: Total spend
Interoperability | 45
Format: X,XXX.XX
To realize these benefits each of our DataOps tools requires the abil‐
ity to both preserve (pass through) this metadata, and the ability to
enrich it. For example, if we’re lucky, the source system already con‐
tains this metadata on the data element. We now require our extrac‐
tion tool to read, preserve, and pass this metadata to the unification
tool. In turn, the unification tool preserves and passes this metadata
to the cataloging tool. If we’re unlucky and the source system is
without this metadata, the extraction tool is capable of enriching the
metadata by casting the element to numeric, the unification tool is
capable of mapping to an existing data element with description
Total spend, and the dashboarding tool applies a typical currency
format X,XXX.XX.
In flowing from source, through extraction, to unification and dash‐
board, the metadata is preserved by each tool by supporting a com‐
mon interoperable metadata exchange format. Equally important to
preserving the metadata, each tool enriches the metadata and expo‐
ses it to each other tool. This interaction is in stark contrast to data
integration practices espoused by schema-first methodologies.
Although metadata pass-through and enrichment are certainly a
secondary requirement to that of data pass-through and enrichment
(which even still remains a challenge for many tools today), it is cer‐
tainly a primary and distinguishing feature of a DataOps tool and is
an essential capability for realizing interoperability in the DataOps
stack.
Continuous Automation
If the primary challenge of the DevOps team is to streamline the
software release cycle to meet the demands of the Agile develop‐
ment process, so the objective of the DataOps team is to automate
the continuous publishing of datasets and refreshing of every tool’s
results or view of those datasets.
To illustrate continuous automation in a DataOps project, let’s con‐
sider a data dashboarding tool. We might ask the following ques‐
tions of its API suite. Does the tool have the following:
Automation | 47
The ability of a tool to be automatically updated and refreshed is crit‐
ical to delivering working data and meeting business-critical timeli‐
ness of data. The ease at which the dataset can be published and
republished directly affects the feedback cycle with the data con‐
sumer and ultimately determines the DataOps team’s ability to real‐
ize the goal of shortened lead time between fixes and faster MTTR.
Batch Automation
With most tools already providing sufficient capability to be contin‐
uously updated in an automatic, programmatic manner, it is easy to
overlook the almost distinctly Agile need to spin up and tear down a
tool or set of tools automatically.
Indeed, it is especially tempting to devalue batch automation as a
once-off cost that need only be paid for a short duration at the very
beginning when setting up a longer-term, continuous pipeline.
However, the ability of a tool to be automatically initialized, started,
run, and shut down is a core requirement in the DataOps stack and
one that is highly prized by the DataOps team. Under the tenet of
automate everything, it is essential to achieving responsiveness to
change and short time to delivery.
To illustrate, let’s consider batch automation features for a data uni‐
fication tool. Does the tool have the following:
Automation | 49
CHAPTER 7
Embracing DataOps:
How to Build a Team and Prepare
for Future Trends
Mark Marinelli
So far, we’ve covered the catalyst behind and need for DataOps, the
importance of an Agile mindset, principles and components of the
DataOps ecosystem, and what to expect from a DataOps tool.
But for an organization looking to build a DataOps team, where do
you start? And what are the major trends expected for the future
that your organization should prepare for?
51
izational structure that formalizes some new functions and resour‐
ces them accordingly.
We begin by identifying the key functions in the DataOps team. If
you’re versed in data management, the functions might sound famil‐
iar, but the DataOps methodology calls for different skill sets and
working processes than have been traditionally employed, just as the
DevOps approach to software development restructured existing
development teams and tooling. The focus on agility and continu‐
ous iteration necessitates more collaboration across these functions
to build and maintain a solid data foundation amid constantly shift‐
ing sources and demands.
Every large company buys lots of products and services to run their
business, and all of them would like to improve the efficiency of
their supplier relationships, so let’s use supplier data as an example,
looking at three of these functions.
Data Supply
Who owns your internal supplier management systems? Who owns
your relationships with external providers or supplier data? The
answer to these questions, typically found in the CIO’s organization,
is the data source supplier, of which you probably have dozens. As
we transition from database views and SQL queries to data virtuali‐
zation and APIs, from entity relationship diagrams (ERDs) and data
dictionaries to schema-less stores and rich data catalogs, the expect‐
ations for discoverability and access to these sources have increased,
but requirements for controlled access to sensitive data remain. In a
DataOps world, these source owners must work together across
departments and with data engineers to build the infrastructure nec‐
essary so that the rest of the business can take advantage of all data.
A great data supplier can confidently say: “Here are all the sources
that contain supplier data.”
Data Preparation
End users of corporate data don’t care about physical sources, they
care about logical entities: in this case, suppliers. So, they need
someone to prepare data for them that combines raw data from
across all available sources, removes duplication, enriches with val‐
uable information, and can be easily accessed. Effective data prepa‐
ration requires a combination of technical skill to wrangle raw
52 | Chapter 7: Embracing DataOps: How to Build a Team and Prepare for Future Trends
sources and business-level understanding of how the data will be
used. DataOps expands traditional preparation beyond the data
engineers who move and transform data from raw sources to marts
or lakes to also include the data stewards and curators responsible
for both the quality and governance of critical data sources that are
ready for analytics and other applications. The CDO is the ultimate
executive owner of the data preparation function, ensuring that data
consumers have access to high-quality, curated data.
A great data preparation team can confidently say: “Here is every‐
thing we know about supplier X.”
Data Consumption
On the “last mile” of the data supply chain, we have everyone
responsible for using unified data for a variety of outcomes across
analytical and operational functions. In our supplier data example,
we have data analysts building dashboards charting aggregate
spending with each supplier, data scientists building inventory opti‐
mization models, and data developers building Supplier 360 portal
pages.
Modern visualization, analysis, and development tools have liber‐
ated these data consumers from some of the constraints of tradi‐
tional business intelligence tools and data marts. However, they still
must work closely with the teams responsible for providing them
with current, clean, and comprehensive datasets. In a DataOps
world, this means providing a feedback loop so that when data
issues are encountered, they aren’t merely corrected in a single dash‐
board but are instead communicated upstream so that actual root
causes can be uncovered and (optimally) corrections can be made
across the entire data community.
A great data consumer can confidently say: “Here are our actual top
10 suppliers by annual spend, now and projected into next year.”
54 | Chapter 7: Embracing DataOps: How to Build a Team and Prepare for Future Trends
The good news: filling these roles is becoming easier. The skill set of
the average data professional is increasing. More college students are
learning data analytics and data science, and more traditional infor‐
mation workers are upskilling to include data analytics and data sci‐
ence in their repertoire.
56 | Chapter 7: Embracing DataOps: How to Build a Team and Prepare for Future Trends
machine learning in cleaner user interfaces, abstracting away the
arcana of their underlying algorithms, we’ll see the barriers to adop‐
tion of these newer techniques lower. These factors combine to form
a powerful dynamic that will accelerate progress in the domain.
A Final Word
Traditional data management techniques are adequate when data‐
sets are static and relatively few. But they break down in environ‐
ments of high volume and complexity. This is largely due to their
top-down, rules-based approaches, which often require significant
manual effort to build and maintain. These approaches are becom‐
ing extinct, quickly.
The future is inevitable—more data, technology advancements, and
an increasing need for curation by subject matter experts. Data uni‐
fication technology will help by connecting and mastering datasets
A Final Word | 57
through the use of human-guided machine learning. The future is
bright for organizations that embrace this new approach.
58 | Chapter 7: Embracing DataOps: How to Build a Team and Prepare for Future Trends
About the Authors
Andy Palmer is cofounder and CEO of Tamr, a data unification
company, which he founded with fellow serial entrepreneur and
2014 Turing Award winner Michael Stonebraker, PhD, adjunct pro‐
fessor at MIT CSAIL; Ihab Ilyas, University of Waterloo; and others.
Previously, Palmer was cofounder and founding CEO of Vertica Sys‐
tems, a pioneering Big Data analytics company (acquired by HP).
During his career as an entrepreneur, Palmer has served as founding
investor, board of directors member, or advisor to more than 50
startup companies in technology, health care, and the life sciences.
He also served as global head of software and data engineering at
Novartis Institutes for BioMedical Research (NIBR) and as a mem‐
ber of the startup team and chief information and administrative
officer at Infinity Pharmaceuticals. Additionally, he has held posi‐
tions at Bowstreet, pcOrder.com, and Trilogy.
Michael Stonebraker is an adjunct professor at MIT CSAIL and a
database pioneer who specializes in database management systems
and data integration. He was awarded the 2014 A.M. Turing Award
(known as the “Nobel Prize of computing”) by the Association for
Computing Machinery for his “fundamental contributions to the
concepts and practices underlying modern database systems as well
as their practical application through nine startup companies that he
has founded.”
Professor Stonebraker has been a pioneer of database research and
technology for more than 40 years, and is the author of scores of
papers in this area. Before joining CSAIL in 2001, he was a professor
of computer science at the University of California Berkeley for 29
years. While at Berkeley, he was the main architect of the INGRES
relational DBMS, the object-relational DBMS POSTGRES, and the
federated data system Mariposa. After joining MIT, he was the prin‐
cipal architect of C-Store (a column store commercialized by Ver‐
tica), H-Store, a main memory OLTP engine (commercialized by
VoltDB), and SciDB (an array engine commercialized by Para‐
digm4). In addition, he has started three other companies in the Big
Data space, including Tamr, oriented toward scalable data integra‐
tion. He also cofounded the Intel Science and Technology Center for
Big Data, based at MIT CSAIL.
Nik Bates-Haus is a technology leader with more than two decades
of experience building data engineering and machine learning tech‐
nology for early-stage companies. Currently, he is a technical lead at
Tamr, a machine learning–based data unification company, where
he leads data engineering, machine learning, and implementation
efforts. Prior to Tamr, he was director of engineering and lead archi‐
tect at Endeca, where he was instrumental in the development of the
search pioneer, which Oracle acquired for $1.1 billion. Previously,
he delivered machine learning and data-integration platforms with
Torrent Systems, Thinking Machines Corporation, and Philips
Research North America. He has a master’s degree in computer sci‐
ence from Columbia University.
Liam Cleary is a technical lead at Tamr, where he leads data engi‐
neering, machine learning, and implementation efforts. Prior to
Tamr, he was a post-doctoral associate at MIT, researching quantum
dissipative systems, before working as an internal consultant at Ab
Initio, a data integration platform. He has a PhD in electrical engi‐
neering from Trinity College Dublin.
Mark Marinelli is a 20-year veteran of enterprise data management
and analytics software. Currently, Mark is the Head of Product at
Tamr, where he drives a product strategy that aligns innovation in
data unification with evolving customer needs. Previously, Mark
held roles in engineering, product management, and technology
strategy at Lucent Technologies, Macrovision, and most recently
Lavastorm, where he was Chief Technology Officer.