From Data Lake To Data-Driven Organization
From Data Lake To Data-Driven Organization
Mandy Chessell
Ferd Scheepers
Maryna Strelchuk
Ron van der Starre
Seth Dobrin
Daniel Hernandez
Redguide
Executive overview
In 2014, IBM® and ING published the IBM Redguide™ publication Governing and Managing
Big Data for Analytics and Decision Makers, REDP-5120. That publication laid out a vision of
a governed data lake (referred to as the data reservoir at that time) and put the concept of a
metadata and governance catalog at the heart of the data lake. The catalog controls the
engines that manage the data within the data lake. It also defines the visibility and access that
people and applications have to this data.
Further publications include Designing and Operating a Data Reservoir, SG24-8274, and the
online blog, Building a data reservoir to use big data with confidence, both of which provide
more details about how a governed data lake is built and operated.
Since then the concepts and design patterns have been successfully adopted by many
organizations from different industries.
This new guide looks back on the key decisions that made the data lake successful and looks
forward to the future. It proposes that the metadata management and governance
approaches developed for the data lake can be adopted more broadly to increase the value
that an organization gets from its data. Delivering this broader vision, however, requires a new
generation of data catalogs and governance tools built on open standards that are adopted by
a multivendor ecosystem of data platforms and tools.
Work is already underway to define and deliver this capability, and there are multiple ways to
engage. This guide covers the reasons why this new capability is critical for modern
businesses and how you can get value from it.
This guide covers details of ING’s incredible transformation, driven by the idea that the way to
deliver the best customer experience is to align the business around the data that supports
them.
This is not just a single company’s story. IBM is itself transforming to become data centric and
is also working with visionary organizations from many industries on a similar journey.
Despite the different industry processes and regulations and the variability in the types of
data requiring focus, we have all found similar challenges and solutions. Thus, this guide is
representative of many organization’s experiences but told through the eyes of a global
organization involved in its own transformation, and a technology and business innovation
company supporting them.
The data lake consolidates an organization’s data into a governed and well-managed
environment that supports both analytics development and production workloads. It
embraces multiple data platforms, such as relational data warehouses, Apache Hadoop
clusters, and analytical appliances, and manages them together through a common
governance program. These data platforms can be distributed geographically. Access to the
data platforms is restricted to the data lake services and the engines that manage the data.
Applications and people access the data through the data lake services.
The data lake allows organizations to innovate with data in a safe and properly governed way.
For the IT teams, the data lake provides the following opportunities:
Complexity reduction: Over the years the analytical landscape might have become
complex with numerous data warehouses and data marts with complex sets of interfaces.
To provide agility and flexibility, these environments need to be aligned and made more
consistent.
Cost efficiency: As IT budgets came under increasing pressure, complex IT landscapes
need to become more efficient and cheaper to run and maintain.
Transparency: An ever-increasing regulatory pressure required a new approach on how to
manage data and demanded an analytical platform that had governance by design.
New sources of data: Existing structures of data were not ready for new semi- and
unstructured data sources. Heterogeneous information virtualization is required to
provisioning data in a simple way to the consumer.
3
Figure 2 shows the major groupings of data lake services.
The catalog data lake service is the heart of the data lake controlling what data people can
find and access and controlling the processing of the various engines operating inside the
data lake. The catalog consists of the following tightly integrated types of metadata:
Governance metadata: Defines the governance program and the glossaries of business
terminology that describes the types of data held and used by the organization.
Technical metadata: Provides the inventory of the data assets of the organization. These
data assets are used by numerous run times, such as applications, data movement and
transformation engines, and databases and reporting platforms.
Operational metadata: Provides transparency on the operation of the information supply
chains as they copy data between the systems and data platforms, which is often referred
to as lineage.
The Enterprise IT Data Exchange services enable data to flow in and out of the data through
both batch and real-time interfaces. The data lake is a hub—not a data pit where data is
thrown, never to be returned.
Finally many different types of people need self-service access to the data in the data lake.
We typically divide these types of people into two broad groups. The data scientists and
business analysts are building new analytics and executable rules that will be deployed into
the production systems. They need access to raw data, just as it appears in the production
systems so that they can produce analytics that work on real data. Other users tend to need
data that has had some level of processing to make it simple to use in different tools. Thus,
the data lake has two different self-service access points. The access points determine the
scope of the data that the person can see. The metadata in the catalog determines exactly
what a specific individual is allowed to see. The self-service access points enforce these
restrictions.
Why is it so challenging for companies to become data driven and to obtain value from their
data?
The complexity of the IT landscape in many organizations is greater than any individual can
understand. Almost any process in a given large-scale organization depends on dozens or
more applications.
5
Figure 3 shows an example of the ING systems in use at the start of the data lake project for
a single process in a single country.
Often these applications include a mix of purchased applications that are heavily modified to
fit into the existing landscape and self-developed applications that have a history spanning
decades. These application typically have their own internal data model, their own way of
defining their interfaces, and their own set of experts.
A good example is the definition of a customer. For different domains, a customer can be
defined differently. There is the obvious difference between a wholesale or retail customer but
also regarding the moment somebody actually moves from being a prospect to a customer.
Also consider does a family count as one customer or multiple customers? Must a customer
be active (that is, have at least x products or transactions)? There are many other reasons the
definition of a customer might not be the same throughout the system. However, after you
start to exchange customer data between systems, the mismatch in definitions is not always
known or understood. It can often lead to big issues in integration tests, bad data quality, and
inconsistent data sent to regulators.
With this complex landscape in mind, it is good to think about the challenges you might face
on top of just maintaining the existing complexity of the architecture.
The big data hype has led to more data that is less understood
Even though big data is no longer the buzzword it was 5 years ago, big data is more of a
reality in most companies today. The amount of data that companies generate and gather has
gone up, and many companies have built big (Hadoop) clusters, both for exploration purposes
and for production use cases. Many companies have a group of data scientists analyzing the
data, understanding the many data sets they receive (often via emails), and examining ad hoc
deliveries of all kinds of data. Some estimates suggest data scientists spend over 70% of their
time on data preparation and making sense of data and less than 25% of their time on what
they really need to focus on, which is creating new analytic models. And which company truly
knows what data the data scientists have access to? Most data science occurs in a private
sandbox, with data that is not under control or well understood.
Several years ago it was BCBS239, mainly a challenge for financial institutions. Now GDPR
demands that every company with any European citizen’s data must implement clear
measures that control this data. And many more of such regulations will surely be coming.
7
Creating the data lake architecture was a major step in our journey
ING realized this dilemma early on. The starting point for our data lake journey was the need
to create an overall architecture that addressed simplifying the existing data landscape and
enabled the change to a data-driven organization by giving access to data for all employees.
But creating an architecture that is new and is not based on a reference that has already
proved its merit is potentially high risk. Nobody wants to end up with a data architecture that is
not in line with where the market is moving, that is not supported by technology in the market,
and that leads to building everything yourself.
To avoid these issues, we decided to develop the architecture together with IBM, using the
knowledge of IBM to challenge our thinking and to make sure we ended up with an
architecture that can be mapped on real technology products but in such a way that the
architecture itself is vendor agnostic. And from day one, we decided to publish our common
architecture openly and not keep it to ourselves—again to make sure that we created
something that is seen as a preferred practice, that can be adopted by others, and that can
help steer the direction of the market.
Simply creating information and system architectural slides for the business stakeholders isn’t
always successful. It is essential to translate the story into the language of the stakeholder to
make easier to understand and sell. We have found that using the analogy of a library, where
data is represented by books, works well (as illustrated in Figure 4). The need to find your
book using a catalog, the need to have common definitions to find information about the book,
how to explain metadata using the analogy of a library card, all represent common methods
that business stakeholders can relate to complex architectures.
Another challenge is the scale of the problem and, thus, the scale of the solution. It is a
journey of many years, and the real benefits come after an investment of both time and
money. Helping stakeholders to understand that this is not a quick journey that will pay off in a
few months or even a year is critical. The low hanging fruits are limited, and the impact on the
organization is huge. But so are the benefits.
Building and rolling out the ING Data Lakes as a multi-year journey
With an agreed architecture and budget, the longest part of the journey started—building the
ING Data Lakes and rolling them out. To prove each key aspect of the data lake, a technology
or concept was delivered first. People understand technology much better when they can see
it running and experiment with it. The data lake team set up a regular delivery of
demonstrations of the following different aspects of the data lake:
The catalog as central data management point of the data lake
The definition of glossary terms and how they relate to data assets
Lineage across the different Data Lake technologies
Lineage for data flowing between data lakes
Support for ING specific metadata, such as “in the Data Lake” indicator
Support for masking in ingestion jobs based on confidentiality data classification level
The technology proof of concept made one thing clear to the organization. The benefits of
using a governed data lake with a central catalog depends strongly on a consistent
implementation throughout all domains and countries. Using different technologies and
standards would make it impossible to have a single metadata and governance view and
lineage view.
To make sure we created this consistent data lake throughout all countries and delivered our
first value quickly, we made the following decisions:
Scope the initial delivery down to what we called the Data Lake Foundation (DLF). The
initial delivery included the minimal foundational components of the data lake that we
needed for governance.
Build this DLF together with a team of engineers from multiple domains and countries.
This team ensured that all the knowledge of ING worldwide was brought together and
created buy in from everyone in the organization from day one.
In our internal sales pitch, the delivery of this team was called the Data lake CD, effectively
the basic data lake technology on a CD. This technology embedded the governance rules that
were based on metadata settings in the catalog.
Each country took the CD and deployed their data lake from it, supported by a central team
that travelled to all countries to help with the rollout. Specific places in the ingestion jobs
allowed a country to customize the software, and these customizations where checked. The
result was a consistent rollout of a data lake in each country, and a consistent rollout for the
group. The country data lakes feed a subset of data to the group data lake, this subset
determined by the (growing) demand from group level, initially contained the data that was
necessary for the Financial and Regulatory Reporting.
9
of individual point-to-point connections per system is being reduced, and we can indeed show
lineage across the entire chain from system of record (SoR) to the reports.
A second benefit we have already seen is a huge increase in the speed of delivery for
changes to reports. As we have all the data in the data lakes, changes that in the past took
many months to implement can now be implemented in days.
For existing systems that already are connected to the data lakes, having all the data
available outside of the SoR makes decommissioning these systems also a lot easier. This
process is now something we are using to facilitate a faster migration to the target landscape.
An added benefit is that we are building common platform enabled teams from different
countries to collaborate and share technology, data definitions, and glossaries. This
collaboration formed spontaneously as the teams recognized their shared challenges.
A first important deliverable of this new Global Data Management organization was to
establish a common language for ING, something we call the ING Esperanto. Previously,
when we exchanged data between systems, information received from the sender in the
original language had to be translated by the receiving party to be further consumed. This
process was not a scalable solution, because each country would need to “know” multiple
languages. In addition, assuring consistency and reliability was a challenge. Understanding
these issues led to the realization that we needed to have single data-exchange language
(ING Esperanto). ING Esperanto is a global glossary that describes the business terms that
we use frequently within entities and creates one consistent language throughout ING.
Switching to one common language for all of ING’s countries and entities from day one would
not be a feasible task. Instead, it was decided to focus on key terms and definitions that must
be consistent throughout multiple locations, different divisions, and systems. This way, each
ING country or entity can use their local language and configuration on local systems and, at
the same time, use ING Esperanto for the information that must be shared throughout the
entities.
Using ING Esperanto addressed the data-exchange problem and also helped to ensure that
we understand the data, have a single source of truth, and can guarantee that reporting is
accurate, consistent, and timely.
Starting with a set of common business terms, the ING Esperanto now is much more. We
have a glossary of more than 1000 agreed upon business terms, a logical data model of
these terms and their relationships, and a (flattened) physical data model to exchange data
between the data lakes. Currently, we are working on a physical data model for the
information warehouse in the data lake, and we are using the same definitions also to
structure our APIs and events.
Today, the data lakes are a key part of delivering ING’s data-centric strategy, but there is more
to do. Customer, product, and payment information used in the bank’s day-to-day operation
must be trusted if the bank is to respond rapidly to requests and opportunities. Building on the
success of the data lake, ING’s enterprise architecture team is expanding the use of the
catalog and the governance program to include all key operational data and processes. This
expansion is again raising the bar on ING’s governance and data management capability.
An organization’s attitude to data management and its competence to execute are often
plainly visible to its customers and business partners, because poor data management
manifests itself as lost orders, incorrect payments, and inconsistent customer service. An
increasing use of digital services makes data management a front-line capability and the
information governance program a critical driver of an organization’s success.
In fact this focus on value begins with the data strategy. Its alignment with the business
strategy ensures the correct focus is given to the types of data that is targeted along with the
people and processes that are affected. Regulations affecting the organization often require
accurate reporting of the business activity or a demonstration that a particular type of data is
being managed and used appropriately. So they are often relevant to the governance
program. These regulations along with the data strategy combine to define the governance
requirements.
Aligning the data strategy with the regulatory requirements creates synergies rather than
conflicts between the cost of doing business and the desire to deliver value. These synergies
are reflected in the following common types of governance responses:
Governance principles that guide the way that data should be managed
Governance obligations that define the regulatory and corporate rules around managing
and using data
Governance approaches that provide the stake-in-ground decisions around specific best
practices, tools, and related methods that are agreed upon across the organization
Governance responses define “what” the organization is going to do. Governance controls
and measures then start to flesh out the “how.” They define which types of processing, rules,
data collection, and procedures need to be performed on the organization’s data. Often they
are divided into technical and organizational controls. Technical controls are implemented by
technology, and organizational controls are implemented through staff roles, culture, and
procedures.
For effectiveness, the governance controls are expressed in terms of the organization’s data
classifications. Each data classification describes a well-defined characteristic of the
organization’s data. Often there are classifications that describe how confidential a data set
is, the different levels of quality or confidence that should be given to the data, any legal
retention requirements, and the subject area that the data set describes. For example,
consider a data classification called “sensitive personal data,” which describes a data
classification for particularly private data about an individual. The governance controls then
describes how sensitive personal data is managed in different situations. For example, there
might be a control that states that sensitive personal information must be encrypted if copied
11
to a portable storage device and another control that says only individuals with a specific
business need must be given access to the data.
The governance controls identify where change is necessary in the IT systems and
organization. They help to scope implementation projects that encode the governance
program requirements into the roles, processes and technology across the organization.
Figure 5 summarizes this structure and the concepts typically used by governance leaders. It
is drawn as a pyramid to reflect the increasing scope and cost of the governance program as
it rolls out over the organization. Arrows show a natural flow downwards, although a
governance program is built out iteratively and relies on feedback from the more detailed
activities to tune its operation to maximize business value.
Many organizations have an information governance program, typically around their data
warehouse. This program is often experienced by the business as restrictive and bureaucratic
because the data warehouse needs consistent and accurate data from processes that feed
Only a mandatory regulation enables such an asymmetric distribution of cost and benefit in
an organization. If an organization is to be truly data centric, the balance must be restored so
that activities around data occur throughout the organization, and any investment a team
needs to make in managing data is returned to them through increased efficiency and
capability to their operations.
Forrester characterizes this type of operation as data citizenship.1 Their vision describes the
types of job roles that people in a data-centric organization perform and the way their work
delivers value to the organization. For many organizations, although they might not use the
term “data citizenship,” the capabilities that Forrester describes are closely aligned with their
vision. Enabling data citizenship requires a focus on data that challenges a traditional siloed,
process-centric operation. It also changes the way IT systems are developed and operated.
To understand this concept, it is necessary to look behind the tools that support the new data
roles to the IT technology and the processes that feed them.
From a technical perspective, data is formatted and stored in different ways, and that affects
the type of technology needed to manage it. For example, we talk about structured data that
is stored in databases and unstructured data (text, documents, and videos) that is stored in
file systems and document stores.
Many IT organizations are structured around the types of technology they support. The data
for a single subject area is typically distributed across this technology.
This complexity presents challenges for each phase of the governance rollout, both
technically and organizationally. Each move to broaden scope of the program—either by
bringing in new parts of the organization or adding new types of data—typically expands the
list of technology and cultures that need to be integrated.
1
Brief: Establish A Data Governance Journey Toward Data Citizenship”, 27th April 2016,
https://www.forrester.com/report/Brief+Establish+A+Data+Governance+Journey+Toward+Data+Citizenship/-
/E-RES132243
13
Governance technology underpinning the maturity model
Governance brings its own technology into the IT landscape. This technology is typically
coordinating the collection, management, and distribution of metadata. There are also new
tools that support the people with specific responsibilities relating to the management of the
data landscape itself. We characterize the technology and the associated activity into the
following levels of maturity:
Cataloging data.
Defining the governance requirements and linking them to the data descriptions in the
catalog.
Automating the governance requirements in the operational environment.
Enabling the business to manage the governance requirements for data and any changes
are reflected automatically in the operational environment.
Supporting data citizenship roles across the organization.
Each of these levels is covered in more detail in the sections that follow.
Initially the details recorded about each data store are simple, covering its name, location,
short description, and owner. Ideally this cataloging is automated in the data stores and
engines that create the data stores.
Next is a process of looking at the data itself and recording details, such as the structure of
the data and any characteristics of the data itself. This process is called metadata discovery
and is another area where automation is both possible and desirable. Metadata discovery can
also include an initial assessment of the quality of the data. For example, are all of the values
completed and do they match the agreed valid values.
The final stage of cataloging is classification, which needs human input. Typically the owner of
the data or a subject matter expert needs to classify the data in the data store with glossary
terms that define the data element’s meaning and the data classifications that are used in the
definition of the governance controls. Analytical functions can help to identify the logical type
and candidate classifications for the data, but the human expert is needed to confirm the
classification is correct based on their understanding of the business context around the data.
Governance definitions should be created by the governance team through tools that are
linked to the metadata repository that hosts the catalog of data stores.
Control points are places where a decision or judgment needs to be made to resolve an
anomaly or error in the data. A person with appropriate responsibility chooses the best course
of action and their decision is recorded in an audit log, potentially approved by a colleague
and then actioned.
Automation in the operational landscape provides greater coverage and consistency in the
implementation. There are many runtime engines that can be configured to execute
governance rules at particular points in the processing, supporting both the enforcement
points and verification points. These engines include security tools, data movement engines,
and data access APIs. Workflow tools support the automation of exception management and
control points.
Enterprise-wide programs to use common platforms and solution platforms can reduce the
cost of implementing operational governance. Nevertheless, it requires changes to the way IT
solutions are budgeted for and rolled out to ensure the operational governance
implementations are included.
Many business run their digital services continuously, so the metadata for operational
governance must also be continuously available. Digital service need to execute efficiently so
that they cannot make multiple calls to a centralized metadata server, even if it was
guaranteed to always be available.
This process means that the catalog and access to data needs to be a part of the toolset that
is available to all employees. Appropriate governance controls can enforce data protection
15
and auditing for ad hoc data requests and usage. The organization has a culture of data
custodianship and ownership, and individual employees are trained in appropriate use and
governance of data.
This process also requires that most digital services and tools are used by the organization to
make use of the same metadata to locate, access, and govern data. The challenges for an
organization to reach this level of maturity are similar to level 4, but they are magnified by the
increased variety of technology that needs to support the shared metadata.
Each maturity level also expands the number of people who need to be a part of the
governance program. This growth takes influence and leadership to make the necessary
organizational changes.
Making this work requires a common set of APIs, protocols, and message formats for
technology to implement.
Paper documents of this integrated standard are necessary but not sufficient. Organizations
need working software to deploy immediately.
A proprietary solution from a single vendor is not going to get the industry traction, but open
source, with an open, commercially friendly license can provide the base for open,
data-centric technology.
The open software for metadata must be deployable into a wide range of operational
environment from IoT solutions that operate in remote locations and physical assets, such as
cars, to edge servers and cloud platforms and across the wide range of technology already
deploying in enterprises today. It must not rely on a central point of control or central server for
metadata. The protocol must be peer-to-peer, supporting the specific needs of a local
community of data users and their use cases, whilst embracing the greater needs of the
enterprise. Finally it must accept that there is a wide variety of technology that must play in
this ecosystem, many from different, competing vendors. It must allow market differentiation
for these vendors and innovation whilst ensuring metadata interoperability.
A challenging set of requirements, but they form the basis of our strategy for open metadata
and governance.
17
Figure 7 shows the integration of these three metadata repositories through the open
metadata technology. Each repository has a deployment of the OMRS, which might be
embedded in the repository’s server or can run as a repository proxy along side the
metadata repository. The deployed OMRS components communicate through a topic to
exchange details of each other’s capabilities and REST API network address.
After the registration process is complete, each OMRS can issue queries across all of the
repositories through a single request. In the background, the metadata repositories are
also exchanging copies of some metadata to ensure a copy of the metadata used
frequently by their local users is stored locally. This exchange of metadata is designed to
improve performance and availability of metadata for all users.
The Open Metadata Access Services (OMAS)
The OMAS provides specialized APIs and events for data tools, data processing engines,
and governance tools. For example, there is a specific OMAS for the catalog, another for
data movement tools, another for data science tools, and another for security enforcement
points. Each OMAS has a REST API and an event exchange protocol for asynchronous
integration.
These services simplify the access to the broad range of technologies needed to support
the higher levels of governance maturity needed by a data centric organization.
Each OMAS retrieves and stores open metadata through the OMRS so it has access to all
connected metadata repositories. The OMRS can be running in its own server (called a
repository proxy) or embedded in a vendor’s metadata server.
Many organizations are decentralized and siloed to allow them to scale. The OMRS
enables a server to connect to multiple cohorts. Thus, separate cohorts are running in
decentralized parts of the organization and can be linked to create an enterprise view for,
say, the enterprise Chief Data Office.
Figure 9 shows the metadata repository for the Chief Data Office linked to two
cohorts—one for each division of the enterprise. The OMAS services for the Chief Data
Office can potentially access all metadata from both divisions. Whereas the other
deployed metadata repositories can see metadata only from their own cohort.
19
Future developments
The OMRS and OMAS focus on providing access to open metadata. After they are complete,
the next phase focuses on automated metadata discovery and open governance services,
such as stewardship and exception management.
Both of these work streams help to build the adoption ecosystem around the Apache Atlas
base.
Finally we need the buy-in of the standards organizations that have defined the base
standards that underpin open metadata. Our first set of discussions has been with The Open
Group. Their Open Platform 3.0 standards are closely aligned with the data-oriented solutions
that open metadata and governance is focused on. One of their new standards is particularly
important. The Open Group’s Open Data Element Framework (O-DEF) standard provides a
data naming framework plus numeric identifier for data items based on their meaning. The
resulting O-DEF tag can be used to identify how data elements from different sources map to
one another.
By extending the scope of the defined O-DEF tags to cover the full scope of the open
metadata model we have a standardized way to show how each of the existing open
metadata standards maps to the open metadata data model. The practical value of this work
is in simplifying the effort of building bridging software needed to link tools that use the
existing metadata standards with the wider open metadata ecosystem. However, it also
exposes the overlaps and gaps in existing standards landscape that can lead to further
standards work in the future.
Progress to date
To date we have a significant portion of the open metadata capability coded and available in
Apache Atlas. There is a full-time cross-company team that is focused on its delivery and
maintenance going forward. Through our partnerships and progress in developing the
software in Apache Atlas, the understanding and buy-in to the open metadata and
governance strategy is growing.
IBM, ING, and Hortonworks are key players in all aspects of both Apache Atlas and the ODPi,
and we have significant strategic initiatives built on their success.
Unified governance is distributed to reach all systems that work with data; from the small IoT
monitoring station to the popular global public cloud service. It is also connected into a single
logical platform that enables the catalog to draw from metadata collected in many different
systems and the governance to be optimized and distributed to the data access points and
management engines.
IBM’s unified governance offers the ability to govern data consistently, irrespective of how it is
stored or structured. It supports data in an organization’s data centers and in IBM cloud
services. Its integration and internal use of the open metadata and governance standards
extends its reach beyond IBM software and platforms to encompass all of an organization’s
data.
IBM’s unified governance drives to support organizations as they raise their governance
maturity on the journey to becoming a data driven organization.
Conclusions
A data-driven organization is more agile and responsive to its customer needs. The
transformation from a traditional hierarchical organization to a data-driven organization is on
many CEOs’ agenda, but it is a challenging undertaking, affecting processes, organizational
roles and careers, and long cherished customs and practices.
This guide laid out the strategic approach for open metadata and governance that has
emerged from our experiences in transforming organizations to become data centric. It also
covered the activities in progress to make this strategy a reality.
At this stage, although much of the vision and base capability is in place, there are plenty of
areas that need specific focus and contribution. Open metadata and governance is an
ecosystem that will grow in strength through contribution. It will enable organizations to work
confidently with data, making conscious choices in their use of data and the direction of the
data strategy.
As digital technology transforms every aspect of our lives, data volumes will continue to grow.
Every organization and individual is impacted with both new opportunities and obligations.
Strong information governance is in everyone’s interest but is should not be as hard as it is
today. Open standards embedded in tools and technology platforms that automate the proper
cataloging, classification, and use of data is key to maximizing the value we can get from this
data. We have laid out the road map and started the journey. The door is open for you to join.
2
IBM’s Unified Governance,
https://public.dhe.ibm.com/common/ssi/ecm/im/en/imw14949usen/analytics-analytics-platform-im-white-p
aper-external-imw14949usen-20170719.pdf
21
Next steps
If you are interested in learning more:
Talk to an IBM representative about IBM’s contributions to driving open metadata and
governance through its Unified Governance strategy.
Sign up developers to contribute to the Apache Atlas open source project.
Join the ODPi to:
– Drive the adoption of the open metadata and governance standards by your vendors.
– Work with fellow subject matter experts in developing governance collateral to
accelerate an organization’s transformation to a data-centric organization.
Join The Open Group to influence the open standards related to open metadata and
governance.
Authors
This guide was produced by a team of specialists from around the world working at the
International Technical Support Organization (ITSO).
Mandy Chessell is an IBM Distinguished Engineer, Master Inventor, and Fellow of the Royal
Academy of Engineering. Mandy is a trusted advisor to executives from large organizations,
working with them to develop their strategy and architecture relating to the governance,
integration, and management of information. She is also driving IBM’s strategic move to open
metadata and governance through the Apache Atlas open source project. Mandy is the
leader of the ODPi Data Governance PMC.
Ferd Scheepers is the Chief Information Architect for ING. He has worked for ING since 1995
and in that time has held many different roles, ranging from Lead Architect for Development
Environments, Business Intelligence and Middleware, to Enterprise Architect for Payments,
Transaction Services, and Customer Centricity. In his current role, Ferd is responsible for the
Data and Information Architecture for ING Global.
Maryna Strelchuk is an Information Architect at ING, leading a team of developers who are
building key components in Apache Atlas. She is enrolled on the ING International Talent
Program (ITP) and has experience as a Data Scientist as well as in software development.
Ron van der Starre is an Information Management Architect for the IBM Software Group in
the Netherlands. He has over 20 years of experience in IT with the last 15 years within IBM.
He started within the services organization as an IT Specialist for packaged solutions and
after a couple of years switched to work as a consultant in the area of business process
redesign.
Seth Dobrin is Vice President and Chief Data Officer for IBM Analytics. He is responsible for
internal data, data science, and digital transformation as well as influencing offering and
sharing experiences with clients. Seth is also on the board of the ODPi. Previously he was the
Director of Digital Strategies at Monsanto.
Daniel Hernandez is Vice President and Offering Management leader in IBM Analytics. He
leads a talented team that builds, operates, and continuously improves the products and
services in IBM’s data, unified governance and analytics portfolio.
Find out more about the residency program, browse the residency index, and apply online at:
ibm.com/redbooks/residencies.html
23
24 The Journey Continues: From Data Lake to Data-Driven Organization
Notices
This information was developed for products and services offered in the US. This material might be available
from IBM in other languages. However, you may be required to own a copy of the product or product version in
that language in order to access it.
IBM may not offer the products, services, or features discussed in this document in other countries. Consult
your local IBM representative for information on the products and services currently available in your area. Any
reference to an IBM product, program, or service is not intended to state or imply that only that IBM product,
program, or service may be used. Any functionally equivalent product, program, or service that does not
infringe any IBM intellectual property right may be used instead. However, it is the user’s responsibility to
evaluate and verify the operation of any non-IBM product, program, or service.
IBM may have patents or pending patent applications covering subject matter described in this document. The
furnishing of this document does not grant you any license to these patents. You can send license inquiries, in
writing, to:
IBM Director of Licensing, IBM Corporation, North Castle Drive, MD-NC119, Armonk, NY 10504-1785, US
This information could include technical inaccuracies or typographical errors. Changes are periodically made
to the information herein; these changes will be incorporated in new editions of the publication. IBM may make
improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time
without notice.
Any references in this information to non-IBM websites are provided for convenience only and do not in any
manner serve as an endorsement of those websites. The materials at those websites are not part of the
materials for this IBM product and use of those websites is at your own risk.
IBM may use or distribute any of the information you provide in any way it believes appropriate without
incurring any obligation to you.
The performance data and client examples cited are presented for illustrative purposes only. Actual
performance results may vary depending on specific configurations and operating conditions.
Information concerning non-IBM products was obtained from the suppliers of those products, their published
announcements or other publicly available sources. IBM has not tested those products and cannot confirm the
accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Statements regarding IBM’s future direction or intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
This information contains examples of data and reports used in daily business operations. To illustrate them
as completely as possible, the examples include the names of individuals, companies, brands, and products.
All of these names are fictitious and any similarity to actual people or business enterprises is entirely
coincidental.
COPYRIGHT LICENSE:
This information contains sample application programs in source language, which illustrate programming
techniques on various operating platforms. You may copy, modify, and distribute these sample programs in
any form without payment to IBM, for the purposes of developing, using, marketing or distributing application
programs conforming to the application programming interface for the operating platform for which the sample
programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore,
cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are
provided “AS IS”, without warranty of any kind. IBM shall not be liable for any damages arising out of your use
of the sample programs.
The following terms are trademarks or registered trademarks of International Business Machines Corporation,
and might also be trademarks or registered trademarks in other countries.
Redbooks (logo) ® Redbooks®
IBM® Redguide™
Other company, product, or service names may be trademarks or service marks of others.
REDP-5486-00
ISBN 0738456667
Printed in U.S.A.
®
ibm.com/redbooks