Data Governance With Oracle PDF
Data Governance With Oracle PDF
SEPTEMBER 2015
Disclaimer
The following is intended to outline our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any
material, code, or functionality, and should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or functionality described for Oracles products
remains at the sole discretion of Oracle.
Table of Contents
Disclaimer
Introduction
10
11
12
12
16
16
17
17
19
20
22
Introduction
Data governance is the formulation of policy to optimize, secure, and leverage information as an
enterprise asset by aligning the objectives of multiple functions. Data governance programs have
traditionally been focused on people and process. In this whitepaper, we will discuss how key data
governance capabilities are enabled by Oracle Enterprise Metadata Manager (OEMM) and Oracle
Enterprise Data Quality (EDQ). This paper will introduce specific examples of how to leverage OEMM
and EDQ tools within a well-defined data governance process and will also provide specific technical
examples of how to deploy these solutions on Oracle and Intel based engineered systems.
HIPAA
Corporate Governance
GLBA
DPA
Basel III
SB1386
Dodd-Frank
Information Governance
Data Management
Document eDiscovery
Information Governance
International
Data Privacy
Legal / Data
Infrastructure
Marketing
Analytics
Business
Intelligence
Master Data
Management
Data governance is more important today than it has ever been before. While always critical for
Regulatory, Legal and Corporate Governance initiatives, the role of data governance has expanded to
become a foundation for driving Business Growth initiatives as well. Since 2010, the industry focus on
Analytics, Business Intelligence, and Big Data programs has elevated the impact of data governance
by driving requirement to make data easier to find, associate data with applications and business line
growth programs, and improve the confidence and trust that data is accurate enough for business
decision-making.
For the remainder of this paper well take a look at a pragmatic step-by-step approach you can use in
order to get started with an operational data governance program using Oracle tools.
1. Define the Business Problem
2. Identify the Business and Executive Sponsors
3. Manage and Classify the Business Terms
4. Identify the Critical Data Elements for the Business
5. Setup Policies, Rules and Allowable Values
6. View Metadata Lineage: Business and Technical
7. Manage Stewardship Process and Workflows
8. Integrating Glossary and Metadata from Big Data and NoSQL
After we explain the steps for using the Oracle Enterprise Metadata Management (OEMM) and Oracle
Enterprise Data Quality (EDQ) tools with a data governance process, we will also specifically explain
how these solutions can fit together with other Oracle tools such as Oracle Data Integrator, Oracle
GoldenGate, Oracle Data Relationship Management, and Oracle Master Data Management. Finally,
we will conclude by providing specific guidance on how to deploy OEMM and EDQ into Oracle and
Intel environments such as Exadata, Exalogic, Big Data Appliance, Oracle Public Cloud and Oracle
Managed Cloud.
Lets get started!
1Selling Information Governance to the Business, Sunil Soares (MC Press, 2010)
100,000
5%
5,000
$100
$1,000,000
Table 1: Business benefits from improved ship-to addresses at a medical device manufacturer.
For example, the vice president of credit risk at a bank sponsored a data governance initiative to reduce the amount
of capital that needed to be set aside to meet regulatory requirements. The data governance team profiled the
columns in the collateral database. It found that a number of secured loans actually had null values in the
COLLATERAL_VALUE field. For example, the bank might have approved a loan for $10,000 against an automobile
with a resale value of $20,000. However, due to clerical errors, the value of the collateral was never entered into the
COLLATERAL_VALUE field. As a result, the bank had to consider the loan to be unsecured, which meant that it had
to assign a higher level of regulatory capital than if the loan were fully secured. Once the data governance program
identified the list of loans with data quality issues, it was able to engage the operations team to enter the correct
value of the collateral. As a result, the bank was able to reduce its regulatory capital by five percent with respect to
those loans.
In another example, a European telecommunications operator had first-hand experience with inconsistent business
terms when different functional areas within the organization could not agree on a consistent definition for the term
active subscriber. For example, the billing department defined an active subscriber as someone who had received
a bill in the previous 30 days. The network department defined an active subscriber as someone who had used the
carriers network in the previous 30 days. The key difference between these two definitions related to subscribers
2 Selling Information Governance to the Business, Sunil Soares (MC Press, 2010)
who had signed up for service but who had switched SIM cards and were roaming outside the carriers network on a
full-time basis. The data governance team was able to establish a consistent set of definitions in this case. It got
people out of their silos to share information and see the broader business concepts and drivers. The data
governance organization then re-used these common definitions across others parts of the organization as well.
Figure 2: OEMM business glossary contains the definition for the term registration type.
Another reason that organizations will frequently consider tools rather than spreadsheets is that audit requirements
demand to be able to track and report the history of changes to business terms that are used to drive reporting and
operational decisions. For example, OEMM maintains a change log of data steward activities on the business
objects. Beyond simple audit ability requirements, this type of tool capability is absolutely crucial for maintaining and
verifying the organizational and staff accountability of specific KPIs, reports, or reference data. Among the most
important of all data objects to maintain accountability and audit ability for are Critical Data Elements.
identify these and focus efforts on supporting the sustainability and the effective use of these assets. Data stewards
often have a significant role in identifying CDEs. Figure 3 shows that registration type has been flagged as a CDE in
OEMM.
The data governance tool should also support the mapping of business terms to the associated business rules. As
shown in Figure 6, the data steward links the Registration Type business term to business rule BR-001 in OEMM.
Documenting the business rules in this manner is the first step in enforcing compliance with business processes
when data is gathered in upstream applications. In this case, the upstream application should implement a data
validation rule that driver license, national ID and passport are the only acceptable forms of evidence for
registrations.
For larger and more complex business rule sets, data stewards can create a new domain in OEMM and associate
those terms in relation to it. This way, youll have a hierarchy that is more searchable and traceable. Additionally, this
will restrict the user to select only from the available domains. Within OEMM, this approach will also leverage the
mapping-specification functionality, which is the ability to semantically map those terms to actual physical attributes;
this way, one is able to make a semantic analysis for a specific physical attribute (or entity), and make use of the
definitions and business rules mapped to it.
2.
3.
Business lineage
A business user can view the lineage from business terms and CDEs to report semantics. For example, the data
steward should be able to use OEMM to trace the Profit per Customer metric in the profitability report to the
definitions of profit and customer. The definition of profit must also be traced to the business terms representing
individual line items in the companys profit and loss statement. This same functionality can be used to show where
terms and rules are used across many business reports. For example, given a term in the repository, show all business
reports which use a data element classified by (semantically linked to) that term in the glossary. The business user
may search for terms in the glossary. Once they find what they are interested it, one click shows all the reports which
have fields that relate to the term by (a) direct Glossary to Report mappings and (b) Glossary to Data Element
classification where that data element is then connected to the report field by pass-through (or very simple) data flow
4.
Figure 10 shows a new business term called SAMP_REVENUE_F that is in draft status in OEMM.
As shown in Figure 11, the OEMM workflow routes the business term to the data steward who adds an approval.
Figure 11: Data steward approves the business term as part of an OEMM workflow.
https://www.linkedin.com/pulse/oracle-metadata-management-omm-business-glossary-issam-hijazi
As shown in Figure 12, the status of the business term has been changed to approve in OEMM.
Since Hadoop is very often used to store raw data that is by definition ungoverned, a critical first step to
governance is cataloging the data. OEMM provides the metadata harvesting capabilities to harvest metadata from
Apache, Cloudera, Hortonworks and MapR Hadoop distributions. Metadata may be harvested from HDFS, Hive,
Impala and HBase among others.
Other governance functionality such as data profiling, data cleansing, and data enrichment are covered by other
Oracle tools such as Enterprise Data Quality and Big Data Preparation.
In similar fashion, the data steward created two additional data quality rules in OEMM associated with an invalid
character check for Last Name and a valid email address format in the Email Address field.
In Figure 16, the data steward then associates First Name with the No Data Check processor in EDQ.
Figure 16: Associate First Name with No Data Check processor in EDQ.
In Figure 17, the data steward then associates Last Name with the Invalid Character Check processor in EDQ.
Figure 17: Associate Last Name with Invalid Character Check processor in EDQ
Finally, the data steward associates Email Address with the pre-built Email Check processor in EDQ as shown in
Figure 18.
Figure 18: Associate Email Address with Email Check processor in EDQ.
The data steward executes the processes in EDQ. As shown in Figure 19, the Invalid Character Check processor
shows that 4,362 records and 278 records have valid and invalid last names respectively.
Figure 19: View results of Invalid Character Check on Last Name in EDQ.
As discussed so far, data governance involves a number of processes and artifacts including business glossaries,
CDEs, alignment with information security, business rules, allowable values, business and technical lineage, data
stewardship workflows, Big Data Governance, data quality rules, data quality dashboards and data issue resolution.
OEMM and EDQ provide a robust set of software tools to support the creation of these data artifacts as part of the
data governance process.
roles and responsibilities of the different organizations involved. This interconnection of people, process and
technology can requires some dedication and focus, but when implemented successfully, can deliver some
significant business benefits.
The following best practices illustrate how these different elements come together in a successful data governance
program:
There is no right or wrong way to organize around and implement data governance and whole books can, and have,
been written on the subject. Each enterprise needs to consider its own unique circumstances before deciding on the
approach that works best in its environment. Significant factors may include industry, geography, history, culture
and, of course, the specific business objectives at hand. However it is clear that to achieve the speed, scalability and
breadth required of most data governance programs, the selection and incorporation of appropriate software
technology will be an essential foundational element.
Adaptive
Altova
Apache HCatalog
Apache Hive/HQL
Apache HDFS
Borland
CA ERwin
Cloudera Impala
Cloudera Apache
COBOL Copybook
DataStax
Embarcadero
EMC ProActivity
GentleWare
Google BigQuery
Grandite
Hadapt Hive
Hortonworks Hive
IBM Cognos
IBM DB2
IBM DataStage
IBM Discovery
IBM Federation
Server
IBM Lotus Notes
IBM Netezza
IBM Rational Rose
CoSORT
ISO SQL Standard
(DDL)
MapR Hadoop Hive
MicroFocus
Microsoft Access
Microsoft Office
Excel
Microsoft Visio
Microsoft SQL
Server
Microsoft SSIS
Microsoft Visual
Studio
Microstrategy
Magic Draw
OMG CWM
Standard
OMG UML Standard
Oracle BI Answers
Oracle BI Enterprise
Edition
Oracle BI Server
Oracle DAC
Oracle Data
Integrator
Oracle Data Modeler
QlikView
SAP BO Crystal
Reports
SAP BO
Designer
SAP BO Desktop
Intelligence
SAP BO
Repository
SAP BO Data
Integrator
SAP BO Data
Steward
SAP Master Data
Management
SAP Sybase
PowerDesigner
SAP Sybase ASE
Database
SAS Data
Integration Studio
SAS BI Server
SAS Information
Map
SAS Metadata
Management
SAS OLAP
IBM Rational
Architect
Informatica
Metadata Manager
Informatica
PowerCenter
Informatica
Developer
Oracle Database
Oracle Designer
Oracle Hyperion
Applications
Oracle Hyperion
Essbase
Oracle Warehouse
Builder
Pivotal Greenplum
PostgreSQL
Server
Select
Sparx Architect
Syncsort
Tableau
Talend
Teradata
Tigris
Visible
W3C DTD & XSD
Schema
Looking beyond the vast number of metadata harvesting bridges that are directly supported by and compatible with
Oracle Enterprise Metadata Management it also becomes necessary to support a rich import/export API for bringing
glossary and technical metadata into the governance environment. The OEMM tool provides an exceptionally rich
Excel/XLS formatted means to import complex metadata and begin governing metadata from any system even
those that are not formally certified with the Oracle tools.
Oracle Data Integrator (ODI) ODI is a data integration tool for executing data transformations where the
data resides. Frequently during data transformations it is necessary to execute data quality rules, and ODI
is integrated with EDQ both via the ODI Studio user interface, as well as via the runtime execution of the
data flows. Additionally, ODI metadata is fully integrated with OEMM so that all data transformation logic
and data movement activity can be documented as part of an end-to-end data lineage.
Oracle GoldenGate (GG) GG is a real-time replication tool for capturing and moving data transactions
with high speed and reliability. Typically data cleansing operations would be performed at the application
layer or in batch on the data warehouse, but integration with EDQ is possible using the GG user exits.
Additionally, GG replication data flows may be included in OEMM for visually depicting data lineage along
with other data sources.
Oracle Data Relationship Management (DRM) DRM is used to manage changes in reference data and
business hierarchies across operational systems. EDQ can leverage DRM reference data as input to EDQ
cleansing rules via lookup tables and/or via application APIs (e.g., MDM or CRM). DRM reference data
may also be exchanged with OEMM via import/export formats as a way of seeding business taxonomies in
either tool.
Oracle Master Data Management (MDM) MDM tools are used to govern customer, product or citizen
data records for operational and analytic reporting. EDQ is pre-integrated with Oracle MDM via a pre-built
connector that allows EDQ to provide the name and address cleansing features for MDM. OEMM can be
leveraged with Oracle MDM as a way to externalize and govern the CDEs of MDM attributes from directly
within the OEMM Business Glossary.
Oracle has always been committed to an Engineered Systems strategy that aims to make it easier to use Oracle
products together; this is part of the core investment philosophy in the data governance area as well.
Bare Metal Deployments both OEMM and EDQ support software installations on Intel-based Windows
and Linux platform servers from commodity hardware providers.
Oracle Exalogic Deployment Oracle Exalogic Elastic Cloud is hardware and software engineered
together to provide extreme performance, reliability, and scalability. It runs both Intel x86 and Solaris
chipsets, and works with Oracle, Java, and other applications, to deliver low total cost of ownership,
reduced risk, higher user productivity, and one-stop support. Both OEMM and EDQ runtime processes can
run directly on the Exalogic system.
Oracle Exadata Deployment Oracle Exadata is the highest performing and most available platform for
running Oracle Database. Its Intel x86 and Solaris architecture features scale-out industry-standard
database servers and intelligent storage servers, and a high-speed InfiniBand internal fabric that connects
all servers and storage. Both OEMM and EDQ are certified to use an Exadata host for their internal
database repository requirements.
Oracle Big Data Appliance Deployment Oracle Big Data Appliance is an integrated, optimized, and
tuned Hadoop environment running Intel x86 architecture to rapidly deliver comprehensive and secure big
data capabilities to enterprises. The appliance offers dramatically shortened deployment times and
reduced risk result in an overall lower total cost of ownership. OEMM is certified to harvest metadata from
the Cloudera Hadoop instance running on the Big Data Appliance and EDQ may be used in combination
with Oracle Data Integrator to cleanse any data being used in the big data environment.
Oracle Managed Cloud Deployments Oracle delivers dedicated managed cloud services across its
broad portfolio of business applications, middleware, database, and hardware. Oracle Managed Cloud
Services are complete, secure, and customized to fit specific customer business requirements. Both
OEMM and EDQ may be deployed, hosted and managed within the Oracle Managed Cloud environment.
Oracle (Public) Cloud Deployment Oracle Cloud is the industry's broadest and most integrated public
cloud, offering best-in-class services across software as a service (SaaS), platform as a service (PaaS),
infrastructure as a service (IaaS), and data as a service (DaaS). Oracle Enterprise Data Quality is certified
to be deployed, hosted and managed within the Oracle Cloud environment as part of the Oracle Java
Cloud Service.
CONNECT WITH US
blogs.oracle.com/oracle
facebook.com/oracle
Worldwide Inquiries
Phone: +1.650.506.7000
Fax: +1.650.506.7200
Copyright 2015, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the
contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
means, electronic or mechanical, for any purpose, without our prior written permission.
twitter.com/oracle
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
oracle.com
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license
and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo
are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0615
Data Governance with Oracle
September 2015
Author: Sunil Soares
Contributing Authors: Jeff Pollock