Data Vault Modeling Guide
Data Vault Modeling Guide
F o rward
Data Vault modeling is most compelling when applied to an enterprise data warehouse program (EDW). Several key decisions concerning the type of program, related projects, and the scope of the broader initiative are then answered by this designation. In short, the organization contemplating this initiative is committing to an integrated, non-volatile, time variant and business key driven data warehouse program. The data vault principles are specifically well suited for such a program and when applied consistently can provide the organization with some very compelling benefits. These include auditability, agility, adaptability, alignment with the business, and support for operational data warehousing initiatives. To gain these benefits however, the organization will need to commit to both EDW program level factors as well as specific data vault modeling patterns, rules and methods. This guide presents data vault modeling in the context of the EDW.
DATA VAULT MODELING GUIDE | 5/15/2012
I nde x
FORWARD................................................................................................................................1 INDEX .....................................................................................................................................2 THE EDW P ROGRAM .................................................................................................................3 THE DATA VAULT FUNDAMENTALS................................................................................................4 MODELING WITH THE DATA VAULT...............................................................................................6 THINK DIFFERENTLY...................................................................................................................7 THE BUSINESS KEY .....................................................................................................................9 BUSINESS KEY ALIGNMENT......................................................................................................... 10 ARCHITECTURE ....................................................................................................................... 12 SAMPLE DATA VAULT MODEL..................................................................................................... 13 HYBRID TABLES ....................................................................................................................... 14 APPLYING THE DATA VAULT ...................................................................................................... 14 FINAL NOTE ........................................................................................................................... 15
Fig. 1 Hu b
A Link represents a natural business relationships between business keys and is established the first time this new unique association is presented to the EDW. It can represent an association between several Hubs and sometimes other Links. It does maintain a 1:1 relationship with the unique and specific business defined association between that set of keys. Just like the Hub, it contains no descriptive information. The Link consists of the sequence ids from the Hubs and Links that it is relating only, with a warehouse machine sequence id, a load date/time stamp and a record source.
DATA VAULT MODELING GUIDE | 5/15/2012
Fig. 2 Li n k
Notice the similarity between the Hub and the Link. Both represent the first time a core business concept (Hub) or natural business relationship (Link) is introduced to the DW.
The Satellite contains the descriptive information (context) for a business key. There can be several Satellites used to describe a single business key (or association of keys) however a Satellite can only describe one key (Hub or a Link). There is a good amount of flexibility afforded the modelers in how they design and build Satellites. Common approaches include using the subject area, rate of change, source system, or type of data to split out context and design the Satellites. The Satellite is keyed by the sequence id from the Hub or Link to which it is attached plus the date/time stamp to form a two part key. Note that the Satellite then is the only construct that manages time slice data (data warehouse historical tracking of values over time).
Fig. 3 Sat el l i t e
A Satellite does not have a Sequence ID of its own and in fact cannot have a different key than the Hub or Link sequence to which it is attached. Further, a Satellite does not have any foreign key constraints (no snow-flaking, branching or bridging). These three constructs are the building blocks for the DV EDW. Together they can be used to represent all integrated data from the organization. The Hubs are the business keys, the Links represent all relationships and the Satellites provide all the context and changes over time.
DATA VAULT MODELING GUIDE | 5/15/2012
When we look at the Hub and Link together, they form the backbone or Skeletal Structure of the model. This backbone model represents a 1:1 relationship with core Business Concepts and their natural business relationships.
Note that all context (descriptive information) and all history are found in the Satellites.
Mo de ling w it h t he Da ta Va ult
The process of modeling with the Data Vault is closely aligned with business analysis. The first step is to identify the Hubs for the given subject area. Once the Hubs are defined we next model the natural business relationships between these Hubs. Then we design and add the Satellites to provide context to these constructs. STEP
1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3
TASK
Identify Business Concepts Establish EWBK for Hubs Model Hubs Identify Natural Business Relationships
DATA VAULT MODELING GUIDE | 5/15/2012
Analyze Relationships Unit of Work Model Links Gather Context Attributes to Define Keys Establish Criteria & Design Satellites Model Satellites
Fig. 6 St eps t o model i n g wi t h D at a Vau l t
This process is not concerned with separating facts from dimensions, or from separating master entities from events or transactions. The focus is squarely on core business
concepts and their unique business keys. In that regard, all of the above are candidates for Hubs. For example events including transactions are modeled as Hubs.
Cu s t omer
With Data Vault modeling we separate the business keys from the relationships from the context. All of the business keys are modeled as Hubs, all relationships and associations are modeled as Links, and all context and history is provided for through the Satellites. Shown here we can see that the Business Key (Customer_Code) is in a Hub (H_Customer), the relationship (Customer_Class_ID) is in a Link (L_Customer_Cust_Class), and the context is modeled in several Satellites.
Look back to the 3NF model and now consider that all of the same information (the same components of data) about Customer are represented fully in both models. Interestingly both models represent a dependency on a single business key. Actually if we draw circles around each of these models we can see that what is inside each circle is a representation of the same single business key, the same set of attributes and the same relationship. Notice here that the reaching out from the Customer to the Customer Class is modeled through a relationship with a FK inside the 3NF circle. The same is true for the DV circle in that reaching out from Customer to Customer Class is modeled though a relationship (Link) with a FK in that Link and on the perimeter of the circle.
It is important to think of the DV circle in the same way as the 3NF circle. This means that a) all things in either circle are dependent on a single business key, b) relationships pass through the circle directly from table with the BK, c) the only grain shift in either circle is based on Date/Time stamp for the purpose of tracking history.
HINT: As you progress with Data Vault Modeling, this view of thinking differently will become more and more important. We tend to see tables the same way we have always see them. For this reason, we tend to re-combine keys with relationships with context. But as soon as you do, you actually stop vaulting and return to other forms of modeling. So before you change the grain of a satellite, include a relationship FK in a Satellite or Hub, please consider the above circles analysis and reconsider.
The process of identifying and modeling these EWBKs is then closer to a business requirements gathering process than a source system analysis. Balancing the various input factors, with an emphasis on the business versus the technical, effectively summarizes the best practices for this process.
DATA VAULT MODELING GUIDE | 5/15/2012
First the business process designs, business interviews, existing data marts,
business metadata, process metadata, a business glossary if is exists, semantic models, logical models, information models and the master data management artifacts (if existing) and the industry reference model (to the extent certain components are aligned) and then
10
On the other side we know that the enterprise data warehouse (EDW) must be aligned with the organizational view of the business keys/terms (EWBKs). See the top right Business Keys in figure 5. The integration of these Raw keys with the EWBKs represents a core function of the EDW today. In effect, we have been boxed in by the upstream requirements (build a DW that includes all data at the atomic level and with full traceability) and the downstream requirements (align the Business Keys with the organization at the enterprise level using business terminology).
NOTE: We cannot rely on having these transformations happen in the Mart Staging or Data Mart layers as a) the Mart Staging is not intended to be persisted and b) the Data Marts are departmental in scope (not Enterprise Wide). Not sharing this business key alignment through the EDW will result in a failure to integrate around the true business keys and will result in the downstream inconsistencies common to data silos.
The extent to which the sources are already aligned with the EWBKs will determine the scope of integration and alignment that must occur in the DV EDW. See figure 6 below.
The alignment of these keys is facilitated through Links. As can be seen in figure 5 on the previous page, the Raw Key of Person is aligned with the Business Key of Customer using a link structure.
DATA VAULT MODELING GUIDE | 5/15/2012
Note that the naming conventions are not adequate in and of themselves to warrant the separation of Raw and Business key Hubs. If in fact Person and Customer meant the exact same thing to the business and were in fact true business term synonyms, then the raw system load of Person records could populate the Customer Hub directly. However, in this case the Person Raw key does in fact mean something different than the Customer Business key. In this case we assume that there are business rules at play for example a Person record is determined to be a Customer if they were involved in a Sale transaction, there was a non-zero purchase price, the transaction was successfully completed, and the Sale was not cancelled. As you can see here, the raw auditable load is to the Person and the business aligned load is to the Customer. Those tables that are loaded using this type of business processing must be identified as sysgen record source records (generated by us through a business rule driven process).
11
These components of the DV EDW are often referred to as business data warehouse (BDW) or business data vault (BDV) components.
NOTE: Business logic in the BDW or BDV can take many forms and can relate to many types of transformations. The logic specifically targeting the alignment of raw and business keys is a subset of this area and often referred to as the BAR component (Business key Alignment Rules). Why is this important? Because the primary objective of the DV EDW is to integrate and historize data from various sources. The BAR rules are the heart of this activity.
Both the BDW and the BAR layers are part of the business aligned data warehouse.
Arc hitecture
The high level DV EDW architecture includes an EDW that aligns the RAW with the BAR levels. The high level architecture can be represented as seen here.
Fig. 14 D V E D W A rc h i t ec t u re
Note that the business rules represented in the BDW layer can be applied either in the same area as the EDW or as a separate layer. Data Marts can then source the BDW and the EDW layers as appropriate. The Stage and Data Mart areas are not persisted. In a typical scenario, the Stage area is not persisted (data is not kept but rather overwritten). The EDW components are persisted, the BDW layer is commonly persisted however it is not required, and the Data Marts are not persisted. This latter point represents an important distinction between dimensional (federated) data warehouses and the DV data warehouse. Dimensional data warehouses are based on the concept of persisting the dimensions and associated facts. The RAW and BAR layers represent a logical designation. While this separation is often also physical, this factor is not required. In fact one common approach includes a logical separation with Links managing the alignment within the physical layer.
12
13
Hy brid Ta bles
The data vault approach has defined a set of Hybrid tables that can be used to make the overall deployment more efficient. These are applied on a case-by-case basis as appropriate for the specific circumstances. The Point In Time table (PIT) is a modified Satellite table that tracks the valid time slices of the satellites surrounding a particular Hub. This is populated to make the process of associating relative context/descriptive data together for reporting purposes. The Bridge table is a modified Link stable that flattens the relationship between Hubs including important related context/descriptive data (potentially also the business keys) into a single table for ease of access and performance. In all cases, these and other constructs can coexist in the DV EDW provided however that they are always noted as sysgen tables and utilized only for performance reasons. The related historical and auditable data that is used to load these constructs must remain the sole source of the EDW data over time.
14
F inal No t e
The Data Vault approach is growing and adapting from year to year. Incremental changes to the modeling approach, rules and best practices can be expected with some frequency. Please note that this guide should be applied in concert with current updates found online on data vault forums, LinkedIn and from certified practitioners. Online training including updates and current topics is also available online at DataVaultAcademy.com
15