Designing Data Integration The ETL Pattern Approac
Designing Data Integration The ETL Pattern Approac
net/publication/230661908
CITATIONS READS
9 4,077
3 authors, including:
Veit Köppen
Otto-von-Guericke-Universität Magdeburg
96 PUBLICATIONS 373 CITATIONS
SEE PROFILE
Some of the authors of this publication are also working on these related projects:
All content following this page was uploaded by Veit Köppen on 22 May 2014.
The process of ETL (Extract-Transform-Load) is important for data warehousing. Besides data gathering from heteroge-
neous sources, quality aspects play an important role. However, tool and methodology support are often insufficient. Due
to the similarities between ETL processes and software design, a pattern approach is suitable to reduce effort and increase
understanding of these processes. We propose a general design-pattern structure for ETL, and describe three example
patterns.
Context This describes the situation where the problem occurs. Yes
Data Quality Which data quality issues are addressed and which data No
quality dimension/s is/are improved.
Implementation For various ETL tools, the solution is put into practice No
differently, therefore different implementations are referenced
here.
“
problem. For pattern users, it is necessary to identify prob-
lem, context, and solution in an easy way. Therefore, tem-
plates should be used to structure all patterns uniformly.
Although the ETL process
We derive our pattern structure from software engineer- can be performed in any
ing patterns because of the similarities between Software
Design and ETL processes. A template consists of different
individually programmed
elements such as name and description. For examples of application, commercial
templates in object-oriented software design see [10], for
software architecture design patterns see [11], and for the ETL tools are often used
”
52 CEPIS UPGRADE Vol. XII, No. 3, July 2011 © Novática
Business Intelligence
“ A pattern is described as a
three-part rule consisting of
text of master data can be done according to the dimen-
sions.
Problem: Master data changes only occasionally, but
they do sometimes change (such as the last name of a per-
the relations between context, son). These changes should be taken into account in the
problem, and solution
3.1 The Aggregator Pattern
” dimension tables. However, challenges occur due to the use
of domain keys that change over time, thus they cannot be
use as primary keys. This is in contrast to the modeling of
dimension tables in the star schema. Another problem is
Name: Aggregator Pattern the use of domain keys if redundancy is required.
Intention: Data sets should be aggregated via this pat- Solution: An important challenge is to store old and
tern within ETL processes. new data in the DW system. Furthermore, a relation of fact
Classification: Elementary task table and dimension data is necessary. For this purpose, the
Context: From a database or file data on a fine-grained dimensional table has to be extended by additional attributes.
level are loaded into the DW. In a first step, a virtual primary key is added, together with
Problem: The DW data model does not require data at one or more attribute/s storing current or up-to-date infor-
a fine-grained level. If data from the operational system is mation. The attributes valid_from and valid to are used to
not needed at a fine-grained level, two problems may oc- store the information about when the data was valid. This
cur: more storage is required in the DW, and performance is described differently in the literature, for example as
decreases due to more data having to be processed. changes of type II dimensions [10] or as snapshot history
Solution: An ETL operator is used that collects data [13]. For every data set, a decision has to be made: either it
from the sources and transforms them into the desired granu- is a new dataset, an updated one, or a data set that already
larity. existed in the dimension tables of the DW. For this com-
Resulting Context: A performance increase can be ob- parison, a key should be used that is persistent in time, such
tained, in the DW system as well as in the ETL process, as the domain key. Every source data set is mapped with
through the reduction of data. Furthermore, the required stor- this key to dimensions. If this is not possible, a new entry is
age space is reduced. However, one disadvantage is that identified. If all attributes are equal for the source data set
there exists no inverse operation, so the inference to origi- compared to a data set in the DW, an existing one is identi-
nal data is not possible. If data granularity changes, infor- fied. Otherwise an updated data set is detected. A new data
mation loss may result. set has to be stored in the dimension tables and the attributes
valid_from and valid_to as well as the virtual key have to
3.2 The History Pattern be generated and timeliness set to true. For an update, the
Name: History Pattern timeliness and valid_to information of the already existing
Intention: Data sets in the dimension tables should be dataset have to set before the source dataset can be en-
marked and cataloged. tered into the DW.
Classification: Composite task in the category of di- Resulting Context: All data are historicized, however this
mension loading for star schema. influences performance due to the increase of the data amount
Context: Product, Location, and Time are dimension in the dimension tables. The domain key has to be unique;
in the DW that can change over time. Analyses in the con- otherwise, duplicate detection has to be performed first.
Data Quality: All available information (data complete- Solution: Duplicates have to be identified and deleted.
ness for dimensions) is accessible for analysis with the help As a first step, data have to be homogenized. This includes
of the history pattern. Data timeliness is another advantage conversions, encodings, and separations of all comparative
for data quality issues, as long as the loading is performed attributes. Partitioning of data reduces comparison effort,
at short, regular time intervals. but must be chosen with caution in order not to miss dupli-
Composite: Before an ETL task from the History Pat- cates. The comparison is based on similarity measures that
tern is performed, patterns from the categories Loading Di- help to identify duplicates. There exist different methods
mension and Transformation may be applied. The History and measures based on the data context.
Pattern can be followed by patterns from the Loading Facts A data fusion of identified duplicates has to be carried
and Loading Dimension categories. out. Aspects of uncertainty and inconsistencies have to be
considered in this context. Inconsistency means that seman-
3.3 The Duplicates Pattern tically identical attributes have different values. Uncertainty
Duplicate detection is a common but complex task in occurs if only null values are available. Data conflict avoid-
ETL processes. With our pattern template, we briefly de- ance can be carried out via the survivor strategy [17], where
scribe the solution, although in practice this should be de- a predefined source entry is favored against all others, or
scribed more comprehensively, see [14][15][16] for more via set-based merge [9], where the disjunction of all val-
details. ues is stored. In contrast, data conflict resolution can be
Name: Duplicates Pattern carried out via a decision strategy, where an entry is deter-
Intention: This pattern reduces redundancy in the DW mined from the sources, or a mediation strategy, where new
data; in the best case, it eliminates redundancy completely. values can be computed.
Classification: Composite task in the category transfor- Resulting Context: Duplicates are only partially de-
mation. tected. Due to complexity of the duplicate detection, the
Context: Data from heterogeneous sources (e.g., appli- ETL designer has to carefully consider data context and
cations, databases, files) have to be loaded into the DW. appropriate methods for measuring similarities or partition-
Problem: A data hub for the integration of data is not ing strategy.
always available, therefore master data redundancy occurs Data Quality: The data quality issue non-redundancy
in different business applications. A duplicate are two or is supported with this pattern.
more data sets that describe the same real-world object. Data Composite: The Duplicates Pattern can be preceded by
in the DW should give a consolidated view and must be free patterns from the Transformation category as well as from
of duplicates. the category Harmonization & Plausibility Check. The cat-
egories Transformation, Updating, and Loading Dimension
“
include patterns that can be used for subsequent tasks, see
The creation of complex ETL Figure 4.
”
structure the required work and support software architects
to software engineering and developers. We propose ETL patterns for the support
of ETL designers. This provides an adequate structure for