A Road Map To Data Migration Success
A Road Map To Data Migration Success
Data migration plans and schedules typically are driven by larger projects for example, a master data management initiative, a new business process, or a data consolidation effort that supports a new reporting environment. Any changes to the parent project schedule affect the data migration schedule, including changes to production deployment and changes to test and beta deployment schedules.
content
4 5 5 5 6 6 6 7 8 8 8 8 9 9 10 10 11 12 14 14 15 15
Executive Summary Planning and Scoping the Migration Allowing for Dependency Management scenario: Larger projects scenario: Knowing Your Data scenario: changing source systems and Business rules establishing Data governance and stewardship Building the Migration team Project Planning Considerations Mapping Business needs Assessing infrastructure current Architecture target Architecture planning for testing Understanding Data evaluate Data sources and targets perform a Data quality Assessment perform a Data Mapping Assessment perform a Data Migration Assessment perform a postmigration needs Assessment Data Migration Success sAp Data Migration services
executive SuMMARy
Data migration is the one-time movement of data from a legacy source, or multiple sources, to a new target database. this simple concept and requirement can drive a scope that is much larger than expected. A data migration requirement can be driven by a range of initiatives, such as an application replacement or upgrade, the need to consolidate data within a data warehouse, or a requirement to create a single view of product within an organization. in cases where migrated data is transformed for new
uses, your project team will encounter some very specific management and technical challenges. for example, a team that has been writing extraction, transformation, and load (etL) code for a data warehouse faces a new set of challenges when migrating data to a live, operational system. Although a 2% error rate may be acceptable for aggregate reporting, it is not acceptable for customer contact data in this example, we would fail to recognize one out of 50 customers when they call.
Many significant business initiatives and large it projects depend upon a successful data migration. Your goal is to minimize as much of your risk as possible through effective planning and scoping. the objective of this sAp thought leadership paper is to provide insight into the issues that are unique to data migration projects and to offer advice on how to best approach them.
involving business users and data experts in the migration project is the single biggest factor driving your success. to fulfill your data migration needs, you must include a data governance strategy that provides leadership and direction for user input.
With any project, success depends on a good plan. Data migration projects have a distinct methodology and project approach. too often, project managers make the costly mistake of thinking that migrating data is a simple task within a larger project and hand it off to a development team. the table below lists the migration tasks that must be considered.
Migration Element Business needs mapping Infrastructure Identifying data sources and targets
the following sections detail two more project management considerations that, in addition to these tasks, are essential to any data migration dependency management and data governance planning.
Gap analysis between source and target Impact of multiple sources of data
Mapping assessment
Migration assessment
ensure that your migration plan is driven by the expectations and needs of the users sort unique infrastructure requirements as early as possible in the project timeline Validate the system of record for your source data and identify the data needs of your target database Assess the quality of your data to meet the target application requirements and business needs identify and plan to mitigate any gaps between available source data and target application data requirements estimate the challenges of consolidating similar data from several sources or integrating dissimilar data Understand the effort required to accurately identify source data at column-level detail, including transformation specifications Understand the effort required to design, code, test, and implement the data migration
plan, followed by a family service plan, and so on. What may be a straightforward change for the larger project could exponentially increase the scope of the data migration effort. it is not uncommon for resources to shift roles during a project. for example, the priorities of the larger project could drive key players away from the migration tasks, which are deemed less important than other development. thus, your schedule is delayed and your risks increase. scenario: Knowing Your Data the success of your migration project largely depends upon how well you know the data content and business rules. Your data experts help with identifying business needs, analyzing source systems, and testing, covering the full life cycle of your project. experts likely will be from the business and tasked to the migration project on an as-needed or part-time basis. scenario: changing source systems and Business rules the source data and the business rules that govern it may change during the course of your project. there could be other initiatives in the organization, new product offerings, or business-process changes that impact the data. the source system itself could change, due to application or server upgrades. Application upgrades are particularly risky, since the application features and data model of the source may be altered.
sharing and capturing content detail about source system data and business processes testing and confirming application functionality that is supported by migrated data the approach to establishing a data governance team is specific to your organization. if possible, leverage the talents of an existing team or committee. often a data warehouse users group is an excellent source of data knowledge, or you may be able to leverage a team that supported a previous project. consider a metadata tool for the project team. the data migration project is a unique opportunity to capture metadata about your source systems and to initiate a business process for managing data in the target application. What can you leverage that you have in place now? What are the requirements for managing the metadata and the business rules for the data migration project? What are the requirements going forward? consider the following: the need to share content knowledge and business rules for multiple team members testing against identified business rules and mapping and transformation specifications historical documentation (audit trails) of the decisions made, and logic behind them, for the data migration historical documentation about the source of data, especially useful when merging data from multiple sources identification of data owners and experts for the migration and future data management requirements
consider using consultants for key roles on a migration project, since it is a onetime, custom effort and your team may not have the specific planning, strategy, or development experience. focus your project plan on growing the skills needed
in your organization, which may include future data migration, data integration, or data quality projects.
Database administrator
Data architect
coordination and communication of task status Management of issues creation and maintenance of project plan Management of dependencies to larger project, target, and source application schedules coordination of the creation of a data governance strategy and team identification and validation of business needs Analysis of target and source systems Assessment of data quality performance of tests Assessment of data mapping and business rules for data transformation infrastructure support, including database sizing, optimization, and life-cycle management; works closely with the database analyst Data support, including data analysis, data quality assessment, mapping, and migration assessment initial sizing for staging database(s) planning for life-cycle management of database objects Management planning for schema changes and user security Architecture and methodology source and target application data analysis Data quality assessment, mapping assessment Data governance planning configuration management configuration standards and quality assurance Architecture and programming standards Data quality, data mapping, and data migration assessment source-to-target gap analysis and multiple systems impact assessment
the following sections address planning considerations for your data migration project.
Additionally, as unexpected data quality and gap analysis issues arise (and they will), knowing the business needs aids in identifying priorities for the project team. for example, if the customer age is not fully populated or has inaccurate data (230 years old), the team can reference the known uses of that data to determine the extent of data cleansing needed. in another example, the medical history for an automated patient chart must be of the highest quality when migrated. the business need, which is driven by legal requirements, must be clearly understood by the entire migration team. ideally, the project team will include data experts and business users as part of a data governance strategy to ensure that this process is as straightforward as possible.
too many organizations wait until the beta or production conversion to discover performance issues that can set the schedule back by weeks or months. current Architecture What is your current architecture? Do you have data quality and data integration tools needed to accomplish a data conversion? is there a database or server available to host your data and code? When and how will you test the full conversion to ensure that your performance will meet the schedule needs? the team should evaluate what servers, databases, and software tools are available to host a migration project. Leverage what you can, and do not reinvent the wheel. specifically, review: requirements documentation and capture Metadata and data integration mapping repositories Data integration tools Data quality tools code management tools and processes test tools and processes given that the data migration is a onetime project, you should evaluate what code management processes can be bypassed or customized for this effort. this may sound like a surprising recommendation, but consider the environment surrounding the data migration project. Your schedule is not your own; rather, it is dictated by the larger project. Also, the priorities and phased approach can change at any time. the data migration team needs to be very agile. the approach to analysis, coding, and testing should be lightweight and mutable.
Assessing Infrastructure
A data migration project demands a dedicated infrastructure. Dont make the mistake of using your existing development and test environments. for example, do not underestimate the planning needed for a dedicated staging area or for load testing (yes, load testing). You might be converting many years worth of data in a single weekend. far
consider using consultants for key roles on a migration project, since it is a one-time, custom effort and your team may not have the specific planning, strategy, or development experience. focus your project plan on growing the skills needed in your organization, which may include future data migration, data integration, or data quality projects.
target Architecture the dedicated infrastructure for the project should be planned as part of your scope phase. Do not make the error of waiting until the development phase of the project to discover that you need additional resources for integration testing, such as servers or new database instances. the planning for this infrastructure should include: A staging database for analysis and development. impact on production services is minimized by the use of a separate migration server and by the use of configured staging and process areas within the service to manage multiple data versions, sources, and status values. Your team can also use the staging area to create needed reference data for the target system. A sizing estimate for the staging database, which includes any needed replication required to support both migration code development and application A code life-cycle management plan performance test requirements Backup and recovery planning scalable connectivity requirements and approaches (for example, how will you extract and load data to the source and target systems?) planning for testing the test requirements for a data migration project are unique, as you may have multiple conversions during parallel and beta testing of an application. Adding to the complexity is the fact that the migration team is supporting the application test requirements of the larger project and also testing the conversion code itself.
Your migration team needs to first consider the basics of audit and reconciliation. this is missed more often than it should be. Are you looking at the right set of data? can you trace back to your source system at a record level? You should include basic record counts and validation of etL processes. Many of these processes will be accomplished multiple times, as testing and reloads occur. the audit statistics need to be reported and evaluated each time. Also, if the project is a phased implementation, you may be running similar data across parallel databases. Audits and reconciliation are needed to ensure that the data is consistent across these applications. You should also plan for performance testing. Many projects are delayed because the development team has successfully converted a test set of data but waited until the last minute to run the conversion of the full set. A customer consolidation project for one of the largest global computer manu-
facturers was delayed by months when the development team attempted to convert (at the time of going live) a years worth of customer data using code that had been developed using a days worth of test data. for your conversion, there may be millions of records to process, with complex referential integrity. performance issues for data processing at this magnitude can require significant code changes. What should you do? estimate your load and include this consideration when doing unit and integration testing. participate in peer reviews, or contract with experienced consultants, to address code optimization, data query tuning, and database-tuning challenges. And include these options in your earliest planning. the primary reason for capturing the metadata around your business rules is to test the mapping and transformations. All phases of testing unit, integration, and target application should include a validation of the business rules.
Data migration is all about the data. Data-related tasks, especially those around quality, will be the bulk of your work. plan enough time to evaluate the source data, to compare the available data to the needed data, and to drill down to the detail needed for source-to-target mapping. Your goal is to ensure that the data migration development is the shortest task on your project plan.
a high level, the team will identify subject areas, data entities, and supported functionality. for example, customer address data will come from the billing system, customer phone contact data will come from the data warehouse, and so on. the planning team needs to look for the following risks and complexities. the commonly agreed to, or popular, system of record may not be the best source of data. consult as many data experts as possible, and look for reconciliation statistics, compliance numbers, and other forms of audit to identify a trusted source. Multiple sources increase your schedule exponentially. never assume that a second source of data will make your project twice the effort. if you have one source of data, you may have to look for duplicate data, but if you have two sources, you have to look for duplicate data, harmonize reference data, and run record-matching algorithms to determine correct mapping. (A requirement for sharing master data across multiple applications is a project in itself. the need to harmonize reference data is complex enough to warrant its own thought leadership paper.) certain types of projects introduce their own challenges. for example, the approach to managing a data migration that is part of an MDM project will be distinct from other kinds of migrations, with a focus on dependency management, data quality, and data governance.
As early as possible in your migration schedule, allow the business users to test the data in the target application. Does it support the features and functionality as expected? Data migration mapping is very abstract, especially if the users have never been part of a similar project. they may not get it until the converted data is actually supporting the application features and processes that they are implementing. Allow time in your planning for changes to the mapping, business rules, and reloads. it is not uncommon for there to be confusion during beta testing and implementation as to where errors originate. its all too easy for bad data to be blamed for every test error. What can you do as part of your planning to avoid this? implement the migration with as much metadata and reconciliation as possible. not only are you ensuring the integrity of your data, you are creating the lineage necessary to support the integration test process.
Understanding Data
Data migration is, in fact, all about the data. Data-related tasks, especially those around quality, will be the bulk of your work. plan enough time to evaluate the source data, to compare the available data to the needed data, and to drill down to the detail needed for source-to-target mapping. Your goal is to ensure that the data migration development is the shortest task on your project plan. You must understand the source data and target needs before you start coding. You dont want to experience the very costly surprise of learning that the source data is not fit to use at integration testing or implementation. evaluate Data sources and targets As part of the planning process, your team will take an initial look at the source and target systems. the objective is to know enough about each system to make an estimate of the effort to complete mapping and migration. At
10
An MDM implementation requires initial data migrations but also ongoing integration and reconciliation. the business rules, matching criteria, and governance processes that are established for migration will be propagated to support the live application. Also, mergers and acquisitions require data consolidation, with a focus on reference data and business processes (that are represented in the data structures and content). even if the two organizations have the same software, the mapping may not necessarily be straightforward. Another challenge is that a data warehouse migration can seem less complicated because the data is relatively static in the target database; however, the expectations of the business users
can be extensive. in a well-designed warehouse, one data mart will support multiple business units. Also, the data migration process is likely to be the initial load of an ongoing etL process, and there should be reuse of many components. You should conduct a gap analysis. the features that the business expects from the target system may not have a data source at all, and the migration team may be the designated messenger to carry this bad news. for example, in an MDM customer data project, one region out of five has never captured customer contact history, and, when consolidating the customer data, a subset will not have a full history. the business requirement to establish customer segmentation based on contact history is seriously impacted.
Your planning should include the time needed and the approach for resolving gaps in the source to target data mapping (and a cost benefit of the effort). there are several options, and each will increase the scope of your project: the business users and data experts could manually create data. this is not uncommon when creating reference data, such as product types or customer status. the project plan will include the time needed to work with the governance team and facilitate the data entry, updates, and testing. You may have the option to source data externally for example, using Dun & Bradstreet data to create more consistent customer data. You could default the data to an unknown value and plan to populate it over time, as part of the target application. Using the customer segmentation example above, if it requires a years worth of customer contact history, the business will accept the 12-month delay before having a 100% correct segmentation calculation. perform a Data quality Assessment the biggest factor in your success and your biggest risk is the quality of the source data. Many projects fail, or are significantly delayed, upon discovering that the data is not fit to use. the most important phase in a data migration project involves the tasks needed to understand the data content. You would not put dirty gasoline in your new car why would you map bad data into your new software application?
Figure 1: Sample Data Profiling Report Showing the Frequency Distribution of Values
11
its unlikely that anyone on your team really understands the current state of the data at the level of detail needed for your project. And even if the source data is pristine, that doesnt mean that it is fit for the requirements of the new application. experience shows that unexpected data conditions encountered in load programs can significantly impact the timelines of data migration projects. early assessment of data quality reduces project risk. Your team will do an initial data quality assessment as part of the planning process. the sooner you look at the data, the more likely your success is. for this initial pass, you are looking for the big-ticket items missing data, data that is not fully populated, column names that are inconsistent with content, and so on. Your goal is to know enough to estimate the level of effort to get the migration job done. You have done a review of business needs, and this will drive the type of data you review. for example, if the objective is to create a consolidated view of product data across six regional sAp applications, you may pull a subset of product data from each and attempt to match them. how good is the data? What are the challenges? this should give you a pretty clear idea of the project requirements for mapping and migration. Another example is pulling customer data from two systems, such as billing and crM. You want to run some high-level reconciliations and audits between the two systems, looking for deltas. What if you have customers who have products but theyre not getting billed? it has happened.
What if you have historical data for billing that isnt represented in your crM application? When capturing metadata, a tool is a good option but when assessing data quality, a profiling tool is a must. Use data profiling technology to systematically scan the tables of interest to quantify any data quality and data integrity issues, such as: Missing values pattern analysis frequency analysis ranges and outliers redundant data analysis foreign key integrity analysis statistics relevant to numeric fields parsing requirements to enable mapping of source columns to the target columns perform a Data Mapping Assessment hand-in-hand with data quality is data mapping. What are we moving, and what are the rules? how much time will it take the team to complete data mapping? given your planning, you should have a general idea of what it will take to create the final design and implementation. the captured business needs, the source and target system metadata, and the data profiling results from the data quality assessment all create the information needed to understand the mapping effort. Much of the mapping will be data analysis work, if done right. it is not uncommon for teams to accomplish this mapping based on guessing which column names match in a spreadsheet, or worse, jotting a few notes down on a hard copy of the data definition language (DDL)
listing before coding. Be aware that there are development teams capable of this behavior. Where possible, identify the data challenges. one example illustrates how complex your mapping can become. An insurance premium discount for a new membership system is stored in a single column on a single record. however, in the custom source systems, it is stored in multiple locations and averaged according to complex business rules. these rules are captured in the source system code. there are no data experts who understand fully how the logic works. the data migration team must recreate this logic in the data integration code in order to populate the target system value. to make it even more complicated, the data conversion is an ongoing process that happens at policy renewal, and getting the calculation wrong will cost the company money. if the discount is too large, the organization loses money; if its too small, the organization may lose customers. finally, there is the importance of capturing this logic in the metadata or design of the data migration for auditors, actuaries, and the membership support teams. some of the things to consider when estimating the time it will take to accomplish data mapping include the following recommendations. Know that some of your target requirements will require significant transformations or the creation of new data content. for example, the mapping scope should include the effort to create reference data for the target application,
12
such as product types, customer status, and vendor segmentation types. its not uncommon for the team, working with the data experts, to create these and set up some cross-reference tables to support the conversion of the data. this should be planned early in the project, and data quality validation should be included, so that all records are converted. Also, matching and consolidating is required when sourcing from multiple systems. it could be that you need to make a match decision on the best record from multiple source systems rather than having a single system of record, or that your reference data in your target system is such that you need to combine multiple records to make a best of instance. Data quality matching could also be used to eliminate duplicate data within a single source. this logic could be implemented to select a best of record that will later be evaluated by a data expert. the logic and the process to identify the good record should be part of your mapping specification. Allow time to look at the data. Dont allow your team to fall into the trap of mapping based on column names, copybooks (remember those?), or DDL. for each subject area, you should allow at least a week of time for the analyst to gather information about the meaning of the data, how it is populated (data profiling), and how it supports the target application. for high-priority data content, the team should learn as much as possible about
the business processes that create the data and the target applications business processes. Data experts and business users are an invaluable source for this information. for example, a patient record in a hospital radiology application has some very basic requirements for radiologist review. there may be scheduling and workflow functionality included in the application. if that data is mapped to an automated patient chart, it will be used by physicians other than radiologists to make broader diagnoses and evaluations. What are the differences in these processes and does the data support the new needs represented by the automated patient chart? the planning should include the evaluation of a mapping tool or mapping functionality within a data integration tool. Key things to look for are the ability to easily connect to and analyze the source data
as part of the mapping process, the ability to share analysis results across a larger team, and how well the mapping drives the development of code. include validation and reconciliation steps in your mapping. for any matching that you do, or any consolidation, verify that each source record is present in the source systems as designed. each subject area mapped must have a reconciliation base to the source, validating that all records extracted were processed. for example, if there are 2,000 customers on a billing system, then there should be 2,000 customers on the new crM system. external to the migration process, you should also have a process that is designed for a checks-and-balances approach.
13
Another approach is to create a highlevel mapping to a database that is wholly external to the migration process, thus creating a further audit validation. for example, you may map your in-patient data in an automated chart application to hospital admissions data that is captured by a financial system. is this overkill? that entirely depends on your business user community and the extent to which you want to guarantee trusted data. perform a Data Migration Assessment effective planning will allow you to mitigate the risks associated with your project. if you have done the work of scoping the infrastructure, the mapping, and the data quality assessment, then the migration itself will be your shortest task. When it comes time to migrate your data, the people and the processes will make it work. its more of a team effort than any other type of it project. throughout the migration process, you will have checks and balances in place, including a team of data experts available to address the expected data content decisions and to help resolve the unexpected. this can include team members who are clearly responsible for tracking a set of audit and reconciliation metrics to evaluate such issues as: Do the record counts match from source to target through each phase of your processing? can you reconcile counts and reports to other systems in the organization that host the same data for example, numbers of active customers or product sales? this is especially important if you are running systems
in parallel. can you track the deltas between the parallel systems, and can you understand them? Do the data experts have access to review the data for quality and consistency? can you create an audit back to your source system that would satisfy financial compliance requirements? Do you have an approach to a phased project, and how will you keep parallel systems in sync as you migrate forward? this can include ongoing integration or replication between applications for the duration of the project. Agility is key. this is a custom development project, and you should apply the rigor that you would to any other development project. however, keep the implementation approach as lightweight as possible. there are many dependencies to other projects and applications within the organization. plan for the inevitable need to change scope, schedules, and road maps as quickly as possible. its not unusual for a new application to source from multiple databases, flat files, or external feeds. Additional sources will increase your complexity exponentially. consider your approach to rationalizing the data across these sources. Also, what is your cutover strategy? And how will these sources impact your schedule? if you are deduping or consolidating data across sources, consider what your record-matching requirements are and be certain to evaluate your data integration software with this in mind.
finally, map all of your data integration requirements and data quality needs against an off-the-shelf tool set. simple things, such as populating a gender column based on customer name, come out of the package and save significant amounts of time. the more complicated needs, such as matching and consolidation, are infinitely easier using dedicated tools. Dont make the mistake of thinking you are saving money by hand coding you are risking your schedule and the quality of your product. perform a postmigration needs Assessment finally, as part of your planning process, do not forget the needs of your team going forward. these include the following: Data that is not converted must be archived, or it needs to be reported on in its historical structure. the ongoing reconciliation and balancing between the new application and parallel applications must be considered. for example, if you implement a new customer data mart, it should include a frequent reconciliation back to its source (lineage) and to another unrelated customer application (trust). its likely that the data governance team has determined that resolution of a set of data quality issues can be deferred until postproduction. plan to capture as much detail as possible over the course of your project.
14
if you have successfully implemented your data migration, you should have data governance, standards, integration, and quality processes in place. Leverage these on your next project. Use the migration effort as a route to growing your overall information management skills, which will make subsequent data projects easier, faster, and cheaper.
effective planning will allow you to mitigate the risks associated with your project. if you have done the work of scoping the infrastructure, the mapping, and the data quality assessment, then the migration itself will be your shortest task. When it comes time to migrate your data, the people and the processes will make it work. its more of a team effort than any other type of it project.
15
50 092 958 (08/12) 2008 by sAp Ag. All rights reserved. sAp, r/3, xApps, xApp, sAp netWeaver, Duet, partneredge, ByDesign, sAp Business ByDesign, and other sAp products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of sAp Ag in germany and in several other countries all over the world. Business objects and the Business objects logo, Businessobjects, crystal reports, crystal Decisions, Web intelligence, xcelsius, and other Business objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business objects s.A. in the United states and in several other countries all over the world. Business objects is an sAp company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. national product specifications may vary. these materials are subject to change without notice. these materials are provided by sAp Ag and its affiliated companies (sAp group) for informational purposes only, without representation or warranty of any kind, and sAp group shall not be liable for errors or omissions with respect to the materials. the only warranties for sAp group products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. nothing herein should be construed as constituting an additional warranty.
www.sap.com /contactsap