Why do so many data migration projects end in failure?

Financial Institutions including Banks are faced with the challenge of rising consumer expectations, mergers & acquisitions, continued digital disruption and intense competition from FinTechs like never before. With a growing focus on the implementation of newer systems and processes, banks are continuously introduced to fresh data sources and diverse formats. Data Migration in financial institutions most commonly involves the transfer of data from legacy systems to newer ones, which is often easier said than done.  

Data Migration is one of the key activities for any system implementation or upgrade that entails significant amounts of risk. Migration is rarely straightforward, and there are very few situations in which everything can be validated. Validation of migrated data is an assessment and balancing act between enterprise risks versus business priorities.

Here are some of the most common data migration scenarios:

1. Transferring data from one system to another while changing the storage, database or upgrading to a higher version of the application.
2. Data Migration after a merger or acquisition (M&A)
   

To help you get started with your data migration validation we have put together some tips and actionable insights.

7 Keys to get Data Migration Testing Right

1. Data Quality at Source:  To optimize risk efforts banks must take charge of data quality at source ensuring its accuracy, completeness, and relevance. Data inaccuracies that go unattended can not only cost banks irreversible loss of reputation but also put them at a regulatory risk. This process begins with defining and establishing appropriate data standards and processes for measuring, monitoring, and reporting conformance to acceptable levels of data quality.
2. Functional Testing with Migrated Data (For a system upgrade): Going by the principle of testing, an application must be tested based on how it will be consumed by end users and not the way it was built. While Functional testing helps validate the new system, ensuring it meets the business objectives and needs, it is conducted on simulated data in most cases. It's possible to pass all the tests only to find out that some of the key functionalities are lost with migrated data, or they don't work according to the standards expected. Therefore, it's imperative to perform functional testing on migrated data at some level.
3. Automated Validation of Data on UI: Identifying combinations that could distort the UI can prove very difficult, as validating sizable records manually is not feasible. This is where automated validation can prove to be extremely beneficial. The use of functional automation tools like UFT & Selenium amongst many more can help you save time and improve coverage of verified data. Moreover, the automation of data migration brings with it a high ROI on each script, as it is executed multiple times. Automated validation of data on UI is strongly encouraged during an implementation project.  
4. Exhaustive Data Profiling right from the start

• Identify classes
• Test design techniques
• Combinations cause problems
• Cover all record sets-classes

‍‍‍5. Balancing Enterprise Risk versus Business Priorities: From a business perspective, testing is all about achieving operational excellence and debunking the most common misconception that all defects need to be fixed. Some may have workarounds or can be handled via new transformation scripts, while some can be ignored. While deciding what to fix, it’s important to never lose sight of its impact on business.
6. Migration Waves in Rapid Succession:   While balancing of data volume in a wave and not just the number of customers can help minimize the risks and also mitigate any issues that may lead to customer dissatisfaction.  A wave list may typically go through 4 to 5 iterations depending on the complexity of parameters. Here are some of the criteria’s that help establish migration waves

• Migration requirements / requests of integrating application partners like PeP+, Payplus/ Wires gateway, and ARP SMS.
• Timelines of other projects and their migration testing plans.
• Migration mechanism of interfacing applications
• Balancing customers of various groups like government, VIP customers, business customers etc.
• Location of customers to evenly distribute across geo-territories. In some cases global customers may have to be grouped together for language support.

‍7. Verifying the Data Migration Mapping Document:  Before you get started make sure you have the data migration mapping document verified for accuracy and complete coverage. It’s an integral piece of your migration process helping you mitigate your risk efforts. Take a look at our checklist to make sure nothing derails your data migration plan.

Here are some more tips to help you validate your data migration program.

1. Building Validations around Data Transformations: Before you begin to think of data migration you need to start identifying the type of transformations required and build test cases for each transformation.  Quick Tip: It’s a good practice to automate test cases executed with different sets of data.
2. Proof of Concepts (PoCs) : The best way to get your migration to speed is to perform POC testing covering all entities and different classes of transformation in place of one entity covering all validations.
3. Periodic Assessments: A key element of the data migration journey – from business case to actual implementation is risk assessment and mitigation.  As a product goes through migration waves, risk assessments and reviews will have to be regular and current.
   

‍

Conclusion

‍

The Four Pronged Approach

As Data and User migration are infinitely more complex than functionality or usability, it requires a four-pronged approach. Identify your customer conversion risks up front, in any situation where data is being moved from one or multiple products to another or one database to another.  The Four-Pronged Approach helps to strengthen the planning, analysis, and implementation of data migration programs. This leads to better risk reduction, data accuracies and preparedness.

Stage 1 - Data Intelligence

• Product based identification of entities and data-sets
• Profiling by grouping, sub-grouping and clustering
• Checksum based reconciliation

Stage 2 - Machine Recon

• Automated data-loading and shifting
• Complex condition induced code for efficient comparison
• Recursive learning scripts for increasing result efficiency

Stage 3 - Supervised Feedback

• Supervised review of results to remove false-negatives
• Knowledge-based data sampling to identify anomalies
• Customer impact enabled defect analysis

Stage 4 - Experience based Testing

• Rich experience-based tests to calibrate data usage
• Simulating live customer interaction on migrated data
• Infuse confidence to migrate in Live

Go-Live Faster’s Data Readiness platform is paving the way to making financial institutes "Data Ready.” If you’d like to discuss further and learn how Go-Live Faster can help you get “Data Ready” and alleviate migration risks in any situation, please send us an email on reachout@golivefaster.com.

‍