Data migration projects tend to be time-consuming and costly, often requiring extensive application downtime. In fact, according to IDC (IDC Storage and Data Migration Services Overview, November 2013) data migrations represent up to 60% of all large enterprise IT projects and only 60% are completed on time.
At a basic level, data migration affects several key IT infrastructure components. For example, it’s often an expected step when companies acquire new hardware, deploy new applications, or upgrade existing applications. It also can play a role in process redesign, especially when the movement and placement of data affect process performance.
Because data migration is an integral step in these frequent IT operations, any migration problems hinder your agility and responsiveness. If your data migration fails or slows processes, your costs increase. You have no choice but correct the migration, draining your IT resources.
It’s paramount to get the data migration right the first time, which requires teamwork. Gather an internal team that includes storage and networking personnel, as well as database administrators and application people who know the data, testing and quality assurance folks to validate quality, and compliance staff to approve documentation and audit provisions.
Next, here are some steps to take:
- Understand, select, and locate the data to migrate. You’ll need to know what data you’re migrating, where it resides, and what form it takes. Start your data migration by knowing about the sources of data you’ll move to your new system. The best place to get this information is by talking to the people that use the data. When you know where to locate the data and understand how it’s stored, you can determine which elements from each of your sources you’ll require. Finally, consider the form your data currently takes and the form it will take when it arrives at its new destination.
- Extract, clean, and transform the data. Almost all data contains some problems. Take your data migration as the perfect opportunity to clean your data—improve the quality by identifying any errors, omissions, or issues. This might include standardizing formats or enforcing naming conventions. Then you’ll need to transform your data. You’ll want to assess the format of the data and the format required for the new system, at which point you’ll need to determine how to get the old data into the new format. Fortunately, ETL and other automated tools can handle most of this process.
- Move the data systematically. Once you’ve determined how to transform your data, you begin the largely manual process of actually transforming it. Quite often, people perform data migrations ad hoc, taking various approaches at various times. It’s best to establish and enforce policies for migrating data. You might want to restrict data migrations to overnight hours when network usage is low, so it doesn’t interfere with regular business operations. Automated tools are available that can also help confirm a consistent process. Although it may seem obvious, you should also try to avoid irreversible transformations, and definitely store a backup copy of the original data. If the migration doesn’t work, you’ll want the ability to undo mistakes.
- Test and validate the migrated data. Migrating data over the network itself can result in errors. For this reason, we recommend that you check and test the migrated data to verify that it’s an accurate representation of the original data, and it’s in the expected format. In their rush to complete a data migration, organizations often skip this step. But without testing and validating the migrated data, you can’t be confident of its integrity. Ultimately, you want to be prepared for the most rigorous test available—user acceptance testing.
- Audit and document the process thoroughly. Once you’ve ensured the integrity of your data, you’ll need to prove it. In this age of compliance mandates, you should document what you did at each stage of the migration process. Then you should create and preserve a clear, traceable audit trail of who did what to which data and when. In effect, you want to document what you did and preserve the evidence that you did it.
While you may not migrate data often, it’s worth doing right the first time. By implementing these five steps you can dramatically reduce the likelihood of a data migration failure. At the same time, you can help your company become more responsive and avoid compliance problems. The best outcome from a data migration that goes well, however, is solid data integrity!