Successful Database Migration
October 28, 2008
One of the issues that companies are facing in their operations is database migration to new hardware, or a different operating system platform, or even a new database vendor. There are many steps and factors involved in this process. This article will describe the overall process and interviews with companies that went through this process.
According to a recent Bloor Research survey, the data migration market is estimated to be over $5 billion and growing. Companies perform their database migration in many different ways, such as extract-transform-load (ETL), replication, and manual scripts. This task gets more complex as the data volume grows on one hand and available downtime is shrinking on the other one.
Statistics presented in Data Migration in the Global 2000 by Bloor Research in September 2007 shows the following picture:
Migration has to be carefully planned and executed to minimize production downtime and maintain data integrity and database performance, because many companies have locations worldwide and have to be operational 24x7.
Our interviews with people from a wide range of industries on the topic of database migrations have shown that legacy system migrations are one of the top reasons to switch over to new systems according to the interviewees. Total cost of ownership (TCO) of new systems is typically lower as out-of-date systems are more expensive to maintain. Some companies perform database migration due to the equipment lease ending and being replaced by a newer system. Once migration is complete, the target system will be up-to-date and supported by hardware and software vendors. The new system will have a wider range of features and will be more environmentally friendly.
Since databases are more difficult to migrate than a set of files, the migration process and resource allocation have to be carefully planned and customized to the specific environment to minimize production downtime.
There are various types of database migration such as:
All these types of data migration are independent of each other, but some are often combined together to make it more efficient and to take advantage of the overall process. One of the most common types of migration is the database upgrade to the latest release. This is usually done in conjunction with hardware upgrade and / or OS upgrade. Any of these data migration paths involve installation of the new database server on the target system. Usually, it is the latest database release.
Most of the interviewees said that the source and the target platforms are different. One representative of a global provider of enterprise-class solutions and technologies said their target system is Windows, while the source is a combination of Windows, Linux, and UNIX. Mohican Laine from Overstock.com mentioned that their target platform is 64-bit Linux, while the source one is 32-bit. According to Daniel Suciu at Vodafone target and source systems had different operating systems and even databases.
Regardless of the migration type, there are common steps to be performed to ensure a successful migration. These steps are:
Additional tasks may have to be performed depending on the type of the migration. These tasks may be creating the new database structure when moving to a different database platform or data cleansing to remove obsolete or redundant information. Data cleansing and consistency were the top challenging issues named by the respondents among compatibility and time constraints.
There are also third party solutions, such as GoldenGate Software that provides a transactional Data Management (TDM) platform to enable high volumes of transactional data to be moved continuously between enterprise systems with sub-second speed, preserved data integrity, and with minimum impact to those systems.
Overstock.com selected GoldenGate Software to move customer data from their Oracle 9i databases supporting their retail site into the Teradata Warehouse running the Teradata V2R6 database. Mohican Laine said that they evaluated Quests Shareplex software and also Oracles embedded database replication software. We went with GoldenGate due to implementation simplicity and ease of use.
Another solution is by Informatica that offers Data Migration Suite. It is a set of several products that can be used for various scenarios and has the option of a perpetual license or project-bound license. The project-bound license is limited in term to 6, 12, or 18 months.
Celona offers data migration software that can be used in various migration scenarios, such as event based, incremental, bulk load and others.
Danes John said that when working for Vaman Technologies they analyzed most of the migration tools available, so in some cases we can use third party tool and [in] others we developed in-house.
There are other vendors that provide similar solutions that may be more applicable in your situation. These solutions can assist with various data migration scenarios and reduce downtime or eliminate it entirely.
Database migration is a complex project that has to be thoroughly analyzed. Determine the migration type and the necessary tools and resources to complete it. Various approaches have to be fully reviewed and understood prior to choosing one.
Migration can be done in-house or outsourced to a consulting practice. Consider using third-party solutions to prevent hand-coding migration scripts that will be time consuming and rather complex if data cleansing is involved. In addition, these solutions can reduce or eliminate the downtime and facilitate the replication between the systems. Outsourcing will bring in the best practices and appropriate tools to complete the project.
About the author
Alex Polishchuk is the founder and president of Advanced Computer Consulting (www.advcomputerconsulting.com) that provides database consulting services. Alex has over fifteen years of professional experience administrating, designing, developing, and implementing database applications in various industries and companies ranging from small to Fortune 50 corporations. Alexs primary areas of expertise are in database security and performance optimization and tuning. He can be contacted at firstname.lastname@example.org.