Ask the Experts: Data Purging and System Migration

  • Written By:
  • Published:

One reader recently wrote in with this question:

"... I wondered if you can point me at any related sites/groups/publications for more established users that discusses, for example

  • migrating between ERP systems,

  • purging ERP data,

  • integrating data from acquired companies into ERP and PLM"

We thought we’d take this on ourselves—but see the bottom of this post for more resources!


When migrating between systems, it is crucial to define the scope of implementation, as well as to outline each stage of the project and the resources that will be needed. A failed implementation will paralyze the operational capabilities of an organization, but the right methodology will help ensure a successful implementation.

In the issues related to the areas of ERP and PLM integration, we’ll highlight relevant areas of consideration. Furthermore, you’ll learn what steps can be taken to safeguard purging and data retention. This is a legal and mandatory business consideration.

We’ll assume for the purposes of this blog post that a new system exists, and that we are migrating data from an existing legacy system to a new ERP/PLM system. This can be viewed as an in-house system upgrade, or as migration of data from a purchased company.

Purging and Data Retention

When production databases become too large, they impact productivity by slowing access to information, and by extending the time required for system backups or for system restores.

Depending upon the industry (for example, medical, government, etc.), the need for data retention varies based on regulatory compliance. Some industries have long duration product guarantees, which results in the necessity to retain data.

Archiving has evolved into a discipline known as information lifecycle management (ILM). ILM helps organizations maximize the business management of storage from creation to disposal. Management is understandably reluctant to perform data purges due to the unknown operational risks, and it is therefore often done in stages.

Unstructured data populates file servers and typically includes e-mails, drawings, and user- and application-generated files in hundreds of unique formats. Purging can be by date, by type (internal or external), and by inbound or outbound status. Nevertheless, while many IT shops are only archiving e-mail to a less expensive tier of storage, they are still unwilling to permanently purge e-mail for legal or operational reasons.

The usual approach consists of transfer of data from active tables to online historical backups on a monthly basis. Since historical data is essentially invariant for long periods, it does not require being re-backed up if it had no changes. The backup facility may also make a second copy to non-rewritable storage. In the process of creating archives, an accompanying step is often taken to create summary data into a data-warehousing product for business intelligence studies. Summary data allows a look at a product’s sales figures for a given time period, by examining a single entry in a table rather then summing up individual sales order lines.

System Migration

Defining Your Needs
A migration project often starts with a feasibility study, which takes place before an implementation project gets off the ground. The approval process can include the board of directors or a high-ranking officer who sponsored the feasibility study for the project. Subsequently there is a lengthy process to build a cutover plan. This is necessary because errors or oversights at the beginning of an implementation project can be very costly to the organization down the line.

The feasibility study will address the following considerations:
System Upgrades
Upgrades are the migration from an older version of a product to a newer one. The vendor will usually provide a set of utilities (for example, to move from version X to X+1), but not always. A “not-always” situation would be in the migration from version X to version X+3 or X+4, where intervening versions must be jumped over.

Interfacing with a new system requires the maintenance of the old system along with major testing of the new system. Don’t neglect the importance of ensuring your web interfaces are also up-to-date. As I mentioned, the old system will have valid data for some time.

It may be that the switchover is done on a midnight when the decision taken is to remain on the live system, and not do a fall back. In this case, you will need to address the problem of what to do with your web site interfaces if something goes wrong. For this reason, you should only update these interfaces once you are sure the implementation is a success.

(Yes, some go-lives fail the first time. This happens mainly with large systems because they are of a more complex product design).

Historical Data Migration
Normally, ERP systems have a built-in provision for creating historical data. When is data transferred to historical archives? Some businesses archive data 24 months old, others do it at 36 months.

The move to historical tables is to reduce the number of records in “active tables” so that system response times are reasonable and system backups can be done in the window of time reserved for this operation.

Documentation and training
This is an important critical part of the migration or upgrade process. The usual practice is to make the key users responsible for guaranteeing the training of their respective groups. Any vendor-based courses deemed necessary are held a week or two before going live, sometimes on-site or at other times, off-site.

The last stage prior to “going live” is user acceptance testing (UAT) where the client tries out the system to ensure that everything is working properly and that the developer has fixed all the application bugs.

Once the cutover has been completed, the employees begin to work with the new system, performing their day-to-day tasks. If problems arise at this point, the key users/champions will be informed immediately. It is likely to take some time for the solution to work properly; in the meantime workarounds will be set in place until the problem is resolved.

Post-migration Review

After go-live, a post-project assessment is performed. This assessment is a checkpoint to determine if the system’s performance aligns with the project charter, as well as a way to audit the vendor. Did the vendor meet the requirements stated in the request for proposal (RFP)? If not, additional enhancements may be recommended as a secondary phase of the project. (Often it is not possible to deliver all software in the migration process.) The priority is given to completing migration rapidly and with essential software only, to mitigate the financial drain surrounding migration activities. In the post-implementation period, the applications or reports that are not mission-critical are evaluated, and coding can be scheduled for these deliverables.

The finance department examines the costs; if the migration is amortized over several years, implementation costs are accrued, and life starts. There may be a period of time when transactions such as sales order entry are “frozen” (i.e., orders are manually held for entry in the new system after the implementation).

Data Integration Considerations
Integrated system configuration is an area common to both ERP and PLM. Data compatibility with the combined PLM/ERP system requires common units of measure (UoM) and decimal precisions.

A UoM converter identifies how items are stored and how items are converted. Converters are introduced as pairs or words with multipliers (e.g., hammers are purchased by packages of 12, but sold individually). Sample UoM conversions are “dozen to each” and  “gross to dozen,” “box24 to ea,” etc.

In the migration process, the source quantity is converted to the target quantity, with a corresponding change to costs and re-order amounts. If the target quantity is not defined, the item will be rejected and have to be reprocessed after corrections are made.

Other preparatory work is required before importing the source data. It includes part numbers, customer details, bills of materials, and finance mappings from source system values to the target system standards. The system has to be able to incorporate these changes as part of its functionality.

There are a number of departments affected by the system implementation. Co-ordination of open sales orders, as well as open purchase orders and open production orders, must be completed prior to go-live.

Open production orders are works in process, and existing open production orders are allowed to complete on the old system in order to maintain a level of operational control for the business. New material requirements planning (MRP) runs may be done in the new system, to insure inventory is correct, by taking into account the open production orders active in the shop floor.

A Final Thought
We have only touched the surface concerning the topics of data purging and system migration. With our overview we hope you’ll gain an idea of the complexity of the processes and considerations involved.

Additional Resources

Harwood, Stephen. 2002.  ERP: The Implementation Cycle. Publisher: Butterworth-Heinemann

For project management templates, see These templates need some cosmetic reformatting in order to be useful for a migration project.
comments powered by Disqus