Data migration — Moving and cleaning the data the whole program depends on

This is the ninth article in the series "Inside a Large ERP Program." The previous chapters followed the program through build, testing, and authorizations. This one turns to the workstream the whole go-live rests on, and the two things that make it the most dangerous part of the program even though no one underestimates its importance.

A few weeks before go-live, the third mock load fails. Not on something exotic, on the data that runs the business: article master data, open items on the customer and vendor accounts. The kind of data you can't operate without, because without it there's no clearing, no payment run, no collection. The business impact is real and immediate, and for the first time the question on the table isn't "how do we fix this" but "do we still go live." When data migration derails, it doesn't push a task back by a few days. It pushes the go-live itself.

That's why it's the riskiest and most important stream on the program. And the interesting part is that almost nobody underestimates the migration itself. What gets underestimated is what makes it fail.

What isn't the risk: moving the data

Start by clearing away the obvious suspect. Migrating data, in the narrow sense of extracting it from one system and loading it into another, is a tooled and well-understood problem. The mechanics are not where programs get hurt. Everyone knows the migration is critical, it gets staffed and planned as critical, and the transport itself rarely fails on its own terms. The failure pattern practitioners keep describing isn't the transport, it's the scheduling: migration gets treated as the last ten percent of the project when the profiling, cleansing and reconciliation belong in the first thirty to forty, and once that work is deferred to the final weeks every defect becomes a go-live blocker with no time left to fix it. The point is consistent across post-mortems of failed implementations, where the root cause traces back to the data layer rather than the software.

So if the part everyone watches isn't the part that breaks, the risk is sitting somewhere people aren't looking, underneath the transport itself.

The real risk factor: legacy data quality

What gets underestimated is the state of the data in the legacy system. The cleaning is always longer and more expensive than planned, because the assumption going in is that the data is cleaner than it actually turns out to be. Years of workarounds, half-retired records, fields quietly repurposed for something other than their original meaning, duplicates nobody ever reconciled: all of it surfaces the moment you try to move it into a system that enforces rules the old one let slide.

Cleaning that data happens in two stages, with two different owners. First, the business teams clean at the source, inside the legacy system, wherever the data can be corrected there. They know what a record is supposed to mean, so they're the ones who can decide whether it's right. Then IT takes over for the mass cleansing, applying rules that can't be enforced in the legacy system at all, transforming the data inside the ETL tool. The two stages aren't interchangeable. Skip the business-led cleaning at the source and IT is left mass-transforming data it doesn't fully understand the meaning of.

That's the risk on a brownfield conversion. On a greenfield build, there's a second layer on top of it.

Greenfield: you don't transport, you transform

On a greenfield program you're not moving data into the same shape it had before. The target is a redesign, and that changes what migration even means.

It's not only a new chart of accounts or a new organizational model, though it's often those too. It's the transactional business rules themselves. If the project moves the company onto IFRS, the data has to be restated to fit. If it decides that flows which used to run through an external tool now run inside the ERP, that data has to be brought in and reshaped to rules it was never captured under. Each of those decisions turns migration from a transport problem into a transformation problem. You're not carrying the data across, you're rebuilding it to fit a structure and a set of rules it never lived in. That's the greenfield surcharge, and it sits on top of the cleaning.

Cleaning and transformation would already make migration the heaviest stream. But what makes it the most dangerous is where it sits relative to everything else.

The vise: caught between quality below and rules above

Here is what makes migration unique on the program. It's exposed on two sides at once, and it controls neither.

On one side, it's the foundation downstream. The migrated data is the base of the transactional flows every other stream runs on, so a defect in migration surfaces as someone else's problem, a test postponed or failed, in build, in the functional streams, anywhere that needs real data to prove a process works. The migration doesn't get to fail quietly in its own corner. When it's wrong, it blocks everyone.

On the other side, it's the receptacle upstream. The migration is where the business rules land. So when a rule changes during testing, because the business refines a point, decides a mapping was wrong, adjusts a transformation, the migration is hit in return. And this happens often. It isn't a rare event you can treat as an exception. The consequence is concrete and brutal for the migration team: they rework the same objects far more times than there are mock loads. The "three mocks" is a floor, a planning convenience. The real count of how many times an object gets rebuilt is set by how many times the rules above it move.

So the migration sits in a vise. The quality below it is worse than assumed, and the rules above it won't hold still. Neither pressure originates in the migration stream, and both are paid for there.

What the mock loads reveal, in order

The mock loads are where that vise becomes visible, one turn at a time. Three are usually enough, and they aren't three repetitions of the same exercise. They're three different things.

The first mock is technical. Its job is to prepare and shake out the tooling and the transformation rules, to prove the pipes work, not to land business-perfect data. The two that follow are business mocks, and each one aims to land as close to the go-live target as it can, on cleaned data and real volumes this time. Each pass uncovers a layer of reality the previous one hid: the first business mock exposes what the technical mock's clean assumptions glossed over, and the last one is meant to be the dress rehearsal.

Which is exactly why the third mock failing is the moment the whole chapter opened on. By then there's no margin left to absorb it, the data that fails is the data the business can't operate without, and the rules may have moved again since the second mock. The vise closes at the worst possible time, on the data that matters most.

The way out: make rule changes visible and costly

The mistake is to treat this as a technical problem and ask the migration team to be faster or more careful. The pressure on them isn't technical in the first place, so a technical answer won't relieve it. What relieves it is governance.

Every mid-project change to a business rule, for the migration and for every other stream, has to go through a committee, typically a Design Authority Board that arbitrates gaps. The lever that protects the program is how you classify those changes. You push them through as project gaps, a change of scope, rather than as business gaps, an expected new feature the business is simply entitled to. That distinction does two things at once. It protects the IT teams from unbounded rework, because a change now has a visible cost and an owner. And it forces the business to think hard about whether a rule change is genuinely worth it, because changing a rule is no longer free, it's a decision someone has to defend in committee.

It's the same principle that runs through the reporting stream: the discipline that cuts rework isn't applied in the stream that suffers it, it's applied upstream, on the decisions that destabilize it. We will cover this part deeply in the next chapter.

What to take from this

Data migration is not a technical transport problem you execute at the end of the project. It's the most exposed stream on the program, caught between legacy data quality that's worse than anyone assumed and business rules that won't stop moving through testing. You don't secure it by asking the team to migrate better. You secure it by governing what destabilizes it: cleaning the legacy data early and at the right level, and arbitrating every rule change so the migration doesn't silently absorb the cost of a business decision nobody had to defend.

A program that lets the rules drift without arbitration pays for it in migration rework, and eventually pays for it in the date. And whatever comes through this stream, in whatever state, is the same data the next stream has to display: that's where the reports get built, and rebuilt. Reporting — Building the same reports more than once.