Testing — Proving the system works before you turn it on

This is the seventh of thirteen chapters in "Inside a Large ERP Program." The earlier chapters set out the triple lens the book reads each phase through, then walked from scoping to the build. This one covers testing, the phase a program treats as proof it is ready, and the phase whose proof is worth exactly as much as the decisions nobody watched being made.

Testing presents itself as the objective part of a program. After all the judgment calls of scoping and design, here at last is a measure: the system either passes or it doesn't. A green UAT, and you're ready.

It doesn't work like that. A green UAT does not say "the system works." It says "what we decided to test worked." And everything that matters is hidden inside that decision, what got tested, who was responsible for it, which defects were allowed to wait, because that decision is made earlier than people think, more quietly than they'd like, and often for worse reasons than they'd admit. Testing isn't a measurement. It's the only insurance a program buys itself, and like any insurance, its worth is decided when the coverage is written, not when the claim comes in.

The cascade and the level that gets sacrificed

The textbook cascade is well established, and worth stating once so we can talk about where it bends. Unit testing checks each piece of code or configuration on its own. System integration testing, SIT, checks that the modules actually talk to each other. UAT puts the business in front of real scenarios on production-like data. Regression confirms that what worked before still works after a change. Four levels, each catching what the one below it couldn't.

On paper. In practice, when the schedule tightens, and it always tightens, one level gets sacrificed before the others, and it's almost always SIT. The reason is visibility. UAT has a business audience watching, so cutting it is politically loud. Unit testing is the developers' own floor, cut it and the build visibly falls apart. Regression has the weight of "don't break production" behind it. SIT has none of that. No business audience, no immediate go-live pressure, just consultants checking integration in a room nobody important is watching. So it's the first thing pared back when days go missing.

Which is the worst possible choice, because SIT is the level that tests the seams between modules, and tightly-coupled SAP breaks at the seams more than anywhere else. Cut SIT to save a week and you've reduced your coverage in exactly the place the risk is highest and the visibility is lowest. The insurance gets thinner precisely where the claims will come from, and nobody in the room feels it happen, because the thing you stopped testing is the thing whose failure shows up two months later.

Two things called UAT

Here's a fight I've watched up close, inside a large consulting group, between managers who could not agree on what a UAT even is.

One camp held that UAT is essentially a copy of the SIT. The business sits down and replays the scenarios the functional consultants just ran, confirming the same flows give the same results. The other camp held that SIT only tests what the consultants understand of the flow, and that the business should build its own UATs from its own view of how the work really goes. The two definitions sound close. They are not. They certify completely different things.

My own position is the second one, firmly, and the reason is about responsibility. A UAT that just replays the SIT lets the business off the hook entirely. It never forces them to go looking for what the system will actually be like once they're living in it after go-live. They confirm the consultant's flow and sign. Whereas a UAT the business builds itself forces them to imagine their own real day, the awkward customer, the split delivery, the credit note against an invoice that was already partly paid, and to test whether the system survives it. The first kind proves the consultant can re-run their own demo. The second kind proves the system holds up against reality. Same three letters, two utterly different insurance policies, and most programs never notice they bought one when they thought they were buying the other.

This is the heart of why a green UAT can mean so little. If "UAT passed" was generated by the copy-the-SIT model, the certificate attests to almost nothing about the post-go-live world. The coverage was written narrow, and no one read the policy.

The happy path certifies only good weather

The narrow policy has a signature shape, and the testing literature even has a name for the trap: testing only the happy paths creates false confidence. Feed the system the clean case, the one everything was designed around, and of course it passes. The cases that break a system are the ugly ones, the null value, the out-of-sequence event, the quantity that doesn't divide evenly, and those are exactly the ones a happy-path script never contains.

This connects straight back to the two UATs. The copy-the-SIT model is a happy-path machine by construction, because the consultant's SIT scenario is the designed case, the one built to work. Hand that to the business as their UAT script and they walk the golden path, tick every box, and certify a system that has never once been shown a problem. A green UAT built only on happy paths is an insurance policy that pays out only when the weather is good, which is to say when you didn't need it. The whole value of a test is in the cases you were afraid of, and a script handed down from the people who built the thing contains none of them, because they built it not to have any.

Triage: deciding what the policy won't cover

At some point every program decides, explicitly or not, which defects it will fix before go-live and which it will carry past it. This is the moment the coverage gets its exclusions written, and it deserves to be done well.

Done well, it has a real criterion, and mine is concrete. A defect can wait past go-live when three things hold: the close is only weakly impacted, invoicing remains possible, and the manual workaround is manageable. Those three, close, invoicing, workaround, are the load-bearing functions of a running business. If a defect leaves all three standing, the business can live with it for a while, and deferring it is a legitimate call against real risk.

Done badly, the same triage runs on the wrong fuel. A defect gets stamped "minor" because the schedule can't absorb a fix, or because the stream that owns it has more political weight than the stream that reported it, not because the risk is actually low. And here's the mechanism that makes deferred defects come back to bite: the manual workaround that justified the deferral as "temporary" almost never is. It quietly becomes permanent, and the cost of it lands on the run team, every close, every month, as a tax nobody priced when they waved the defect through. Deciding what your insurance won't cover is a normal, necessary act. Doing it for schedule or politics instead of risk is how a program insures itself against the wrong things.

The risk nobody remembers to insure: legacy regression

There's a category of risk that sits outside the visible project, which is exactly why it goes uncovered. The new system doesn't land in a vacuum. The legacy doesn't disappear, as the architecture chapter argued, and the moment you plug the new system into the existing landscape, you can break something in a peripheral system that nobody thought to retest, because it wasn't on the project's map.

The testing literature is blunt about why: because SAP modules are so tightly integrated, even a small change, a pricing condition, a tax setting, can ripple into processes far from where it was made. Now extend that beyond SAP's own modules to every interfaced system around it, and you have a regression surface far larger than the project's test scope, which was drawn around what the project built, not around everything the project touches.

There's a subtlety worth knowing, too. In the SAP world, full regression testing is often positioned for the Run phase after go-live, aimed at catching what version upgrades disturb. Which means a project frequently doesn't run complete regression against the connected legacy before go-live at all, it discovers those regressions in production, once real transactions start crossing the interfaces. The coverage gap isn't an oversight by careless people. It's structural: the risk lives in the space between systems, and the test scope was drawn around a single system. You insure what you can see, and the connected legacy is the part you've half-forgotten is still load-bearing.

Who decides the coverage

Step back to who is actually making these calls day to day, because it isn't the program director. It's the test manager and the stream leads, in the unglamorous flow of triaging defects, signing off levels, and deciding what "tested enough" means this week.

That's the point worth ending on. Every one of those daily decisions is a coverage decision on the program's insurance, and most of them get made implicitly, absorbed into the schedule rather than named. The test manager who concedes the SIT to hold a date has made a decision about coverage without calling it one. The lead who accepts a copy-the-SIT UAT because the business is busy has narrowed the policy without telling anyone the policy got narrower. The good ones make these calls explicit and at the right level, on a criterion of risk: this is what we're choosing not to test, here's why, here's who carries the consequence if we're wrong. The weak ones let the calendar make the calls and call the result a test plan.

A program isn't judged by whether its UAT went green. It's judged by what that green was built to cover, and whether the people who narrowed the coverage did it on purpose, for reasons they could defend, or let it happen because a date was coming.

Once the system is tested for what you decided it should prove, the next gap opens around the people who'll use it, and the rights they need to do their jobs, rights that can only be designed once the processes behind them stop moving. That's where the next chapter goes: Authorizations — Assigning roles too early, and rebuilding them later.