Config Releases Ship Data, and Data Doesn't Get Code Review

Three Highest-priority defects landed in UAT on the same day. All on the same new product family.

Two wrong promotion configurations on two variants, and an activation fee discount granted to a variant that should never have received one.

Two days later a second new product family broke in the same environment. This time the agent could not add the product to the cart at all.

The Discount Applied Correctly

Take the activation fee discount case on its own.

A functional test asks: did the discount apply? Run it and the answer is yes. The discount applied. It applied cleanly, at the right value, on the right line, and it flowed through to everything downstream exactly as designed.

The defect is that the product was never supposed to get it.

There is nothing wrong with the process logic. The engine did what an engine does. What was wrong was a mapping — which product variant qualifies for which promotion, which one qualifies for a fee discount — and a mapping is data, not behaviour.

Functional testing routinely walks straight past that layer, because every question it asks is phrased in terms of mechanics. Does the calculation work. Does the discount appear. Does the total update. All yes. All useless here.

The Assertion That Was Missing

The question nobody had written down was: should this product receive this discount at all?

That reframes the test in a way that matters. "Discount calculated correctly" is a statement about the pricing engine. "This variant qualifies and that one does not" is a statement about configuration data, and it needs its own assertion, including the negative half.

The negative half is where these defects live. It is easy to check that an eligible product gets its promotion — you can see it, the number changes, the case passes. Checking that an ineligible product gets nothing requires knowing which products are ineligible, which requires reading the eligibility matrix rather than the story.

Two variants of a family look identical on screen. One qualifies. One does not. Nothing in the cart tells you which, and a test case written from the story will happily exercise whichever one the tester picked first.

Two Families in One Week Is Not Bad Luck

One product family failing in UAT is an incident. You fix it, you retest it, you move on.

Two families in one week, failing at completely different layers — promotion eligibility on one, basic cart addition on the other — is not a coincidence you can fix ticket by ticket. Those two defects have nothing in common technically. They have everything in common procedurally.

New product configurations were reaching UAT without any QA pass at all. Not an inadequate pass. None.

Which means the cheapest possible check in the entire process — add each new SKU to the cart once, see what happens — belonged to nobody.

Why Configuration Escapes Every Gate You Own

Code has a pipeline. Someone writes it, someone reviews it, unit tests run against it, it gets deployed as an artefact with a version and a history.

Configuration data has almost none of that. Someone types values into records or metadata, often under time pressure, often from a spreadsheet that was itself assembled by hand. There is no pull request. There is no reviewer asking "should this variant really map to that promotion?" There is no compile step that fails when a mapping is wrong, because no mapping is syntactically wrong.

Then it ships. And the first time anyone executes it end to end is when a business user in UAT tries to sell the product.

The perception layer makes it worse. Config releases get described as "just config", which sets the expected risk to near zero, which sets the allocated test effort to match. The word "just" is doing an enormous amount of work in that sentence, and none of it is true — a wrong mapping produces exactly the same customer-facing damage as a wrong line of Apex, and it arrives faster because nothing slowed it down.

The Gate That Would Have Caught All Four

This does not need a test plan. It needs three checks per new SKU, run once, before UAT sees it.

• Add it to the cart. One action, and it would have caught the second family's failure entirely, before a business user ever hit it. If a product cannot be added, nothing else about it matters.
• Verify eligibility in both directions. Confirm the promotion and any fee discount apply where they should — and confirm they do not apply where they should not. The negative assertion is the one that catches a mis-mapped variant, and it costs one extra cart.
• Confirm the orchestration plan completes. Configuration that works in the cart can still fail on the way out, and the plan finishing is the cheapest available proof that the SKU is wired into the downstream flow.

Three checks. A handful of minutes per SKU. Against four Highest-priority defects in one week, two of which reached business users.

The Real Finding

The bugs were the symptom. The finding is that a mandatory activity had no owner.

Nobody decided to skip the smoke pass on config releases. There was no meeting where that risk was accepted. The activity simply did not exist in anybody's definition of done, so it did not happen, and its absence was invisible until two product families failed within a week of each other and someone read the tickets side by side.

That is what an unowned gate looks like from the inside. Not a decision — an empty space between two roles, discovered by the defects that fall through it.

When a config-only release goes out on your project, who runs the first execution — and are you sure the answer is not "a user in UAT"?