The Spec You Can Read Is Not the Spec That Runs

Three artefacts described one formula. All three disagreed.

The story came with a written acceptance criterion, an attached spreadsheet labelled "calculated example", and the shipped Apex class. Step 2.1 of the criterion divided Total Contract Value by Total Days in contract. The workbook did not divide at all. Neither did the code.

The human-readable requirement was the one that was wrong.

Why That Is Uncomfortable

Testers are trained into a specific arrangement. The acceptance criterion is the specification. The build is the thing under test. You compare the second against the first, and where they differ, the build loses.

That arrangement works right up to the moment the criterion is the defective artefact. Then it inverts silently, and nothing in your process announces the inversion.

Trusting the prose here would have baked a wrong expected value into every test case on that story. Every one would have failed. Every failure would have looked like a real defect in a class that was doing exactly what the business wanted.

Or — and this is the outcome worth worrying about — someone notices the tests failing, decides the test data must be off, adjusts the numbers until things go green, and never records that the criterion and the build were describing two different formulas.

What Actually Settles It

Only one of the three artefacts executes.

The criterion is a person's summary of a calculation. The workbook is a person's arithmetic. Both are readable, both are useful, and neither of them is what runs when an order goes through.

So the oracle is a golden vector: one concrete input set, one expected output, reproduced against the actual class. Not derived from the prose. Not copied out of the workbook. Executed.

Once you have that, the disagreement stops being a mystery and becomes a list. Here is what the code does with these inputs. Here is what the criterion says it should do. Here is what the workbook says. Two of those are documentation defects, and documentation defects get tickets like anything else.

Two Out of Three Is Not a Vote

It is tempting to read this as majority rule — the workbook and the code agreed, so the criterion was the odd one out.

Resist that, because agreement between two artefacts often means one was produced from the other. A worked example built by opening the class and following it is not independent evidence. It is the same claim, restated in a spreadsheet.

The question to ask before you count votes: was this example derived from the build, or from the requirement? If nobody remembers, the example carries no weight at all, and you are back to reproducing the vector yourself.

The Case Where No Artefact Wins

A harder version of the same problem showed up in a discount popup.

The screen displays a recurring price of £22 — the promotional price for that product. Apply a 40% custom discount and the engine calculates against the £57 catalogue price instead. The result drives the cart to zero.

That reads like a pricing bug. It is not, or not only. The reporter could not write a clean Expected Result, and said so on the ticket: it is genuinely ambiguous whether a manual percentage should apply to the price the agent is looking at or to the catalogue price underneath it.

Nobody had written that rule down. Not in the criterion, not in the design, not anywhere.

So "Expected: £13.20" is unwriteable. It is not a fact about the system, it is one tester's intuition about which base a discount stacks on — and if it goes on the ticket as an expected result, that intuition quietly becomes the specification.

The Two Layers

These are two different failures, and they need two different reflexes.

• When artefacts disagree, find the one that executes. Reproduce a golden vector against the build, then raise the non-matching documents as wording defects. Never adjust your test data to match prose you have not reproduced against.
• When no artefact answers the question, do not invent one. File "which of these two numbers is the source of truth?" rather than "wrong result", and get the answer in writing before anybody writes an expected value.

The second one is harder because it does not look like testing. It looks like being blocked. A ticket that asks a question instead of asserting a failure feels weaker in triage, and there is real pressure to convert it into something that reads like a defect.

That pressure is the mechanism. An escaped defect is very often an assumption nobody wrote down, and the fastest way to keep it undocumented is for a tester to fill the gap with their own reasonable guess.

What This Changes in Practice

Before you write expected values for any formula, spend the twenty minutes.

Take one input set. Compute the expected output by hand from the criterion. Compute it from the worked example. Run it against the build. Three numbers on one line. If they match, you have a formula you can test against with confidence, and you have spent twenty minutes buying that confidence.

If they do not match, you have found the most valuable thing you will find on that story, and you found it before writing a single test case rather than during execution when everything is expensive.

The output of that exercise is not a pass or a fail. It is a decision about which artefact is authoritative, recorded where the next person can find it.

The Deliverable Nobody Asks For

Naming the ambiguity is the work.

A tester who reports "the calculation is wrong" against a build that is doing what the business wanted has produced noise. A tester who reports "the criterion, the example and the class describe three different formulas, the class is authoritative for these inputs, and the criterion needs a wording fix" has produced the thing the team actually lacked.

Same investigation. Different deliverable. The second one survives after the ticket is closed.

When your test fails, how do you decide whether the build is wrong — or the sentence you tested it against?