Prove Uniformity by Query, Spend Execution on the Outliers

The scope said verify 78 configuration records. I ran somewhere between six and eight test cases.

The gap between those two numbers is not a shortcut. It is a piece of work, and it has to be written down somewhere specific or it reads as laziness.

The trap that config-driven features set

A feature driven by configuration hands you a beautiful, terrible list. Every configured record looks like a test case. They are enumerable, they are countable, and counting them produces a scope number that survives every planning conversation intact, because nobody can argue with "there are 78 of them."

So you plan 78 executions, you get part of the way through, and the sprint ends.

What that plan assumes — silently, without anyone stating it — is that the 78 records might differ from each other in ways that matter. That assumption is usually false, and more importantly it is checkable.

Records produced by the same deployment carry the same clause. That is not a hope. It is a property of how the configuration got there, and it can be proven from the data in minutes.

What the query showed

I extracted every instance of the gate clause — all 84 places it appears — and compared them against each other.

Byte-identical. Every one.

And two structural facts that matter more than the sameness: the clause is never nested under an OR, and it is always effectively ANDed at the top level.

Those two properties are what make the uniformity load-bearing. A clause that is identical everywhere but sits under an OR in six places is not uniform in behaviour — the surrounding structure can neutralise it. Identical text plus top-level AND means the same input produces the same decision in all 84 places. That is a proof about behaviour, not just about strings.

Proving the same thing by execution would have taken weeks. And it would still have missed the record that has no clause at all, because a record with nothing to check does not announce itself while you are busy checking the ones that do.

What the scope became

Once uniformity is proven, the question changes shape.

It is no longer "does each of the 78 records behave correctly." It is "does the mechanism work, and what are the exceptions."

That resolves into: prove the mechanism once per entry route, plus the handful of exceptions the same query surfaced. Six to eight test cases, estimated.

The mechanism is what you actually want executed. Does the clause, evaluated for real by the platform, produce the outcome the acceptance criteria describe — once per distinct route into the feature, because the routes are what genuinely differ. Everything else was already answered by the comparison.

The query that justifies the reduction is the same query that finds the exceptions. That is the sentence worth keeping. You do not do one piece of work to save time and a second piece of work to get coverage. Extracting all 84 instances to prove they agree is exactly the operation that surfaces the ones that do not, and those are the highest-value executions in the whole scope — far better than whatever fraction of identical records you would have got through by hand.

The reconciliation half

Uniformity of the clause is one axis. The other is whether the records that were declared to exist are the records that actually got deployed.

That comparison ran too: declared against implemented, reconciling 38 of 38 in one set and 40 of 41 in the other.

Note what 40 of 41 is. It is not a rounding error and it is not noise. It is one record that does not line up, found by a comparison that took minutes, sitting in a population that manual execution would very likely never have reached.

And note what 38 of 38 is. It is a genuine, evidenced statement that a whole population is accounted for — which is a stronger claim than "we tested a sample of them and they were fine."

Now write it where the coverage is judged

Here is where this practice usually dies.

The analysis happens. The tester convinces themselves, correctly, that 78 executions are unnecessary. They run their six to eight, they attach the results, and the ticket shows a small number of executions against a large configured population.

Read by anyone who was not in the tester's head, that is thin testing. Not deliberately priced scope — thin testing. The reasoning that makes it rigorous is the one part that never left the tester's brain.

So the proof goes into the ticket, next to the executions, in the same place the coverage is judged. Not in the wiki, not in a chat thread, not in a stand-up sentence. The same artefact.

Including an explicit sentence saying the executions are not being run and why: the gate clause is identical in all 84 places it appears, never under an OR, always top-level ANDed; declared and implemented reconcile 38/38 and 40/41; the 78 per-record executions are therefore not being run, and the budget goes to the mechanism per entry route plus the exceptions the extract surfaced.

That sentence is the deliverable. Without it there is a proof and a small number of runs in two different places, and whoever reviews the coverage only ever sees one of them.

The mirror

There is a companion rule to this one, and they only work as a pair.

One runtime execution is what prices a finding from static analysis — the diff nominates, the run tells you whether it matters. This is the same relationship running the other way: the static proof is what licenses running fewer executions.

Analysis without execution is a claim nobody has to accept. Execution without analysis is expensive and still incomplete. The pair is what makes each half credible, and publishing both halves in the same place is what makes that visible to somebody who was not there.

Neither is a way of doing less. Both are ways of choosing what the expensive thing gets spent on.

Look at your current test scope. How much of it is one property you could prove by query in ten minutes — and where would you write the proof so it counted?