A Static Finding Is a Candidate. One Run Prices It.

A configuration diff told me there was a billing exposure. It was right about the record and wrong about the consequence.

One order fixed that. Not one hour of analysis. One order.

What the analysis found

Comparing the declared coverage against what was actually deployed surfaced a single mismatch. One orchestration scenario was declared as covered by the gate, and carried no gate clause of its own.

That is a real fact, and the inference from it is the obvious one. An order entering through that scenario is not gated. An ungated order proceeds. Proceeding means the billing system gets something it should not have got.

Declared as covered, not covered, money involved. If you have ever read a config diff, you already know how compelling that reads on the page.

I filed it as an exposure and assigned it a severity on that basis.

What one run showed

Then I built one order and entered through that exact scenario.

Every item ended skipped. Same outcome as the designed path — indistinguishable from it.

The reason is a level down. Every item definition beneath that plan carries the gate clause itself. The scenario is the entry, but the items are where the work would have happened, and each of them refuses independently. An order entering through the ungated scenario walks into a plan whose every step declines to run.

So the missing clause is not an exposure. It is a redundant record — a mismatch between what the coverage document claims and what the configuration says, with no behavioural consequence at all. Cosmetic. Worth fixing for the sake of the next person reading the config, worth nothing as a risk.

Reliable about facts, unreliable about consequences

That gap is the whole mechanism, and it generalises well beyond this feature.

A configuration diff reads records. It is excellent at that — better than a human, faster, and it does not get bored on instance 60 of 84. When it says a clause is absent from a record, the clause is absent. Take that as fact.

What it cannot see is an order.

It has no model of what happens to something that actually enters through that record: what evaluates next, what sits beneath it, which of the layers below would independently refuse. Those consequences live in the runtime, and the runtime is the only thing that knows them.

So static analysis produces facts with an implied severity attached, and the implied severity is the tester's own inference. The inference is where the error is. Not the finding.

The rule

Treat every finding from configuration analysis as a candidate carrying an assumed severity, and let one runtime execution price it.

Candidate, not defect. Assumed, not assessed.

That is not a demotion of static analysis — the analysis is what found the thing, and no amount of execution would have surfaced a missing clause in a record nobody thought to enter through. It is a statement about what kind of evidence it is. It nominates. It does not price.

The run prices it, and pricing runs both directions. A run can confirm the assumed severity, and then you have a finding with a demonstration attached, which is a far stronger ticket than a screenshot of a diff. It can also do what this one did.

How to do it without spending your week

The cost of pricing a static finding is much lower than it looks, because you are not testing the feature. You are answering one question.

• Write the assumed severity down when you file, and label it assumed. One line: "flagged by configuration comparison, severity assumed pending one execution." This is what stops the number hardening into a fact while the ticket sits in a queue.
• Pick the single entry that exercises the finding and nothing else. Not a scenario suite, not a regression pass. The narrowest order that enters through the flagged record, run once, with the outcome of every step recorded.
• Look one level down before you conclude anything. This is the specific lesson from this case. Configuration features are usually layered — a plan and its items, a page and its fields, a gate at entry and a gate at execution. A clause missing at one level tells you nothing until you know whether the level beneath carries it.

Publish the downgrade

Here is the part people skip, and it is the part that decides whether any of this survives.

When the run downgrades your own finding, say so. In the ticket, in the same place the original severity is recorded, in your own words: analysis flagged it as X, one execution showed Y, revised to cosmetic.

The temptation is to quietly close it. Nobody enjoys writing down that their exposure was a redundant record, and a downgraded finding feels like it makes the analysis look bad.

It does the opposite. A tester whose static findings are always confirmed is a tester nobody checks; a tester who publishes both the confirmations and the downgrades is one whose next finding gets read at face value. The credibility of everything you flag by analysis rests on the visible fact that you price it before you defend it.

And the reverse is what actually destroys the practice. File four config findings that turn out to have no behavioural consequence, never correct any of them, and the fifth one — the real exposure — arrives into a room that has learned to discount you.

Why this escapes

Because static findings arrive feeling finished.

There is a record, there is a rule, there is a discrepancy between them. It has the texture of evidence — written down, repeatable, and it took real skill to find, which makes it expensive to treat as provisional.

Meanwhile the run that would price it feels like a chore. You already know the answer, and building one order to confirm what the configuration plainly says looks like ceremony.

It is not ceremony. Here it was the difference between an exposure and a typo, and it cost one order.

Look at the last finding you raised from configuration analysis. Did anything actually run through it — or did the diff decide the severity on its own?