Your Coverage Number Is a Definition, Not a Measurement

One sentence rewrote a programme's entire coverage report. No test case was deleted. No test was re-run.

The sentence was: only Approved test cases count as coverage.

Applied to the actual export — 4,626 test cases, 81% of them sitting in Draft — most of the reported coverage stopped being coverage and became what it always was: a count of documents someone had started writing.

Why Nobody Catches This

Coverage figures get attacked on one axis: did we test the right thing?

Everyone braces for that argument. It is the interesting one, the one that gets a slide in the retrospective, and it is genuinely hard to settle because it depends on risk, judgement and hindsight.

Almost nobody asks the cheaper question first: which record states does this number quietly include?

So a figure computed over unreviewed, never-dry-run drafts travels through weekly reporting for years. It is not a lie. Nobody constructed it to deceive. It simply counts rows in a system where the qualifying condition was never stated, and in the absence of a stated condition, the tool counts everything.

The Cheapest Correction Available to a QA Lead

Defining the qualifying status costs one meeting and produces no new artefacts.

It also usually halves the number — and if a definition can halve it, then the number was never the one anyone thought they were reading.

That is worth saying out loud to whoever receives the report, and it is worth saying before someone else discovers it. A coverage figure that drops from 68% to 34% because you tightened the definition is a governance win. The same drop discovered by an auditor is an incident.

Three questions to put to your own report this week:

• Which record states does this count? Draft, In Review, Approved, Deprecated — name them.
• Has anyone dry-run these cases? A case that has never been executed by a second person is a plan, not a test.
• Where is the zero that the average is hiding? Which brings us to the second axis.

The Zero Behind the Average

On the same programme, a gap analysis found 101 stories with no test cases at all. That is 32.2% of everything delivered — which sounds like a margin you could live with.

Then we split those 101 into functional gaps (60) and technical or configuration items (41), and the picture changed completely. Entire functional areas had nothing. Salesforce security enforcement: 16 stories, zero cases. Change of ownership: 5 stories, zero cases.

The headline number was 67.8% coverage. Acceptable, on paper.

That average was hiding a zero in the area where a defect is least visible and most expensive — the one where a permissions mistake does not throw an error, it just quietly shows the wrong records to the wrong person.

Coverage without risk classification is a vanity metric. It answers how much did we write and lets everyone hear how safe are we.

What to Report Instead

A single percentage cannot carry this. Replace it with three numbers that fit on one line each.

• Qualified coverage. Stories with at least one Approved, dry-run test case. This is the number you defend.
• Zero-coverage areas, by functional area. Not by story count — by area. "Security enforcement: 0 of 16" tells a stakeholder something that "67.8%" never will.
• Draft debt. How many cases exist but do not qualify, and how long they have been sitting there. This is the number that says whether the first one is improving.

None of that requires a new tool. It requires deciding what qualifies before you count.

The Part That Is Uncomfortable

Reclassifying most of your programme's coverage as fiction is not a popular thing to do in a status meeting. It reads as bad news, and it will be received as your bad news, even though you did not create it.

The alternative is worse. Every week you report the old figure is a week someone plans a release around it.

The number was already wrong. Defining it just made the wrongness visible on a date you chose.

What status do the test cases in your coverage report actually have — and when did anyone last check?