The Safest Story in the Sample
Two stories were picked as the safe baseline in a documentation audit because both had 100% approved coverage. One turned out to be the most defect-dense item in the sample.
Every sample needs a control. Something you already believe is fine, so you can see what "fine" looks like.
I picked the two stories with the best test coverage in the set. Both were wrong choices, and one of them was the worst item in the whole sample.
Why the Control Existed
The pilot was a documentation audit: ten items, read the requirement against what was actually built, record where they diverge.
An audit like that needs a baseline. If everything you inspect turns up drift, you cannot tell whether you have found a real problem or simply invented a very sensitive instrument. So two of the ten were chosen as controls — the two with the highest test coverage in the sample, both sitting at 100% Approved.
The reasoning was not lazy. High coverage means cases exist, they were executed, and somebody signed them off. If any item in a ten-story sample deserves to be treated as the safe end of the range, it is that one.
What the Control Actually Showed
Both control stories carried real drift between the documented requirement and the delivered build.
One of them turned out to be the single most defect-dense item in the entire sample. Four linked live bugs, including a discount that silently failed to apply on the customer's bill.
Read that pairing again, because it is the finding. The story with the best coverage numbers in the sample was the story carrying the most defects in the sample. Not despite the coverage — alongside it, entirely undisturbed by it.
What Coverage Was Actually Reporting
The instinct being tested here is that high coverage is a proxy for low risk. Almost everyone in delivery shares it, including me, including whoever reads the coverage report in a steering meeting and feels reassured.
Coverage answers one question precisely: were tests written against this story, and were they run and approved.
It says nothing about what version of the requirement those tests were written against. Nothing about whether the requirement was complete when the cases were authored, or still agreed by the time the build happened, or whether a decision taken in a call three weeks later ever made it back into either the document or the suite.
So a story can sit at 100% Approved and be fully covered against a requirement that no longer describes what was built. Every case passes. Every case is honest. The whole set is aimed slightly to the left of the target.
That is the shape of drift, and coverage is structurally incapable of showing it. The measurement is taken from inside the test suite, and the suite is precisely the thing that has drifted.
What This Was Not
It was not a bad test suite. The cases were written and executed by competent people, and they did what they said they did.
It was not a coverage-definition problem either — nobody was inflating the number or counting the wrong things. The figure was accurate, and irrelevant to the question everyone was using it to answer.
And it was not one unlucky story. Both controls drifted. Two out of two.
The Second Axis
The conclusion I took from the pilot is that an escape-analysis process needs two axes, not one.
The first is coverage: did we test it. Cheap to measure, already reported, and the only one most programmes have.
The second is fidelity: did we test the right thing — does the documented requirement still match what was built, and does the suite still match the requirement. That one cannot be computed from a test management tool. It requires somebody to read the design next to the org and write down where they disagree.
A programme with only the first axis will always place its confidence where the numbers are highest. Which, on this evidence, is not obviously where the risk is lowest.
Why the Blind Spot Prefers That Corner
There is a reason drift hides best behind good coverage, and it is not statistical.
Attention follows suspicion. Stories with gaps in the suite get re-read, re-tested and questioned in every review they appear in. Stories at 100% get skipped — by the reviewer, by the escape analysis, by the person deciding what to spot-check before a release.
A fully covered story is the one place in your backlog where nobody is looking. That is exactly the property that makes it a good hiding place, and it is why I would now start an audit there rather than end it there.
If you had to pick one story from your last release to re-read against its design, would you pick the one with the best coverage?