The Design Doc Changed and Nobody Told Me

I opened the technical design for a second time, three weeks after the first read.

New title. New authors. A section I had built test cases from was gone.

The change log at the bottom still said: v0.1, initial draft.

What Actually Changed

The document had been near-totally rewritten. Not edited — rewritten, and with a change of purpose that nobody announced.

The first version described intended design: what the feature must do, written before it existed. The second version described the build: what the org currently does, written after.

Those are different documents wearing the same filename.

The rewrite also deleted the entire section documenting the checkout enforcement gate — the mechanism that actually blocks an agent from proceeding with an unapproved discount. And the one open item it had been carrying was deferred to a section that was never written.

None of this was flagged. The change log was not updated, because updating a change log is a discipline and not a system constraint.

Why This Costs Real Money

I had test cases sourced from the deleted section. One of them covered five entry points into the feature. Those test cases still existed, still looked valid, and now pointed at nothing.

If I had not re-read the document, I would have executed them. Some would have passed for the wrong reason. Others would have failed and been debated for a day before anyone worked out that the requirement they were built from had been quietly withdrawn.

There is a second, worse consequence.

An as-built document is structurally useless as a test oracle. It records what the system does, not what must be true. So "the doc matches the build" becomes trivially, permanently guaranteed. You can never fail a test against it. The moment a specification turns into a description, your source of truth stops being a source of truth and becomes a mirror.

That transition is silent. Same page, same link, same name in your test case's Reference field.

The Read That Found a Defect Before Any Test Ran

There is an upside to reading documents properly, and it showed up in the same session.

The rewritten design listed the approve/reject notification emails — required by the story's own acceptance criteria — under a heading called required additions (flow delta). Meaning: not built.

The story was already in Testing.

That is a defect found without touching the org, and it is a specific kind of defect worth naming: a technical design that lists AC-required behaviour as a future delta is a bug report the development team wrote about themselves. It just has not been filed by anyone.

The same read predicted two test case failures before execution, which changed the conversation from "these tests failed" to "these tests will fail, here is why, do you want to fix the build or the AC first."

The Rejected Finding That Got Quietly Fixed

One more thing surfaced in that diff.

A finding I had raised earlier — and which had been rejected with "the design doc is just a reference" — had been silently corrected in the rewrite. Exactly as reported.

So the finding was not wrong. It was inconvenient. And the fix arrived through a channel that does not count towards anyone's QA metrics.

Log those. Not to win an argument, and not to keep score against individuals — but because "rejected" and "invalid" are different states, and the gap between them is the only evidence you will have at the next process retrospective when someone claims QA raises too much noise.

What To Do About It

Four habits, none of them expensive.

• Snapshot every design document on first read. Copy it to your own workspace with a date. Not a link — a copy. Links show you today's version and tell you nothing about what you read three weeks ago.
• Diff it on every return visit. Even a quick eyeball of section headings will catch a rewrite. The change log is a claim, not a fact, and treating it as a fact is how you end up testing against a document that no longer exists.
• Flag the migration from specification to as-built. The moment a document starts describing the build, say so out loud and ask where the expected results now come from. Usually the answer is the acceptance criteria plus a business decision nobody has made yet — and finding that out is more valuable than any test you were about to run.
• Record the reference at section level in your test cases. "TDD §7.3" survives a rewrite as evidence. "See the TDD" does not.

Reading Is a Test Level

The industry treats document review as preparation for testing. On this evidence it is testing — with its own defect yield, its own techniques, and defects that are cheaper to fix than anything you find by execution.

The read that found the missing notification cost forty minutes and no environment. The alternative was finding it in UAT, after a build, a deployment and three people's time.

When did you last compare a design document against the version you originally read — and would you be able to, if you wanted to?