Testability Is a Property of the Org, Not of Your Test Steps

Nine mappings in production. Zero in the sandbox where the fix was about to be verified.

The scenario would have passed. It would have passed cleanly, in about eight minutes, and the result would have been worth nothing at all.

What the Org Said When Somebody Asked It

The fix changed how case ownership is assigned. The new logic resolves the user's Role and routes from there, so the code path depends on Role-to-queue mappings existing in the org.

Rather than run the journey and see, a diagnostic script read the org's local queue metadata and counted those mappings. Production had around nine. The sandbox in front of me had none.

That is not a data gap. It is a coverage statement. Every branch that reads a mapping is unreachable in this environment, so every scenario that exercises them resolves through the fallback and reports success.

The test steps were fine. The test steps are almost always fine. The org could not run them.

Pass and Not-Tested Look Identical in Every Report

This is the part that makes the failure durable rather than embarrassing.

A test that cannot exercise its code path does not error. It does not warn. It produces the same green tick as a test that exercised the path perfectly, and that tick travels into the run summary, the sign-off email and everyone's mental model of what is covered.

Nothing downstream can tell the two apart. The distinction exists only in the head of the person who ran it, for about a day, and then it stops existing anywhere.

Which is why the environment check has to happen before the verdict, not after the failure. Once "pass" is written down, it is indistinguishable from the real thing forever.

The Second Shape: The Path That Was Never Sick

A different fix, a few weeks earlier, failed the same way from the opposite direction.

The environment could run the journey. It just could not run that journey. The original defect lived on the external self-service buying journey, and that channel can only be exercised in the integration and acceptance environments — one of which the team reaches roughly once per release.

The fix was scheduled for production anyway, verified on QA.

Read that carefully, because it sounds reasonable. The fix was tested. The test passed. The change is the same change regardless of which channel calls it.

Except the defect was channel-specific. That was the whole finding. So "verified on QA" answers the question "does this work on the channel that was never broken", and everybody hears it as "the fix works where it broke". Those are different sentences and only one of them was true.

The green tick was on the path that was never sick.

Two Questions Before the Verdict

Neither of these takes long. Both of them change what your result means.

• Ask whether the org holds the configuration this code path needs. Not whether the feature is deployed — whether the metadata, queues, roles, price entries or record types the branch reads actually exist here. A count from the org beats a memory of the last release every time.
• Ask which channel or environment the original defect lived in, and whether that exact path is reproducible where you are testing. If it is not, you are not verifying the fix. You are verifying everything except the fix.

The second question is the one that gets skipped, because by retest time the defect has been reduced to a title and a root cause, and the channel it came from has fallen off the ticket.

Say It Out Loud in the Sign-Off

You will not always win the argument. Sometimes the release goes anyway, because the alternative is waiting a full cycle for an environment the programme cannot open on demand.

That is a legitimate business call. It stops being legitimate the moment it is made silently.

So the sign-off carries the limitation in its own sentence, not buried in a comment thread:

"Verified on QA. The channel where the defect was reported is not reproducible in this environment, so that path stays unverified until the integration window."

One sentence. It converts an invisible gap into an accepted risk with a name on it, and it survives into the release notes, where the person deciding whether to push can actually read it.

It also protects the next person. When the same defect reappears in production, that sentence is the difference between "the fix didn't work" and "the fix was never tested where it mattered", and those two conclusions send investigations in completely opposite directions.

Testability Belongs to the Environment

The habit worth building is small. Before judging any fix pass or fail, spend five minutes establishing that the org in front of you can express the failure at all.

Testability is not a property of your test steps. It is a property of the org configuration, the channel availability, and the data that happens to exist in that sandbox this week. Your steps can be immaculate and still be a walk-through of a path that has nothing behind it.

The uncomfortable consequence is that a test result is only as trustworthy as somebody's knowledge of the environment it ran in — and that knowledge is usually nobody's documented job.

Of the tests you signed off last sprint, how many could the environment have actually failed?