Stop Racing the System

The same test can pass and fail on the same build. The build is not the variable.

The variable is the moment you looked.

Three Shapes of the Same Mistake

I have hit this in three different forms in one journey, and only after the third one did they stop looking like three separate problems.

The work has not finished yet. The broadband provisioning item in the activation orchestration plan closes asynchronously. A test that asserts Order Completed or Orchestration Item Closed immediately after submit is not asserting anything stable — it produces a pass or a fail depending on how fast the environment was that morning. Both results are meaningless, and the pass is the more dangerous one, because nobody investigates a pass.

The work finishes in stages. The equipment shipment item closes in two steps: a dispatch update closes the first batch of tasks, and proof of delivery closes the rest. To anyone who has not walked the full flow, those two look like one event. Assert "closed" after dispatch and you get a green tick against a half-closed item — and the system never raises an error for that state, because the state is legal. It is simply not the end.

The work finishes and then reverses. In the externally initiated ordering journey, a termination raised before the point of no return returns an API success. The termination order is then auto-cancelled immediately afterwards. The same endpoint, called against an already-activated connection, returns error 1212. A tester running the termination case against a fresh pre-activation account sees a 200, marks it passed, and never learns that the order they created no longer exists.

What Makes All Three Silent

None of these throws an error. That is the property they share, and it is the property that lets them survive review.

Each one has an intermediate state that is entirely valid. An orchestration item mid-run is not broken. A half-closed shipment is not broken. An HTTP 200 followed by an auto-cancel is not a failed call — the call genuinely succeeded, and then a rule fired.

So there is nothing for the system to complain about. The only thing that is wrong is where the assertion landed, and an assertion has no way to tell you it landed early. It tells you what it saw. What it saw was true at the time.

That is the part worth sitting with. These are not defects in the system. They are defects in the test, and they report as evidence about the system.

The Rule

Name the terminal state before you write a single step.

Not "the order is submitted." Not "the item closed." The final, settled state that this scenario is supposed to end in, and the observable record you will read it from.

If you cannot name it, you are not ready to write the test — you are about to write a test whose result depends on timing, staging or a rule that fires after the response you were watching.

Everything else follows from having named it.

Four Steps That Come Out of That

• Give async work a documented wait and an explicit refresh. The wait needs a number in the test case, not the word "wait." The refresh needs to be its own step, because a page that was rendered before the item closed will keep telling you the old answer for as long as you leave it open.
• Split a multi-stage closure into one step per stage. Two stages, two steps, two assertions. A merged "update and then check" step is structurally incapable of detecting the half-closed state, because the only moment where the difference is visible falls inside a step that has already been marked as one action.
• Assert on the record, never on the transport. The response code tells you the request was accepted. The order status tells you what the system decided to do with it. When a flow has a post-call rule that can reverse the action, only the second one is a result.
• Write down what the pre-condition changes. The pre-activation termination and the post-activation termination look identical at the API call step and have opposite outcomes. If the test case does not state which state the account is in, whoever runs it next has a coin flip in their hands.

Why This Escapes Good Testers

Because the synchronous mental model is the correct one almost everywhere else.

Click, respond, verify. That loop is how forms work, how validation works, how most of a Salesforce org behaves. Orchestration plans break it quietly: the UI hands back control immediately and the actual work continues behind the screen, in a different transaction, at a speed nobody documented.

The first time you meet it, async behaviour looks like a bug. You raise it, someone explains that the item closes on its own a moment later, and the explanation resolves the confusion without ever becoming a rule. Nothing gets written down, because it was not a defect. So the next person meets it fresh, on a different item, and does the same thing.

And there is a second reason, less comfortable. A test that races the system passes most of the time. Most of the time is enough to build confidence in it, enough for it to survive review, and enough for its occasional red result to get re-run instead of investigated. Flaky is the word we use for tests whose assertion is in the wrong place, and calling it flaky is how it stays.

Take the last async assertion you wrote. Do you know how long the system needs — or do you know that it usually works?