Your Regression Suite Is a Playlist

874 test cases in the repository. Something has to run tonight, and it is not all of them.

Ask most teams how their regression suite got its membership and the honest answer is: accumulation. Cases were added when someone had a bad release, and nothing was ever removed, because removing a test feels like accepting a risk while adding one feels free.

Ask the same question on a team that has been burned by a slow suite and you get the other answer: somebody trimmed it, by feel, one afternoon.

Both answers produce the same artefact. A set of tests whose membership nobody can explain. That is not a suite. It is a playlist — a collection someone liked at the time.

What a Selection Rule Actually Buys You

The point of a documented criterion is not tidiness. It is that you can be challenged.

Sprint review, someone asks why regression takes four hours, or why a particular area is not covered. With a rule, both questions have answers, and they are the same answer: here is the criterion, here is which side of it this case falls on. Without a rule you are defending a taste.

Applying five criteria to 874 cases produced a baseline of 57 — thirteen smoke, forty-four core regression. Every one of those 57 got in because it satisfied at least one named rule, and — this is the part that matters more — the other 817 were excluded for a stated reason rather than by omission.

The Five Rules

• Keep anything that moves money. Pricing, discounts, charges, credits, anything that ends up on a bill. A financial defect is not just expensive to fix, it is expensive per day it survives, and it is the class most likely to escalate outside the programme.
• Keep anything that creates or changes a contractual obligation. What the customer is committed to and what you are committed to them — term, activation, cancellation, renewal. These are the states you cannot quietly correct after the fact.
• Keep anything that crosses a system boundary. Every integration point is a place where two teams' assumptions meet and only one of them was updated. Integrations break from changes made somewhere you were not told about, which is exactly what regression exists for.
• Keep anything that fails without announcing itself. Non-obvious failures — a field populated with the wrong source, a discount with no end date, a status that never advances — are the ones no user reports and no error log records. If a defect would be invisible for a month, it belongs in regression.
• Keep the gateways. A gateway is a step everything downstream depends on: the journey entry point, the submit, the state transition that unlocks the rest of the process. If it is broken, the other cases cannot even be executed, so it earns its place by being the thing you find out about first.

Notice what none of these rules mention: how long the case takes to run, how recently it was written, or whether the area feels risky. Feelings are what the rules are there to replace.

The Suite That Never Changed

There is a second way to end up with a playlist, and it is more comfortable than the first because the suite looks stable.

On one project the regression pack was executed every cycle from the same set of scripts. Unchanged. New features shipped, journeys were re-cut, fields moved — and the scripts stayed as they were written.

The team's own verdict, said openly: these tests are useless.

They were right, and the reason is worth stating precisely. The suite had stopped describing the system. It was still describing a version of the system that had been true once. So it could not fail for any of the reasons the system was now capable of failing.

That suite was not protecting the product. It was protecting peace of mind, and it was doing that job very well — a green run every cycle, no matter what.

A regression suite that does not evolve with the system does not defend the system.

Which means the selection rules are not a one-off exercise. Every delivered story asks the same question of the baseline: does anything here satisfy one of the five rules, and if so, what comes out to make room?

The Suite That Ran Too Late

The third failure mode is about placement, not content.

Regression on one project ran only on UAT. The cases were fine. The coverage was reasonable. And the defects it found arrived too late to fix before the release, every time.

The reason was structural rather than lazy: the SIT environment had no backlog of its own and no capacity allocated for regression, so regression was always deferred to the one place where it definitely had to happen.

A regression suite that runs only in UAT is not regression. It is a last line of defence with no reserve behind it. Every finding becomes a release decision instead of a bug — fix it and slip, or ship it and note it — and both of those outcomes are decisions taken under time pressure by people who would rather not be taking them.

The smoke subset exists precisely for this. Thirteen cases is not a four-hour job. It is something that can run early, in a lower environment, on a build nobody has blessed yet, and it buys back the days that make a UAT finding fixable.

Why This Escapes

Because an unjustifiable suite still produces evidence. It runs, it reports, the numbers go into the pack, and nothing about the output distinguishes 57 defensible cases from 57 cases somebody liked.

The failure only becomes visible at the two moments when it is expensive: when a defect escapes through an area nobody chose to exclude, and when someone asks you to cut the suite in half by Friday and you have no principled way to decide what goes.

Write the rules down before either of those moments. Five lines, in the same document as the suite.

If you had to defend every test in your regression pack tomorrow, one by one — how many of them have a reason, and how many just have history?