How to Define Success Before the Test Runs

Here is the most common way an experimentation programme corrupts itself, and it does not feel like corruption from the inside. A test runs. The hypothesis does not pan out, but a secondary metric moved nicely. The team, under pressure to show impact, reports the secondary metric as the result. The experiment is remembered as a win. The hypothesis was never validated, but nobody records that, and the insight that enters the repository was never actually tested. Do this a few times and every result the programme reports becomes unfalsifiable. If success is defined after the results arrive, every test wins, and a programme where every test wins has stopped learning. The protection is simple and old: define the call before the data exists.

What to register

Before launch, write down five things.

The primary metric. One metric, named in advance, that the hypothesis makes a claim about. Not a family of metrics to be chosen from later.

The decision threshold. What size and direction of change counts. "An improvement" is not a threshold; "a relative increase of at least two percent" is.

The duration or sample. How long the test runs or how many users it needs, decided from a power calculation or an honest estimate, not from watching the dashboard until the number looks good. Stopping when significance first appears is choosing the result.

The decision rule. What happens in each outcome. If it wins, we do X. If it loses, we do Y. If it is inconclusive, we do Z. A test with no pre-agreed consequence for losing is not a test; it is a formality on the way to a decision already made.

The hypothesis link. What belief this test examines, so that the outcome is recorded against the belief, not against whichever number moved.

Where to record it

Anywhere that is timestamped and unedited after launch. The brief, the ticket, the log. The medium does not matter. The immutability does. A success definition that can be quietly revised after results arrive provides no protection at all; it just adds a paper trail to the goalpost-moving.

What to do with secondary metrics

Secondary movements are not worthless. They are hypotheses, not findings. The honest handling is to label them exploratory, record them as candidate beliefs, and test them properly if they matter. The dishonest handling is to promote them to results. The difference between a learning programme and a theatre programme is exactly this label.

When the test loses

A pre-registered loss is not a failure of the programme; it is the programme working. The belief was examined, the answer was no, and the organisation now knows something it did not. Record the outcome against the hypothesis, apply the decision rule you committed to, and resist the retrospective search for a metric that flatters. The credibility of every future win you report is built out of the losses you reported honestly.