The Gap Between Running Experiments and Making Better Decisions

Most organisations treat experimentation as a testing function. The team runs tests, produces results, and shares them. Whether those results change any decisions is treated as someone else's problem. The assumption is that good evidence will naturally lead to good decisions. It does not. Evidence and decisions live in different systems, owned by different people, on different timelines. An experiment concludes on a Tuesday. The product decision it should inform was made three weeks ago. Or the result sits in a dashboard that the decision-maker never opens. Or the finding is communicated in a meeting where the relevant stakeholder was not present. The evidence existed. It just never reached the decision. This is the gap that determines whether your experimentation programme is worth the investment. Not whether it runs tests well, but whether the tests change anything.

Where the Gap Shows Up

Evidence is produced but not connected to decisions. The team completes an experiment and logs the result. The result says what happened. It does not say what should happen next. A decision protocol, written before the test launched, would state the planned action for each possible outcome. Without one, the result lands and someone eventually decides what to do with it, or nobody does, and the result quietly expires.

Ask your team: For the last ten completed experiments, show me the decision that followed each one. Not the result. The decision. If fewer than half have a documented decision, the programme is producing evidence that disappears into a gap between 'we know this' and 'we did something about it.'

Decisions are made without available evidence. This is the inverse problem. The organisation has experimentation data that is relevant to a decision being made, but the decision-makers do not know the evidence exists. A product team is redesigning checkout. The experimentation programme tested three checkout variations last year. But the product team does not know those results exist because they are stored in a tool they do not use, tagged in a way they would not search for, or remembered by someone who is not in the room.

Ask your team: When a product decision is being made, how does the decision-maker check whether we have relevant experiment results? If the answer is 'they would need to ask the experimentation team', evidence discovery depends on people knowing to ask. Most do not. The evidence sits unused.

Impact is claimed but not verified. The programme reports that a winning experiment will generate a projected revenue increase. That projection is based on the test data and a set of assumptions that may or may not hold in production. Six months later, nobody checks whether the projection was accurate. The claimed impact sits in a report as a fact that was never verified.

Ask your team: Of the impact figures we have claimed in the last year, how many have been validated against actual business performance? If the answer is none, the programme's impact case is built on projections that nobody has confirmed. That is a fragile foundation for a budget conversation.

Why It Persists

The gap persists because closing it requires work that is nobody's explicit job. The experimentation team's job is to run experiments and produce results. The product team's job is to build features and improve the product. The strategy team's job is to set direction and allocate resources. Connecting experiments to product decisions to strategic outcomes spans all three, which means it belongs to none of them, which means it does not happen reliably. In programmes where it does happen, it happens because one person, usually the programme lead, takes it on as a personal mission. They chase down decision-makers. They attend product meetings to surface relevant findings. They manually connect experiment results to business objectives in a spreadsheet. It works as long as that person is there and has the energy to keep doing it. When they stop or leave, the connections stop with them.

Ask yourself: Is the connection between experiments and decisions maintained by a system, or by a person? If a person, what happens when that person's attention is elsewhere?

What Closing the Gap Requires

Decision protocols before tests launch. Every experiment plan should state, before the test begins, what action will be taken for each possible outcome. If the test wins, what gets implemented and by whom? If it loses, what does the team do next? If it is inconclusive, who decides whether to extend, iterate, or abandon? This is not a bureaucratic exercise. It is a commitment to act on evidence before the evidence arrives. Without it, the gap between evidence and decision is filled by whatever happens to occur after the result comes in, which is usually nothing. Structured connections between experiments and objectives. Every experiment should be linked to a specific business objective. Not loosely. Explicitly. This connection allows two things. First, leadership can see at a glance what the programme is working on and whether it aligns to strategy. Second, at the end of a quarter, the programme can report impact by objective rather than by individual test, which is how leaders think about investment.

Ask your team: Can you show me every experiment connected to our Q3 objective of improving customer retention? If that query requires manually reviewing every experiment and mentally assessing relevance, the connections are not structured. They are implied.

Evidence surfaced at the point of decision. The knowledge the programme produces needs to be findable by people outside the experimentation team, at the moment they are making a decision. Not a week later when someone thinks to ask. Not in a monthly meeting. At the moment the decision is being made. This requires the programme's knowledge to be searchable by product area, customer segment, and business objective, and accessible to anyone in the organisation. If it lives in a tool that only the experimentation team uses, it is functionally invisible to everyone else.

The Question That Matters

The question is not 'are we running experiments well?' Most teams are. The question is 'are experiments changing decisions?' That is a harder question because it requires tracking something that most programmes do not track: the connection between what was learned and what was done about it. If your programme cannot demonstrate that connection with specifics, documented and auditable, the evidence it produces is valuable in theory and invisible in practice. The gap between experiments and decisions is where programme value goes to die, quietly, without anyone noticing, until someone asks 'what has this programme actually done for us?' and the answer takes too long to assemble.