Five Questions That Tell You Whether Your Experimentation Programme Is Working

Most leaders evaluate their experimentation programme by asking how many tests the team ran last quarter. This is like evaluating a research department by how many papers it published without reading any of them. Volume tells you the team is busy. It tells you nothing about whether the programme is producing knowledge that changes decisions. These five questions cut through the activity reports. You do not need a technical background to ask them. You just need to watch how your team responds.

1. What decisions have changed because of experimentation this quarter?

Not what was tested. Not what the data showed. What decisions were made differently because of what the programme produced. If the answer is a confident list with specifics, your programme is influencing the business. If the answer is a pause followed by 'we shared the results with the product team', you do not actually know whether the programme changed anything. Results were produced. Whether anyone acted on them is a separate question, and the fact that your team cannot answer it quickly tells you that nobody is tracking the connection between evidence and decisions.

Ask your team: Show me the last five decisions that were directly informed by experiment results. Where is that documented?

2. What has the programme learned that we did not know twelve months ago?

Not individual test results. Patterns. Themes. Accumulated knowledge. A mature programme should be able to say something like: 'We have learned that our users respond strongly to social proof in checkout but are resistant to urgency messaging. We have tested urgency five times across three product areas and it has never outperformed control. We have also learned that our mobile users behave fundamentally differently from desktop users on pricing pages, which has changed how we approach pricing experiments.' If the answer is a list of individual test results rather than synthesised knowledge, the programme is generating data points without connecting them. Each experiment exists in isolation. Nobody is looking across the full body of work to identify what the organisation is actually learning.

Ask your team: What are the three most important things we have learned from experimentation in the last year? Not results. Learnings.

3. If the programme lead left tomorrow, what would we lose?

This question makes people uncomfortable, which is why it is useful. If the answer is 'nothing, everything is documented and searchable', your programme has institutional memory. If the answer involves a long pause, or 'they have a lot of context', or 'it would take a while to get someone up to speed', then a significant portion of the programme's knowledge lives in one person's head. That is not a programme. It is a dependency.

Ask your team: Without speaking to anyone, show me every experiment we have run in the checkout area in the last eighteen months, including what we learned and what decisions followed. Time how long it takes. That duration is the answer to this question.

4. How do we know we are not retesting things we have already tested?

Duplication is one of the most common and most invisible wastes in experimentation. A team of eight people across two years will inevitably test something that has already been tried, especially if people have joined or left the team in that time. The answer you want is: 'We have a searchable system that anyone can query by product area, theme, or objective. Before any experiment is approved, the plan is checked against past work.' The answer you probably have is: 'We try to remember' or 'we ask around.' That means duplication prevention depends on whoever happens to still be on the team and what they can recall. It also means nobody can quantify how often it happens, which means nobody can tell you how much resource it has wasted.

Ask your team: How many experiments have we run that tested something we had already tested before? If you do not know the answer, how would you find out?

5. What does the programme need from me that it is not getting?

This is the question most leaders never ask, and it is the one that reveals the most. If your team gives a diplomatic non-answer, they either do not trust you with the real answer or they have learned that asking does not produce results. Both are problems. If they give specific, concrete asks, like budget for tooling, executive sponsorship for a governance process, or air cover to push back on rushed tests, you are hearing what the programme actually needs to move from activity to impact. The uncomfortable follow-up: if the answer involves the programme needing infrastructure that connects experiments to decisions, tracks learnings over time, and makes knowledge accessible to everyone, and the team has been solving that problem with spreadsheets and willpower, the programme is being held together by individual effort rather than systems. That works until the individuals change.

Ask your team this one in private, not in a group setting: What is the single biggest thing holding this programme back that nobody talks about?

What These Questions Reveal

If your team answered all five quickly, with specifics, and could point you to documented evidence for each answer, your programme is in strong shape. If several answers required hedging, reconstruction, or phrases like 'I think' and 'probably', the programme is doing work but the work is not structured in a way that makes it accountable, searchable, or durable. That does not mean the team is doing a bad job. It means the infrastructure underneath the team is not doing its job. The people are working hard. The question is whether what they produce will still be accessible and useful in twelve months, or whether it will leave with whoever produced it. You do not need to understand experimentation methodology to evaluate your programme. You just need to ask questions that the programme can only answer well if it has the right infrastructure underneath it.