How to Build a Quarterly Experimentation Report for Executives

Try building your quarterly executive report right now. Just the first section: progress against this quarter's experimentation objectives. For each objective, state how many experiments ran against it, what was learned, and what decisions were made. How long did that take?

What to Include

1. Programme summary against objectives.

Checkpoint: Can you produce this section right now? If doing so requires opening your testing tool, a strategy document, and a spreadsheet and manually piecing them together, the connection between experiments and objectives is not maintained during the quarter. It is reconstructed at the end.

2. Key learnings. Three to five most significant things the organisation now knows.

Checkpoint: Can you name the top three learnings from this quarter without reviewing individual experiment results?

3. Decisions and actions.

Checkpoint: How many decisions were made as a result of experimentation this quarter? If the answer is 'I'm not sure', you are not tracking the most important output of the programme.

4. Impact. Quantify rigorously. Do not inflate.

5. Programme health. First-pass approval rate, decision completion rate, objective coverage, learning documentation rate.

Checkpoint: Can you produce any of these metrics right now?

6. Next quarter outlook.

What Good Looks Like

A strong quarterly report takes less than a day to assemble because the underlying data is already structured. If the checkpoints revealed that assembling the report requires significant manual effort, the problem is not the report format. It is that the data the report draws from is scattered.