How to Run an Experiment Review Board

Think about the last experiment your team launched. Before it went live, did anyone review whether the hypothesis was falsifiable? Whether the measurement plan would actually detect the effect size claimed? Whether a decision protocol had been written? If the answer is 'yes, we have a review process', try the sharper question: is that review based on a consistent checklist, or does it depend on whoever is reviewing? If quality assurance varies with the reviewer, you do not have governance. You have opinion. Three to four regular members. Weekly, fortnightly, or ad hoc depending on velocity.

What Gets Reviewed

Hypothesis quality, evidence base, measurement plan, decision protocol, objective alignment, duplication check, feasibility and risk.

Checkpoint: When the reviewer wants to verify the evidence cited in a plan, can they find the original source quickly? Or does verification require asking the plan author to dig them up?

Checkpoint: Look at your team's last ten completed experiments. How many had a decision protocol written before launch, not after?

Checkpoint: When the review board asks 'has this been tested before?', what happens? Does someone search a system and get an answer in seconds? Or does the room go quiet while people try to remember?

Running the Review

Share plans 24 hours in advance. Provide a consistent review checklist.

Checkpoint: Does your team have a written review checklist that every experiment is assessed against?

Three to five minutes present, five to seven minutes questions, two minutes decision (approved, approved with conditions, sent back). Ten to twelve minutes per experiment.

After the Review

Record decisions. Track quality trends: first-pass approval rate, common reasons for rejection, quality trends over time.

Checkpoint: Can you answer any of those questions right now? If your review board has been running for more than a quarter and you cannot state the first-pass approval rate, you are not tracking the output of the process.

What Good Looks Like

A well-run review board improves plan quality without slowing the programme down. The question is whether the quality criteria are embedded in the experiment template, enforced at the point of capture, and consistently applied, or whether they exist only in the heads of the people on the board.