Example

What you get at the end

Five questions about a change you are considering, and the design review hands back the page below — the question in plain words, the metric that decides it, how long it needs, and what everyone agreed to do at each outcome. Written before the data arrives, which is the only thing that makes it worth anything afterwards.

This is real output, not a mockup. The bet is invented — a size guide on a product page — but the record was generated by the same engine that answers the wizard, so what you see here is exactly what you would get.

Before the dataShip, keep, stop and abandon are agreed in advance, so the result cannot be re-argued into whatever was already wanted.

For whoever asksOne page you can send when someone asks why you shipped it — or why you spent six weeks not shipping it.

Names the weak spotEvery number you gave is re-run wrong on purpose, so you know which guess the answer actually depends on.

The record

Decision record: Put a size guide on the product page

Recorded 8 August 2026, before any data. A pre-registration, not a report: the bet, the bar and the rules for reading the result are fixed here in advance.

Fix this first. This traffic can only prove a win bigger than the change is likely to be. Run it as designed and you spend ~7 days and 895k visitors on a test whose best case is a shrug.

The call

Fix it first

A win means

+5% or better on add to cart on the product page

Time to an answer

~7 days

Fix this first

1. This traffic can only prove a win bigger than the change is likely to be

We can't price your ambition without knowing what kind of product this is: effect sizes differ about 60x between content and e-commerce, so borrowing the wrong corpus is worse than having none.

Skip it and: you spend ~7 days and 895k visitors on a test whose best case is a shrug.

Change this: Name the kind of product you run, or tell us roughly how much you expect this change to move the metric.

Also worth doing before you start

Arms. How many versions to run depends on how much effect there is to find, which is the same missing number. > Change this: Same fix: name the kind of product, or give your own expected effect.

Where else you could point this

Same change, same bar, different place to measure it. 2 of the 3 you named can answer inside your window; the rest cannot, at any effect size you would care about.

Place	Can it answer?	Time to an answer	Smallest it could prove
add to cart on the product page ← measuring here	yes	~7 days	2.6%
checkout completion	yes	~15 days	3.7%
orders from the home page	no	~6 weeks	6.1%

add to cart on the product page is measuring here because it answers fastest. The next best, checkout completion, would take ~15 days.

What would change our mind — agreed before the data

Ship it when the conservative end of the estimate clears +5% on add to cart on the product page.
Keep what we have if the new version is credibly worse.
Stop early if site-wide orders degrades past 2% — regardless of how good the main metric looks.
Abandon if the effect is real but credibly smaller than +5%: not worth more traffic.

These rules do not change once data arrives. That is the whole point of writing them down now, and they stand for the fixed version of this bet too.

What this will not tell us

Whether the business is better off. People substitute and shift behaviour elsewhere, so a gain on add to cart on the product page is an upper bound on the business-wide gain, never a translation — site-wide orders is the guardrail watching for that.

Which of your numbers this rests on

One of the four. We re-ran the whole review with each one wrong by half; the ones marked below change what it finds, so each is worth a minute before you spend anything.

What we assumed	You said	If it were wrong
Share of visitors who meet this change	about 1 in 3 visitors	no change — safe to leave as an estimate
Traffic to that surface	120k per day	no change — safe to leave as an estimate
Current rate on that surface	6.4%	no change — safe to leave as an estimate
Smallest improvement worth shipping	5%	halving the bar changes what this review finds, so this number is doing real work — it should be a business judgement rather than a habit

The design, gate by gate

Scope — Deciding on the surface itself: ~7 days to resolve a 5% improvement — 26x cheaper than deciding on "site-wide orders", which would need ~16 weeks.

Ambition — We can't price your ambition without knowing what kind of product this is: effect sizes differ about 60x between content and e-commerce, so borrowing the wrong corpus is worse than having none.

Change this: Name the kind of product you run, or tell us roughly how much you expect this change to move the metric.

Arms — How many versions to run depends on how much effect there is to find, which is the same missing number.

Change this: Same fix: name the kind of product, or give your own expected effect.

Risk shape — Compiled to a move-fast policy: reversible in minutes or hours, so priced risk beats a long wait for certainty; "site-wide orders" guards the downside at a 2% harm threshold rather than trying to prove the upside.

How these numbers were reached

Every figure is reproducible from your own inputs; the corpus figures come from open datasets you can check yourself.

Time to an answer. At a 6.4% rate and 120k units/day, resolving a 5% effect needs about 895k units in total — ~7 days. This bound holds under continuous monitoring, so you may look whenever you like without inflating the false-positive rate. (Verified: 0 false ships in 150,000 real-traffic A/A splits across 538M units.)

The scope penalty. about 1 in 3 visitors meet this change, so a 5% local effect appears as 1.75% business-wide. Traffic scales with the square of the effect, so the same change needs about 23.0M units on site-wide orders against 895k on add to cart on the product page. Measured across real e-commerce surfaces this penalty runs from 8x to 17,826x.

Registration — the technical channel

{
  "experiment_id": "put-a-size-guide-on-the-product-page",
  "hypothesis": "Put a size guide on the product page",
  "variants": [
    {
      "id": "control",
      "expected_share": 0.5
    },
    {
      "id": "treatment",
      "expected_share": 0.5
    }
  ],
  "unit": "unit",
  "primary_metric": {
    "name": "add to cart on the product page",
    "kind": "binary",
    "direction": "increase",
    "baseline": 0.064
  },
  "guardrail_metrics": [
    {
      "name": "site-wide orders",
      "kind": "binary",
      "direction": "increase",
      "harm_threshold": 0.02,
      "baseline": 0.021
    }
  ],
  "decision_policy": {
    "preset": "move_fast",
    "alpha": 0.1,
    "min_effect": 0.05,
    "tau": 0.01,
    "burn_in": 250,
    "max_duration": "672h"
  },
  "expected_units_per_day": 120000
}

Do this for your own bet

Five questions, about two minutes. No account, nothing stored — the numbers you type are answered and discarded.

Design a test See the evidence first