Earlycallthe calculator that's safe to peek at →

One provable winner in 7,644 experiments

What the largest public archive of A/B tests actually proves — and why the answer should change how product teams make decisions, not how Upworthy wrote headlines.

Between 2013 and 2015, Upworthy — then one of the fastest-growing media sites in history — ran 32,487 randomized experiments on its headlines and images. The full results were released to researchers as the Upworthy Research Archive (Matias et al., Scientific Data, 2021): 150,817 experiment arms, over half a billion impressions, every impression and click preserved. It is the largest public record of real A/B tests in existence, and we replayed all of it through a modern decision engine.

In 7,644 of those experiments, Upworthy’s system formally declared a winner.

We asked one question: how many of those winners can actually be proven — not “had the best numbers,” but provably better than the runner-up, under statistical guarantees that survive the way people actually watch experiments?

The answer is one.

Not one percent. One experiment, out of 7,644. And here is the part we didn’t see coming: in that single provable contest, both variants had the identical headline. The winning arm differed in exactly one field — the photograph. The only thing 7,644 declared victories can prove, under honest statistics, is that one picture beat another picture, 5.17% to 3.39%, across 36,000 impressions.

Two years. Thirty-two thousand experiments. One certifiable win, and it wasn’t even a headline.

How we measured, in one paragraph

For each declared winner, we compared it against every rival arm in its test using an always-valid confidence sequence — a method whose 5% error guarantee holds at every moment you look, not just at a single pre-planned glance. We set the winning bar as low as it can meaningfully go: the winner merely had to be provably better at all, by any margin. Note that this test is biased in the winners’ favor: winners were selected for having the highest observed click-through rate, which inflates their apparent lead over the runner-up. Even with that head start, 7,643 of 7,644 could not clear the bar. (Thirty-eight — half of one percent — were provably better than at least one weaker rival.) Every number in this article is reproducible from the public archive with the open-source commands in the appendix.

Why almost nothing is provable here

Not because our statistics are exotic. Because of arithmetic.

A typical Upworthy test split traffic four to twelve ways, giving each arm a few thousand impressions at a baseline click-through rate around 1.3%. At that sample size, the statistical noise on a measured lift is roughly ±20–25% relative. Real headline effects — and the archive shows they are real, with a spread among the largest ever measured — are mostly smaller than that noise floor. When the noise is as big as the signal, no method of any school can certify individual winners; the information simply is not in the data. A once-only textbook test at the planned end would certify more than our deliberately strict standard, but only modestly more — and “check once and never look again” describes no experimenter who has ever had access to a live dashboard.

The archive’s own creators said as much in their release paper: most arms are statistically indistinguishable. What replaying every test adds is the exact number — and the number is one.

Here’s the twist: Upworthy was doing it right

It would be easy to read this as an indictment of Upworthy. It’s closer to the opposite.

Upworthy faced what decision theorists call a forced choice with cheap wrongness. A headline had to be published — there is no “keep running the test forever” option for a news story. And picking the second-best headline out of a nearly-tied field costs almost nothing. Under that loss function, “run a quick test, take the leader, move on” is close to optimal behavior. When we replay their tests under a priced-risk decision rule — pick the leading arm once the expected regret of doing so drops below a tolerance — the corpus looks genuinely good: roughly half of tests reach a confident priced decision within the traffic they actually spent, and the picks bank real expected lift. Judged as decisions, Upworthy’s process passes.

What doesn’t pass is the story attached to those decisions. The system called them winners — a word that claims proof. The data supported a much humbler and perfectly respectable claim: “best guess, cheaply obtained, probably about as good as anything else we wrote.” Upworthy made fine decisions and told a false story about them.

At Upworthy, the false story was mostly harmless. The trouble is what happened next.

The ritual escaped the newsroom

The “test it, find the significant winner, ship it” ritual became standard practice across product management — copied into a world with a completely different loss function. A product team’s wrong pick isn’t a slightly-worse headline that expires in a day. It’s engineering weeks, migration costs, a regression shipped to every user, and a rollback two months later. Cheap-wrongness rituals are expensive when wrongness stops being cheap.

And the standard tooling makes it worse. The significance calculators most teams use assume you check your experiment exactly once, at a pre-planned sample size. Checked weekly — the way every real team behaves — their false-positive rate isn’t 5%; it’s 25–50%. Every one of those false winners is a ship, a celebration, and a quiet rollback later. This is the mechanism by which a reasonable newsroom practice became an unreasonable engineering one: the certainty theater transferred, the cheap stakes didn’t.

The lesson is not “demand more certainty”

The naive response to “almost nothing is provable” is to demand bigger sample sizes and stricter thresholds everywhere. That’s wrong too — it would have paralyzed Upworthy, whose fast cheap picks were correct for their stakes.

The actual lesson is what we’d call decision hygiene: match the claim to the instrument, and the instrument to the stakes.

The failure mode connecting all three is the same one: deferring decisions to certainty that isn’t there. A “significant” badge on an unprovable pick doesn’t make the decision better. It makes it unexaminable.

One picture out of 7,644 contests is what proof actually looked like at Upworthy’s scale. Everything else was judgment — which is fine, as long as you call it judgment, price it honestly, and spend your certainty budget where it can actually buy something.

Methodology & reproducibility

Data: the Upworthy Research Archive (exploratory, confirmatory, and holdout releases; CC BY 4.0; Matias, Munger & Wright, Scientific Data 8, 195, 2021) — 32,394 tests with at least two arms of ≥1,000 impressions; 7,644 tests carrying a formally marked winner; 25,231 winner-versus-rival comparisons.

Method: two-arm always-valid confidence sequences (empirical-Bernstein, time-uniform; α = 5% per comparison), applied to each test’s final counts; the winner is “provable” against a rival when the conservative bound of the interval clears zero. Engine, audit tool, and scripts are open source; the full corpus audit runs in seconds:

```

comparisons built from the public CSVs; see scripts/ in the repo

bin/audit -comparisons winner_comparisons.json -min-effect 0.001 ```

Honest caveats: (1) Units are impressions, not unique viewers. (2) Our bound is deliberately conservative; the sharpest possible anytime-valid method would certify a small multiple more — a rounding difference against a four-order-of- magnitude gap. (3) None of this claims the declared winners were wrong — many were surely directionally right. It claims they were unproven: indistinguishable, on the data collected, from selection on noise. (4) The winner-vs-runner-up test is biased toward certification (winners were selected for high observed CTR), which makes the headline count an overestimate of provability, not an underestimate.

Written by the team behind Earlycall, the A/B calculator that’s safe to peek at — the same engine that produced every number above. Try it →