What moved (right is better for every metric; bars are 95% intervals for the primary, 90% for guardrails)
Change the numbers
Counts per arm. "Users" is the denominator: everyone assigned whose window has closed, or for a subset metric, only the users who got through that step. Thresholds are relative to control: the minimum worthwhile effect for the primary, the margin for a guardrail.
Leave planned users or days empty if the test had no written plan.
| Metric | Role | Better | Control count | Control users | Test count | Test users | Threshold % | Subset |
|---|
Segments, tracking audits and the full request as JSON
Segments (for the sample ratio check by segment) and tracking audits (counts of events no real user can produce) are easiest to edit here. The format is documented in the README.
Size a test for every check, not just the primary
A real winner ships only if the primary wins and every guardrail rules out harm. So the test needs enough users for each check. Following Spotify's correction (Schultzberg, Ankargren and Frånberg, 2024), each of the checks gets an equal share of the miss rate.
| Guardrail | Control rate % | Margin, % relative |
|---|
How it was tested
136 simulated A/B tests on a couples app, generated from raw event logs. Each test has a true effect known exactly. They were frozen in git before any engine code existed. 48 were used for development; 88 were held out and run once with the engine frozen. Three of the eleven case types appear only in the held-out set. The comparison is the rule most teams use (ship if the primary is up with p < 0.05), with and without a significance veto on guardrails. How the cases were built · Every run
By case type
How it decides, in order
- Can the data be trusted? A sample ratio check (chi-square against the planned split, overall and by segment, failing at p < 0.001) and tracking audits: counts of events no real user can produce, such as a shared game with no partner, a user in both arms, or a partner in the other arm. Any failure means "fix the data first".
- Is the test done? Before the planned days and 90% of the planned sample, the only call allowed is kill, for clear harm (p < 0.001). Otherwise: keep running.
- Did the primary move, per assigned user? A 95% interval for the difference. A primary measured on a subset chosen after assignment ("D1 retention of activated couples") is replaced by its per-assigned version, because filtering out weaker users raises a subset rate on its own.
- Can we rule out harm? Each guardrail passes only if the worst plausible harm (top of a 90% interval, a one-sided 5% non-inferiority test) is within the margin agreed before the test.
- The call. Roll out: the primary wins and every guardrail passes. Iterate: it wins but a guardrail is breached or can't be ruled out, or the result can't tell. Kill: the primary is worse, the test rules out a worthwhile gain, or there's harm with no proven gain.
The rule is the one in Spotify's "Risk-aware product decisions in A/B tests with multiple metrics" (2024): superior on the success metric, non-inferior on every guardrail. The engine is one JavaScript file, and the page runs the same file the benchmark scored.
Code · Product requirements · Example test plan for a couples app · SQL template