A/B Test Significance Playground
Live significance test + power planner · no sign-up · shareable link
Most "winning" A/B tests are noise read too early. Enter visitors and conversions per arm and the playground runs the two-proportion z-test live: lift with confidence bounds, a verdict in plain language, and — the part teams actually need — how many visitors per arm an 80%-power test required to detect that lift. Drag the numbers and watch significance arrive (or evaporate). Runs entirely in your browser.
Two-proportion z-test (normal approximation) + 80%-power sample-size planner
Drag to feel significance arrive — then check the planner before trusting it
4.81% → 5.65%
Lift +17.7% · z = 1.73 · p = 0.0841
10,783/arm
You are at 38% of plan.
Not significant (p = 0.084). At 38% of the 10,783/arm needed — keep running or redesign for a bigger swing.
Traffic required per arm vs the lift you want to detect (smaller swing = much longer test)
What each field wants
- Visitors and conversions per arm
- Unique visitors (or sessions, consistently) and the conversion events in the same window. Same traffic split, same period, no peeking adjustments — the test assumes both arms ran concurrently.
How this number is derived
Two-proportion z-test, stated plainly
Pooled rate p̂ combines both arms; the standard error comes from p̂(1−p̂)(1/nA + 1/nB); z is the rate difference over that error. Two-sided p follows from the normal CDF. Below ~100 conversions per arm the normal approximation wobbles — the tool warns you rather than pretending otherwise.
The planner inverts the test
Given your baseline rate and the lift you want to detect, the per-arm sample size for 80% power at 95% confidence is 2·p̄(1−p̄)·(z_α/₂ + z_β)² / δ². Read it before launching: if the planner says 40,000 visitors per arm and you get 2,000 a week, that test needs five months — redesign for a bigger swing or a closer-to-checkout metric.
Peeking invalidates everything
Checking daily and stopping at the first significant day roughly doubles the false-positive rate. Decide the sample size up front from the planner, run to it, then read the verdict once. The playground shows live p-values for learning — production discipline still means one read at the end.
The checkout button that "won" on Tuesday
- Control
- 4,120 visitors · 198 conversions (4.81%)
- Variant
- 4,085 visitors · 231 conversions (5.66%)
Lift = (5.66 − 4.81) ÷ 4.81 = +17.7%
z ≈ 1.73 → p ≈ 0.084 → not significant at 95%
Planner: detecting +17% off a 4.8% base needs ~10,800/armA +17% lift that is still noise — the test is at 40% of the traffic it needs. The team shipped it anyway last quarter and watched the "win" evaporate over the next month. Run to the planned sample, then decide.
How long tests really take
At a 3% baseline conversion rate, per arm, 80% power, 95% confidence.
| Detect +30% lift | ~3,400/arm | Big swings on high traffic: a week or two for most stores. |
|---|---|---|
| Detect +15% lift | ~14,000/arm | Typical button-copy territory. Often a month+. |
| Detect +5% lift | ~130,000/arm | Effectively untestable for most D2C — redesign for bigger swings. |
| Under 100 conversions/arm | Do not read | Approximation wobbles; the tool says so out loud. |
What this assumes, and what it doesn't model
Assumptions
- Randomised concurrent arms with independent visitors (no cross-contamination, no shared carts).
- Normal approximation: trustworthy past ~100 conversions per arm; flagged below that.
- Two-sided test at 95% confidence, 80% power for the planner.
Deliberately not modelled
- No sequential-testing correction — peeking inflates false positives and the tool does not adjust for it.
- Does not model novelty/carryover effects; a variant can win the first week and regress.
- Ratio metrics (revenue per visitor with wild AOV skew) need a t-test, not this one — use conversion-rate tests here.
- SRM (sample-ratio mismatch) from broken randomisation is not detectable from these four numbers alone.
Common questions
p = 0.06 — can I call it at 90% confidence?
You can, but you should decide that before running, not after seeing 0.06. Moving the goalposts post-hoc is peeking with extra steps. Pre-register 95%, run to the planned sample, read once.
How do I test revenue, not conversion rate?
Carefully: revenue per visitor has huge variance from a few big orders, so the same test needs far more traffic or a variance-reduction method. As a rule, test conversion rate here and read revenue as a guardrail metric (variant must not tank AOV).
What is SRM and should I worry?
Sample-ratio mismatch — the split is not what you configured (e.g. 60/40 instead of 50/50). It signals broken randomisation, often from redirect tests dropping slow connections. Check the visitor counts first; a big imbalance voids the test before any p-value matters.
Can I share the result?
Yes — all four numbers live in the URL, so the link restores the exact verdict for the team thread.
Last reviewed September 6, 2026.