Your winning ad might be a coin flip (check before you scale)
Chinmay Raibagkar · has scaled a “winner” that turned out to be noise🪙 with an A/A coin-flip simulator, not a lecturePublished October 4, 2026
The 60-second version
- Small tests produce big fake winners — two identical ads regularly differ by 10–25% on a week of modest traffic.
- Size the test before launch: catching a 20% lift takes roughly 400 conversions per ad, not 40.
- The fix: use real split tests, don’t peek or stop early, wait for conversion lag, and record the confidence range.
Friday afternoon, the creative test results are in. Ad B — the new hook — converted at 2.20%. Ad A, the old one, at 1.80%. A 22% lift. The team pauses Ad A, moves the budget to B, and writes "new hook wins" in the learnings doc.
Four weeks later, B is converting at 2.0%, exactly where A always was. The hook didn't stop working. It never worked better in the first place. One week of data on 6,000 visitors per ad simply can't tell a 22% gap from luck.
Small tests produce big fake winners. heads, heads, heads… Every team running creative tests has crowned one, scaled it, and filed a "learning" that was just noise. The fix is not more tests. It's knowing how much data a test needs before anyone is allowed to call it.
The winner that wasn't
Two identical ads, one "winner"
The simulator below runs an A/A test: both ads have exactly the same 2% conversion rate. Any gap you see is pure chance. Press re-run a few times at the small traffic level, then switch to the large one.
A 24% gap between two identical ads. A dashboard would call this a winner; it is a coin landing heads a few extra times.
At 400 visitors a day, gaps of 10–25% between identical ads show up all the time. At 4,000 a day, they shrink to a few percent. Nothing about the ads changed — only the amount of data. The smaller the test, the bigger the lies it can tell.
Why ad platforms crown winners too early
Uneven spend isn't a test
Put four ads in one ad set and the platform picks a favourite in the first day or two, then starves the rest. The loser never got enough traffic to lose fairly.
Peeking at the results
Checking daily and stopping the first day it looks 'significant' gives luck many chances to win. Commonly cited simulations put the fake-winner rate at 20–30% instead of 5%.
Too many variants
Test ten ads against a control and, by chance alone, one will usually look like a winner at the 90% level. More arms need more data — or a stricter bar.
Use the platform's real split-test tools for creative tests. Meta's A/B test feature and Google Ads experiments split people into separate groups, so each version gets a fair share of comparable traffic. A normal ad set or ad group is built to optimise, not to compare.
How much data a test needs
The amount of data depends on two things: how often people convert, and how small a difference you want to catch. Low conversion rates and small lifts need a lot of traffic. Try your numbers:
Relative lift: 20% means 2.0% → 2.4%
This test won't give a trustworthy answer within a month. Test bolder ideas (bigger expected lifts), pool traffic into fewer variants, or test on a higher-volume metric like click-through rate.
The rule of thumb worth remembering: to catch a 20% lift, you need roughly 400 conversions per ad. Not 40. Not 100. That's why bold concepts are easier to test than small tweaks — a 50% lift needs only about 60 conversions per ad.
Wait for conversion lag. If customers often buy two or three days after clicking, the last days of a test are missing conversions. Read the test only after the lag window has closed, or late conversions will reshuffle the result.
Reading the result honestly
Once the test has reached its planned size, check significance once. This query computes the lift, confidence and a range for the true lift from a simple test table — the same maths as the significance tool below.
A z_score above 1.96 (or below −1.96) clears the 95% bar. More useful still is the range: if diff_low_pp is below zero, the true effect could be nothing — or negative. For the "winner" above, z is 1.56 and the range runs from −0.10 to +0.90 points. That range includes zero, so the test proved nothing.
A test protocol that survives
Before, during and after a creative test
Process FlowWrite the question and the metric
One hypothesis, one primary metric — conversion rate, CPA or click-through rate. Decide before launch, not after you see the numbers.
Size it before you start
Use the calculator above: the lift worth catching sets the visitors and days needed. Too long? Test a bolder idea instead.
Split traffic properly
Use the platform's A/B test or experiment feature, with two to three variants at most.
Don't peek, don't stop early
Let it run to the planned size, plus the conversion-lag window. Daily checks are for broken links, not for winners.
Read once, record the range
Write the lift, the confidence and the range in the learnings doc. 'Not proven' is a valid result — it saves you from scaling luck.
Testing habits worth keeping pin these to the test tracker →
- No winner below ~400 conversions per arm for a 20% lift — or below whatever your size check says. Smaller tests are 'directional', never 'proven'.
- Test big swings, not button colours — new angles and formats produce lifts large enough to measure with normal budgets.
- Re-check winners after scaling — compare the first four weeks at full budget with the test result. If the lift vanished, update the learnings doc.
Quick gut-check
One question. If you get it, the whole post clicks. 30 seconds, no stats degree
A new ad shows a 29% higher conversion rate after 3 days, on 45 conversions vs 35 from similar traffic. The team wants to scale it today. What do you say?
Frequently asked questions
How many conversions do I need for an ad A/B test?
It depends on the lift you want to detect. As a rule of thumb, catching a 20% relative lift needs roughly 400 conversions per ad; a 50% lift needs about 60; a 10% lift needs about 1,600. Smaller lifts need far more data.
Is 90% confidence good enough for ad tests?
It can be, for low-risk decisions you can reverse cheaply. Just know what it means: about a 1-in-10 chance of a gap that size between identical ads. If you run many tests, some of your "winners" will be luck. For decisions that move large budgets, use 95%.
Can I test creatives inside one ad set?
You can compare them, but it isn't a fair test: the platform shifts spend to its early favourite, so the others never get equal traffic. Use the platform's A/B test or experiment tools when you need a real answer.
The summary
- Two identical ads regularly show 10–25% gaps on small samples — big lifts on small tests are usually luck.
- Size every test before launch: low conversion rates and small lifts need a lot of traffic.
- Catching a 20% lift takes roughly 400 conversions per ad.
- Use proper split tests, don't peek, and wait for conversion lag before reading results.
- Record the confidence range, not just the lift — "not proven" is a result.
Takeaways for your next report
- Small tests produce big fake winners: identical ads often differ by 20% in a week.
- Rule of thumb: ~400 conversions per ad to confirm a 20% lift at 95% confidence.
- Stopping the first day a test looks significant can turn a 5% false-winner rate into 20–30%.
- Ad sets optimise, they don't compare — use the platform's A/B test or experiment tools.
- Test bold ideas: bigger expected lifts need far less traffic to prove.
A/B Test Significance Playground
Enter visitors and conversions for control and variant to get a verdict, confidence, lift bounds and the sample size you actually needed.
Chinmay Raibagkar
About author →Founder of DataLens AI. He helps non-technical teams read their ad and database numbers with confidence — which number to trust, what to do next, and what to ignore.