How the test works
This calculator runs a two-proportion z-test. It pools both variants to estimate the conversion rate if there were no real difference, measures how far apart A and B are in standard errors (z), and converts that into a two-sided p-value.
If the p-value is below 1 − your confidence level (for 95%, below 0.05), the difference is statistically significant: it is unlikely to be random chance.
Sample size and stopping early
“Sample needed” estimates how many visitors each variant needs to detect the difference you are currently seeing with 80% power at your chosen confidence level. If you are far below it, keep the test running.
Avoid stopping a test the first moment it turns significant — results swing a lot early on. Decide the sample size up front, run full weeks to cover weekday and weekend behaviour, and only then read the result.
Frequently asked questions
What does statistically significant mean?
It means the observed difference between A and B would be unlikely if the two variants actually performed the same. At 95% confidence, there is less than a 5% chance of seeing a difference this large by luck alone.
What is a p-value?
The p-value is the probability of seeing a difference at least as large as yours if there were no real difference. Smaller is stronger evidence; 0.05 is the usual cut-off.
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the uplift you want to detect. Small uplifts on low conversion rates need thousands of visitors per variant. The “sample needed” figure gives an estimate for the difference you are seeing.
Can I test more than two variants?
This tool compares two variants. When comparing several, test each challenger against the control and use a stricter confidence level (for example 99%) to limit false positives.
Does it work for ad CTR and email open rates?
Yes. Use impressions (or emails delivered) as visitors and clicks (or opens) as conversions.
