Analytics
Free A/B test statistical significance calculator
By Charles Summers · Updated · Free, no signup
Short answer
This runs a two-proportion z-test on your control and variant counts and returns the actual numbers behind the verdict: both conversion rates, the pooled proportion, the standard error, the z-statistic, a two-tailed p-value computed from the normal CDF, and a confidence interval on the absolute difference. It tells you whether the result clears your chosen threshold, and warns you when either arm has too few conversions for the normal approximation to mean anything.
Use the a/b test significance calculator
What does this tool actually do?
This runs a two-proportion z-test on your control and variant counts and returns the actual numbers behind the verdict: both conversion rates, the pooled proportion, the standard error, the z-statistic, a two-tailed p-value computed from the normal CDF, and a confidence interval on the absolute difference. It tells you whether the result clears your chosen threshold, and warns you when either arm has too few conversions for the normal approximation to mean anything..
It runs entirely in your browser. Nothing you type is sent to a server, no account is required, and there is no usage limit, because there is no cost per run to control.
What a p-value of 0.03 actually claims, and the sentence you must not say
The p-value answers one narrow question: if the variant and the control converted at exactly the same underlying rate, how often would random assignment alone hand you a gap at least this large in either direction? A p-value of 0.03 says three times in a hundred. That is a statement about the frequency of data given a hypothesis. It is not a statement about the probability of the hypothesis given the data, and the two are not interchangeable.
Which is why "we are 95% confident, so there is a 95% chance the variant is better" is wrong, and wrong in a direction that costs money. The frequentist machinery you are running contains no probability distribution over "the variant is better" at all. To get that number you would need a prior: how often do changes of this kind actually move this metric on this site? Most teams have that evidence sitting in their own test archive and never use it. If eight of your last ten tests were flat, a single p = 0.04 result is more likely to be one of the false positives you were budgeting for than a genuine win, because the pool of true effects it could have come from is small.
The confidence interval is the honest version of the same calculation, and it is why this tool prints it. A 95% interval means that the procedure, repeated across many experiments, brackets the true difference 95% of the time. An interval running from +0.1 to +4.8 percentage points is technically a win and practically useless: it is compatible with a rounding error and with a transformational result, and you cannot ship a roadmap against that range. A significant p-value with a wide interval means you have detected direction, not magnitude.
Note also that this tool computes the test statistic from the pooled proportion (the correct denominator under the null hypothesis of no difference) but the interval from the unpooled standard error, which is standard practice. Very close to the threshold the two can disagree by a hair: a p-value of 0.049 alongside an interval whose lower bound just grazes zero. That is not a bug, it is two slightly different variance estimates, and a result that fragile was never a decision anyway.
Peeking: why most reported wins are not wins
A fixed-horizon z-test, which is what this is, assumes you decided the sample size before you started and looked once at the end. Every dashboard in the industry violates that assumption by design. When you check the running total each morning and stop the moment the significance badge turns green, you are not running one test at the 5% error rate, you are running a series of correlated tests and taking the maximum. The false positive rate for that procedure is much higher than the number on the screen.
The mechanism is simple enough to see without any mathematics. Conversion rates wander early, when denominators are small, so the observed difference takes a random walk that starts wide and narrows. Stopping when the walk happens to be far from zero is selecting on noise. Published simulations of repeated interim looks at a nominally 5% test typically land in the 15 to 25 percent range for a handful of checks, and with genuinely continuous monitoring the rate keeps climbing as the sample grows, because a random walk crosses any fixed boundary eventually. The exact inflation depends on how many times you look and how you space the looks, which is precisely why no single number gets quoted reliably.
There are three defensible responses and one indefensible one. The indefensible one is to keep peeking and pretend you did not.
- Fix the horizon in advance. Compute the sample size before launch, write the stop date down somewhere the team can see, and read the result once. Boring, free, and correct.
- Use a sequential method. Always-valid inference, mSPRT and group-sequential boundaries all exist to make continuous monitoring legitimate. They cost sample size in exchange for the right to look, which is a real trade, not a free lunch.
- Look, but only to abort. Checking for a catastrophic drop or a broken tracking pixel is operations, not inference. Set that as a one-directional guardrail and never let it flip into a reason to declare a win early.
- Never extend a test that just missed. Adding a week because p came in at 0.07 is peeking with extra steps, and it biases the result in exactly the direction you were hoping for.
Significance is a filter, not a decision
Statistical significance tells you the difference is probably not zero. It says nothing about whether the difference is worth having. On a page taking two million sessions a month, a 0.2 percentage point lift will clear any threshold you like and may still be worth less than the engineering time to maintain the variant. On a page taking four thousand sessions, a genuine 30% improvement can sit at p = 0.2 for a month, and calling that "no effect" is a reporting failure, not a finding.
Decide the practically significant effect before you look at the data, and state it in money. The chain is short: absolute lift in percentage points, times monthly traffic to that page, times average order value or lead value, times close rate if the conversion is a lead. Run that arithmetic on the lower bound of the confidence interval rather than the point estimate, because the lower bound is the version of the result you can defend to finance. If the lower bound is not worth the deployment cost, the test succeeded at showing you something and the answer is still no.
The reverse error is treating a non-significant result as evidence of no difference. Failing to reject the null is not accepting it. A test that ends at p = 0.4 with a confidence interval spanning minus three to plus four percentage points has told you almost nothing, and the correct write-up says "underpowered, no conclusion" rather than "no impact". The interval width is what separates those two readings, and it is why an experiment log that records only the p-value throws away most of what each test cost you to learn.
Multiple comparisons, segments, and where the normal approximation gives up
Test four variants against one control and you have three comparisons, each with its own chance of a false positive; at a nominal 5% the probability that at least one fires by luck is roughly 1 minus 0.95 cubed, about 14%. Slice the winner by device, browser, new versus returning and traffic source afterwards and you are running dozens of implicit tests, which is why post-hoc segment findings replicate so poorly. The Bonferroni correction (divide your alpha by the number of comparisons) is crude and conservative, but it is defensible and takes ten seconds, which is more than can be said for the usual alternative of not correcting at all.
The normal approximation itself has a floor. The z-test approximates a binomial with a normal distribution, and that approximation degrades when the expected counts are small. The conventional rule is that n times p and n times (1 minus p) should both exceed about 10; in conversion-rate terms that means roughly 30 or more conversions per arm before the p-value is worth quoting to two decimal places. Below that, use an exact test (Fisher's exact, or a binomial test) or, more usefully, accept that you do not have an experiment yet and stop reading the dashboard.
Two failure modes have nothing to do with statistics and will beat all of the above. Sample ratio mismatch is the first: if your 50/50 split arrives as 52/48 across a large sample, something in the assignment or logging is broken and the numbers you are testing are not the numbers you think. Check the split before you check the result. The second is the novelty and primacy effect, where regular users react to any change simply because it is a change; that reaction decays over one to three weeks, so a test ended after four days on a site with a weekly usage rhythm has measured the reaction, not the design. Run at least one full business cycle regardless of what the p-value does on day three.
Numbers worth knowing
| Metric | Typical | What it means |
|---|---|---|
| Two-tailed z critical values | 1.645 / 1.960 / 2.576 | For 90%, 95% and 99% confidence. These are the numbers your |z| has to beat, and they come from the normal distribution, not from your data. |
| Conversions per arm before the maths holds | about 30 | Derived from the np and n(1-p) greater than 10 convention. Under this, the normal approximation to the binomial is loose and an exact test is the honest choice. |
| False positive rate when you peek | well above the nominal 5% | Simulations of a few interim looks typically land in the 15 to 25 percent region, and continuous monitoring climbs further. The exact figure depends on how often you look. |
| Family-wise error, 4 variants vs control | about 14% | Three comparisons at 5% each: 1 minus 0.95 cubed. Bonferroni would test each at 0.0167 instead, which is conservative but defensible. |
| Minimum runtime regardless of significance | one full business cycle | Usually 7 or 14 days. Weekday and weekend traffic convert differently, and novelty effects on returning users decay over one to three weeks. |
Mistakes that quietly cost you results
- Stopping the test the morning the significance badge turns green
- That converts a 5% error rate into something much larger, because you are taking the maximum of a series of correlated looks. Fix the sample size and the stop date before launch, or switch to a sequential method that is built for monitoring.
- Reading 95% confidence as a 95% chance the variant is better
- The p-value is the probability of data this extreme given no true difference, not the probability of a true difference given the data. Getting the second number requires a prior, and your own archive of mostly flat tests is that prior.
- Reporting a win from the point estimate and ignoring the interval
- Multiply the lower bound of the confidence interval by traffic and value instead. If the pessimistic end of the range does not pay for the change, the significant result has not made the decision for you.
- Hunting for a segment where the flat test finally shows a lift
- Ten segments is ten more chances to find noise, and post-hoc segment wins famously fail to replicate. Pre-register the one segment you actually have a hypothesis about, and treat everything else as a question for the next test.
- Calling a non-significant result "no impact"
- Failing to reject is not evidence of equivalence. Check the interval width: if it spans minus three to plus four points, the honest write-up is underpowered and inconclusive, and the finding is that you need more traffic or a bolder change.
What does the output look like?
This is the exact output the tool produces from the example inputs. It is generated by the same code that runs when you click the button, so what you see here is what you get.
Frequently asked questions
Which test is this, exactly?
A two-tailed two-proportion z-test. The test statistic is the difference in observed rates divided by the standard error computed from the pooled proportion, which is the right denominator under the null hypothesis that both arms share one underlying rate. The p-value is twice the upper tail area beyond the absolute value of z. Two-tailed is deliberate: you almost never genuinely know in advance that a change cannot hurt, and a one-tailed test halves your p-value for free by assuming you did.
How is the p-value computed without a lookup table?
The normal cumulative distribution function is evaluated in the browser from the Abramowitz and Stegun 7.1.26 rational approximation to the error function, which has a maximum absolute error around 1.5 times ten to the minus seven. That is several orders of magnitude finer than any decision you would make from a p-value, so the number you see matches what R, Python or a statistics textbook would give you to the digits displayed.
Why does the confidence interval use a different standard error than the z-test?
The test asks whether both arms could share one rate, so it pools the conversions to estimate that shared rate. The interval estimates the size of a difference it does not assume is zero, so it uses each arm's own variance. This is textbook practice and the two agree almost everywhere, but within a hair of the threshold they can disagree slightly. If your result is that fragile, the answer is more sample, not a different formula.
What if one arm has fewer than 30 conversions?
The output says so plainly and you should not quote the p-value. The z-test approximates a binomial distribution with a normal one, and that approximation needs both the expected successes and expected failures to be reasonably large, conventionally above about 10. With a handful of conversions the tails are wrong in a way that usually flatters the result, so use an exact test or keep the experiment running.
Can I use this for revenue per visitor or average order value?
No. This test is for binary outcomes, where each visitor either converted or did not. Revenue is continuous and heavily skewed by a small number of large orders, so it needs a t-test on the mean, a bootstrap, or a comparison of medians. Running a proportion test on revenue data will produce a number, and the number will be meaningless.
Related free tools
- Marketing Budget Allocator Analytics
- A/B Test Sample Size Calculator Analytics
- UTM Campaign Tagger Analytics
- AI LinkedIn Hook Optimizer for consultants Social
- AI LinkedIn Hook Optimizer for recruiters Social
Some links on this site are affiliate links, which means Hacking Demand may earn a commission if you buy through them at no extra cost to you. This does not influence which tools are listed. The tools on this page are free and have no affiliate relationship of any kind.