Analytics

Free A/B test sample size and duration calculator

Name: A/B Test Sample Size Calculator
Author: Charles Summers

By Charles Summers · Updated 8 August 2026 · Free, no signup

Short answer

This computes visitors needed per variant from the standard power formula, using your baseline conversion rate, the relative lift you want to be able to detect, your chosen power and significance level, then converts that into days and whole weeks at your traffic. It also solves the reverse question: given the traffic you actually have, the smallest effect a four-week test could reliably detect. Sample size scales with the inverse square of the effect, so halving the detectable lift quadruples the traffic bill.

Use the a/b test sample size calculator

What does this tool actually do?

It runs entirely in your browser. Nothing you type is sent to a server, no account is required, and there is no usage limit, because there is no cost per run to control.

Power is the half of your error budget nobody buys

Significance controls false positives: the chance of declaring a winner when the two arms are identical. Power controls false negatives: the chance of noticing a real effect that is genuinely there. Teams argue endlessly about whether to use 95% or 99% confidence and then accept 80% power without a word, which means they have quietly agreed to miss one in five of the improvements they actually built. On a roadmap of twenty tests where six changes genuinely work, 80% power expects you to ship four or five of them and bin the rest as flat.

The number in the formula is Z_beta, and it is the only place power enters the arithmetic: 0.8416 for 80% power, 1.2816 for 90%. It gets added to the significance z-value before squaring, which is why moving from 80% to 90% power at 95% confidence takes the bracket from 2.8016 to 3.2416 and inflates the required sample by about 34%. That is the honest price of halving your miss rate, and it is a third more traffic rather than a doubling, which makes it a better deal than most teams assume.

Power also explains the most common bad outcome in experimentation, which is not a false win but a portfolio of inconclusive tests. If you keep running experiments that end at p = 0.3 with intervals four points wide, the problem is almost never the ideas. It is that the sample was never large enough to separate a plausible effect from noise, and every one of those tests cost the same engineering time as a conclusive one. Compute the number before you build the variant, not after the test disappoints.

The inverse square rule, and why small lifts are so expensive

The detectable difference sits in the denominator of the formula, squared. Everything follows from that. Want to detect a lift half as large? You need roughly four times the sample. A third as large? Nine times. This single relationship explains why a startup can validate a redesigned pricing page in a fortnight while a mature site cannot detect the 2% improvements that are all it has left, and why the mature site is right to move to holdout groups and long-run measurement instead of insisting on classical tests it cannot afford.

The baseline rate matters almost as much, through the variance terms. Binomial variance is p(1-p), which peaks at p = 0.5 and collapses towards the extremes, but the difference you are trying to detect shrinks faster than the variance does when the baseline is low. In practice, detecting a 20% relative lift on a 1% baseline takes far more traffic than the same relative lift on a 10% baseline, and that is the arithmetic reason demo-request tests take a quarter while add-to-cart tests take a week. If your primary metric converts under about 2%, look for a legitimate upstream proxy that converts more often and validate that it correlates with the money before you rely on it.

A few consequences worth internalising before you plan a testing programme:

Relative and absolute effects are different inputs. A 20% relative lift on a 3% baseline is 0.6 absolute points. Confuse the two and your sample size is out by an order of magnitude.
Two variants against one control is not free. The control arm is shared, but you need the full per-arm sample in each variant and you are now making multiple comparisons, so the alpha you plug in should already be corrected.
Traffic is not the same as eligible traffic. Only visitors who actually reach the tested element and could plausibly convert belong in the denominator. Counting all sessions makes the test look affordable and then dilutes the effect until it vanishes.
The formula assumes a fixed horizon. The n it produces is only valid if you look once, at the end. If you plan to monitor continuously, you need a sequential design, which will ask for a somewhat larger commitment in exchange.

Setting the minimum detectable effect from the business, not from hope

The MDE is not a prediction of how well the variant will do. It is a threshold: the smallest effect you are willing to spend this much traffic to be able to see. Set it too small and the test runs for six months. Set it optimistically at 30% because that is what you hope happens, and you have built a test that will only detect miracles and will be inconclusive about the ordinary 8% improvement you probably shipped.

The defensible way to pick it is to work backwards from money. Take the annual value of the page, decide the smallest annual gain that would justify building and maintaining the change, convert that back into a percentage lift, and use it. If that number turns out to require more traffic than you have, you have learned something useful before spending anything: this page cannot be optimised by classical A/B testing at your current volume, and your options are to test bolder changes, to test further up the funnel where the numbers are bigger, or to accept a lower confidence level and treat results as directional.

Lowering confidence from 95% to 90% is a legitimate move that people treat as cheating. It shifts the significance z from 1.960 to 1.645, which cuts the required sample by roughly a quarter, in exchange for doubling the false positive rate from 5% to 10%. For a low-risk, easily reversible change on a page you will retest anyway, a one in ten chance of shipping a dud is often a fair trade for getting an answer this quarter. For a change that is expensive to reverse, or one that will be quoted as evidence for a year, it is not. The point is to make the trade deliberately rather than accepting 95% because it is the default in the tool.

Why the duration rounds up to whole weeks

Traffic is not exchangeable across days. Weekday and weekend visitors differ in intent, device mix, source and conversion rate, often by a factor of two on B2B sites and less dramatically but still materially on consumer ones. Payday cycles, Monday planning behaviour and Friday afternoon drop-off all impose structure on the week. A test that runs for ten days contains two Mondays and one Saturday, so its sample is not a random draw from your traffic, it is a random draw plus a systematic overweight of one part of the week.

This does not usually bias the comparison, because both arms experience the same days. What it does is inflate variance and make the result brittle, and it becomes a real bias the moment the composition differs between arms, which happens whenever one arm was deployed a few hours later or a bot filter removed traffic unevenly. Running whole weeks makes the composition identical by construction and removes the argument entirely. It costs a few days of traffic and buys a result you do not have to caveat.

Two further scheduling rules earn their place. Do not start a test on the day of a promotion, a product launch or a public holiday, and if one lands mid-test, note it and consider whether the affected days belong in the analysis at all, decided in advance rather than after seeing which way it helps. And treat the first week as partly a measurement of novelty: returning users react to change as change, and that reaction decays over roughly one to three weeks. On a site where most traffic is returning, a two-week minimum is not conservatism, it is the shortest window in which you are measuring the design rather than the surprise.

Finally, if the calculated duration comes back longer than about six to eight weeks, treat that as an answer rather than a schedule. Long tests accumulate cookie deletion, cross-device drift, seasonality and organisational impatience, all of which degrade the experiment in ways the formula does not model. Better to test a bolder change with a larger expected effect, combine several small changes into one variant and accept that you will not know which one worked, or move the measurement to a switchback or geo holdout design that does not need per-visitor assignment.

Numbers worth knowing

Metric	Typical	What it means
Z values used in the formula	1.645 / 1.960 / 2.576 and 0.842 / 1.282	The first three are two-tailed significance at 90, 95 and 99 percent. The last two are Z_beta for 80% and 90% power. They add before squaring.
Cost of halving the detectable effect	about 4x the sample	The difference is squared in the denominator, so sample size scales with the inverse square. This is the single most important fact in test planning.
Cost of moving 80% to 90% power	about 34% more sample	At 95% confidence the bracket goes from 2.8016 to 3.2416, and 3.2416 squared over 2.8016 squared is 1.34. Cheaper than most people assume.
Cost of moving 95% to 90% confidence	about 25% less sample	Z drops from 1.960 to 1.645. You double the false positive rate to 10% in exchange. Reasonable for reversible changes, not for load-bearing ones.
Practical maximum runtime	6 to 8 weeks	Beyond this, cookie churn, seasonality and cross-device behaviour degrade the assignment itself. If the formula asks for longer, change the test, not the deadline.

Mistakes that quietly cost you results

Entering the absolute lift where the tool asks for a relative one: A 20% relative lift on a 3% baseline is 0.6 absolute points, not 20. Since the difference is squared in the denominator, mixing these up changes the answer by a factor of roughly a thousand and produces a test plan that is either trivially short or absurdly long.
Choosing the MDE from what you hope the variant will do: Derive it from the smallest annual gain that would justify building and maintaining the change. Hope-based MDEs produce tests powered only for miracles, which then report the ordinary real improvement you shipped as inconclusive.
Counting all site traffic instead of eligible traffic: Only visitors who actually reach the tested element belong in the denominator. Including everyone makes the plan look affordable, then dilutes the measured effect across a population that never saw the change, which is how a real lift ends up flat.
Running the calculated sample and then stopping mid-week: Round up to whole weeks. Weekday and weekend traffic differ in intent and conversion rate, so a ten-day test overweights part of the week and inflates variance for no reason. The extra days are cheaper than the caveat you would otherwise have to write.
Using the same sample size for a four-variant test: You need the full per-arm figure in every variant, and three comparisons against control at 5% each carry roughly a 14% chance of at least one false positive. Correct the alpha first, which will raise the per-arm number, then multiply by the number of arms.

What does the output look like?

This is the exact output the tool produces from the example inputs. It is generated by the same code that runs when you click the button, so what you see here is what you get.

REQUIRED SAMPLE 8,509 visitors per variant, 17,018 total across a two-arm test. Runtime at 1,200 visitors per variant per day: 8 days, which rounds up to 2 weeks (14 days). WHAT YOU ASKED IT TO DETECT Baseline rate 3.20% Target rate 4.000% (a 25.0% relative lift) Absolute difference 0.800 percentage points Significance 95% confidence, alpha 0.05, Z = 1.9600 Power 80%, Z_beta = 0.8416 HOW THE NUMBER WAS PRODUCED n = (Z_alpha/2 + Z_beta)^2 x (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2 n = (1.9600 + 0.8416)^2 x (0.030976 + 0.038400) / 6.4000e-5 n = 7.8490 x 0.069376 / 6.4000e-5 = 8,509 WHY IT ROUNDS UP TO WHOLE WEEKS The raw arithmetic says 7.1 days of collection, 8 once you round to a whole day. That is 1 full week plus 1 day, so 1 weekday would appear one more time than the other 6. Weekday and weekend visitors differ in intent, device mix and conversion rate, so a partial week is a systematic overweight of part of your traffic rather than a random draw from it. Both arms see the same days, so this rarely biases the comparison outright, but it inflates variance and it turns into a genuine bias the moment anything removes traffic unevenly between arms. Running 2 whole weeks makes the day composition identical by construction, at a cost of 6 extra days of traffic. WHAT DIFFERENT AMBITIONS COST (same baseline, power and confidence) MDE sample per arm runtime 10% 49,774 per arm 6 weeks 20% 13,012 per arm 2 weeks 30% 6,034 per arm 1 week 50% 2,351 per arm 1 week Notice the shape: the difference is squared in the denominator, so halving the effect you want to see roughly quadruples the traffic bill. THE REVERSE QUESTION At 1,200 visitors per variant per day, four weeks buys you 33,600 per arm, which is enough to detect a 12.3% relative lift (0.394 absolute points) at these settings. Anything smaller than that is invisible to a four-week test here, no matter how real it is. FEASIBILITY: 2 weeks is a comfortable window. Fix the stop date now, before the first result appears and starts arguing with you. CONDITIONS THIS NUMBER ASSUMES 1. You look once, at the end. This is a fixed-horizon calculation; monitoring continuously needs a sequential design instead. 2. Two arms only. A third arm needs its own full 8,509, and the extra comparison needs a corrected alpha, which raises the per-arm figure again. 3. The traffic entered is eligible traffic, meaning visitors who actually reach the tested element. 4. The baseline of 3.20% is measured, not assumed. If it is out by a fifth, so is this plan.

Frequently asked questions

Which formula does this use?

n per variant equals (Z_alpha/2 + Z_beta) squared, times the sum of p1(1-p1) and p2(1-p2), divided by the square of (p2 - p1). That is the unpooled-variance version of the two-proportion power formula. Some calculators pool the variance under the null instead, which shifts the answer by a few percent in either direction. Neither is wrong; the difference is far smaller than the uncertainty in your baseline rate estimate, so do not spend time reconciling the two.

Why does a lower baseline rate need so much more traffic?

Because a fixed relative lift on a small baseline is a tiny absolute difference, and the absolute difference is what the denominator squares. Twenty percent on a 10% baseline is 2 whole points. Twenty percent on a 1% baseline is 0.2 points, ten times smaller, and squaring that puts you roughly a hundred times worse off before the variance terms partly offset it. This is why demo-request tests take a quarter and add-to-cart tests take a week.

Should I use 80% or 90% power?

Use 90% when missing a real improvement is expensive: a change you will not get engineering time to revisit, or a decision that will be quoted for a year. Use 80% when you are screening lots of ideas quickly and can afford to retest the promising near-misses. At 95% confidence the upgrade costs about a third more traffic, which is usually a better purchase than tightening significance from 95% to 99%.

What if the calculated runtime is six months?

Then classical A/B testing is not the right instrument for that page at that traffic, and knowing this before you build the variant is the point of running the calculation. Your options are to test a bolder change with a larger expected effect, bundle several changes into one variant and accept you will not know which one worked, move testing upstream where the numbers are larger, or switch to a geo or time-based holdout design that does not need per-visitor assignment.

Does the required sample include both arms or just one?

The headline figure is per variant. A standard two-arm test needs that number twice, which the output states explicitly. The daily traffic input is also per variant, so if you have 2,400 daily visitors to split evenly, enter 1,200. Getting this the wrong way round is the most common way to end a test at exactly half the power you planned for.

Related free tools

A/B Test Significance Calculator Analytics
UTM Campaign Tagger Analytics
CAC Payback Period Calculator Analytics
Cold Calling Scriptwriter for SaaS sales Outbound
Cold Calling Scriptwriter for insurance agents Outbound

Some links on this site are affiliate links, which means Hacking Demand may earn a commission if you buy through them at no extra cost to you. This does not influence which tools are listed. The tools on this page are free and have no affiliate relationship of any kind.