Home
Book a Discovery call
Book a Discovery call

Plan and read A/B tests with confidence

Find out how many visitors and weeks your test needs, then check whether the winner is real or just luck. Check as often as you like: the maths allows for it.

[ Questions ]

A/B testing questions

Rheed MacPherson, founder of Signal Studio

Can’t find your answer?

Book a Discovery call and ask us directly.

Book a Discovery call
How long should I run an A/B test?

Until you reach the sample size from your plan, and always in whole weeks. A minimum of one full week covers differences between weekdays and weekends. Tests longer than about eight weeks start to suffer from people clearing cookies and switching devices, so if your plan says longer, test a bolder change or a busier page.

What sample size do I need for an A/B test?

It depends on your current conversion rate, the smallest lift you want to detect, your confidence and power, and how many versions you test. For a 3% conversion rate and a 10% relative lift at 95% confidence and 80% power, a classic test needs about 51,000 visitors per version. Use the planner above for your numbers.

Can I check my A/B test results early?

Only if you use a sequential test. A classic test is only valid when you look once at the planned sample size; checking daily can push the false winner rate from 5% to over 25%. Sequential tests stay valid however often you look, at the cost of a larger maximum sample.

What's the difference between Bayesian and frequentist A/B testing?

Frequentist tests ask how surprising the data would be if there were no difference, and give a p-value. Bayesian analysis gives the probability each version is best and how big the lift probably is. This calculator uses a sequential frequentist test to decide when a result is real and Bayesian numbers to describe it, because each answers a different question well.

What does "chance it's better" (chance to beat) mean?

It's the probability, given your data and the prior, that the version's true conversion rate is higher than the original's. A 97% chance to beat means that if you could rerun the world many times, the version would be better in about 97 of 100.

What is a sample ratio mismatch?

A sample ratio mismatch happens when the share of visitors in each version differs from what you set up, beyond what chance explains. It usually points to a redirect, caching, bot or tracking problem. Results from a test with a mismatch shouldn't be trusted until the cause is fixed.

Why is the sequential sample size bigger than the classic one?

The sequential number is a maximum. To stay valid at every look, the sequential test needs stronger evidence early on. If the lift is real, sequential tests usually stop around the classic sample size or sooner, and much sooner when the lift is larger than planned.

Rheed MacPherson, founder of Signal Studio

Can’t find your answer?

Book a Discovery call and ask us directly.

Book a Discovery call
Signal Studio
Our Process
Services Agentic Websites Headless CMS Websites Webflow Websites
Case Studies
Resources Tools Migrations About
Book a Discovery call
Book a Discovery call
AB-TEST-CALCULATOR.MD AB-TEST-CALCULATOR

# A/B Test Calculator

 

Work out A/B test sample size and duration, then check if your winner is real with sequential and Bayesian statistics. Free, runs in your browser.

 

## What it does

 

Plan a test: from the current conversion rate, the smallest lift worth detecting, daily visitors, the share of traffic in the test and 2 to 4 versions, it gives the visitors needed per version and how many weeks that takes, for a sequential test (check any time) and a classic one (look once), with what each could detect week by week. Read results: from visitors and conversions per version, it says whether there is a winner, the lift, the chance each version beats the original and is best, the risk of choosing it, and whether the traffic split is broken.

 

## The maths

 

Classic sample size: a two-sided two-proportion z-test, the same formula as Evan Miller's calculator. Sequential: a mixture sequential probability ratio test, giving always-valid p-values and confidence intervals, so results can be checked at any time. Bayesian: Beta-Binomial posteriors from 80,000 seeded Monte Carlo draws, with a flat, weak or strong prior. Sample ratio check: chi-square goodness of fit, flagged at p < 0.001. Three or four versions use a Bonferroni-corrected threshold. A winner needs the sequential test to pass; a broken split overrides everything.

 

## Private

 

Everything is calculated in the browser. Nothing typed is sent anywhere; a shareable link keeps the numbers in the part of the address after #, which is never sent to a server.

 

## How long should I run an A/B test?

 

Until you reach the sample size from your plan, and always in whole weeks. A minimum of one full week covers differences between weekdays and weekends. Tests longer than about eight weeks start to suffer from people clearing cookies and switching devices, so if your plan says longer, test a bolder change or a busier page.

 

## What sample size do I need for an A/B test?

 

It depends on your current conversion rate, the smallest lift you want to detect, your confidence and power, and how many versions you test. For a 3% conversion rate and a 10% relative lift at 95% confidence and 80% power, a classic test needs about 51,000 visitors per version. Use the planner above for your numbers.

 

## Can I check my A/B test results early?

 

Only if you use a sequential test. A classic test is only valid when you look once at the planned sample size; checking daily can push the false winner rate from 5% to over 25%. Sequential tests stay valid however often you look, at the cost of a larger maximum sample.

 

## What's the difference between Bayesian and frequentist A/B testing?

 

Frequentist tests ask how surprising the data would be if there were no difference, and give a p-value. Bayesian analysis gives the probability each version is best and how big the lift probably is. This calculator uses a sequential frequentist test to decide when a result is real and Bayesian numbers to describe it, because each answers a different question well.

 

## What does "chance it's better" (chance to beat) mean?

 

It's the probability, given your data and the prior, that the version's true conversion rate is higher than the original's. A 97% chance to beat means that if you could rerun the world many times, the version would be better in about 97 of 100.

 

## What is a sample ratio mismatch?

 

A sample ratio mismatch happens when the share of visitors in each version differs from what you set up, beyond what chance explains. It usually points to a redirect, caching, bot or tracking problem. Results from a test with a mismatch shouldn't be trusted until the cause is fixed.

 

## Why is the sequential sample size bigger than the classic one?

 

The sequential number is a maximum. To stay valid at every look, the sequential test needs stronger evidence early on. If the lift is real, sequential tests usually stop around the classic sample size or sooner, and much sooner when the lift is larger than planned.