Is the lift real, or just noise?
Plug in your variant data. We run a two proportion Z-test and tell you, in plain English, whether you can trust the number on your dashboard.
Control (A)
The current version
Variation (B)
The new version
p-value: 0.0091 · Variation B wins by 24.0%. You can roll it out.
Confidence ·
Rate A
Rate B
Relative lift
How it works
A two proportion Z-test, in 10 seconds
An A/B test splits traffic between the current page (control) and a candidate (variation). We compare the two conversion rates using a two proportion Z-test, which asks: if there was really no difference, how often would random chance produce a gap this big?
The p-value is the answer to that question. A p-value of 0.03 means the observed gap would happen by chance 3% of the time. Below 5% (0.05), we stop calling it luck.
Methodology
The math underneath
Formula
Z = (pB − pA) / √(P(1 − P)(1/nA + 1/nB))
- pA, pB
- Conversion rate for each variation.
- P
- Pooled conversion rate across both groups.
- nA, nB
- Visitor count per variation.
- Z
- Z-score, the distance between the two rates in standard error units. Converted to a p-value via the normal distribution.
Testing tips
Boring advice that wins tests
- 1
Run the test until significance. Stopping early inflates false positive rates more than most people realize.
- 2
Test one variable at a time. Two changes, significant result, zero learning: you cannot attribute the lift to either change.
- 3
Aim for ≥1,000 visitors per variation as a floor. Smaller samples need effect sizes big enough to be obvious from the dashboard.
- 4
Decide your significance threshold before the test starts. 95% is the industry default. Moving the goalposts later is the oldest trick in the fake science book.
- 5
Document every test. Losers teach as much as winners; the institutional memory compounds.
Run the math once. Run the form forever.
Try WittyForm freeMore calculators