Treatment group conversion rate: 4.3%. Control group: 4.1%. Difference: 0.2%. Is that real, or random noise? You need statistics to answer that question, and most engineers implement A/B testing without really understanding what their stats are telling them.

The Core Problem#

If you flip a fair coin 10 times and get 6 heads, you don’t conclude the coin is biased. You know 6 out of 10 is within the range of normal random variation. Conversion rates work the same way: random variation in which users were exposed to which experience produces differences even if the treatment has no real effect.

Statistical significance testing asks: “If the treatment had no real effect, how likely is it that we’d observe a difference this large or larger by chance?” The p-value is that probability. Convention says p < 0.05 is “significant”: less than 5% chance this is random noise.

Sample Size Matters#

The smaller your effect size (the difference you’re trying to detect), the more samples you need. A 10% relative improvement in conversion rate is easy to detect; a 0.1% improvement requires a very large sample.

Minimum Detectable Effect (MDE): the smallest difference you care about. If an improvement smaller than 0.5% absolute isn’t worth shipping, your MDE is 0.5%.

Required sample size formula: roughly proportional to (baseline_rate * (1 - baseline_rate)) / (MDE^2). For a baseline rate of 4% and MDE of 0.2%, you need roughly 100,000 users per variant to detect the difference reliably.

Baseline rate: 4% (0.04)
MDE: 0.2% (0.002)
Required N ≈ 16 * 0.04 * 0.96 / 0.002^2 ≈ 150,000 per variant
graph TD A[Define MDE: smallest effect worth detecting] --> B[Calculate required sample size] B --> C[Run experiment until sample reached] C --> D{p-value less than 0.05?} D -->|Yes| E[Effect is statistically significant] D -->|No| F[Insufficient evidence: do not ship] E --> G{Effect size meets MDE?} G -->|Yes| H[Ship treatment] G -->|No| I[Effect is real but too small to matter] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff

Peeking and Multiple Testing#

The most common mistake: checking the results every day and stopping when you see p < 0.05. This is called peeking, and it inflates your false positive rate dramatically. If you check 20 times, the probability of seeing p < 0.05 at least once by chance is nearly 64%, even if the true effect is zero.

The fix: decide the sample size before starting, then stop the experiment when you reach it. Don’t look at results until the sample is complete. If you need early stopping for business reasons, use sequential testing methods (like SPRT) that account for peeking.

Multiple comparisons: testing 20 metrics in one experiment means 1 of them will likely show p < 0.05 by chance. Apply Bonferroni correction (divide the significance threshold by the number of comparisons) or define one primary metric before the experiment starts and treat others as secondary.

At Salesforce#

We “ran A/B tests” by enabling features for some orgs and monitoring error rates. We peeked constantly. We ended up shipping features that showed “improvement” in our peeks that turned out to be random variation. The 80% reduction in review cycles I reference in my Salesforce metrics was measured over a proper timeframe with a clear control group. Experiments I didn’t run properly had much worse track records.

What I’m Learning#

Statistical significance is only the starting question. Effect size, practical significance, and proper experimental hygiene (no peeking, pre-registered primary metric) determine whether your A/B test results are trustworthy.

Have you seen experiments at your company that were called successful based on flawed statistical analysis?