A/B testing and experimentation

The last lesson built the test; this one builds the experiment — because in industry the statistics is the easy half. An A/B test is a randomized controlled trial run on a product: split traffic, treat one arm, compare. Done right it is the only instrument that reliably converts “we changed X” into “X caused Y”. Done casually it manufactures confident nonsense at scale — and the single most common way to do it casually is so seductive, so universal, and so quantifiable that this lesson hands it to you as a simulation you can run until convinced.

Randomization is the active ingredient

Everything an A/B test can promise flows from one act: units assigned to arms by coin flip. Randomization severs every confounder, known and unknown — the causal-inference lesson ahead handles the world where you can’t randomize; here you can, so use it cleanly. The design details that decide validity before any statistics runs:

• Unit of randomization = unit of analysis. Randomize users but analyze page-views and your $n$ is a lie (a heavy user contributes 500 correlated rows — the CLT lesson’s correlated-samples warning, now with a dashboard attached).
• Hash-based assignment (hash(user_id) % 100 — the reproducibility lesson’s split trick) keeps a user in one arm across sessions and devices, forever, deterministically.
• A/A tests as calibration: run the machinery with no treatment difference; it should fire at exactly α. If your platform’s A/A tests “find” effects 15% of the time, every A/B result it ever produced is suspect. This lesson’s widget is, precisely, an A/A test battery.
• Guardrail metrics: alongside the success metric, monitor what must NOT regress (latency, errors, unsubscribes) — a treatment that wins its target by hurting a guardrail is a loss wearing a win’s clothes.

The sample-size arithmetic is last lesson’s power formula with business numbers in it. Detecting a 1% relative lift on a 5% conversion rate (δ ≈ 0.0005 absolute, binary outcome) needs roughly $n \approx 16\sigma^2/\delta^2 \approx$ 3 million users per arm — a number that shocks every product team the first time, explains why experiment platforms obsess over variance reduction (CUPED and friends: regress out pre-experiment behavior, shrink $\sigma$, cut $n$ by 30–50%), and sets up this lesson’s central sin. Because when the honest $n$ is millions and the dashboard updates hourly… people look.

The peeking problem, demonstrated rather than asserted

Fixed-horizon testing’s guarantee — false positives at rate α — holds for a test performed once, at a pre-registered $n$. Watch the dashboard daily and stop “when it’s significant”, and you are running a different procedure with a very different error rate:

The fixes form a ladder. Discipline: pre-register $n$, look once — free, fragile against human nature. Alpha spending / group-sequential designs (O’Brien–Fleming): schedule a few interim looks with adjusted thresholds so early stopping is priced in — the clinical-trials standard. Always-valid inference (mSPRT and kin — what mature experiment platforms actually run): confidence sequences that remain valid under continuous monitoring, costing wider intervals in exchange for the right to peek freely. And the Bayesian reframe (posterior probabilities updated continuously) dissolves the stopping problem but re-imports the prior-choice one — the Bayes lesson’s trade, in production. The meta-lesson is the module’s oldest: the validity of an inference depends on the procedure that produced it, including the procedure’s stopping rule — which no p-value printed at the end can see.

Beyond the single test, three realities of experimentation at scale, so they’re on your map: interference (one user’s treatment affecting another’s outcome — marketplaces, social feeds — breaks the independence every formula above assumed; cluster or switchback designs respond); novelty and primacy effects (week-one lifts that decay — run long enough to cover a full behavioral cycle); and the winner’s curse (effects estimated from the experiments that passed the threshold are biased upward — the same selection artifact as the best-seed reporting trap, and the reason meta-analyses of shipped wins keep “finding” less lift than the launch decks claimed).

import numpy as np
rng = np.random.default_rng(0)

# the peeking law, in eight lines: A/A tests with continuous monitoring
def aa_test(peek_every, n=1000):
    x = rng.standard_normal(n).cumsum()
    z = x / np.sqrt(np.arange(1, n + 1))
    looks = z[peek_every - 1 :: peek_every]
    return np.abs(looks).max() > 1.96

for pe in [1000, 200, 50, 10]:
    fp = np.mean([aa_test(pe) for _ in range(2000)])
    print(f"peek every {pe:4d}: {fp:.1%} false positives")
# 1000 → ~5%.  10 → ~38%.  Same data, same math, different LOOKING.

Exercises