Hypothesis testing

The bootstrap lesson closed this module’s estimation arc; this lesson opens its decision arc, because industry’s daily statistical question is not “what is the effect?” but “is this difference real, or noise?” — asked of a model comparison, a metric movement, an experiment. Hypothesis testing is the classical machinery for that question, and it is simultaneously the most-used and most-misused mathematics in applied work. This lesson builds it honestly: what a p-value is (precisely, because the precise version is the only one that survives contact with practice), what power buys, and the multiplicity trap that manufactures discoveries out of nothing.

The logic: argument by contradiction, statistically

A test is a proof by contradiction with noise in it. Assume the boring world — the null hypothesis $H_0$: no effect, no difference. Compute how surprising your data would be in that world. If it would be very surprising, doubt the world.

The test statistic standardizes the evidence. For comparing two group means with $n$ per group (the workhorse case — model A vs model B, control vs treatment):

— the difference in units of its own standard error, and the CLT lesson already did the heavy lifting: under $H_0$ this is standard normal. The p-value is then $\Pr(|Z| \ge |z_{\text{obs}}| \mid H_0)$: the probability of evidence at least this extreme, assuming the boring world.

Every word of that definition is load-bearing, because the famous misreadings all break one of them. A p-value is not the probability $H_0$ is true (that’s $\Pr(H_0 \mid \text{data})$ — a Bayesian quantity needing a prior; the Bayes lesson’s direction-of-conditioning lesson, again). $p = 0.04$ does not mean “96% chance the effect is real”. And $p > 0.05$ does not mean “no effect” — it means this experiment couldn’t tell, which brings in the concept that completes the picture:

Power: the other error, priced

Two worlds, two mistakes. Rejecting a true null (Type I, rate $\alpha$ — you control this by construction). Failing to reject a false one (Type II, rate $\beta$); power $= 1 - \beta$ is the probability of detecting an effect that is really there. Power is where testing stops being philosophy and becomes engineering, because it is a function of three dials you choose or estimate — effect size $\delta$, sample size $n$, threshold $\alpha$:

The practitioner’s power habits: compute $n$ before collecting (afterward it’s an autopsy); use the minimum effect you’d care about, not the effect you hope for; and read null results through the power lens — “we had 80% power to detect δ ≥ 0.2 and saw nothing” is information, “p = 0.3, n = 12” is a shrug.

The multiplicity trap

Run one test at $\alpha = 0.05$ and the false-positive rate is 5%. Run twenty — twenty metrics, twenty model variants, twenty subgroups — and the chance some test fires under a fully-boring world is $1 - 0.95^{20} \approx 64\%$. This is not a subtle effect; it is the engine of most false discoveries, and it hides everywhere flexibility does: testing many metrics, trying several model seeds and reporting the best (the Evaluation lesson’s warning, now with its mechanism), slicing users into subgroups until one “responds”, or adding data and re-testing until significance (next lesson’s entire subject). The defenses, in working order: Bonferroni (divide α by the number of tests — blunt, safe, fine for small $m$); Benjamini–Hochberg (control the false discovery rate instead — the standard when testing hundreds); and above all deciding the analysis before seeing the data, because no correction can price the tests you ran silently in your head. The RL track’s evaluation lessons and the backtesting lesson each met a costume of this trap; this is the underlying law.

Two closing calibrations. Statistical vs practical significance: with $n$ huge, a 0.01% difference gets $p < 10^{-6}$ — significance measures detectability, never importance; always report the effect size and its confidence interval alongside (the estimation arc and the decision arc are two views of one object: the test rejects at level α exactly when the CI excludes zero). And the test is downstream of the design: a significant difference between groups that weren’t randomized measures the assignment process as much as the treatment — which is why the next lesson is about running experiments, not just testing them.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
# model A vs model B on n=200 prompts each; B is truly +0.3σ better
a = rng.normal(0.0, 1, 200); b = rng.normal(0.3, 1, 200)
t, p = stats.ttest_ind(b, a)
print(f"t={t:.2f} p={p:.4f}")                    # detected, comfortably

# the multiplicity engine: 20 boring metrics, best p reported
boring = [stats.ttest_ind(rng.normal(0,1,200), rng.normal(0,1,200)).pvalue
          for _ in range(20)]
print(min(boring), "← the 'discovery' among 20 nulls")

# power planning: n per group for 80% power at delta=0.2, alpha=.05
from statsmodels.stats.power import TTestIndPower
print(TTestIndPower().solve_power(effect_size=0.2, alpha=0.05, power=0.8))

Exercises