Estimators and uncertainty

The module closes where working data science spends its days: you computed a number from a sample — a mean, a median, an AUC, a treatment effect — and someone asks the only question that matters: how much should we trust it? This lesson gives the classical vocabulary for judging estimators, the precise (and precisely misread) contract of a confidence interval, and then the great modern equalizer — the bootstrap, which buys an uncertainty estimate for nearly any statistic with compute instead of theory.

Judging an estimator

An estimator is any rule $\hat\theta(X_1, \ldots, X_n)$ turning data into a guess. Because the data is random, $\hat\theta$ is a random variable with its own distribution — the sampling distribution — and judging an estimator means judging that distribution. Three criteria carry the load:

• Bias: $\operatorname{bias} = \mathbb{E}[\hat\theta] - \theta$ — is it centred on the truth? The sample mean is unbiased; the variance MLE $\frac1n\sum(x_i - \bar x)^2$ is not (short by the factor $\frac{n-1}{n}$ — the $n{-}1$ affair from lesson 3’s exercise, now with its proper name).
• Variance: how much does it wobble across datasets you could have drawn?
• Consistency: does it converge to the truth as $n \to \infty$? (The biased variance MLE is still consistent — bias $\to 0$; small-sample bias and large-sample correctness coexist happily.)

And the criterion that binds the first two is an old friend wearing new clothes:

— the bias–variance decomposition, exactly as the Supervised Learning module stated it for models, because a fitted model is an estimator and generalization error is estimation error. And the Bayesian lesson’s shrinkage now slots in as strategy, not accident: accepting a little bias (toward the prior, toward zero) to cut a lot of variance is often a net win — ridge regression and the James–Stein phenomenon are both this trade executed deliberately.

The confidence-interval contract, read carefully

The CLT gives the workhorse interval: $\bar x \pm 1.96\, s/\sqrt n$ covers the true mean in 95% of repeated experiments. Every word of that contract is load-bearing, and one misreading is nearly universal, so here is the contrast laid flat:

• A 95% confidence interval (frequentist): the procedure traps the fixed, unknown truth in 95% of the datasets it could be run on. About this dataset’s interval, the theory says nothing probabilistic — it either contains $\theta$ or it doesn’t.
• A 95% credible interval (Bayesian, last lesson’s band): given this dataset and a prior, the parameter lies inside with probability 0.95 — the statement people actually want to make.

In large samples with weak priors the two often nearly coincide (the Bernstein–von Mises result, informally), which is why practitioners get away with the sloppy reading — until small samples, strong priors, or boundary parameters split them apart. Report which one you computed; they answer different questions.

The bootstrap: uncertainty by resampling

The classical route needs a formula for the sampling distribution — fine for the mean, hopeless for medians, ratios, AUCs, or a fine-tuned model’s win-rate. Efron’s 1979 move: the sample is your best available estimate of the population, so resample from it. Draw $n$ points with replacement from your data, recompute the statistic, repeat $B$ times; the spread of those $B$ values estimates the sampling distribution you cannot derive.

No new data, no new theory — just the plug-in principle plus compute. Watch it audit itself against the one case where theory has the exact answer:

The fine print, before it bites: the bootstrap trusts the sample to represent the population, so it inherits every flaw of the sampling (a biased sample bootstraps into confidently biased intervals — resampling cannot manufacture information that was never collected). It needs care with dependent data — time series and grouped users must be resampled in blocks or by cluster, or the CIs are fiction (lesson 2’s correlation warning again, in operational form). And it struggles with extreme-tail statistics (a max, a 99.9th percentile) that hinge on the few points a resample often misses.

The estimation arc, closed

Where this leaves you operationally — the practitioner’s uncertainty toolkit in one paragraph: error bars on any metric (bootstrap the test set: resample rows, recompute accuracy/AUC/F1, report the percentile CI — the honest version of lesson 2’s benchmark arithmetic, and why cross-validation’s fold-to-fold spread in the k-fold widget was drawn at all); comparisons as intervals on the difference, since two overlapping single-model CIs do not settle a comparison — bootstrap the paired difference instead; and calibrated humility about what n can and cannot resolve, priced by $\sigma/\sqrt n$. That closes the module’s estimation arc: you can now attach an honest error bar to any quantity. What you cannot yet do is turn an error bar into a decision — ship or hold, significant or noise — and that is exactly where the next lesson picks up: the decision arc of hypothesis testing, A/B experiments, and causal inference.

import numpy as np
rng = np.random.default_rng(11)

x = rng.lognormal(0.0, 0.8, size=60)              # one skewed sample
boot = rng.choice(x, size=(1000, len(x)), replace=True)

means = boot.mean(axis=1)
print(means.std(), x.std(ddof=1) / np.sqrt(len(x)))          # bootstrap SE vs s/√n
print(np.percentile(means, [2.5, 97.5]))                     # percentile CI

medians = np.median(boot, axis=1)                            # no formula needed
print(np.percentile(medians, [2.5, 97.5]))

Exercises