Evaluation and cross-validation

Every model in this module was judged by “test accuracy” in a widget. This closing lesson is about earning that number. Evaluation is where most real-world ML failures actually live — not in the estimator, but in a leaked feature, a reused test set, or a metric that answered the wrong question. The theme throughout: an evaluation is an estimate with a variance and a bias of its own, and it deserves the same scrutiny as the model.

What we are estimating, and with what error

The target is generalization error $\mathbb{E}[\ell(y, \hat f(x))]$ over the data distribution. A held-out test set of size $m$ estimates it by an average of $m$ i.i.d. draws — so it carries sampling error. For accuracy $\hat p$, the standard error is $\sqrt{\hat p(1-\hat p)/m}$: at $m = 300$ and 90% accuracy that is ±1.7 percentage points, which quietly means most “0.5% improvements” on small test sets are noise. Rule one of the lesson: never report a metric without asking what its error bar is. Rule two: every time you make a decision by looking at a test set (pick a model, tune a knob, stop early), you spend some of its validity — adaptive reuse turns an unbiased estimate into an optimistically biased one. Hence the three-way discipline: train (fit parameters), validation (choose hyperparameters, early-stop), test (touch once, at the end, to report).

k-fold cross-validation

Small data makes a fixed validation split both wasteful and noisy. Cross-validation recycles: partition into $k$ folds; for each fold, train on the other $k{-}1$ and validate on it; average.

Every sample validates exactly once; every fit sees $\tfrac{k-1}{k}$ of the data (a slight pessimistic bias, since the final model will see all of it). The $k$ dial: small $k$ = more bias, cheap; $k = n$ (leave-one-out) = nearly unbiased but expensive and — counterintuitively — high-variance, because the $n$ fits are almost identical and their errors highly correlated (the forest lesson’s $\rho\sigma^2$ term, back again: averaging correlated things helps less than it looks). $k \in \{5, 10\}$ is the standard compromise. Stratify folds for classification so each preserves class proportions.

Two structural rules complete the tool. Model selection: if CV chose the hyperparameters, the winning CV score is biased upward (it won a tournament); reporting it as the performance estimate is self-deception — wrap an outer loop (nested CV) or keep a untouched test set for the final number. Structure-aware splitting: i.i.d. folds lie when data are not i.i.d. — time series must split past→future (forecasting track’s backtesting), grouped data (many rows per patient/user) must keep groups intact, or you are testing on the training distribution’s twins.

Leakage: the silent killer

Leakage = information available at training/evaluation time that will not exist at prediction time. It produces beautiful validation scores and production faceplants, and it is overwhelmingly a pipeline property, not a modeling one. The canonical traps, all field-tested:

• Preprocessing before splitting. Standardizing, imputing, or selecting features on the FULL data lets test-set statistics into training. Everything that learns from data — scalers included — must fit inside each training fold only (sklearn.Pipeline inside cross_val_score exists precisely for this).
• Feature-selection leakage. Screening 10,000 features against the full data’s labels, then cross-validating on the survivors, yields spectacular fake accuracy — the selection already saw every fold’s answers.
• Target leakage. A feature that is a downstream consequence of the label (the “days_in_icu” column predicting mortality; the refund flag predicting fraud). Symptom: one feature with implausibly dominant importance.
• Duplicate/near-duplicate rows straddling the split; temporal leakage (future aggregates in features); group leakage (same user both sides).

The professional habit: treat a too-good validation score as a bug report, not a win. Kaufman et al.’s survey catalogues how routinely this bites even published work.

Metrics under imbalance: ROC vs precision–recall

With 99%-negative data, accuracy is a broken instrument (the all-negative model scores 99%). The confusion matrix’s honest derivatives, as threshold $t$ sweeps (lesson 3 separated the model from the threshold; here is the payoff):

The ROC curve traces (FPR, TPR) over all thresholds; its area (AUC) is threshold-free and has a clean meaning — the probability a random positive scores above a random negative. But both its axes are within-class rates, so it literally cannot see class imbalance: a flood of false positives divides by the enormous negative count and vanishes into a small FPR. The precision–recall curve puts the false positives where you feel them — in precision’s denominator — which is why it is the honest picture whenever positives are rare and false alarms are the operational cost. Same model, same scores, two very different-looking judgments:

Complete the metric toolkit with what each is for: F1 (harmonic mean) when you must collapse precision/recall to one number at a fixed threshold; log loss / Brier when probability quality matters (proper scores — lesson 3); average precision as the PR curve’s area. And resist the single-number reflex: the curve is the information; the number is a lossy summary for leaderboards.

The reporting standard

What “we evaluated the model” should mean, in one box — and what this module now lets you defend line by line:

1. Split with the data’s structure respected (time, groups, strata).
2. All preprocessing inside the training folds (pipelines, not scripts).
3. Hyperparameters chosen on validation/CV; test touched once.
4. Metrics matched to the decision and the imbalance; curves shown, not just areas.
5. Uncertainty attached: ± across folds and across seeds (the SGD lesson’s variance is part of your result).
6. A baseline that would embarrass you if beaten narrowly — majority class, last-value, logistic regression — because “good” is only defined relative to one.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

pipe = make_pipeline(StandardScaler(), LogisticRegression())   # scaler INSIDE
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(pipe, X, y, cv=cv, scoring="average_precision")
print(f"AP {scores.mean():.3f} ± {scores.std():.3f}")          # mean AND spread

Exercises