Regularization and the bias–variance trade-off

Gauss–Markov crowned OLS the best unbiased linear estimator, and the previous lesson’s widget showed a degree-12 “best unbiased” fit thrashing wildly between data points. The resolution of that apparent contradiction is the single most important idea in classical ML: expected error decomposes into bias plus variance, and only their sum matters. Regularization is the art of trading one for the other on purpose.

The decomposition, derived

Fix a query point $x$ . The data set $\mathcal{D}$ is random (drawn from the world), so the fitted model $\hat f_{\mathcal D}$ is a random function. For a new observation $y = f(x) + \varepsilon$ , $\operatorname{Var}(\varepsilon) = \sigma^2$ , add and subtract $\mathbb{E}_{\mathcal D}[\hat f(x)]$ inside the square and take expectations — the cross-terms vanish:

\mathbb{E}_{\mathcal D, \varepsilon}\big[(y - \hat f_{\mathcal D}(x))^2\big] = \underbrace{\sigma^2}_{\text{irreducible}} + \underbrace{\big(\mathbb{E}[\hat f(x)] - f(x)\big)^2}_{\text{bias}^2} + \underbrace{\mathbb{E}\big[(\hat f(x) - \mathbb{E}[\hat f(x)])^2\big]}_{\text{variance}} .

Bias: how wrong your average fit is — the cost of a model class too rigid for the truth. Variance: how much the fit wobbles across data sets — the cost of capacity spent memorizing noise. Flexibility moves the two in opposite directions, and the minimum of their sum sits at neither extreme.

The definition involves an expectation over data sets you never get to see — which is why the concept stays abstract in most courses. Here you can just run the expectation:

At degree 1: every faint fit is nearly identical (variance ≈ 0) and identically wrong (all bias) — the average curve misses the truth’s bends. At degree 10 with tiny λ: the average curve hugs the truth (bias ≈ 0) but the faint bundle is a storm — any single data set hands you one arbitrary member of it. Now hold degree at 10 and raise λ: the bundle tightens dramatically while the average barely moves — you are watching ridge sell a little bias for a lot of variance, the readout numbers making the trade explicit. Find the (d, λ) minimizing bias²+var: it will not be the most flexible model, and it will not be the least.

Ridge: what shrinkage actually does

Ridge regression penalizes the coefficient norm:

\hat\beta^{\text{ridge}} = \arg\min_\beta \; \tfrac{1}{2}\lVert y - X\beta \rVert^2 + \tfrac{\lambda}{2} \lVert \beta \rVert^2 \quad\Longrightarrow\quad \hat\beta^{\text{ridge}} = (X^\top X + \lambda I)^{-1} X^\top y .

(Convention: the intercept is never penalized — centre $y$ and the columns of $X$ , fit without an intercept, restore it after.) Adding $\lambda I$ makes the problem $\lambda$ -strongly convex: solvable even with collinear or $d > n$ designs, and better-conditioned — the optimization module’s $\kappa$ improves to $(L + \lambda)/(\mu + \lambda)$ . But the sharpest description comes from the SVD $X = U D V^\top$ , which diagonalizes the estimator:

\hat y^{\text{ridge}} = \sum_{j=1}^{d} u_j\, \frac{d_j^2}{d_j^2 + \lambda}\, u_j^\top y .

OLS keeps each singular direction of the design fully ( $d_j^2/d_j^2 = 1$ ); ridge multiplies direction $j$ by the shrinkage factor $d_j^2 / (d_j^2 + \lambda)$ — close to 1 where the data are informative (large $d_j$ ), close to 0 along the thin directions where OLS’s variance explodes (recall the $1/(1-\rho^2)$ blow-up from lesson 1: those are exactly the small- $d_j$ directions). Ridge is a per-direction volume knob that turns down precisely the directions statistics cannot support. The effective degrees of freedom follow immediately: $\operatorname{df}(\lambda) = \sum_j d_j^2/(d_j^2 + \lambda)$ , sliding continuously from $d$ (OLS) to $0$ — capacity as a dial rather than an integer.

Ridge also wears two other hats worth knowing: it is the Bayesian posterior mean under a Gaussian prior $\beta \sim \mathcal N(0, \tau^2 I)$ with $\lambda = \sigma^2/\tau^2$ , and its λ-path is intimately related to early-stopped gradient descent on OLS — regularization and optimization are the same subject wearing different clothes.

Lasso: the geometry of exactly-zero

Swap the penalty’s norm and the character changes completely:

\hat\beta^{\text{lasso}} = \arg\min_\beta \; \tfrac{1}{2}\lVert y - X\beta \rVert^2 + \lambda \lVert \beta \rVert_1 .

The constrained view explains everything: minimize the loss subject to $\lVert\beta\rVert_1 \le t$ . The loss’s elliptical contours expand until they first touch the constraint set. An $\ell_2$ ball is round — the touch point is generic, no coordinate favoured, so ridge shrinks everything but zeros nothing. The $\ell_1$ ball is a diamond, and expanding ellipses overwhelmingly first touch it at a corner — where coordinates are exactly zero. Sparsity is not a numerical accident; it is the geometry of corners.

For orthonormal designs (exercise 5 last lesson pays off) both penalties act coordinate-wise on the OLS solution, and the contrast is exact:

\text{ridge:}\;\; \hat\beta_j = \frac{\hat\beta_j^{\text{OLS}}}{1+\lambda} \qquad \text{lasso:}\;\; \hat\beta_j = \operatorname{sign}(\hat\beta_j^{\text{OLS}})\big(\lvert\hat\beta_j^{\text{OLS}}\rvert - \lambda\big)_+

— proportional shrinkage versus soft-thresholding: slide toward zero and stop there. The kink at zero costs the closed form in general (the objective is convex but non-smooth — the optimization module’s lasso caveat); the standard solver is coordinate descent applying exactly that soft-threshold one coordinate at a time, which is what the widget below runs.

With ridge, sweep λ upward: the curve calms, and the β list shrinks smoothly — every coefficient small, none exactly 0. Switch to lasso at the same λ: coefficients vanish one by one (the readout counts nonzeros) until a handful carry the whole fit — automatic feature selection happening in front of you. Push λ high enough and lasso returns the constant model. For the honest comparison, pick the λ minimizing test MSE under each penalty — and notice how unbothered ridge is by the exact value, versus how sharply lasso’s support set changes.

Choosing in practice

Situation	Reach for	Why
Many weakly-informative correlated features	ridge	spreads weight across correlated groups; stable paths
You believe few features matter, want them named	lasso	exact zeros = selection; interpretable support
Correlated groups AND sparsity	elastic net ( $\alpha\lVert\beta\rVert_1 + \tfrac{1-\alpha}{2}\lVert\beta\rVert^2$ )	lasso alone picks one of a correlated group arbitrarily
$d \gg n$	any of the above — OLS is not even defined	penalty supplies the missing strong convexity

Two habits that separate practitioners from tutorials: standardize features before penalizing (both penalties compare coefficient magnitudes across features — that is meaningless across units; the widget’s features are standardized internally), and choose λ by cross-validation, never by eye on training error — training error is monotone in capacity, so it will always vote for λ = 0. The evaluation lesson closes that loop.

Exercises

Work these before the next lesson

Complete the bias–variance derivation: expand $(y - \hat f)^2$ with the add-and-subtract, and show both cross-terms have expectation zero (which independence assumptions do you use, and where?).
Solution
Worked solutions are part of Premium — unlock all of them for £5/month →
4 more exercises — each with a worked solution — are part of Premium. Unlock everything for £5/month →

References

A. Hoerl & R. Kennard, “Ridge Regression: Biased Estimation for Nonorthogonal Problems”, Technometrics 1970.
R. Tibshirani, “Regression Shrinkage and Selection via the Lasso”, JRSS-B 1996.
H. Zou & T. Hastie, “Regularization and variable selection via the elastic net”, JRSS-B 2005.
ESL §3.4 (shrinkage, the SVD picture) and §7.3 (bias–variance) — the backbone of this lesson.
J. Friedman, T. Hastie, R. Tibshirani, “Regularization Paths for Generalized Linear Models via Coordinate Descent”, J. Stat. Software 2010 — the glmnet paper; the solver the lasso widget imitates.