The singular value decomposition

The spectral theorem was magnificent and narrow: it demanded symmetry. Data matrices are rectangular; weight matrices are whatever training made them. The singular value decomposition is the theorem with the asterisk removed — every matrix factors into rotate → stretch → rotate — and it is, without much competition, the most useful single fact in applied linear algebra: PCA is an SVD, LoRA truncates one implicitly, recommender systems factorize by it, and the best-rank-$k$ theorem it carries is the mathematical license for the word “compression” in half of ML.

The statement

For any $A \in \mathbb{R}^{m \times n}$:

with $U$ ($m \times m$) and $V$ ($n \times n$) orthogonal, and $\Sigma$ diagonal with non-negative entries $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$ — the singular values. Geometrically: $V^\top$ rotates the input space, $\Sigma$ stretches along perpendicular axes (padding or dropping dimensions if $m \ne n$), $U$ rotates the output space. Shear does not exist — what looked like shear in lesson 1’s widget was only ever a stretch caught between two rotations. The unit sphere maps to an ellipsoid whose axes are the columns of $U$, with lengths $\sigma_i$.

Where it comes from: $A^\top A$ is symmetric and positive semidefinite, so the spectral theorem hands us orthonormal eigenvectors $v_i$ with eigenvalues $\lambda_i \ge 0$. Define $\sigma_i = \sqrt{\lambda_i}$ and $u_i = A v_i / \sigma_i$; a short computation (exercise 1) shows the $u_i$ are themselves orthonormal, and $A = \sum_i \sigma_i u_i v_i^\top$ follows. Two corollaries you have already used without the receipts: the singular values of $X$ are the $d_j$ in ridge regression’s shrinkage factors $d_j^2/(d_j^2+\lambda)$, and $\kappa(X^\top X) = (\sigma_{\max}/\sigma_{\min})^2$ — the “conditioning squared” warning from the Linear Regression lesson, now a one-line proof.

The outer-product reading, and Eckart–Young

The decomposition rewrites as a sum of rank-one layers:

— each layer a full-size matrix built from one column and one row, weighted by $\sigma_i$. Since the weights are sorted, truncating the sum keeps the most important structure, and the remarkable fact is that this greedy move is optimal:

Eckart–Young theorem. Among ALL matrices of rank $\le k$, the truncated SVD $A_k = \sum_{i \le k} \sigma_i u_i v_i^\top$ minimizes $\lVert A - B \rVert$ (in Frobenius and spectral norms), with error $\lVert A - A_k \rVert_F^2 = \sum_{i > k} \sigma_i^2$.

“Best possible rank-$k$ approximation” is a strong claim over an enormous set, and the SVD just is it. Storage tells the practical story: a rank-$k$ version of an $m \times n$ matrix needs $k(m + n + 1)$ numbers instead of $mn$. Whether that is a bargain depends entirely on how fast the $\sigma_i$ decay — which is a property of the data, and the widget lets you inspect it directly:

The SVD in the ML you already know

PCA is the SVD of centred data. Centre the data matrix $X$ (rows = samples); its covariance is $\Sigma = X^\top X / n$, whose eigenvectors (= right singular vectors of $X$) are the principal components — perpendicular directions of maximal variance, with variances $\sigma_i^2/n$. Projecting onto the top $k$ is Eckart–Young applied to data: the $k$-dimensional view that loses least. (The next lesson builds the projection geometry; the Unsupervised module will use it in anger, including its failure modes.)

LoRA is a rank bet, and the SVD is why it’s plausible. The RL track’s fine-tuning lesson wrote updates as $\Delta W = BA$ with rank $r$. The empirical finding that motivates it — measured fine-tuning updates have rapidly-decaying singular spectra — is precisely “the widget’s bar chart, but for weight changes.” When the bet holds, a rank-16 update captures almost everything a full update would; Eckart–Young says a low-rank representation can; the spectrum decides whether it does.

Recommenders, embeddings, denoising. Latent-factor recommendation approximates the (sparse) user–item matrix by a low-rank product — users and items each get a $k$-vector whose inner product predicts affinity (lesson 1’s similarity, again). Classical LSA embedded words by SVD of word–document counts a decade before word2vec. And keeping top-$k$ while discarding the small-$\sigma$ tail is a denoiser whenever signal is structured and noise is not — the flat-tail intuition from the widget, used in production.

Numerics. The pseudoinverse $A^+ = V \Sigma^+ U^\top$ solves least squares even for rank-deficient designs (np.linalg.lstsq is an SVD under the hood), and $\sigma_{\max}/\sigma_{\min}$ is the condition number — the quantity this curriculum has been paying taxes to since the gradient-descent lesson.

import numpy as np

A = np.random.randn(48, 96)
U, s, Vt = np.linalg.svd(A, full_matrices=False)

k = 10
A_k = U[:, :k] * s[:k] @ Vt[:k]                      # best rank-k, by theorem
err = np.linalg.norm(A - A_k, "fro") ** 2
np.testing.assert_allclose(err, np.sum(s[k:] ** 2))  # Eckart–Young, verified

storage = k * (A.shape[0] + A.shape[1] + 1) / A.size # the compression ratio

Exercises