Matrix calculus

Every training loop in this curriculum runs on derivatives of scalar losses with respect to vectors and matrices, and yet most people compute them by pattern-matching against half-remembered identities. This closing lesson makes the patterns principled: three definitions, five identities, one chain rule — enough to derive by hand every gradient this curriculum has used, and to see (in the widget) what the Hessian’s eigenstructure means before the Deep Learning module automates all of it with autodiff.

Three derivative objects, one convention

For $f: \mathbb{R}^n \to \mathbb{R}$, the gradient $\nabla f \in \mathbb{R}^n$ collects the partials; for $F: \mathbb{R}^n \to \mathbb{R}^m$, the Jacobian $J \in \mathbb{R}^{m \times n}$ has $J_{ij} = \partial F_i / \partial x_j$; for scalar $f$, the Hessian $H_{ij} = \partial^2 f / \partial x_i \partial x_j$ is the (symmetric, by Schwarz) matrix of curvatures. The honest warning that saves hours of sign errors: two layout conventions exist in the wild (numerator vs denominator layout — whether a gradient is a row or a column), and mixed sources mix them silently. This curriculum uses column gradients ($\nabla f$ has the shape of $x$) and $m \times n$ Jacobians, matching PyTorch/JAX. Check shapes first, always; the correct expression is usually the only one whose shapes compose.

The best mental model is not partial derivatives at all but the linear approximation: the derivative of a map at $x$ is the matrix that best linearizes it,

Read against lesson 1: near any point, a differentiable map is a linear map (its Jacobian), so everything this module established about linear maps — rank, eigenstructure, conditioning — applies locally to nonlinear ones. That single observation is why linear algebra governs the training of deeply nonlinear networks.

The identities that do all the work

Derived once from the linearization view (perturb, expand, keep first order), then used forever:

Worked example, because it is the module’s recurring character: perturb $f(x) = \tfrac12\lVert Ax - b\rVert^2$ by $\delta$, expand $\tfrac12\lVert A x - b + A\delta \rVert^2$, and the first-order term is $(Ax - b)^\top A \delta$ — so $\nabla f = A^\top(Ax - b)$, and the Hessian (differentiate again) is $A^\top A$: the matrix whose eigenvalues gave least squares its condition number, whose SVD gave ridge its shrinkage, and whose positive semidefiniteness made the problem convex. One computation, four modules of consequences.

The chain rule in matrix form is Jacobians multiplying in composition order:

For a deep network $\ell(W_L \sigma(W_{L-1} \cdots \sigma(W_1 x)))$, the gradient of the loss with respect to early weights is a product of many Jacobians — and backpropagation is nothing but evaluating that product right-to-left (vector-times-Jacobian at each step, never materializing full Jacobians), which is why it costs only ~2× the forward pass. Two curriculum call-backs land here with a click: products of matrices compound their spectra, so many Jacobians with singular values below 1 give vanishing gradients and above 1 exploding ones — the saturation-plateau story from Non-convex Landscapes, now stated as linear algebra; and initialization schemes (Deep Learning module, ahead) are precisely spectrum-control for these products.

Seeing second order: the quadratic form under your hands

The Hessian’s Taylor term $\tfrac12 \delta^\top H \delta$ is a quadratic form, and its geometry — bowls, saddles, ravines — is fully described by the spectral theorem: eigenvectors are principal axes, eigenvalues are curvatures along them. You have classified these algebraically since the Optimization module; here is the object itself, with the entries of a symmetric $A$ on sliders:

What autodiff changes, and what it doesn’t

Frameworks compute all of this mechanically — reverse-mode autodiff is the chain rule with bookkeeping — so why learn it by hand? Three durable reasons. Shapes and conventions still break code daily, and the person who can write $\nabla_W \lVert XW - Y\rVert^2 = 2X^\top(XW - Y)$ debugs in minutes what shape-guessing debugs in hours. Second, custom losses, custom layers, reparameterization tricks and implicit-differentiation setups all require exactly these derivations at the boundary where the framework can’t see. Third — and this is the module’s closing thought — the derivative objects carry the geometry: the Jacobian’s spectrum explains vanishing gradients, the Hessian’s spectrum explains optimizer behaviour, the gradient’s covariance explained SGD’s noise ball. Autodiff gives you the numbers; this module gave you their meaning.

import torch

X = torch.randn(50, 3)
W = torch.randn(3, 2, requires_grad=True)
Y = torch.randn(50, 2)

loss = 0.5 * ((X @ W - Y) ** 2).sum()
loss.backward()
manual = X.T @ (X @ W.detach() - Y)          # this lesson, by hand
torch.testing.assert_close(W.grad, manual)   # autodiff agrees — it is the chain rule

Exercises