Non-convex landscapes

Every guarantee in this module so far leaned on convexity, and every interesting model you will train violates it. This closing lesson maps the territory honestly: what the critical points of a non-convex surface look like, which of them actually bite in high dimensions (the answer is not the one 2-D intuition suggests), and what a decade of loss-landscape research established about why the convex-case toolkit — SGD, momentum, Adam — trains hundred-billion-parameter networks anyway.

The taxonomy of flat places

A critical point is any $x$ with $\nabla f(x) = 0$ — everywhere gradient descent can converge to, since it only ever sees the gradient. The Hessian’s eigenvalues $\lambda_i$ classify it:

Two of these dominate discussion; only one deserves it, as the next section argues.

Why high dimensions demote local minima and promote saddles

The classic fear — “we’ll get stuck in a bad local minimum” — is a low-dimensional intuition. A counting argument dissolves it. At a random critical point of a generic high-dimensional function, model each Hessian eigenvalue’s sign as an independent coin flip: the probability that all $d$ of them come up positive — a local minimum — is $2^{-d}$. In $d = 10^6$ dimensions, a critical point with no downhill direction is exponentially rare — a lottery no one wins — except in one systematic place: where the loss is already low, because near the bottom of the landscape there is little room left to curve downward. Random-matrix analyses and empirical Hessian studies of neural networks (Dauphin et al., Choromanska et al.) sharpened this into the modern picture:

High-dimensional loss surfaces are overwhelmingly populated by saddle points, and the local minima that do exist concentrate at loss values close to the global one. The practical enemy is not a deep trapping basin — it is the vast, nearly-flat neighbourhoods around saddles, where the gradient is small in every direction and progress crawls.

This inverts the naive worry. The two-wells surface from the gradient-descent lesson — genuinely distinct basins with different loss — is what low-dimensional non-convexity looks like. Overparameterized networks look more like an enormous connected family of good solutions (mode-connectivity experiments find low-loss paths between independently trained minima) punctured by saddles you must pass, not escape.

Saddle dynamics: why plain GD stalls and noise saves it

Near a saddle, take the diagonal model $f(x, y) = \tfrac{\lambda_+}{2} x^2 - \tfrac{\lambda_-}{2} y^2$. Gradient descent gives, per coordinate,

The stable direction contracts; the escape direction grows geometrically — but only in proportion to how far along it you already are. Arrive with $y_0 \approx 0$ (and a symmetric initialization can arrange exactly that) and the exponential amplifier has nothing to amplify: GD spends an arbitrarily long time creeping along the stable manifold, gradient norm shrinking, loss curve flat, everything looking converged. The escapes are equally mechanical:

• Noise (SGD’s free gift): a random kick puts mass on the unstable direction, and the $\,(1 + \eta\lambda_-)^t$ amplifier does the rest. Perturbed GD escapes strict saddles in polynomial time (Jin et al.) — one of the few clean theorems in this area.
• Momentum: velocity accumulated before the saddle carries the iterate across the flat region rather than parking in it.
• Strict saddles ($\lambda_- > 0$ strictly) are provably not where GD converges from random init (Lee et al.) — the stable manifold has measure zero. The cost is time, not final destination; the flat crawl is the price actually paid.

Ravines, plateaus, and where they come from in real networks

The remaining pathologies are the degenerate row of the taxonomy table, and they are manufactured systematically by architecture:

• Ill-conditioning at scale. Real Hessian spectra show a bulk of near-zero eigenvalues plus a few large outliers — a ravine in millions of dimensions. This is why the adaptive-methods lesson’s per-coordinate scaling is near-mandatory for transformers, and why normalization layers (which condition the surface) were adopted as trainability tools before anyone argued about their regularization effects.
• Saturation plateaus. Squashing activations (sigmoid, tanh) have exponentially small derivatives in their tails; chains of them multiply into vanishing gradients — a plateau not near a critical point but across swaths of parameter space. ReLU-family activations and careful initialization scales (Glorot/He — a later module derives them) exist precisely to keep early training out of these dead zones.
• Symmetry manifolds. Permuting hidden units, or scaling one layer up while scaling the next down (with homogeneous activations), leaves the function unchanged — entire flat directions of the loss along which the gradient is exactly zero and along which nothing needs to move. Much of a network’s “non-convexity” is this benign redundancy rather than genuine multi-modality.

Flat vs sharp minima. Within the good-solutions family, basins differ in width, and small-batch noise biases SGD toward wide ones (the noise ball cannot sit in a basin narrower than itself — the SGD lesson’s cloud, reinterpreted as a selection principle). Wide minima correlate with better generalization often enough that an optimizer (SAM) explicitly seeks them — but the correlation has real caveats: sharpness is not reparameterization-invariant (Dinh et al. construct sharp minima with identical generalization), so treat “flat = generalizes” as a useful heuristic with known counterexamples, not a law.

The practitioner’s diagnostic table

The point of landscape theory is reading training curves without superstition:

The through-line of the whole module, one last time: the learning rate fights curvature ($\kappa$, the gradient-descent lesson), noise fights the ball radius (the SGD lesson), memory and per-coordinate scaling fight the geometry (the momentum lesson), and high dimensionality — surprisingly — fights on your side (this lesson). With that map, the next modules’ architectures and initialization schemes stop being recipes and start being countermeasures aimed at specific, nameable geometry.

Exercises