Maximum likelihood

Where do loss functions come from? So far this curriculum has treated them as givens — squared error here, cross-entropy there — with occasional hints that something deeper was choosing them. This lesson is the reveal. Maximum likelihood is a single principle — pick the parameters under which the observed data would have been least surprising — and applied to different noise models it generates the curriculum’s losses one by one. After this lesson, “choosing a loss” and “asserting a noise model” are the same act, and you will know which assertion you are making.

The principle

A model with parameters $\theta$ assigns the observed data a probability — the likelihood, read as a function of $\theta$ with the data held fixed:

The maximum likelihood estimator is $\hat\theta = \arg\max_\theta L(\theta)$ — in practice always via the log (products become sums, tiny numbers become sane, optima are unchanged by monotone transforms):

And here is the bridge this curriculum has been building toward: minimizing negative log-likelihood is an optimization problem — the Foundations Optimization module’s machinery applies verbatim, and “training” is MLE (or its regularized cousins) running on gradient descent.

Two derivations that unmask the curriculum’s losses

Bernoulli → cross-entropy. Coin flips with $\Pr(x{=}1) = p$: with $s$ successes in $n$ flips,

The sample frequency — reassuringly. Now let $p$ depend on features through a sigmoid, $p_i = \sigma(\beta^\top x_i)$, and $-\ell$ is the logistic-regression loss from Supervised Learning, term for term. Cross-entropy was never a choice; it was Bernoulli MLE all along — and the language-model training objective (next token as a categorical variable) is the same derivation with more categories.

Gaussian → least squares. Model observations as signal plus Gaussian noise, $y_i = f_\theta(x_i) + \varepsilon_i$, $\varepsilon_i \sim \mathcal N(0, \sigma^2)$:

Maximizing over $\theta$ ignores the constant and minimizes the sum of squared errors: Gauss’s least squares — the method that found Ceres in the Start Here story — is exactly MLE under Gaussian noise, which is the argument Gauss himself ran (in the other direction: he derived the normal distribution as the noise law that justifies the mean). The corollaries write themselves: absolute-error loss is MLE under Laplace noise (heavier tails → robust regression), and the CLT from last lesson explains why the Gaussian assertion is so often defensible — noise that is a sum of many small effects is Gaussian-ish by theorem.

What MLE promises, and what it doesn’t

The classical guarantees, stated at working precision: under regularity conditions, the MLE is consistent ($\hat\theta \to \theta^\ast$ as $n \to \infty$), asymptotically normal ($\sqrt n(\hat\theta - \theta^\ast) \to \mathcal N(0, I(\theta^\ast)^{-1})$ with $I$ the Fisher information — the peak curvature from the widget), and asymptotically efficient (no consistent estimator does better, by the Cramér–Rao bound). That trio is why MLE is the default estimator of working statistics.

The fine print is equally load-bearing:

• Small-n MLE overfits. The likelihood rewards fitting the data you have; with little data that means fitting its accidents. Three flips, three heads → $\hat p = 1$: a confident absurdity. The fixes are this curriculum’s two favourite medicines, now unified: regularization is literally MLE plus a penalty, and lesson 4 shows the penalty is a prior in disguise.
• MLE is optimization, with everything that entails. Bernoulli and Gaussian gave closed forms; logistic regression needed gradient descent; deep networks give non-convex likelihoods where “the” MLE is a fiction and you get whatever the optimizer finds — the Non-convex Landscapes lesson, now with a statistical reading.
• The model can be wrong. MLE finds the best parameters within the family you wrote down (formally: it converges to the KL-closest member). Gaussian MLE on heavy-tailed data confidently estimates a mean the outliers own — the robust losses exist precisely because the noise assertion behind squared error is sometimes false.

One more unification, because it completes the picture: maximizing likelihood is minimizing $-\frac1n \ell$, whose population limit is the cross-entropy between the true distribution and the model — i.e., MLE minimizes KL divergence to the truth. That information-theoretic phrasing is the one the LLM track will use, and it is also this module’s quiet claim to the throne: every “loss curve” any student of this platform has ever watched descend was a KL divergence being ground down.

import numpy as np
from scipy.optimize import minimize_scalar
rng = np.random.default_rng(3)

x = rng.binomial(1, 0.7, size=40)                    # hidden truth p = 0.7
nll = lambda p: -(x.sum() * np.log(p) + (len(x) - x.sum()) * np.log(1 - p))
res = minimize_scalar(nll, bounds=(1e-6, 1 - 1e-6), method="bounded")
print(res.x, x.mean())                               # optimizer meets closed form

Exercises