Bayesian inference

Maximum likelihood answers “which parameters make the data least surprising?” and gave a confident absurdity on three heads in three flips. The Bayesian move is to add the one ingredient MLE lacks: what you believed before the data arrived. This lesson builds the machinery — prior, likelihood, posterior — on the cleanest possible example, watches it learn live, and then delivers the payoff this curriculum has been promising since the Regularization lesson: the penalties you have been adding to losses are priors, term for term.

Bayes’ theorem, promoted from formula to learning rule

As a formula it is one line of conditional-probability algebra from lesson 1:

Read as a learning rule it says: beliefs are distributions over parameters, and data updates them by reweighting — multiply prior belief by how well each $\theta$ explains the observations, renormalize. Three consequences deserve immediate notice. The output is a whole posterior distribution, not a point — uncertainty comes built in rather than bolted on. The update is sequential: today’s posterior is tomorrow’s prior, and processing data one batch at a time provably gives the same answer as all at once. And the denominator $p(\mathcal D)$ — the evidence — is a constant in $\theta$ you can usually ignore, until model comparison, where it becomes the star (it automatically penalizes complexity, the “Bayesian Occam’s razor”).

The cleanest example in statistics: Beta–Bernoulli

Estimate a coin’s bias $p$ from flips. Take the prior $p \sim \mathrm{Beta}(\alpha, \beta)$, with density $\propto p^{\alpha-1}(1-p)^{\beta-1}$. Multiply by the Bernoulli likelihood $p^{s}(1-p)^{f}$ for $s$ heads and $f$ tails, and the posterior is

— the same family back, with the data simply added to the parameters. This is conjugacy: prior and likelihood matched so the update is closed-form bookkeeping, and it hands you the exact reading of the prior: $\mathrm{Beta}(\alpha, \beta)$ is $\alpha{+}\beta{-}2$ imaginary flips you saw before the real ones. The posterior mean makes the compromise explicit:

— a weighted average that starts at the prior and slides to the MLE as real flips outnumber imaginary ones. Watch that slide happen:

The reveal: regularization is a prior

Now the payoff. Suppose instead of the full posterior you just want its peak — the maximum a posteriori (MAP) estimate. Take logs of Bayes’ rule:

The log-prior is an additive term in the objective — a regularizer. Make it concrete with a Gaussian prior $\theta \sim \mathcal N(0, \tau^2 I)$: $\log p(\theta) = -\frac{1}{2\tau^2}\lVert\theta\rVert^2 + \text{const}$, so MAP estimation with Gaussian noise and a Gaussian prior is

— ridge regression, exactly, with $\lambda = \sigma^2/\tau^2$. A Laplace prior yields the $\ell_1$ penalty — lasso — whose sharp peak at zero is why it zeroes coefficients out. The whole regularization toolkit is unmasked: weight decay asserts “parameters are probably small,” sparsity penalties assert “most parameters are probably zero,” and choosing $\lambda$ is choosing how strongly you believe it. Even the SVD lesson’s ridge-shrinkage formula $\sigma_i^2/(\sigma_i^2 + \lambda)$ is now a statement about posterior means under that Gaussian prior. This is the curriculum’s tightest unification so far: MLE, MAP and regularized loss minimization are one family, differing only in what they assume about $\theta$ before seeing data.

Credible vs confident, and what full posteriors buy

Two honesty notes, both load-bearing. First, vocabulary: the widget’s band is a credible interval — “given prior and data, $p$ lies here with probability 0.95,” the statement civilians think confidence intervals make (next lesson draws the frequentist contrast precisely). Second, cost: the Beta posterior was free only because of conjugacy. For neural networks the posterior is a billion-dimensional object nobody can normalize — hence the working hierarchy: MAP (cheap: just regularized optimization — what almost all deep learning does), MCMC (sampling from the posterior; gold standard, expensive), variational inference (fit a tractable approximation by optimization — the idea inside VAEs, ahead in the Deep Learning track). When does paying for a full posterior matter? Whenever the tails drive the decision: medical calls, pricing risk, exploration bonuses in RL (Thompson sampling literally acts by sampling this lesson’s posteriors — the bandit connection returns in the RL track), and any setting where “how sure are we?” changes what you do next.

import numpy as np
from scipy import stats

rng = np.random.default_rng(7)
flips = rng.binomial(1, 0.75, size=100)          # hidden truth 0.75

a, b = 2.0, 2.0                                   # Beta(2,2) prior
for i, x in enumerate(flips, 1):
    a, b = a + x, b + (1 - x)                     # one-line Bayesian update
post = stats.beta(a, b)
print(post.mean(), post.interval(0.95))           # ≈ 0.73, tight around truth
# sequential == batch: same (a, b) as a + sum(flips), b + n - sum(flips)

Exercises