Random variables and distributions

Machine learning is a probability discipline wearing an optimization costume. A model’s output is a conditional distribution; a loss is a negative log-likelihood in disguise (that reveal comes in lesson 3); every evaluation metric is an expectation estimated from samples. This module builds the probability a practitioner actually leans on daily — not measure theory, but fluency: the operators, the standard distributions, and above all conditioning, which is what the word “prediction” formally means.

Random variables, and the two operators that matter

A random variable $X$ assigns a number to each outcome of a random process; it is described by its distribution — a pmf $p(x)$ for discrete values, a pdf $f(x)$ for continuous ones (heights of a density, not probabilities themselves: only areas $\int_a^b f$ are probabilities — the classic beginner trap, since densities happily exceed 1). Two functionals summarize a distribution, and they behave like operators with algebra worth memorizing:

The rules — provable in two lines each, used ten thousand times each:

• Linearity of expectation: $\mathbb{E}[aX + bY] = a\,\mathbb{E}X + b\,\mathbb{E}Y$ — unconditionally, independence not required. The single most-used fact in this curriculum’s proofs (the unbiasedness of minibatch gradients was exactly this).
• Variance algebra: $\operatorname{Var}[aX] = a^2 \operatorname{Var}[X]$, and $\operatorname{Var}[X + Y] = \operatorname{Var}X + \operatorname{Var}Y$ only under independence (in general add $2\operatorname{Cov}(X,Y)$ — forgetting the covariance term is how the forest lesson’s $\rho\sigma^2$ floor surprises people).
• The consequence that runs statistics: an average of $n$ i.i.d. draws has $\mathbb{E}[\bar X] = \mu$ and $\operatorname{Var}[\bar X] = \sigma^2/n$ — the $1/\sqrt n$ standard-error law, formally born here and starring in the next lesson.

The zoo, and how to choose from it

Distributions are modeling assumptions with names. Choosing one is asserting a data-generating story, and each standard story earns its slot:

Joint, marginal, conditional: the grammar of “given”

Real problems involve several variables, and three constructions relate them. The joint $p(x, y)$ says everything; the marginal $p(x) = \sum_y p(x, y)$ integrates the rest away; the conditional re-normalizes a slice:

Conditioning is the central operation of machine learning — the whole discipline of supervised learning is the art of estimating $p(y \mid x)$: the logistic model was $\Pr(y{=}1 \mid x)$, a language model is $p(\text{next token} \mid \text{context})$, and “prediction” means computing a conditional. Two structural facts complete the grammar. Independence ($p(x,y) = p(x)p(y)$ — knowing one tells nothing about the other) is the assumption behind every “i.i.d.” in this curriculum, and its conditional cousin — $X \perp Y \mid Z$ — is subtler and stronger machinery: Naive Bayes assumes features independent given the class; Markov chains assume the future independent of the past given the present (the Markov property that carried the entire RL track). And the chain rule $p(x_1, \ldots, x_n) = \prod_i p(x_i \mid x_{<i})$ factorizes any joint into conditionals — the identity that makes autoregressive language models possible at all.

The inversion of conditioning — $p(x \mid y)$ from $p(y \mid x)$ — is Bayes’ theorem, which gets lesson 4 to itself. Here, just the shape of the trap it resolves: $\Pr(\text{disease} \mid \text{positive test})$ and $\Pr(\text{positive test} \mid \text{disease})$ differ by the base rate, and confusing them (the prosecutor’s fallacy) is possibly the most consequential statistical error civilians make.

Expectation as the universal interface

A closing reframe that pays rent across the curriculum: nearly every quantity ML optimizes or reports is an expectation —

— and none of them can be computed exactly, because the underlying distribution is the world. Everything is estimated by sample averages, which is why the next lesson — on what sample averages do and how fast — is arguably the load-bearing wall of the entire practice: it prices every training batch, every test set, and every Monte Carlo rollout you will ever run.

import numpy as np
rng = np.random.default_rng(0)

# linearity needs no independence; variance addition does
x = rng.normal(2.0, 1.0, 200_000)
y = 0.8 * x + rng.normal(0, 0.5, 200_000)          # correlated with x
print(np.mean(x + y), np.mean(x) + np.mean(y))     # equal (linearity)
print(np.var(x + y), np.var(x) + np.var(y))        # NOT equal (covariance ≠ 0)
print(np.var(x + y), np.var(x) + np.var(y) + 2 * np.cov(x, y)[0, 1])  # fixed

Exercises