Entropy and information

This curriculum has been paying Shannon royalties for twenty lessons without naming him: cross-entropy losses, perplexity as “effective branching factor”, KL in t-SNE and RLHF, “training a language model is compression”. This module collects the debt properly. The claim to be earned today: information is a measurable quantity, its unit is the bit, and probability distributions have an exact price tag — after which every loss function you have ever minimized will read differently.

Surprise, axiomatized

How surprising is an event of probability $p$? Demand three obvious things — certain events carry no surprise ($p=1 \Rightarrow 0$), rarer is more surprising (decreasing in $p$), and independent surprises add ($h(pq) = h(p) + h(q)$, since independent probabilities multiply) — and the logarithm is forced, uniquely:

A fair coin flip: 1 bit. A one-in-a-million event: ~20 bits. The entropy of a source is then just its average surprise:

— maximal ($\log_2 n$) when all $n$ outcomes are equally likely (you know nothing), zero when one outcome is certain (nothing to learn), and concave between. If expectation is the Probability module’s universal interface, entropy is its reading on the “how uncertain” dial.

The operational meaning: entropy is a price

What makes $H$ more than a formula is Shannon’s source coding theorem: entropy is the minimum average number of bits per symbol any code can achieve for the source — attainable (to within a rounding bit) by giving each symbol a codeword of length $\approx -\log_2 p_i$: short codes for common symbols, long for rare. Morse had the instinct (E is one dot); Shannon proved the limit. Feel the price move:

Cross-entropy: coding with the wrong beliefs

Now the bridge this curriculum crosses daily. Suppose reality emits symbols from $p$, but you built your code for $q$ — lengths $-\log_2 q_i$. Your average cost is the cross-entropy:

always at least the true entropy, with the excess — the price of wrong beliefs — getting its own lesson next. Read your training losses again with this decomposition in hand: minimizing cross-entropy loss is minimizing the coding cost of the data under your model’s probabilities. The MLE lesson proved cross-entropy = negative log-likelihood algebraically; Shannon supplies the meaning: a classifier or language model is literally a codebook, its loss the bits-per-symbol it would spend, and the gap above the data’s own entropy is exactly what remains learnable. Perplexity ($2^{H}$, the LLM track’s metric) is this same invoice quoted as a branching factor, and the irreducible term $E$ in the scaling-laws lesson is $H(p)$ itself — the entropy of human text, the part no model can ever compress away.

Two boundary stones before the next lesson. Differential entropy (continuous distributions) exists and appears in derivations, but is a subtler object — it can be negative and it shifts under changes of units, so treat “the entropy of a Gaussian” as relative, not absolute. Entropy measures the distribution, not the meaning: a shuffled deck and an encrypted secret have identical entropy readings; Shannon priced description, not importance — the same honest boundary the likelihood-truth gap drew in the LLM track.

import numpy as np

p = np.array([0.5, 0.25, 0.125, 0.125])
H = -(p * np.log2(p)).sum()
print(H)                              # 1.75 bits — vs 2.0 for the naive code

# the price of wrong beliefs: cross-entropy under a mistaken q
q = np.array([0.25, 0.25, 0.25, 0.25])
print(-(p * np.log2(q)).sum())        # 2.0 = H + KL: the wrong-code penalty

# entropy of English letters vs uniform — why text compresses ~50%
# (frequencies rounded from any corpus table)

Exercises