Anomaly detection

Fraud, sensor faults, intrusions, data-pipeline corruption: the highest-value unsupervised problem in industry is usually not “find the groups” but “find the things that belong to no group”. It is also where sloppy definitions cost real money, because anomalous is not a property of a point — it is a property of a point relative to a model of normal. Change the model, change the anomalies. This lesson builds the three workhorse models of normal, shows them disagreeing on the same dataset, and finishes with the part most treatments skip: thresholds, budgets, and base rates — the difference between a score and a system.

Model 1: normal = one Gaussian cloud (Mahalanobis)

Fit mean $\mu$ and covariance $\Sigma$; score a point by how many “shape-aware standard deviations” it sits from the center:

The $\Sigma^{-1}$ is the whole point. Plain Euclidean distance treats the data cloud as a sphere; Mahalanobis whitens it first, so a point can be close in raw distance yet wildly anomalous because it violates the correlation structure — a transaction whose amount is normal and whose hour is normal, but whose amount-at-that-hour is not. Under a Gaussian model, $d_M^2 \sim \chi^2_d$, which even hands you a p-value. The cost: one Gaussian is a strong bet, and $\Sigma$ itself is corruptible by the very outliers you seek (robust variants — MCD — exist for exactly that).

Model 2: normal = near a low-dimensional structure (reconstruction)

PCA (or an autoencoder, its nonlinear heir in the Deep Learning track) learns the subspace normal data lives near; the score is the reconstruction error $\lVert x - \hat x \rVert^2$ — the residual the Projections lesson taught you to read as signal. This catches a different species of weird: points that are off the manifold, even at modest distance from the mean. The two scores dissociate cleanly — far along the principal axis is unremarkable to reconstruction but may alarm Mahalanobis; far from the axis alarms reconstruction first:

Model 3: normal = hard to isolate (Isolation Forest)

Both models above are geometric. Isolation Forest is combinatorial: grow random trees that split on random features at random thresholds; anomalies — being few and different — get isolated into their own leaf in few splits, so short average path length = high anomaly score. Its virtues are practical: no distance metric (so mixed scales hurt less), near-linear cost, robust in moderate dimensions, and no distributional assumption at all. Its blind spots: axis-aligned splits (tilted structure is partly invisible — the exact thing Mahalanobis is best at), and local anomalies inside a dense region of another density. One-class SVM and kNN-distance methods (and DBSCAN’s noise output, from last lesson) round out the classical menu; deep variants replace the geometry with learned representations but keep these same score semantics.

From score to system: thresholds, budgets, base rates

Every method above outputs a ranking. Production needs a decision, and the decision math is where anomaly detection is won or lost:

• Threshold = quantile of the score on normal data, i.e. an alarm budget: “flag the top 0.5%”. Set it from the review capacity you actually have — the widget’s slider is precisely this dial.
• Base rates dominate. At 1-in-10,000 true anomalies, a detector with a 1% false-positive rate drowns each true catch in ~100 false alarms — the Probability module’s base-rate lesson, now with an on-call rotation attached. Precision at your operating point, not AUC alone, is the metric that predicts operator experience.
• Evaluation without labels is the chronic ailment: standard practice is a small labeled audit set, precision-at-k spot checks, and monitoring the score distribution for drift. When normal itself shifts (new product launch, season change), yesterday’s model of normal manufactures alarms — scheduled refits and drift alarms on the inputs are part of the system, not an optional extra.
• An anomaly is not an explanation. Operators act on “amount 40× this user’s median, at 3 a.m., new device” — per-feature contributions to the score (easy for Mahalanobis and reconstruction: read the residual vector) are worth more than two points of AUC.

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.covariance import EmpiricalCovariance

rng = np.random.default_rng(0)
X = rng.multivariate_normal([0, 0], [[2.0, 1.3], [1.3, 1.0]], size=2000)
outliers = np.array([[3.0, -2.0], [-3.0, 2.2], [4.0, 0.0]])   # break correlation
Xa = np.vstack([X, outliers])

m = EmpiricalCovariance().fit(X)
dm = m.mahalanobis(Xa)                      # χ²₂ under the Gaussian story
print(np.argsort(dm)[-3:])                  # the planted three, found

iso = IsolationForest(random_state=0).fit(X)
print(np.argsort(iso.score_samples(Xa))[:3])  # isolation's top-3 — compare!

Exercises