Logistic regression and calibration

Classification’s workhorse is routinely introduced as “regression squashed through a sigmoid,” which gets the formula right and the ideas backwards. Logistic regression is a probability model fitted by maximum likelihood; the sigmoid, the cross-entropy loss, the gradient’s beautiful form, and the entire calibration story all fall out of that starting point rather than being bolted on.

The model: log-odds are linear

For binary $y \in \{0, 1\}$, model the conditional probability:

The honest reading inverts the sigmoid: the model asserts that the log-odds are linear,

which gives every coefficient a precise meaning — one unit of $x_j$ adds $\beta_j$ to the log-odds, i.e. multiplies the odds by $e^{\beta_j}$ — and makes the geometry immediate: $p(x) = \tfrac12$ exactly on the hyperplane $\beta^\top x = 0$. The decision boundary is linear, always; the sigmoid curves probabilities, never the boundary. Probability contours are parallel hyperplanes, packed tightly when $\lVert\beta\rVert$ is large (a confident, sharp model) and spread out when it is small.

The loss: maximum likelihood becomes cross-entropy

Each observation is a Bernoulli draw, so the likelihood of the data set is $\prod_i p_i^{y_i} (1 - p_i)^{1 - y_i}$. Maximizing its log = minimizing the negative log-likelihood:

— the cross-entropy (log) loss, not chosen for convenience but derived from the probability model. Differentiate (the sigmoid’s identity $\sigma' = \sigma(1-\sigma)$ collapses everything):

Identical in shape to least squares’ $X^\top(\hat y - y)$ — prediction error times features — except $p$ depends nonlinearly on $\beta$, so there is no closed form and we are back in the Optimization module: $\mathcal{L}$ is convex (its Hessian $X^\top W X$ with $W = \mathrm{diag}(p_i(1-p_i))$ is PSD), so gradient descent or Newton’s method (a.k.a. iteratively reweighted least squares) finds the global optimum. Two practical notes with mathematical roots: on linearly separable data the MLE does not exist — $\lVert\beta\rVert \to \infty$ as the loss chases certainty — so an $\ell_2$ penalty is not optional hygiene but what makes the optimum finite; and that same penalty is why library defaults (sklearn’s C) regularize whether you asked or not.

Why not just fit least squares on the labels? Squared loss on 0/1 targets penalizes confidently correct predictions ($p$ pushed past the label), and its linear-model version happily predicts probabilities outside $[0,1]$. Cross-entropy’s asymmetric punishment — a confidently wrong prediction costs $-\log(\text{tiny})$, i.e. an unbounded loss — is exactly the incentive structure a probability estimate should face. That property has a name worth knowing: log loss is a proper scoring rule, minimized in expectation only by the true conditional probability.

The threshold is not part of the model

The fitted model outputs $p(x)$. Turning that into a decision requires a threshold $t$ — predict 1 when $p(x) \ge t$ — and nothing in the fitting procedure chose $t = 0.5$. Decision theory does: predicting 1 is optimal when expected cost is lower,

so 0.5 is right only when false positives and false negatives cost the same — almost never true in fraud, medicine, or moderation. Geometrically, changing $t$ slides the decision hyperplane parallel to itself through the probability contours; it never rotates it. Separating “fit the best probability model” from “choose the operating point” is the clean mental model, and the evaluation lesson builds ROC analysis on exactly this separation.

Calibration: do the probabilities mean anything?

Accuracy asks “is $p \ge t$ on the right side?” Calibration asks something stronger: among all cases where the model said 70%, does the event happen 70% of the time? Formally, $\Pr(y=1 \mid p(x) = q) = q$. A model can be accurate yet badly calibrated (confidence systematically inflated), and calibrated yet weakly accurate (honest but vague). The standard diagnostic is the reliability diagram — bin predictions, plot observed frequency against mean predicted probability, hope for the diagonal — summarized by expected calibration error (ECE) or, better, by proper scoring rules: log loss and the Brier score $\tfrac1n\sum_i (p_i - y_i)^2$, both of which reward honesty about uncertainty, not just correct ordering.

Field notes worth carrying: logistic regression trained on log loss is typically well-calibrated in-distribution (it is optimizing a proper score); max-margin and ensemble methods — SVMs, boosted trees, bagged forests — typically push probability mass away from 0 and 1, an under-confident sigmoidal distortion at the extremes (Niculescu-Mizil & Caruana’s classic measurement), which is exactly why a sigmoid is the corrective: Platt scaling (fit a 1-D logistic regression on the model’s scores — yes, logistic regression as a patch for other models) or isotonic regression (fit a monotone step function; more flexible, needs more data). Modern deep networks are miscalibrated in the opposite direction — over-confident — and temperature scaling, Platt’s one-parameter cousin, softens them. And calibration is fragile under distribution shift and under resampling: if you balanced your classes for training, your probabilities are calibrated to the balanced world, not the real one — correct the intercept by the log of the sampling ratio, or recalibrate on unresampled data.

from sklearn.linear_model import LogisticRegression
from sklearn.calibration import calibration_curve

clf = LogisticRegression(C=1.0).fit(X_tr, y_tr)          # C = 1/λ
prob = clf.predict_proba(X_te)[:, 1]
frac_pos, mean_pred = calibration_curve(y_te, prob, n_bins=10)
# plot frac_pos vs mean_pred against the diagonal = reliability diagram

Exercises