Calculus and Taylor approximation

Every module ahead performs the same sleight of hand, usually in a half-sentence: “expanding to first order…”, “to second order, the loss is locally quadratic…”. This lesson is that half-sentence given a full hearing — because gradient descent, Newton’s method, the √d in attention, the Hessian’s bowls and saddles, and every convergence proof in the track are corollaries of one idea: smooth functions are boring up close, and boring is computable.

The derivative: the best local lie

Forget “slope of the tangent” for a moment; take the definition this curriculum actually uses. The derivative at $a$ is the number $f'(a)$ that makes the linear approximation

as good as a linear function can be: the error vanishes faster than $h$ ($o(h)$; with a bounded second derivative — the usual case in this track — the lie shrinks quadratically while the step shrinks linearly). That “best local lie” framing is why the derivative rules the track: an optimizer never sees the true loss surface, only this lie at the current point — and steps as if the lie were true. Small steps, good lie, progress; big steps, stale lie, divergence — you have already felt this trade if you have touched the Gradient Descent lesson’s η slider, and now you know its name: the approximation error of a first-order Taylor expansion.

Three mechanical rules generate every derivative in this curriculum — product, quotient, and above all the chain rule, $(f \circ g)' = f'(g(x)) \cdot g'(x)$: the lie of a composition is the product of the lies. Backpropagation (Deep Learning module) is this sentence, industrialized across a billion parameters; the Matrix Calculus lesson upgrades it to vectors. What deserves emphasis here is the meaning: derivatives compose multiplicatively, which is why deep products of them vanish or explode (the RNN lesson’s theorem) and why “the Jacobian is the local linear map” (Linear Algebra module) makes every linear-algebra fact a statement about nonlinear systems too.

Taylor: lying to higher order

If one linear term makes a good local lie, more polynomial terms make a better one:

— each term corrects the previous lie with the next derivative’s information, and the remainder $R_n$ (Lagrange’s form, with $\xi$ somewhere in the gap) prices the residual dishonesty. Two truncations carry the entire track: first order ($f(a) + f'(a)h$) is what gradient descent believes; second order ($+\tfrac12 f''(a)h^2$) is what Newton’s method and every Hessian argument believes — the loss as a local bowl, curvature and all. Watch the whole machinery, including its famous failure mode:

The track’s expansions, audited in advance

The honest ledger of every Taylor moment you are about to meet, so none of them slips by as hand-waving:

• Gradient descent’s step rule (next lessons): expand to second order, note the first-order term shrinks the loss while the second-order term punishes big steps by curvature — the step-size bound $\eta < 2/L$ falls straight out of $R_1$.
• Newton’s method: trust the quadratic lie completely and jump to its minimum — brilliant where the lie is good, catastrophic where it isn’t (the same radius-of-convergence morality as the widget).
• The Hessian’s geometry (Matrix Calculus): the second-order term $\tfrac12 h^\top H h$ IS the bowls-saddles-ravines picture; the quadratic-form widget you may already have met is a Taylor term with the truncation taken seriously.
• Momentum, Adam, curvature arguments (Optimization module): every “the loss is locally an ill-conditioned quadratic” premise is a second-order expansion plus the hope that third-order terms stay small over a step.
• Softmax saturation, sigmoid gradients, log(1+x) tricks across the DL and LLM tracks: one-term expansions used exactly within their trusted windows — now you can check the window yourself.

One closing distinction worth carrying: this lesson’s guarantees are local (Taylor’s theorem says nothing beyond the radius) while the next lesson’s subject — convexity — is the rare global promise. The track’s arc is exactly that tension: local lies you can always compute, global structure you occasionally get for free, and algorithms that stitch thousands of local lies into a path that works anyway.

import numpy as np
from math import factorial

# Taylor for sin around 0, error vs order at x = 0.5 — the widget's readout
x, a = 0.5, 0.0
for n in [1, 3, 5, 7, 9]:
    P = sum(np.sin(a + k * np.pi / 2) / factorial(k) * (x - a) ** k
            for k in range(n + 1))
    print(n, abs(P - np.sin(x)))       # collapses ~2 orders of magnitude per step

# and the failure: ln(1+x) at x = 2.5, outside radius 1 from a = 0
x = 2.5
for n in [2, 6, 12, 20]:
    P = sum((-1) ** (k - 1) / k * x ** k for k in range(1, n + 1))
    print(n, abs(P - np.log(1 + x)))   # GROWS with n — more terms, more wrong

Exercises