Eigenvectors and eigenvalues

Watch any linear map act on the plane and most vectors get knocked off their direction. A few do not: they come out scaled, pointing exactly where they pointed before. Those stubborn directions — eigenvectors — are the transformation’s skeleton, and finding them turns an incomprehensible map into a diagonal one. This lesson earns three things: the decomposition itself, the humble iteration that computes it at web scale, and the spectral theorem — which, count them, already underwrites the condition number κ, the Hessian classification of critical points, ridge’s shrinkage picture, and the covariance geometry of PCA.

The definition, and how to find them by hand

A nonzero $v$ is an eigenvector of $A$ with eigenvalue $\lambda$ when

— the map’s entire action on that direction is a stretch by $\lambda$ (negative: flip; magnitude below one: contraction). Rearranged, $(A - \lambda I)v = 0$ demands a nonzero vector in the kernel of $A - \lambda I$, which exists exactly when that matrix is singular:

For a 2×2 this is a quadratic with a form worth memorizing, $\lambda^2 - \operatorname{tr}(A)\,\lambda + \det(A) = 0$, so $\operatorname{tr} A = \lambda_1 + \lambda_2$ and $\det A = \lambda_1 \lambda_2$ — the area factor from last lesson is the product of the stretches, as it must be. Beyond 3×3 nobody solves characteristic polynomials (they are numerically treacherous); iteration wins, as you are about to see.

If $A$ has $n$ independent eigenvectors, stack them as columns of $V$:

Read the decomposition as instructions: change into eigen-coordinates, stretch each axis independently, change back. The payoff line $A^k = V\Lambda^k V^{-1}$ turns repeated application — the expensive thing — into powering scalars, and it is the exact tool you used in Optimization to solve gradient descent’s dynamics ($x_{t+1} = (I - \eta H)x_t$ decoupling per eigendirection into factors $(1-\eta\lambda_i)^t$: that was this lesson, smuggled in early).

Power iteration: the web-ranking algorithm in four lines

Apply $A$ repeatedly to almost any starting vector and expand in the eigenbasis:

Every component decays relative to the dominant one at rate $\lvert \lambda_2/\lambda_1 \rvert^k$ — the direction of $A^k x$ converges to the top eigenvector, no polynomial-solving required. Normalize each step and you have power iteration, and you have also just derived PageRank: Google’s founding algorithm builds the web’s link matrix (column-stochastic, damped) and power-iterates it; the stationary importance of every page on the internet is the top eigenvector, and the damping factor 0.85 exists precisely to control the $\lambda_2/\lambda_1$ gap that sets convergence speed. The same iteration scores nodes in every network (eigenvector centrality), finds the top principal component when the data is too big to decompose, and — as the Deep RL chapter’s power method for values hints — is the quiet engine inside many “iterate to a fixed point” algorithms.

The spectral theorem: symmetry buys everything

For general matrices, eigenvectors may be non-orthogonal, complex, or missing. For the matrices ML actually optimizes over, nature is kind:

Spectral theorem. Every symmetric $A = A^\top$ has real eigenvalues and an orthonormal basis of eigenvectors: $A = Q \Lambda Q^\top$ with $Q$ orthogonal.

Symmetric maps are exactly rotate, stretch along perpendicular axes, rotate back — no shear, no complex behaviour, nothing hidden. And the matrices this curriculum lives on are symmetric by construction: $X^\top X$ (Supervised Learning’s normal equations), covariance matrices (PCA, next lessons), Hessians $\nabla^2 f$ (Optimization), kernel matrices (SVMs). For all of them the spectral theorem is why:

• κ was well-defined: the condition number is $\lambda_{\max}/\lambda_{\min}$ of a symmetric Hessian — perpendicular stretch factors of the loss’s local bowl.
• Saddles were classifiable: mixed Hessian eigenvalue signs = up in some perpendicular directions, down in others (the Non-convex landscapes lesson’s taxonomy is the spectral theorem plus adjectives).
• Ridge’s shrinkage was diagonal: $(X^\top X + \lambda I)^{-1}$ shares eigenvectors with $X^\top X$, shrinking each direction by $d_j^2/(d_j^2 + \lambda)$ — the Regularization lesson’s formula, now with its provenance attached.

The quadratic-form view makes the geometry tactile: $f(x) = x^\top A x$ has the eigenvectors as its principal axes and the eigenvalues as curvatures along them — which is precisely the surface classification you will drive by slider in lesson 5’s widget, and precisely what “positive definite” ($\lambda_i > 0$: a bowl) means.

import numpy as np

A = np.array([[1.5, 0.45], [0.35, 0.9]])
lam, V = np.linalg.eig(A)                 # general case

def power_iteration(A, k=50):
    v = np.random.randn(A.shape[0])
    for _ in range(k):
        v = A @ v
        v /= np.linalg.norm(v)            # the whole algorithm
    return v, v @ A @ v                   # direction, Rayleigh-quotient eigenvalue

S = A + A.T                               # symmetrize → spectral theorem applies
lam_s, Q = np.linalg.eigh(S)              # eigh: real λ, orthonormal Q, sorted
np.testing.assert_allclose(Q @ np.diag(lam_s) @ Q.T, S, atol=1e-12)

Exercises