Vectors and linear maps

Machine learning is applied linear algebra with a loss function attached. A dataset is a matrix; an embedding is a vector; a layer is a linear map plus a nonlinearity; “similarity” in every retrieval system on Earth is an inner product. This module exists so those aren’t metaphors, and it starts with the one mental upgrade that makes everything else in it easy: stop reading matrices as tables and start reading them as motions of space.

Vectors: data with geometry

A vector $x \in \mathbb{R}^d$ is a point and an arrow — a customer’s features, a word’s embedding, an image’s pixels — and geometry gives data its first useful vocabulary. The norm measures size, and two norms carry this curriculum:

— you met them as ridge and lasso penalties, where their shapes (sphere vs diamond) decided whether coefficients shrink or die. The inner product

is the single most-executed mathematical operation on the planet right now, because its geometric reading — how aligned are these two directions? — is what “similarity” means computationally. Cosine similarity (the inner product of normalized vectors) ranks documents against queries in every embedding-based retrieval system; attention scores in transformers are scaled inner products of query and key vectors; even the correlation coefficient of classical statistics is a cosine between centred data columns. When $x^\top y = 0$ the directions are orthogonal — carrying no linear information about each other — an idea lesson 4 will build the geometry of least squares and PCA on.

The reframe: a matrix is a transformation

Here is the module’s cornerstone. A matrix $A \in \mathbb{R}^{m \times n}$ is the function $x \mapsto Ax$ — and it is completely determined by where it sends the basis vectors, because linearity forces everything else:

The columns of $A$ are the images of the basis vectors, and every output is just a recombination of the columns. Read that sentence until it is obvious; it is the decoder ring for the whole module. Linearity itself is two visual promises — grid lines stay parallel and evenly spaced, the origin stays put — and everything a linear map can do (rotate, stretch, shear, reflect, project, collapse) is a combination of surprisingly few primitives:

Composition, and why “matrix multiplication” is what it is

Matrix multiplication is function composition: $(AB)x = A(Bx)$, first $B$ then $A$. Every mysterious-seeming rule falls out of that one fact: the row-by-column formula is just “track where the basis goes under both maps”; non-commutativity is obvious (rotate-then-stretch ≠ stretch-then-rotate — try it in the widget); $(AB)^{-1} = B^{-1}A^{-1}$ is “undo in reverse order,” like socks and shoes. And a deep network before its nonlinearities is exactly a composition $W_L \cdots W_2 W_1 x$ — which collapses to a single linear map, which is why networks need nonlinearities at all: without them, a hundred layers has exactly the expressive power of one.

Three matrix species recur so often in ML they deserve name-tags now:

The module’s arc, announced in advance: the next two lessons prove that symmetric matrices are secretly diagonal (spectral theorem), and that every matrix is diagonal between two rotations (SVD). Diagonal is the destination; decompositions are the route.

Rank: how much survives the map

The rank of $A$ is the dimension of its image — how many dimensions of information survive. Full rank preserves everything (invertible, det ≠ 0); rank deficiency crushes some directions to zero, exactly what the collapse preset shows. In ML, rank is a resource dial you have already used: LoRA (RL track) constrains a weight update to rank $r$, betting the change a task needs is low-dimensional; the SVD lesson makes that bet precise with the best-possible low-rank approximation theorem, and lesson 4’s projections describe what the lost directions were. And the Supervised Learning module’s collinearity troubles — exploding coefficient variance at $1/(1-\rho^2)$ — are simply $X$ approaching rank deficiency: the map almost crushes a direction, so inverting it almost divides by zero.

import numpy as np

A = np.array([[1.4, 0.3],       # columns = images of e1, e2 — same as the widget
              [0.5, 1.1]])
x = np.array([2.0, 1.0])
print(A @ x, np.linalg.det(A))          # transform a point; signed area factor
print(np.linalg.matrix_rank(A))         # dimensions surviving the map

Q, _ = np.linalg.qr(np.random.randn(4, 4))      # a random orthogonal matrix
np.testing.assert_allclose(Q.T @ Q, np.eye(4), atol=1e-12)  # rotations: QᵀQ = I

Exercises