t-SNE, UMAP and reading embedding plots

Every ML team’s slide deck contains one: a cloud of colored dots, “the embedding space”, clusters neatly separated, distances apparently meaningful. These plots are made by t-SNE or UMAP, and they are simultaneously the most useful exploratory tool in high-dimensional data analysis and the most systematically misread artifact in the field. This lesson derives what the algorithms optimize — because only the objective tells you what the picture can and cannot say — and then runs t-SNE live on data whose ground truth you know, so you can catch the distortions in the act.

Linear reduction, recapped in one paragraph

PCA (Foundations: Projections + SVD) finds the orthogonal directions of maximal variance — the best linear view, preserving large-scale geometry: distances, angles, densities are distorted only by dropping components. Its weakness is the mirror of its guarantee: a nonlinear manifold (a swiss roll, a cluster wrapped around another) has no good linear shadow. When PCA’s 2-D view explains little variance or shows structureless overlap, the nonlinear tools earn their turn — at the price of giving up exactly the geometric faithfulness PCA guaranteed.

t-SNE: matching neighbor distributions

t-SNE never tries to preserve distances. It converts them to neighbor probabilities and matches those. In the original space, point $j$‘s probability of being picked as $i$‘s neighbor uses a Gaussian kernel:

where each $\sigma_i$ is tuned per point so the conditional’s entropy hits a target — the perplexity, effectively “how many neighbors count”. In the 2-D map, similarities use a Student-t kernel $q_{ij} \propto (1 + \lVert y_i - y_j \rVert^2)^{-1}$ — heavy tails that let dissimilar points sit far apart without crushing gradients (the fix that made the method work). The map is found by gradient descent on

Read the objective the way the MLE lesson taught: KL punishes putting close points far apart (large $p$, small $q$) much harder than the reverse. So t-SNE preserves local neighborhoods and treats everything else — global distances, cluster sizes, densities — as negotiable. Every misreading of these plots is a failure to know that sentence.

UMAP, and how it differs

UMAP builds a weighted k-nearest-neighbor graph, interprets it through fuzzy simplicial sets, and optimizes a cross-entropy that — unlike t-SNE’s KL — has an attractive and repulsive term for far pairs too. Practical differences: UMAP is faster, scales better, has n_neighbors (≈ perplexity) and min_dist (visual packing) as its dials, and tends to preserve slightly more global structure. What it does not do is escape the fundamental trade: it is still a neighbor-graph method, cluster sizes and inter-cluster distances remain unreliable, and its apparent global structure is sensitive to initialization (as is t-SNE’s — modern practice initializes both with PCA for stability, which also makes runs reproducible).

The reader’s contract

What you may and may not conclude from a t-SNE/UMAP plot, condensed:

• May: points in the same island are likely neighbors in the original space; well-separated islands usually reflect real separation; local neighborhood browsing (“what sits next to what”) is the intended use.
• May not: compare island SIZES (equalization artifact), read meaning into DISTANCES between islands, interpret DENSITY differences, or treat axis directions as meaningful (there are none — the objective is invariant to rotation).
• Must: try several perplexities/n_neighbors before believing any structure; check suspected clusters with a method that outputs labels and uncertainty (GMM, HDBSCAN) in the ORIGINAL space; and remember that clustering on the 2-D embedding inherits its artifacts — cluster in more dimensions than you visualize.

One more honest note: distortion is not a fixable bug. A theorem-level fact (no embedding of most high-dimensional metrics into 2-D preserves all distances — Johnson–Lindenstrauss needs $O(\log n / \varepsilon^2)$ dimensions) means every 2-D picture of high-dimensional data lies about something. The only question is whether you know what yours is lying about.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

X, y = load_digits(return_X_y=True)          # 64-D, 10 true classes
Z = TSNE(perplexity=30, init="pca", random_state=0).fit_transform(X)
# rerun with perplexity 5 and 100; overlay y as colors; note what changes
# and what survives — survivors are the trustworthy structure
print(PCA(2).fit(X).explained_variance_ratio_.sum())   # linear view: ~29%

Exercises