t-SNE Deep Dive

The visualization problem

Embeddings, image features, and gene expression profiles routinely live in hundreds or thousands of dimensions. Humans can look at two, maybe three. t-SNE (t-distributed Stochastic Neighbor Embedding), introduced by Laurens van der Maaten and Geoffrey Hinton in 2008, became the go-to method for compressing high-dimensional data down to 2D or 3D for visualization — specifically optimized to keep points that were close together in high-dimensional space close together in the plot.

How t-SNE works

t-SNE operates in two stages.

Stage 1 — Model neighbor probabilities in high-dimensional space. For every pair of points, compute a probability that one would pick the other as its neighbor, based on distance, using a Gaussian distribution centered on each point. A parameter called perplexity controls the effective number of neighbors considered — roughly, how many nearby points matter when deciding local structure.

Stage 2 — Find a low-dimensional layout matching those probabilities. Randomly place points in 2D, then iteratively move them so that the neighbor probabilities in the low-dimensional map match the high-dimensional probabilities as closely as possible, measured via KL divergence. Crucially, the low-dimensional similarities use a Student's t-distribution (hence the 't' in t-SNE) rather than a Gaussian — its heavier tails let moderately distant points spread out in the low-dimensional map rather than collapsing into a crowded blob, which directly solves what's called the 'crowding problem.'

Pitfalls of over-interpreting t-SNE plots

Cluster sizes don't mean anything. t-SNE tends to expand dense clusters and compress sparse ones to roughly similar visual sizes in the plot. A tight, small-looking cluster might represent a much larger or smaller region of the original high-dimensional space than a sprawling one next to it.

Distances between clusters are not meaningful. Two clusters sitting close together in the t-SNE plot are not necessarily more similar than two clusters sitting far apart — global arrangement is essentially arbitrary and can change between runs.

Results are not deterministic. Different random initializations can produce visibly different layouts of the same data. Always check whether a pattern replicates across multiple runs with different random seeds before trusting it.

Perplexity changes the picture a lot. Low perplexity reveals fine local structure but can fragment real clusters into noise; high perplexity merges structure into broad blobs. It's standard practice to run t-SNE at several perplexity values (commonly 5 to 50) rather than trusting a single plot.

Apparent clusters can be artifacts. t-SNE can manufacture visually convincing clusters out of structureless or purely random data, especially at low perplexity — visual separation in a t-SNE plot is not evidence of real structure on its own.

PCA vs. t-SNE vs. UMAP

What's next

t-SNE's main limitations — slow scaling and unreliable global structure — motivated a newer method built on different mathematical foundations. UMAP Deep Dive covers how UMAP preserves both local and global structure more faithfully than t-SNE, and why it has become the default choice for visualizing large embedding spaces.