Embedding Space Geometry
An embedding is a point, but the space it lives in has structure
An embedding maps a discrete or high-dimensional object — a word, an image, a user — to a dense vector in $\mathbb{R}^d$. The useful property isn't the individual vector; it's that the *geometry* of the resulting space — distances, angles, directions — ends up encoding something semantically meaningful, even though nothing in the training objective explicitly specified 'put semantically similar things near each other.' That structure emerges from training, and understanding why it emerges, and what notion of 'distance' actually governs it, is key to using embeddings correctly.
Cosine similarity vs. Euclidean distance
Two different notions of 'closeness' are common and are not interchangeable. Euclidean distance $\|a - b\|_2$ measures straight-line distance and is sensitive to vector magnitude. Cosine similarity $\frac{a \cdot b}{\|a\|\|b\|}$ measures only the angle between vectors, ignoring magnitude entirely. Most modern embedding models (word2vec, sentence transformers, CLIP) are trained with objectives — like dot-product-based contrastive losses — where direction carries the semantic signal and magnitude is comparatively arbitrary or correlated with unrelated factors like token frequency. For these, cosine similarity is almost always the right metric; applying raw Euclidean distance to unnormalized embeddings can give misleading nearest-neighbor results, because a vector with large magnitude can appear 'close' to many things in Euclidean terms simply by virtue of its length.
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# Classic word2vec analogy: king - man + woman ≈ queen
# This works because the model learned a roughly linear "gender" direction
# that composes additively across unrelated semantic axes.
def analogy(embeddings, a, b, c, top_k=5):
target = embeddings[a] - embeddings[b] + embeddings[c]
target = target / np.linalg.norm(target)
sims = {
word: cosine_similarity(target, vec)
for word, vec in embeddings.items()
if word not in (a, b, c)
}
return sorted(sims.items(), key=lambda kv: kv[1], reverse=True)[:top_k]Linear substructure: why vector arithmetic works at all
The famous word2vec result — $\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$ — is a striking empirical finding that embedding spaces trained with certain objectives (skip-gram with negative sampling, in word2vec's case) develop approximately linear substructure: semantic relationships like gender, tense, or capital-city-of correspond to roughly consistent *directions* (not just regions) in the embedding space, and those directions compose additively across unrelated attributes. This isn't guaranteed by every embedding method — it's an emergent property tied to how word2vec's training objective relates to pointwise mutual information between co-occurring words (Levy & Goldberg, 2014), and it's noticeably weaker or absent in some other embedding schemes.
Isotropy and the embedding space degeneration problem
| Metric | Sensitive to magnitude? | Common use case |
|---|---|---|
| Cosine similarity | No | Semantic search, word/sentence embeddings |
| Euclidean (L2) distance | Yes | k-NN with normalized features, clustering |
| Dot product | Yes | Retrieval models trained directly on dot-product loss |
| Manhattan (L1) distance | Yes | Sparse or high-dimensional feature spaces |
High dimensionality distorts intuitive distance
What's Next?
This geometric view connects directly to the manifold hypothesis — the idea that real-world high-dimensional data actually lies on or near a much lower-dimensional surface, which is what allows an embedding space to compress meaning into comparatively few dimensions in the first place.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.