Word Embeddings

The problem with words as symbols

Computers work with numbers. Words are symbols. Before you can do anything useful with language in a neural network, you need to convert words into numbers.

The naive approach: assign each word an integer. cat = 1, dog = 2, car = 3. But this is terrible — it implies dog is twice cat, and car is the sum of cat and dog. Arbitrary numbers encode no meaning.

The better approach: represent each word as a vector of real numbers — a point in a high-dimensional space. Words with similar meanings end up near each other in that space. This is an embedding.

What Word2Vec discovered

Word2Vec (Google, 2013) was the first embedding model to demonstrate that vector arithmetic encodes semantic meaning.

The famous example:
vector("king") − vector("man") + vector("woman") ≈ vector("queen")

This works because the model learned that the relationship between "king" and "man" is the same as the relationship between "queen" and "woman" — royalty minus gender.

How it learns this: Word2Vec trains on a simple task — predict a word from its surrounding context (or predict context from a word). To succeed at this prediction task, the model has to encode meaning into the vectors. The embeddings are a side effect of learning to predict.

Embedding dimensions

A word embedding is just a list of numbers. The length of that list is the embedding dimension.

Common choices:
- Word2Vec: 100–300 dimensions
- GloVe: 50–300 dimensions
- Modern LLMs: 768 to 4096+ dimensions per token

Higher dimensions = more capacity to encode nuance, but more compute and memory.

You can visualize embeddings by reducing to 2D using t-SNE or PCA. When you do, words cluster by meaning: animals together, verbs of motion together, countries together.

From words to tokens to subwords

Modern LLMs don't embed whole words. They embed subword tokens using algorithms like BPE (Byte Pair Encoding).

unhappiness might become: ["un", "happi", "ness"]

Why? Rare words (brand names, technical terms, misspellings) would be out-of-vocabulary in a word-level system. Subword tokenization handles any word by decomposing it into known pieces.

GPT-4 uses ~100,000 tokens. Each token gets its own embedding vector in a learned embedding table.

Why embeddings matter for RAG and semantic search

In RAG pipelines, you embed your documents and queries into the same vector space. Then you find the documents closest to the query — not by keyword matching, but by semantic similarity.

"How do I cancel my subscription?" and "steps to end my account" have completely different words but nearly identical embeddings. Keyword search fails. Semantic search succeeds.

This is why choosing the right embedding model matters. OpenAI's text-embedding-3-large, Cohere's embed-v3, and open-source models like BGE or E5 each have different strengths in different domains.

Embeddings are frozen knowledge

Pre-trained embeddings encode what the training corpus said about word relationships. If your domain uses specialized language (medical, legal, financial), fine-tune your embedding model on domain data or use a model already trained on that domain. General-purpose embeddings underperform on specialized text.