Word2Vec and GloVe Embeddings
From one-hot vectors to dense meaning
Before embeddings, words were represented as one-hot vectors — sparse, high-dimensional, and carrying zero information about meaning. 'cat' and 'kitten' were as unrelated as 'cat' and 'bicycle' under this scheme, because every dimension was orthogonal to every other. Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) solved this by learning dense, low-dimensional vectors (typically 100–300 dimensions) where geometric proximity encodes semantic similarity. These were the first embeddings to make the famous king - man + woman ≈ queen analogy arithmetic actually work.
Word2Vec: predicting context from words (and back)
Word2Vec comes in two architectures. Skip-gram takes a center word and tries to predict its surrounding context words within a window; it works well with small datasets and rare words. CBOW (Continuous Bag of Words) does the reverse — it averages the context words' vectors to predict the center word — and trains faster on large corpora. Both are shallow, two-layer neural networks whose real product isn't the prediction task itself but the learned weight matrix: each row becomes a word's embedding. The objective, for skip-gram, is to maximize $\sum_{t} \sum_{-c \le j \le c, j \ne 0} \log P(w_{t+j} | w_t)$ over the whole corpus.
Negative sampling makes it tractable
GloVe: global co-occurrence statistics
GloVe takes a different route. Instead of predicting local context windows one at a time, it first builds a global word-word co-occurrence matrix $X$ over the entire corpus, where $X_{ij}$ counts how often word $j$ appears in the context of word $i$. It then factorizes this matrix so that the dot product of two word vectors approximates the log of their co-occurrence probability ratio: $w_i^T w_j + b_i + b_j \approx \log(X_{ij})$. This blends the strengths of count-based methods (like LSA) with the local-context sensitivity of prediction-based methods like Word2Vec.
Word2Vec vs GloVe
| Aspect | Word2Vec | GloVe |
|---|---|---|
| Training signal | Local context windows | Global co-occurrence counts |
| Objective | Predictive (skip-gram/CBOW) | Matrix factorization |
| Training speed | Fast, online, incremental | Requires full co-occurrence matrix upfront |
| Rare word handling | Weaker on infrequent words | Benefits from aggregated global counts |
| Typical use today | Superseded by contextual embeddings | Superseded by contextual embeddings |
import numpy as np
def softmax_similarity(target_vec, candidate_vecs):
"""Cosine similarity between a target embedding and candidates."""
target_norm = target_vec / np.linalg.norm(target_vec)
scores = []
for v in candidate_vecs:
v_norm = v / np.linalg.norm(v)
scores.append(float(np.dot(target_norm, v_norm)))
return scores
# Toy example: pretend these are trained 4-dim embeddings
embeddings = {
"king": np.array([0.8, 0.1, 0.9, 0.2]),
"queen": np.array([0.7, 0.6, 0.85, 0.3]),
"man": np.array([0.75, -0.3, 0.1, 0.1]),
"woman": np.array([0.65, 0.5, 0.15, 0.2]),
}
analogy_vec = embeddings["king"] - embeddings["man"] + embeddings["woman"]
sims = softmax_similarity(analogy_vec, [embeddings["queen"], embeddings["king"]])
print("Similarity to queen:", sims[0], " | to king:", sims[1])Limitations: one vector per word
The fundamental weakness of both methods is that every word gets exactly one fixed vector, regardless of context. 'Bank' gets the same embedding whether it means a riverbank or a financial institution — the vector ends up being some blurred average of all the senses seen during training. This limitation is precisely what contextual embeddings (ELMo, then BERT and GPT-style transformers) were built to fix, by computing a different vector for each occurrence of a word based on its surrounding sentence.
Still useful, rarely state-of-the-art
What's Next?
Static embeddings established the geometric intuition — that meaning lives in direction and distance in vector space — that every later embedding model still relies on. The natural next step is seeing how that same geometric idea gets reinvented inside attention mechanisms, where a word's vector is recomputed fresh for every sentence it appears in.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.