The Pre-Transformer Embedding Era: ELMo, ULMFit & BERT

Before contextual embeddings: one vector per word

Through the early-to-mid 2010s, the standard way to represent words numerically was static word embeddings — Word2Vec and GloVe. Each word got exactly one vector, learned from co-occurrence statistics across a large corpus, regardless of context. The word "bank" had the same vector whether it appeared in "river bank" or "savings bank." These embeddings were a real advance over one-hot encodings, but the single-vector-per-word limitation meant downstream models still had to do all the work of disambiguating meaning from context themselves, usually with a task-specific architecture trained from scratch on relatively small labeled datasets.

ELMo: embeddings that depend on context

ELMo (Embeddings from Language Models, 2018) broke the one-vector-per-word assumption. Instead of a lookup table, ELMo used a deep bidirectional LSTM trained as a language model — predicting the next word from the left context and the previous word from the right context — and then used the LSTM's internal hidden states as the word representation.

Because those hidden states depend on the entire sentence the word appears in, "bank" in "river bank" and "bank" in "savings bank" now received genuinely different vectors. ELMo representations were a weighted combination of hidden states from multiple LSTM layers, which let downstream models draw on both lower-level (syntactic) and higher-level (semantic) information. ELMo embeddings were typically plugged into existing task-specific architectures as a drop-in replacement for static embeddings, giving consistent improvements across many NLP benchmarks without redesigning those architectures.

ULMFiT: transfer learning as a recipe, not just a representation

ULMFiT (Universal Language Model Fine-tuning, 2018) made a different contribution: it wasn't primarily about a new representation, but about formalizing a transfer learning *recipe* for NLP that mirrored what had already worked wonders in computer vision (pretrain a CNN on ImageNet, then fine-tune on a smaller task-specific dataset).

ULMFiT's recipe had three stages:

1. General-domain pretraining — train a language model (an LSTM, in this case) on a large, generic text corpus.
2. Target-task fine-tuning — continue training that same language model on text from the target domain, using techniques like discriminative learning rates (different layers fine-tune at different speeds) and slanted triangular learning rates (a warmup-then-decay schedule) to avoid catastrophically forgetting the pretrained knowledge.
3. Classifier fine-tuning — attach a small classification head and fine-tune the whole stack on the labeled task, gradually unfreezing layers from the top down.

This staged, careful fine-tuning approach proved that a single pretrained language model could be adapted to many different downstream tasks with only a small amount of labeled data per task — a dramatic efficiency gain over training from scratch.

BERT: transformers plus bidirectionality plus masking

BERT (Bidirectional Encoder Representations from Transformers, 2018) arrived the same year as ELMo and ULMFiT and combined their best ideas with the transformer architecture. Three choices defined it:

Transformer encoder, not LSTM. BERT replaced ELMo and ULMFiT's recurrent backbones with the transformer encoder, giving it the parallelism and long-range attention advantages covered in the transformers lesson.

Deep bidirectionality via masked language modeling. A standard left-to-right language model can't be bidirectional (it would be cheating to see the answer). ELMo faked bidirectionality by concatenating a separate left-to-right and right-to-left LSTM. BERT instead trained with a masked language modeling objective: randomly mask ~15% of input tokens and train the model to predict them using context from *both* sides simultaneously, in a single unified network.

Next sentence prediction. BERT was also trained to predict whether two sentences were originally consecutive, giving it a notion of sentence-level coherence useful for tasks like question answering.

BERT's pretrain-then-fine-tune workflow (directly inherited from ULMFiT) combined with transformer-quality representations (inherited from the attention literature) made it the dominant NLU model for several years, topping leaderboards across question answering, natural language inference, and classification benchmarks.

ELMo vs ULMFiT vs BERT

What's next

ELMo, ULMFiT, and BERT together moved NLP from static word vectors to the transfer-learning paradigm that every modern LLM uses. From this encoder-focused history, the next natural step is to look at the decoder side and the broader principle behind it — self-supervised learning, the idea that a model can learn rich representations from unlabeled data alone.