You already know why subword tokenization exists: it strikes a balance between word-level tokenization (huge vocabularies, no way to handle unseen words) and character-level tokenization (tiny vocabularies, but very long sequences that make it hard for a model to represent whole-word meaning). What's less often explained is that "subword tokenization" isn't one algorithm — it's a family of three genuinely different algorithms, each with a different mathematical criterion for deciding which subword units belong in the vocabulary: Byte Pair Encoding (BPE), WordPiece, and Unigram. They produce different vocabularies and different tokenizations of the same text, and the choice matters for model behavior.

Byte Pair Encoding (BPE): greedy frequency merging

BPE, adapted from a 1994 data compression algorithm, builds its vocabulary bottom-up. Start with every individual character as a token. Then, repeatedly: count every adjacent pair of tokens across the entire training corpus, find the single most frequent pair, and merge it into one new token. Repeat this merging process a fixed number of times (determined by your target vocabulary size). The result is a vocabulary of characters plus the most frequent character sequences, built up through greedy, repeated merging.

BPE is simple, fast, and deterministic once trained. It's used by GPT-2, GPT-3, GPT-4, and RoBERTa (often in a byte-level variant, covered in the next lesson). Its main limitation is that it's purely frequency-driven — it has no explicit probabilistic model of what makes a good tokenization, it just greedily merges whatever pair happens to be most frequent at each step.

python

WordPiece: likelihood-driven merging

WordPiece, developed at Google and used by BERT, looks similar to BPE on the surface — it also starts from characters and merges pairs iteratively — but it uses a different merge criterion. Instead of merging the most *frequent* pair, WordPiece merges the pair that most increases the likelihood of the training data under a unigram language model built from the current vocabulary. In practice this is well-approximated by merging the pair whose joint frequency, divided by the product of each token's individual frequency, is highest — favoring pairs that co-occur more than you'd expect by chance, rather than pairs that are simply common on their own.

This means WordPiece can prefer a less-frequent-but-more-informative pair over a more-frequent-but-less-informative one, if the informative pair contributes more to how well the resulting vocabulary models the training data. WordPiece also conventionally marks non-initial subword pieces with a ## prefix (e.g., ##ing), making it visually obvious which tokens start a new word versus continue one.

Unigram: prune from a large vocabulary down

Unigram tokenization, used by SentencePiece (and models like T5, ALBERT, and XLNet), works in the opposite direction from BPE and WordPiece. Instead of starting small and merging upward, it starts with a very large candidate vocabulary (all substrings that occur frequently enough) and iteratively removes tokens. At each step, it computes how much removing each candidate token would hurt the overall likelihood of the training corpus under a unigram probability model, and prunes away the tokens whose removal hurts the least — repeating until the vocabulary reaches the target size.

A distinctive feature of Unigram tokenization is that it's inherently probabilistic: because it maintains a probability for every subword in the vocabulary, a given piece of text can be segmented multiple valid ways, and the tokenizer picks the segmentation with the highest overall probability (via the Viterbi algorithm) — or can even sample among several valid segmentations, which has been used as a data augmentation technique during training.

Note

BPE and WordPiece are both bottom-up (start from characters, merge upward to a target vocabulary size), differing only in their merge criterion — frequency vs. likelihood gain. Unigram is top-down (start from a huge vocabulary, prune downward). This directional difference is why Unigram naturally supports multiple valid segmentations of the same text, while BPE and WordPiece produce one deterministic tokenization per input once trained.

python

BPE vs. WordPiece vs. Unigram tokenization

Property	BPE	WordPiece	Unigram
Construction direction	Bottom-up: merge characters upward	Bottom-up: merge characters upward	Top-down: prune a large vocabulary downward
Merge/prune criterion	Most frequent adjacent pair	Pair that most increases data likelihood	Removes tokens that least hurt data likelihood
Segmentation of a word	Single deterministic result	Single deterministic result	Probabilistic; multiple valid segmentations possible
Marks word-continuation	Implicit via merge rules	Explicit `##` prefix on continuations	Explicit `▁` marks word starts (SentencePiece)
Used by	GPT-2/3/4, RoBERTa	BERT, DistilBERT	T5, ALBERT, XLNet (via SentencePiece)
Example: "unhappiness"	["un", "happiness"]	["un", "##hap", "##pi", "##ness"]	["▁un", "happ", "iness"]

Note

You cannot swap a model's tokenizer after the fact — the model's embedding layer and every learned weight are tied to the exact vocabulary it was trained with. Choosing BPE vs. WordPiece vs. Unigram is a decision made once, before pre-training even starts, and it has lasting downstream effects on vocabulary size, how gracefully the model handles rare words and multilingual text, and even token-level metrics like perplexity, which — as the previous lesson noted — aren't comparable across tokenizers.

What's next

All three algorithms described here operate on characters, which already assumes some way of handling the characters themselves — an assumption that breaks down for languages with huge character sets, emoji, and arbitrary Unicode text. The next lesson covers byte-level tokenization, which sidesteps the character-vocabulary problem entirely by operating on raw bytes.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →