Byte-Level Tokenization: Solving the Unknown-Token Problem

The problem every character-level tokenizer eventually hits

Standard BPE, WordPiece, and Unigram tokenizers (as covered in the previous lesson) all operate on characters: their base vocabulary is built from whatever distinct characters appear in the training corpus. This works fine for the languages and scripts the training data was rich in, but it creates a real problem the moment the tokenizer encounters something it has never seen — a rare Unicode character, an emoji, a character from an underrepresented script, or a byte sequence from binary-adjacent text. Character-level tokenizers handle this with an unknown token, usually written [UNK] or <unk>: any character outside the trained vocabulary gets mapped to this single catch-all token, and the model loses all information about what that character actually was.

This is a worse problem than it might sound. If a document is written in a script barely represented in training data, large fractions of it can collapse into [UNK] tokens, making the text nearly unprocessable by the model. It also means tokenizer vocabulary size has to grow to cover every script a model might ever see, which doesn't scale cleanly to the full space of Unicode (over 140,000 characters and counting).

The byte-level fix

Byte-level BPE, introduced with GPT-2 and used by most GPT-family models since, sidesteps the problem entirely with a simple trick: instead of running BPE over Unicode characters, run it over raw bytes. Any piece of text, in any language or script, can be encoded as a sequence of UTF-8 bytes, and there are only 256 possible byte values. This means the base vocabulary — before any merges are learned — is fixed at exactly 256 tokens, one per possible byte value, and every single piece of text in any language, emoji set, or script can be represented using only these 256 base tokens, with zero exceptions and zero unknown-token fallback ever needed.

BPE merges are then learned on top of this byte-level base vocabulary exactly as described in the previous lesson — frequent adjacent byte-pairs get merged into single tokens — so common ASCII text (most English words) ends up represented efficiently as whole-word or near-whole-word tokens, while rare scripts or symbols that didn't appear often enough to earn merges simply fall back to sequences of 2-4 raw byte tokens (since UTF-8 encodes most non-ASCII characters using multiple bytes) rather than to a single meaningless [UNK].

The real tradeoff: efficiency, not coverage

Byte-level tokenization doesn't come free. Because rare scripts and symbols fall back to multiple raw-byte tokens rather than one compact subword token, text in underrepresented languages ends up using noticeably more tokens than equivalent text in well-represented languages, for the same amount of actual content. This has real consequences: more tokens means a longer effective context window is consumed, slower generation (more tokens to produce), and — for API-billed models — higher cost per unit of content, all specifically for users writing in less-common scripts. This is a well-documented fairness and efficiency gap in widely-used tokenizers, not a hypothetical concern.

Word-level tokenization, by contrast, is maximally efficient for whatever language and vocabulary it was built around (one token per common word) but fails completely on any word outside that vocabulary. Character-level tokenization guarantees full coverage of a given character set and loses no efficiency advantage for different scripts relative to each other, but produces very long sequences, since each token carries very little information. Byte-level BPE tries to get the best of both — compact tokens for common text, full coverage for everything else — at the cost of uneven efficiency across languages.

Word-level vs. character-level vs. byte-level BPE

What's next

Tokenization determines what a model reads; what a model is trained to do with what it reads is a separate question. The next lesson shifts from text representation to model training: how instruction-tuning datasets like those behind FLAN and Alpaca-style fine-tuning are built, and why data quality has an outsized effect on instruction-following ability.