Decoder-Only vs Encoder-Decoder Architectures
Two ways to arrange the same building blocks
The original Transformer (Vaswani et al., 2017) was an encoder-decoder model built for machine translation: an encoder reads the full source sentence bidirectionally, and a decoder generates the target sentence one token at a time, attending back to the encoder's output. Since then, the field has split into two dominant architectural families that reuse the same self-attention block but wire it together differently: decoder-only models (GPT, Llama, Mistral) and encoder-decoder models (T5, BART, the original Transformer). The choice has deep consequences for how a model is trained, what it's good at, and how it scales.
Decoder-only: one stack, causal masking
A decoder-only model has a single stack of transformer layers. Every token can only attend to itself and the tokens before it — enforced by a causal (triangular) attention mask. Training is simple: next-token prediction on raw text, with no need for paired input/output data. This uniformity is a huge practical advantage — you can pretrain on virtually unlimited raw text, and the same architecture handles translation, summarization, coding, and chat by just changing the prompt. Essentially all modern frontier LLMs (GPT-4 class, Claude, Llama, Gemini) are decoder-only.
Encoder-decoder: separate stacks, bidirectional input
An encoder-decoder model splits the work into two stacks. The encoder processes the entire input with bidirectional self-attention — every token sees every other token, including ones after it — producing a rich contextual representation. The decoder then generates output tokens autoregressively, using causal self-attention over what it's generated so far, plus cross-attention into the encoder's output. This structure is a natural fit for tasks with a clear input/output split (translate French → English, summarize article → summary) because the encoder isn't forced to process input causally.
Decoder-only vs encoder-decoder
| Property | Decoder-only | Encoder-decoder |
|---|---|---|
| Attention over input | Causal (left-to-right only) | Bidirectional in encoder |
| Parameter count for same depth | Lower (one stack) | Higher (two stacks + cross-attention) |
| Pretraining data | Raw unlabeled text | Often benefits from paired/structured data |
| Best fit | Open-ended generation, chat, few-shot | Fixed input→output tasks (translation, summarization) |
| Inference pattern | Single growing sequence + KV cache | Encode once, then decode with cross-attention cache |
Why decoder-only won the scaling race
import numpy as np
def causal_mask(seq_len):
"""Decoder-only: token i can only see tokens <= i."""
mask = np.triu(np.ones((seq_len, seq_len)), k=1)
return np.where(mask == 1, -np.inf, 0.0)
def bidirectional_mask(seq_len):
"""Encoder: every token sees every other token."""
return np.zeros((seq_len, seq_len))
seq_len = 4
print("Causal mask (decoder-only):\n", causal_mask(seq_len))
print("Bidirectional mask (encoder):\n", bidirectional_mask(seq_len))
# Adding these masks to raw attention scores before softmax
# is literally the only structural difference between the two.When encoder-decoder still makes sense
Encoder-decoder architectures remain competitive when the task has a genuinely fixed input/output shape and you want maximal quality per parameter — the bidirectional encoder can extract more from the input without wasting capacity generating it. They're common in production translation systems, some retrieval-augmented setups (where a long document is encoded once and reused across many decode steps), and specialized summarization pipelines. They also tend to be more parameter-efficient for pure encoding tasks, which is why BERT-style encoder-only models (a third family, used for classification/embeddings rather than generation) still dominate non-generative NLP.
What's Next?
Both architectures depend entirely on how attention is computed and masked — the next layer of understanding comes from looking inside that attention mechanism itself: how positions are encoded, how queries and keys interact, and how that computation is made efficient enough to run at scale.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.