Positional Encoding in Transformers
Attention has no sense of order
Self-attention computes a weighted sum over all tokens in a sequence based on content similarity between queries and keys — but that computation is permutation-invariant. If you shuffled the words in a sentence before feeding them to raw self-attention, the output for 'the cat sat on the mat' would be identical to 'mat the on sat cat the', just rearranged. That's a disaster for language, where order carries most of the meaning. Positional encoding is the mechanism that injects information about token position into the model so attention can learn to use it.
Sinusoidal positional encoding (the original approach)
The original Transformer paper added a fixed, non-learned vector to each token embedding before the first layer, built from sine and cosine functions at different frequencies: $PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})$ and $PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d})$, where $pos$ is the token's position and $i$ indexes the embedding dimension. Lower dimensions oscillate fast (encoding fine-grained position), higher dimensions oscillate slowly (encoding coarse position) — similar to how a clock's second, minute, and hour hands together encode time at multiple resolutions. A key motivation was that this scheme lets the model generalize to sequence lengths longer than anything seen in training, because the function is defined for any position.
Learned absolute positional embeddings
An alternative, used in BERT and GPT-2, is simply a learned embedding table indexed by position — position 0 gets its own trainable vector, position 1 gets another, and so on, added to the token embedding just like sinusoidal encoding. This is simpler to implement and lets the model learn whatever positional structure is useful from data, but it cannot extrapolate past the maximum sequence length seen during training (there's no vector for position 2049 if the table only has 2048 rows).
Absolute position is the wrong abstraction
import numpy as np
def sinusoidal_positional_encoding(seq_len, d_model):
position = np.arange(seq_len)[:, None] # (seq_len, 1)
dim = np.arange(d_model)[None, :] # (1, d_model)
angle_rates = 1 / np.power(10000, (2 * (dim // 2)) / d_model)
angles = position * angle_rates
pe = np.zeros((seq_len, d_model))
pe[:, 0::2] = np.sin(angles[:, 0::2])
pe[:, 1::2] = np.cos(angles[:, 1::2])
return pe
pe = sinusoidal_positional_encoding(seq_len=6, d_model=8)
print(pe.round(3))
# Each row is the positional vector ADDED to that token's embedding.Positional encoding strategies compared
| Scheme | Learned or fixed? | Extrapolates beyond training length? | Used in |
|---|---|---|---|
| Sinusoidal absolute | Fixed function | Yes, in principle | Original Transformer |
| Learned absolute | Learned table | No | BERT, GPT-2 |
| Relative position bias | Learned, added to attention scores | Partially | T5, ALiBi |
| Rotary (RoPE) | Fixed rotation applied to Q/K | Good with scaling tricks | Llama, GPT-NeoX, most modern LLMs |
Why this choice shapes everything downstream
The positional scheme a model is trained with directly determines its context-length behavior: whether it can be extended beyond its original training length, how gracefully it degrades when you do, and how much engineering effort (position interpolation, NTK-aware scaling, YaRN) is required to stretch it further. This is why almost every major open-weight LLM released since 2023 has standardized on rotary embeddings — they turned out to offer the best combination of relative-position awareness and extensibility.
What's Next?
Understanding why relative position matters sets up the specific mechanism — rotating query and key vectors by an angle proportional to position — that made RoPE the default choice across the current generation of large language models.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.