Rotary Position Embeddings (RoPE)
The core idea: rotate, don't add
Rotary Position Embeddings (RoPE), introduced by Su et al. in 2021 and now used in Llama, GPT-NeoX, PaLM, and most modern open-weight LLMs, take a fundamentally different approach from sinusoidal or learned positional encodings. Instead of adding a positional vector to the token embedding, RoPE rotates the query and key vectors by an angle that depends on their position, before computing attention. The rotation is applied as pairs of dimensions, treating each pair as a 2D coordinate and rotating it by $m\theta$ for position $m$ and a frequency $\theta$ that varies across dimension pairs.
Why rotation encodes relative position automatically
The elegant property that makes RoPE work: when you rotate query vector $q$ at position $m$ by angle $m\theta$, and key vector $k$ at position $n$ by angle $n\theta$, their dot product $q_m \cdot k_n$ depends only on the difference $(m-n)\theta$ — not on the absolute positions $m$ and $n$ themselves. This falls directly out of how rotation matrices compose: rotating two vectors by different absolute amounts and then taking their inner product is mathematically equivalent to rotating one of them by the relative difference. The practical result is that attention scores naturally encode 'how far apart are these two tokens' without any extra parameters or lookup tables.
No added vector, no extra parameters
import numpy as np
def rotate_half(x):
"""Split last dim in half and rotate: (x1, x2) -> (-x2, x1)."""
x1, x2 = x[..., ::2], x[..., 1::2]
return np.stack([-x2, x1], axis=-1).reshape(x.shape)
def apply_rope(x, position, base=10000.0):
dim = x.shape[-1]
inv_freq = 1.0 / (base ** (np.arange(0, dim, 2) / dim))
angles = position * inv_freq # theta_i * m
cos = np.repeat(np.cos(angles), 2)
sin = np.repeat(np.sin(angles), 2)
return x * cos + rotate_half(x) * sin
q_pos3 = apply_rope(np.array([1.0, 0.5, -0.3, 0.8]), position=3)
k_pos7 = apply_rope(np.array([0.9, -0.2, 0.4, 0.1]), position=7)
print("dot product depends only on relative distance (7-3=4):", np.dot(q_pos3, k_pos7))Frequency bands, just like sinusoidal encoding
RoPE still uses the same frequency-banding trick as the original sinusoidal scheme: different dimension pairs rotate at different rates, controlled by $\theta_i = base^{-2i/d}$. Early dimension pairs rotate quickly (sensitive to small position differences, useful for local structure), later pairs rotate slowly (sensitive only to large position differences, useful for long-range structure). The constant base (typically 10000, though larger values like 500000 are now common) controls how quickly these frequencies decay, which directly affects how well a model trained at one context length generalizes to a longer one.
Extending context length: interpolation and NTK scaling
Because RoPE is a smooth, continuous function of position, researchers found they could stretch a model's effective context window after training by rescaling position indices — a technique called position interpolation — or by adjusting the base frequency so low frequencies compress more than high frequencies (NTK-aware scaling, later refined into YaRN). These techniques let models trained on, say, 4K tokens be fine-tuned or even used zero-shot at 32K+ tokens with much less quality degradation than would be possible with a fixed learned embedding table that simply has no entries past its training length.
RoPE is not a free lunch at extreme lengths
What's Next?
RoPE fixes how positions are represented inside attention, but attention itself still has a quadratic memory and compute cost in sequence length — the next bottleneck worth understanding is how that cost is actually paid for, and reduced, at the hardware level.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.