Mixture of Experts (MoE)

The scaling problem MoE solves

In a standard ('dense') transformer, every parameter participates in every forward pass. If you want a bigger, more capable model, you add parameters — and every single token now has to pass through all of them. Compute cost and parameter count scale together: a model with 10x the parameters costs roughly 10x as much to run per token.

Mixture of Experts (MoE) breaks that coupling. Instead of one large feed-forward block per transformer layer, an MoE layer has many smaller feed-forward blocks — the 'experts' — plus a lightweight router (or gating network) that looks at each token and decides which small subset of experts should process it. Only the selected experts do any work for that token. The model can have an enormous total parameter count while each individual token only activates a small fraction of it.

How routing actually works

At each MoE layer, the router is typically a small learned linear layer that takes a token's hidden representation and outputs a score for every expert. The top-k highest-scoring experts (commonly k=1 or k=2) are selected, their outputs are computed, and then combined — usually weighted by the router's scores — to produce the layer's output. Every other expert does nothing for that token.

Crucially, which experts get picked varies token by token and layer by layer. Over a whole sequence, many different experts end up contributing, but for any single token, only a small, cheap subset of the network's total capacity is actually used.

Dense model vs Mixture of Experts

The load-balancing problem

A naive router tends to collapse: it discovers a few experts that work reasonably well early in training and keeps routing most tokens to them, starving the rest of gradient signal. Those under-used experts never improve, and the model effectively becomes a much smaller dense model wearing a large parameter count as costume.

MoE training adds an auxiliary load-balancing loss that penalizes uneven routing and encourages tokens to be spread more evenly across experts. This is a genuine trade-off: perfectly even load balancing isn't necessarily what's best for task performance, but some balancing is necessary or the sparse capacity goes to waste. Getting this right is one of the harder parts of training MoE models well.

What experts actually specialize in

It's intuitive to imagine experts specializing cleanly by domain — one expert for math, one for code, one for poetry. In practice, routing patterns that emerge from training are usually messier and less interpretable than that. Experts often end up specializing along more superficial statistical regularities (certain token types, syntactic patterns, or local context shapes) rather than clean semantic domains. The 'expert' terminology is somewhat aspirational — it's an architectural mechanism for sparse capacity, not a guarantee of clean conceptual division of labor.

Costs and trade-offs

MoE isn't free complexity-wise. All experts must typically be loaded into accelerator memory even though only a few are used per token, so memory requirements don't shrink the way compute does. Serving an MoE model efficiently at scale usually requires specialized infrastructure — expert parallelism, where different experts live on different devices, and careful routing to avoid bottlenecking any single device with too many tokens. Training is also more complex: load-balancing losses, routing instability, and uneven expert utilization are real engineering problems, not edge cases.

The net trade-off: MoE buys you more model capacity per unit of inference compute, at the cost of more memory, more infrastructure complexity, and trickier training dynamics.

What's next

MoE makes a single large model cheaper to run per token by activating only part of its parameters. Speculative decoding attacks the inference-cost problem from a completely different angle — using a small, fast model to guess ahead and a large model to verify, speeding up generation without changing the output at all. That's the next lesson.