Mixture of Experts
A model with many specialist sub-networks that only wakes a few per token — how frontier models got enormous without getting proportionally slow.
When not to use it
- When memory is your constraint. You load every expert and use a few. If VRAM is what you're short of, this is the wrong architecture.
- On a single small device. MoE's advantage assumes you can hold the whole thing; that assumption is what makes it a datacentre technique.
- When you need predictable per-token cost. Capacity limits and token dropping make behaviour load-dependent.
- When you're fine-tuning and want it to behave. MoE fine-tuning is less understood, and the router is a component you didn't train and don't control.
Reach for something else instead
- A dense model — simpler, predictable, easier to fine-tune and serve.
- Distillation — get a smaller dense model from a large one, if inference cost is the actual problem.
- Quantization — reduces memory, which is MoE's weakness rather than its strength.
Sources & further reading
- Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — the paper that made it work at scale.
- Fedus et al. (2022), Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — top-1 routing; the simplification that stuck.
- Jacobs et al. (1991), Adaptive Mixtures of Local Experts — the original idea, thirty years early.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Comparing total parameters to a dense model's parameters. The honest comparison is active parameters for compute and total for memory.
- Assuming experts specialise by topic. Routing is per-token and mostly keys on things you wouldn't call subjects.
- Ignoring memory. "It runs like a 17B model" is about compute, not VRAM.
- Not knowing tokens can be dropped under load. Capacity factor couples throughput to quality.
At a glance
FieldLanguage & LLMs
Ideamany experts, a few active per token
Cheap oncompute
Expensive onmemory
Routingper token, not per topic
Needsload-balancing loss
DifficultyAdvanced
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
MoE vs. a dense model — same knowledge for less compute, at the price of more memory and much harder serving. It's a datacentre trade.