Mixture of Experts
A model with many specialist sub-networks that only wakes a few per token — how frontier models got enormous without getting proportionally slow.
Reviewed July 11, 2026Stable
When not to use it
- When memory is your constraint. You load every expert and use a few. If VRAM is what you're short of, this is the wrong architecture.
- On a single small device. MoE's advantage assumes you can hold the whole thing; that assumption is what makes it a datacentre technique.
- When you need predictable per-token cost. Capacity limits and token dropping make behaviour load-dependent.
- When you're fine-tuning and want it to behave. MoE fine-tuning is less understood, and the router is a component you didn't train and don't control.
Reach for something else instead
- A dense model — simpler, predictable, easier to fine-tune and serve.
- Distillation — get a smaller dense model from a large one, if inference cost is the actual problem.
- Quantization — reduces memory, which is MoE's weakness rather than its strength.
Read more on the blog
- What is a mixture of experts (MoE)? Bigger, cheaper AIFrontier AI models now hold hundreds of billions or even trillions of parameters, yet stay affordable enough to run at scale. The trick behind that is the mixture of experts, an architecture that lets a model be enormous in total size while using only a small slice of itself on any given word.
- AI scaling laws: why bigger wins, and whether it's endingOne empirical discovery explains most of the last six years of AI and the hundreds of billions spent on it: model performance improves in a smooth, predictable way as you add size, data, and compute. Here is what scaling laws are, why they reshaped the field, the Chinchilla correction, and the live 2026 question of whether pure scaling is running out.
- How much energy does AI use? Training vs inferenceThe energy cost of AI is discussed constantly and measured badly. The per-query figures in circulation are stale and vary by an order of magnitude, the balance has shifted from training to inference, and the constraint that actually bites is not generating electricity but delivering it to a particular building.
- Why does AI need GPUs? Parallelism and the memory wallAI runs on graphics chips because of a historical accident: neural networks turned out to need the same kind of arithmetic that rendering pixels does. But the constraint has since moved, and modern AI hardware is limited less by how fast it can calculate than by how fast it can fetch the numbers to calculate with.
Mixture of experts routes each token to a few specialists.
Instead of every token passing through the whole network, a router sends each one to just a couple of 'expert' sub-networks out of many. Only those experts activate, so the model can have huge total capacity while doing far less computation per token — the trick behind several frontier models that are enormous on paper but efficient to run.
Further reading
- Shazeer et al. (2017), Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer — the paper that made it work at scale.
- Fedus et al. (2022), Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — top-1 routing; the simplification that stuck.
- Jacobs et al. (1991), Adaptive Mixtures of Local Experts — the original idea, thirty years early.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Comparing total parameters to a dense model's parameters. The honest comparison is active parameters for compute and total for memory.
- Assuming experts specialise by topic. Routing is per-token and mostly keys on things you wouldn't call subjects.
- Ignoring memory. "It runs like a 17B model" is about compute, not VRAM.
- Not knowing tokens can be dropped under load. Capacity factor couples throughput to quality.
At a glance
FieldLanguage & LLMs
Ideamany experts, a few active per token
Cheap oncompute
Expensive onmemory
Routingper token, not per topic
Needsload-balancing loss
DifficultyAdvanced
Often compared with
MoE vs. a dense model — same knowledge for less compute, at the price of more memory and much harder serving. It's a datacentre trade.
Where this sits
A destination. 16 concepts lead here, and nothing in the corpus depends on it.
10Levelsteps in
16Needs firstconcepts
0Opens upnothing further
1Areastays here
Learn these firstTransformer
Computed from the prerequisite graph, not assigned. How this works