What is a mixture of experts (MoE)? Bigger, cheaper AI

Modern frontier AI models are staggeringly large. The leading large language models hold hundreds of billions, and in some cases more than a trillion, parameters, the adjustable numbers that store what a model knows. By the ordinary logic of neural networks, a model that big should be ruinously expensive to run, because normally every parameter does work on every word the model processes. And yet these models are affordable enough to serve to hundreds of millions of people. The trick that makes this possible has an odd name, the mixture of experts, and understanding it explains one of the most important shifts in how modern AI is built. A mixture of experts lets a model be enormous in total size while using only a small fraction of itself on any given token, which decouples how much a model knows from how much it costs to run and is the main reason frontier models could keep growing without becoming impossibly expensive.

This guide explains the problem MoE solves, how it works, why model cards now list two different parameter counts instead of one, the real costs it introduces in exchange for its savings, and why nearly every frontier model is now built this way. The payoff is a clear answer to a puzzle that sits under the whole field: how AI models got so big and stayed so cheap at the same time.

The problem: the dense wall

To see what MoE fixes, start with how a normal model works. A standard neural network, called dense, uses all of its parameters for every input. When a dense language model processes a word, every one of its parameters participates in the computation. This has a direct and unforgiving consequence: the cost of running the model, in both computation and time, is proportional to its size. Double the parameters and you roughly double the work for every single token, forever, on every query.

That collides with one of the most reliable findings in AI, captured by the scaling laws: bigger models are better, and adding parameters predictably improves capability. So there is a painful tension. Quality pulls toward more parameters; cost and latency pull toward fewer. In a dense model you cannot have one without paying for the other, and at the frontier the bill becomes enormous, because the added parameters that make the model smarter also make every response slower and more expensive to produce, at inference time, for the life of the model. This is the wall MoE was designed to get around.

The idea: many experts and a router

The mixture of experts breaks the link between size and cost with a simple change of structure. Inside a transformer, the standard architecture behind modern language models, each layer has two main parts: an attention mechanism, and a feed-forward network that does much of the actual processing. MoE leaves attention alone and replaces that feed-forward network with something new: instead of one feed-forward network, it installs many smaller ones, called experts, alongside a small routing network, called the router or gate.

Here is the key move. When a token arrives, the router looks at it and selects only a few of the experts to handle it, not all of them. A typical setup might have eight experts and use the top two for each token, or, at the largest scale, hundreds of experts with only a handful active at a time. The chosen experts do their work, their outputs are combined, and the rest of the experts sit completely idle for that token. This is called sparse activation: only a small, input-dependent slice of the model runs on any given word, even though the whole model is available. The router learns, during training, which experts to send which tokens to, so the selection is not random but a learned decision about which parts of the model are most useful for the input at hand.

The payoff: two numbers, not one

The consequence of sparse activation is the whole reason MoE matters, and it shows up in a small but telling detail: modern model cards now report two parameter counts instead of one, the total parameters and the active parameters. Total parameters are the model's full size, everything it stores and knows. Active parameters are how many actually run for each token, which determines the cost. In a dense model these two numbers are the same. In an MoE model they come apart dramatically, and that gap is the entire point.

The real figures make it vivid. One prominent frontier model holds around 671 billion total parameters but activates only about 37 billion of them per token, meaning roughly five percent of the model does one hundred percent of the work on each step. Another well-known open model has about 47 billion total parameters but uses under 13 billion at a time. A smaller open MoE model stores 21 billion parameters yet activates under 4 billion, letting it run on a single modest GPU while performing far above its active-parameter weight. In every case the model gets the knowledge and capacity of its large total size while paying, in compute and speed, only for its small active size. That is how a model can be enormous and affordable at once, and it is why comparing models by their headline total size alone is misleading: what a model costs to run is set by its active parameters, not its total.

How it works under the hood

The mechanism is worth seeing clearly, because it is more concrete than the name suggests. For each token, the router produces a score for every expert, turns those scores into weights, and selects the top few experts with the highest scores. The token is passed only through those selected experts, and their outputs are blended together as a weighted sum according to the router's scores, so an expert the router trusted more contributes more. During training, both the experts and the router learn together: the experts gradually specialize, each becoming better at certain kinds of input, and the router learns to send each token to the experts most likely to handle it well. A refinement used in several recent models adds a shared expert that is always active for every token regardless of routing, capturing the common knowledge every token needs, while the routed experts handle specialization on top of that shared base.

It is not an accident that MoE targets the feed-forward network specifically. In a transformer, the feed-forward layers hold a large share of the parameters and account for roughly two-thirds of the computation, so making exactly that part sparse is where the savings are largest. Attention, which handles the relationships between tokens, is left dense and untouched, because it is both a smaller share of the cost and harder to sparsify without hurting the model's grasp of context.

The catches: what MoE costs in return

Sparse activation is not free, and being honest about its costs is what separates understanding MoE from believing the marketing. Several real problems come with it.

The first is load balancing. Left to its own devices, the router tends to collapse onto a few favorite experts, sending most tokens to them while other experts sit unused and become wasted, dead parameters. This defeats the purpose, so MoE training adds an auxiliary load-balancing objective that pressures the router to spread tokens more evenly across experts. Getting this balance right is a persistent and finicky part of training MoE models, and a frequent source of instability.

The second, and arguably the most important, is memory. Even though only a few experts run per token, the model must keep all of its experts loaded in memory, because any token might need any of them. So an MoE model saves on computation but not on storage: you still have to hold the full total parameter count in memory to run it. This is the fundamental trade MoE makes. It converts a compute problem into a memory problem, spending abundant, cheaper memory capacity to save on scarce, expensive computation. That is a good trade at the frontier, but it means MoE models are demanding on memory in a way their small active size can disguise, and it is why techniques like quantization, which shrink the memory each parameter takes, and distillation, which compresses a model's knowledge into a smaller one, pair so naturally with MoE.

The third cost is systems complexity. At large scale the experts are spread across many different processors, so routing tokens to their chosen experts means constantly shuffling data across the network between devices, an all-to-all communication pattern that becomes a serious bottleneck and demands purpose-built infrastructure to run efficiently. The fourth is that the experts are not the interpretable specialists the name suggests. They do specialize, but not along clean human lines; there is generally no identifiable grammar expert or math expert you could point to. The specialization is real but statistical and messy, so reading the word experts as a claim about interpretability is a mistake.

Is MoE simply better than dense?

It is tempting to conclude that MoE is a free win, but the honest picture is more nuanced. The right comparison decides the answer. Compared against a dense model with the same number of active parameters, that is, the same cost to run, an MoE model is clearly better, because it brings far more total capacity to bear for the same compute. That is the comparison that matters in practice, where compute and latency are the binding constraints, and it is why MoE dominates. But compared against a dense model with the same number of total parameters, that is, the same stored capacity, a dense model can still hold a slight edge in quality, because it uses all of that capacity on every token rather than a routed slice. So MoE is not a way to get more capability from the same stored parameters for free. It is a way to get much more capability for the same compute, by trading memory and training complexity for it. When you are limited by how much you can compute per token rather than how much memory you have, which is the situation at the frontier, that trade is decisively worth making, which is exactly why the frontier made it.

Why nearly every frontier model is now MoE

Put these pieces together and the recent history of large models makes sense. The scaling laws said that adding parameters keeps improving models, but dense scaling made every added parameter more expensive to serve, threatening to price frontier models out of practical use. MoE broke that constraint by letting total capacity grow while keeping the active, paid-for portion small, so models could follow the scaling laws into hundreds of billions and trillions of parameters while staying servable. As a result, the mixture of experts went from a niche idea to the dominant architecture behind the current generation of frontier and open-weight models. It is the reason a model can advertise a total size that would once have been unthinkable to run, and the reason the meaningful question about a model's cost is no longer how big it is but how much of it wakes up for each word. MoE is the companion to the scaling laws: scaling says bigger is better, and the mixture of experts is how the field got bigger without paying the full price.

The short version

A mixture of experts is a neural network architecture that replaces the dense feed-forward layer in a transformer with many smaller expert networks plus a router that sends each token to only a few of them. This sparse activation means only a small slice of the model runs on any given token, which decouples a model's total size from its running cost. That is why MoE model cards list two numbers, total parameters and active parameters: the total is everything the model stores and knows, while the much smaller active count is what actually runs per token and sets the cost. A model can therefore have the knowledge of a giant while paying, in compute, only for a small model, which is how frontier systems reached hundreds of billions or trillions of parameters while staying affordable to serve. The trade is not free: MoE must keep all experts in memory even though few run, needs careful load balancing to stop the router collapsing onto a few experts, and adds systems complexity, so it swaps a compute problem for a memory-and-complexity one. Against a dense model of equal running cost it wins clearly, which is why nearly every frontier model now uses it.

The idea to hold onto is that a mixture of experts lets a model be huge in total capacity while activating only a small fraction of itself per token, decoupling what a model knows from what it costs to run, which is how modern AI got enormous and stayed cheap at the same time, at the price of holding all that capacity in memory and the complexity of routing. The next time you see a model described by two parameter counts, you will know that the gap between them is the whole trick.

Common questions

What is a mixture of experts (MoE)? A mixture of experts is a neural network architecture that lets a model be very large in total size while using only a small part of itself for each input. Instead of one feed-forward network in each transformer layer, an MoE model has many smaller expert networks and a router that selects only a few experts to process each token. This sparse activation means most of the model stays idle on any given word, so the model gets the knowledge of its full size while paying, in computation, only for the small active portion. It is the architecture behind most current frontier language models, and the reason they can be enormous yet affordable to run.

How does a mixture of experts work? Inside each transformer layer, MoE replaces the single feed-forward network with a set of expert networks plus a small router. When a token arrives, the router scores all the experts, selects the top few with the highest scores, and passes the token only through those, combining their outputs as a weighted sum based on the router's scores. The other experts do nothing for that token. During training, the experts gradually specialize while the router learns which experts to send which tokens to. Attention is left unchanged; only the feed-forward part becomes a routed set of experts, because that part holds most of the parameters and computation.

Why do MoE models list total and active parameters? Because in an MoE model those two numbers are very different, and both matter. Total parameters are the model's full size, everything it stores and has learned, which determines how much it knows and how much memory it needs. Active parameters are how many actually run for each token, which determines the computational cost and speed. In a dense model these are identical, since every parameter runs every time. In an MoE model only a small subset activates per token, so a model might have 671 billion total parameters but activate only 37 billion. The gap between the two numbers is exactly the efficiency MoE provides.

What is sparse activation? Sparse activation means only a small, input-dependent portion of a model's parameters run for any given input, rather than all of them. In a dense network, every parameter is active for every token, so compute scales directly with size. In a sparse MoE network, a router selects only a few experts per token, so most of the model is inactive at any moment. This is what lets an MoE model have a huge total capacity without a huge per-token cost: the capacity is all there and available, but only the small activated slice consumes computation on each step. Sparse activation is the core mechanism behind MoE's efficiency.

What problem does MoE solve? It solves the tension between capability and cost in scaling models. Scaling laws show that adding parameters reliably improves a model, but in a dense model every added parameter runs on every token, so making the model better makes every response proportionally more expensive and slower. This becomes an engineering wall at the frontier. MoE breaks the link by activating only a fraction of the parameters per token, so total capacity can keep growing while the compute paid per token stays low. It lets models follow the scaling laws to enormous sizes while remaining affordable enough to actually deploy and serve at scale.

Is a mixture of experts better than a dense model? It depends on the comparison. Against a dense model with the same active parameters, meaning the same running cost, an MoE model is clearly better, because it brings much more total capacity to bear for the same compute, which is the comparison that matters in practice. But against a dense model with the same total parameters, meaning the same stored capacity, a dense model can hold a slight quality edge, because it uses all its capacity on every token rather than a routed subset. So MoE is not free extra quality from the same stored parameters; it is much more capability for the same compute, bought by spending more memory and training complexity, which is a strong trade when compute is the limiting factor.

What are the downsides of a mixture of experts? MoE trades its compute savings for several real costs. It is memory-hungry, because all experts must be kept loaded even though only a few run per token, so the full total size still has to fit in memory. It needs careful load balancing, since routers tend to collapse onto a few favorite experts and starve the rest into wasted parameters, requiring extra training objectives to prevent it. It adds systems complexity, because at scale experts live on different processors and tokens must be shuffled between them, creating communication bottlenecks. And the experts do not correspond to clean, interpretable specialties despite the name. In short, MoE converts a compute problem into a memory-and-complexity problem, which is worthwhile mainly when compute is the binding constraint.