What is mixture of experts (MoE)?
Mixture of experts (MoE) is a way of building a large language model so that only part of the network does work on any given token. In a standard dense transformer, every token passes through every layer and every weight. In an MoE model, certain layers are replaced by a set of parallel sub-networks called experts, plus a small router. For each token, the router picks a few experts, often two out of eight or more, and only those experts run. The experts are not specialists in the human sense; the router learns a division of labor during training that rarely maps to recognizable topics.
The result is a model with a large total parameter count and a much smaller active parameter count. A model might hold several hundred billion parameters in total while activating a few tens of billions for each token.
Why mixture of experts matters
MoE changes the relationship between model quality, speed, and hardware. Because only the active parameters do arithmetic, an MoE model generates tokens roughly as fast as a dense model the size of its active count, while drawing on knowledge closer to its total count. The catch is memory. All the experts must be loaded onto the GPUs even though most sit idle for any given token, so the model needs the GPU memory of its total size. For a self-hosted deployment, that often means more GPUs than a dense model of similar output quality would need, offset by higher throughput once it is running. For hosted APIs, the provider absorbs this trade-off and the customer sees it only in pricing and speed.
Mixture of experts in practice
A logistics company evaluating open-weight models for a self-hosted dispatcher assistant compares a dense model with an MoE model that has twice the total parameters but a quarter of them active per token. The MoE model needs more GPU memory to load but answers faster and handles more concurrent dispatchers per server. Because the company already needed two GPUs for the dense model at full precision, the MoE option fits the same hardware and wins on throughput.
What to watch for
- Size GPU memory by total parameters and speed by active parameters; vendors often quote only the flattering number.
- Fine-tuning MoE models is more delicate than fine-tuning dense models, because the router can become unbalanced.
- Inference server support for a given MoE architecture varies, so confirm the serving stack before committing to hardware.
- Quantization applies to MoE models too and can bring the total memory footprint back within reach of fewer GPUs.