Mixture of Experts (MoE) is a neural network architecture that splits a model into many specialized sub-networks called experts, and uses a routing mechanism to activate only a small subset of them for each input token. This means a model can have a very large total parameter count while only running a fraction of those parameters per token, giving high capacity at a lower inference cost than a dense model of the same size. Many leading open-weight models, including recent DeepSeek, Mixtral, and Qwen releases, use MoE designs. The trade-off is added routing complexity and higher memory requirements, since all experts must still be loaded even though only a few run at a time.
Frequently Asked Questions
What is a mixture of experts model?
A mixture-of-experts (MoE) model contains many expert sub-networks and activates only a few per token via a router, giving large capacity at a lower per-token compute cost than a dense model.
Why do labs use MoE?
MoE lets a model scale to a very high parameter count while keeping inference costs down, since only a small fraction of the network runs for any given token.