Mixture of Experts (MoE)

In an MoE transformer, feed-forward layers are replaced by many parallel experts plus a router that activates only a few of them per token. A model can therefore have a very large total parameter count while spending the compute of a much smaller dense model on each token — Mistral's Mixtral and DeepSeek's V-series brought the pattern to open weights, and it is now standard across frontier labs.

The economics explain the popularity: capability tracks total capacity while serving cost tracks active compute, so MoE models deliver near-flagship quality at markedly lower cost per token. Much of the steep decline in intelligence-per-dollar pricing traces to this architecture.

The tradeoff is memory: all experts must sit in accelerator memory even though few fire per token, so MoE favors large serving clusters — one reason self-hosting big MoE models is harder than their active-parameter numbers suggest.