Mixture of Experts

Mixture of Experts (MoE) is the architecture behind most of today's largest open and frontier models — Mixtral, DeepSeek, and (by wide consensus) the GPT-4 class. The core trick: have a huge number of parameters, but only use a small fraction of them for any given token. You get the capacity of a big model at the inference cost of a small one.

Dense vs Sparse

A dense transformer runs every parameter for every token. Doubling parameters doubles the compute per token.

A sparse MoE replaces the feed-forward block in each transformer layer with many parallel expert networks plus a router. For each token, the router picks the top-k experts (commonly 2 of 8, or 8 of 256) and only those run. So a model can have 671B total parameters but activate only ~37B per token (DeepSeek-V3's numbers).

The terms to keep straight:

• Total parameters — what the model "knows," what you must store in memory.
• Active parameters — what actually runs per token, which drives compute and latency.

The Router

The router is a small learned layer that scores each token against each expert and sends the token to its top-k. It's the hard part:

• Load balancing — left alone, routers collapse onto a few favorite experts while the rest starve. An auxiliary load-balancing loss (or loss-free balancing tricks) spreads tokens across experts.
• Routing instability — small changes in router weights can reroute tokens entirely, making training jumpy. This is why MoE training is finickier than dense.
• Token dropping — experts have a fixed capacity per batch; overflow tokens get dropped or passed through unchanged.

Why MoE Won at the Frontier

• Capacity per FLOP — you can grow what the model knows without growing what each token costs.
• Specialization — experts empirically specialize (syntax, domains, languages), though less cleanly than the name suggests.
• Training efficiency — for a fixed compute budget, sparse models reach lower loss than dense ones.

The Catches (and Why an Engineer Should Care)

MoE's cost shifts from compute to memory and systems complexity:

• You still hold all the weights. A 671B MoE needs memory for 671B parameters even though it computes like a 37B model. VRAM, not FLOPs, becomes the constraint — this is why MoE serving wants lots of GPU memory.
• Expert parallelism — experts get sharded across devices, so inference involves all-to-all communication to route tokens to the right GPU. Networking can dominate latency.
• Batching is weirder — different tokens in a batch hit different experts, so utilization depends on routing distribution. Throughput is less predictable than dense.
• Fine-tuning is trickier — routing learned during pre-training is fragile; naive fine-tuning can wreck load balance.

What to Watch For

When you compare models, look at active parameters for latency/cost intuition and total parameters for capability and memory footprint. A "37B active / 671B total" model behaves like a 37B model on your latency budget but needs the hardware of a 671B one. Quoting just one number hides the tradeoff that actually matters for your deployment.