Mixture of Experts
What is it?
A mixture-of-experts model contains many specialized sub-networks ('experts') but activates only a few per input, so it can be huge in total yet cheap to run.
Explain like I'm 5
Why was it created?
Making dense models bigger raises quality but also cost for every token. MoE was developed to grow total capacity while keeping the per-token compute roughly fixed.
Where is it used?
- Large language models
- Scaling capacity efficiently
- Multilingual and multi-domain models
- Cost-sensitive high-quality inference
Why should developers care?
Several frontier models use MoE to get more capability per dollar, so it explains how models keep scaling without runaway inference cost.
How does it work?
A lightweight 'router' looks at each token and picks a small number of experts (e.g. 2 of many) to process it. Only those experts run, and their outputs are combined. Total parameters are large, but active parameters per token stay small.
Real-world example
A token about French grammar gets routed to experts that specialize in language structure, while a coding token goes to different experts — without running the whole network.
Common use cases
- Scaling model capacity cheaply
- Handling diverse domains in one model
- High quality at controlled inference cost
- Efficient very-large models
Advantages
- Huge capacity, modest per-token compute
- Experts can specialize
- Better quality-per-FLOP
- Scales without linear cost growth
Disadvantages
- High memory to hold all experts
- Routing can become unbalanced
- Trickier to train and serve
- Load-balancing overhead
When should you use it?
When you want more model capacity without paying full dense-model compute for every token.
When should you avoid it?
When memory is tight or simplicity matters — a smaller dense model may be easier to train and deploy.
Alternatives
Related terms
Interview questions
Beginner
- What is an 'expert' in an MoE model?
- Why is MoE cheaper to run than its size suggests?
Intermediate
- What does the router do?
- Why can total parameters be huge while active parameters stay small?
Senior
- What is expert load balancing and why does it matter?
- What are the serving/memory challenges of MoE at scale?
Common misconceptions
- "All experts run for every token" — only a few selected ones do.
- "MoE is smaller than dense models" — it has more total parameters, just fewer active per token.
Fun facts
- Only a small fraction of an MoE's parameters activate for any given token.
- The routing idea dates back decades but scaled up dramatically with transformers.
Timeline
- 2017 — Sparsely-gated MoE layers demonstrated at scale
- 2020s — MoE adopted in several frontier LLMs
Learning resources
Quick summary
Mixture of experts routes each token to a few specialized sub-networks, giving large total capacity at low per-token compute.
Cheat sheet
- Many experts, few active per token
- Router picks the experts
- Big capacity, modest compute
- High memory + load-balancing cost