What is mixture of experts in AI?
Mixture of experts in AI is an architecture where a model is split into several specialist sub-networks, called experts, and a router decides which experts handle each piece of input.
The idea is old. A 1991 paper by Jacobs, Jordan, Nowlan and Hinton described "adaptive mixtures of local experts". It became popular for large language models after a 2017 Google paper by Shazeer and colleagues showed a sparsely gated MoE layer with thousands of experts.
Picture a big coaching centre in Kota. Instead of every teacher attending every doubt, a front desk sends each student's question to the two most suitable teachers. The centre has many teachers in total, but each question only uses a couple of them.
If you are new to how these models are built, start with what a transformer model is, since modern MoE LLMs are transformers with a twist.
How does a mixture of experts model work?
A mixture of experts model works by replacing some layers of the network with a set of experts plus a router (also called a gating network) that scores the experts for every token.
In most MoE LLMs, the feed-forward block inside each transformer layer is the part that gets split into experts. The attention mechanism is usually shared. For each token:
- The router looks at the token's internal representation and gives every expert a score.
- It picks the top few experts, often the top 1 or 2.
- Only those experts process the token.
- Their outputs are combined, weighted by the router's scores, and passed to the next layer.
The router is trained along with the experts, so the model learns by itself which experts to use for which kinds of tokens.
What is sparse mixture of experts and why does it save compute?
Sparse mixture of experts means only a small subset of experts is active for each token, so the compute per token stays low even when the total model is very large.
A normal "dense" model uses all of its parameters for every token. A sparse MoE model has many more total parameters, but each token only touches its chosen experts. That is why MoE models are described with two numbers: total parameters and active parameters.
Mixtral 8x7B from Mistral AI is a clear example. It has 8 experts per layer, the router picks 2 for each token, and the model has roughly 47 billion parameters in total while using only about 13 billion per token. In practice, you get the knowledge capacity of a larger model at roughly the inference compute of a much smaller one.
Mixture of experts vs dense model: what are the trade-offs?
MoE models give more capacity for the same compute per token, but they need more memory and are harder to train than dense models.
| Point | Dense model | Mixture of experts model |
|---|---|---|
| Parameters used per token | All of them | Only the chosen experts |
| Compute per token | Grows with total size | Grows with active size |
| Memory needed | Total size | Total size (all experts must be loaded) |
| Training | Simpler and more stable | Needs load balancing, can be unstable |
| Serving | Easier on one machine | Often spread across many GPUs |
The memory point surprises many beginners. Even though only 2 experts run per token, all 8 must sit in memory, because the next token may need different ones. Training also needs extra tricks, such as a load-balancing loss, so the router does not send everything to the same favourite expert.
What are some examples of mixture of experts models?
Well-known open mixture of experts models include Mistral AI's Mixtral 8x7B and Mixtral 8x22B, and DeepSeek-V3 from DeepSeek, which reports far more total parameters than active ones.
Google's Switch Transformer research also used MoE, routing each token to just one expert. Several large closed models are widely believed to use MoE, but companies often do not confirm architecture details, so treat such claims as unverified.
A common myth is that each expert becomes a clean "maths expert" or "Hindi expert". The Mixtral paper found routing followed patterns linked more to syntax and token types than to obvious subjects. I find this the most useful thing to remember for interviews: "expert" is a name, not a job title. MoE also shows up in LLM interview questions, often alongside questions on quantization.