What is an “expert,” really?
Not a specialist you can name. Just one of many parallel feed-forward blocks (MLPs) — and a small router that picks a few of them per token.
Decoder ring — every term on this page, in one line each
- token
- a chunk of text (a word or word-piece) — the unit the model reads and writes
- parameter / weight
- one learned number; a “7B model” has 7 billion of them
- layer
- one repeated stage of the model; dozens to ~100 stacked make the whole thing
- attention
- the part of a layer where tokens look at each other; stays normal in MoE
- feed-forward block
= FFN = MLP - Three names, one thing. FFN = Feed-Forward Network; MLP = Multi-Layer Perceptron. It is the part of a layer applied to each token by itself (no looking at other tokens) — two matrix multiplies with a squashing function between. Holds ~⅔ of all parameters, and an “expert” is one copy of it
- dense model
- the normal kind — every token passes through every parameter
- MoE
- Mixture of Experts — a layer holding many parallel feed-forward blocks, of which only a few run per token
- expert
- one of those parallel feed-forward blocks. Not a subject specialist
- router / gate
- a tiny layer that scores the experts and picks which ones run for this token
- top-k
- how many experts the router keeps per token — commonly 1, 2, or 8
- softmax
- turns raw scores into fractions that add to 1 — the router’s blend weights
- total vs active
parameters - total = what you must store; active = what actually runs for one token
- shared expert
- an expert that always runs, unrouted — some designs keep one
- capacity factor /
token dropping - each expert has a seat limit per batch; overflow tokens get skipped
- all-to-all
- the network step where every GPU ships tokens to whichever GPU holds their expert
- HBM
- High-Bandwidth Memory — the GPU’s on-package memory; parameters live here
Read this once and every panel below is plain English. The only word doing sneaky work is “expert” — panel 3 is about exactly that.
1 · Start with the dense layer — every token walks the same hallway
In a normal (“dense”) transformer, each layer has one feed-forward block, and every token goes through all of it. Want the model smarter? Make that block wider — and every token gets more expensive. Capacity and cost are welded together.
2 · An expert is just a copy of that block — and a router picks a few
Replace that one block with E copies of it (say 8, or 256). Add a tiny router: a linear layer that scores the experts for this token and keeps the top-k (usually 1, 2, or 8). Only those run; their outputs are blended by the router’s weights. Everything else in the layer — attention, norms — stays completely normal and dense.
That is the whole trick: capacity is decoupled from cost. You buy knowledge with memory, and pay compute only for the slice each token uses.
3 · Are experts actually “experts”? — mostly no
Which is why the name is the most misleading word in the field. “Expert” means “one of the parallel blocks the router may pick,” nothing more.
4 · A worked example — one token through one MoE layer
Two costs the picture makes visible. Memory: all 671B must sit in HBM even though 37B ran — MoE buys compute savings with capacity. Network: step ③ is a data-dependent all-to-all — the traffic pattern is decided by the router at runtime, so a popular expert becomes everyone’s bottleneck (a straggler with a fan club).
The mixture is a router choosing a few, per token, per layer.