The first two hands-on posts in the neural-nets-as-a-distributed-system track kept the communication pattern fixed. All-reduce, three ways asked how to realize a fixed contract; consistency models asked how to relax it. A mixture-of-experts layer breaks the premise underneath both: the pattern itself is dynamic.

The topology is a function of the input

In an MoE layer a small learned router scores each token against every expert and sends it to its top-k. The experts are spread across devices, so "which expert" means "which device." Put those together and the dispatch step is an all-to-all whose traffic matrix is computed from the data, every batch. There is no fixed wiring diagram to draw — the diagram is the routing, and the routing changes with every input.

That single fact drags two operational problems into the open, and they are the problems MoE systems spend their complexity budget on.

Skew drives imbalance, drops, and the bottleneck — together

Routing is essentially never uniform. A few hot experts attract disproportionate traffic, and because expert compute and the all-to-all both run at the speed of the busiest device, that imbalance is a straggler — the same enemy from the consistency post, arriving by a different road. Each expert also has a fixed capacity buffer; tokens past it are dropped.

Sweeping routing skew from uniform (0) to heavily skewed (2) in the simulator:

 skew  expert imbal  dropped %  a2a bottleneck  network %
  0.0          1.09       0.0%            2152      75.2%
  1.0          4.15      27.9%            3439      74.9%
  2.0          7.15      55.0%            4627      76.3%

At skew 2, the busiest expert sees ~7× its fair share, more than half of all routed tokens are dropped on the floor, and the all-to-all bottleneck has more than doubled. The network % barely moves — it sits near (D−1)/D, a structural constant when token sources are spread evenly — so the damage is entirely in imbalance, not in raw volume.

It morphs every batch

Left: expert load for uniform vs skewed routing against the capacity line — hot experts overflow and their tokens drop. Middle: the device-by-device all-to-all traffic matrix for one skewed batch, with a hot destination column where the popular experts live. Right: per-device receive load across eight batches, with the lines crossing — the busiest device is different almost every batch.

The middle panel is one batch's dispatch matrix: a hot column, because every source device ships heavily to whichever device hosts the popular experts. The right panel is the punchline of the whole track — same routing distribution, eight different batches, and the lines cross: the busiest device is different almost every batch. You cannot statically place experts to fix this. The imbalance is re-rolled from the data each step; the topology genuinely will not hold still.

What this is in Megatron

This is Megatron-LM expert parallelism. The dispatch/combine all_to_all is the dominant communication in an MoE layer, and its per-device volume is set by the routing — exactly the bottleneck the simulator measures. The two production knobs map straight onto the two failure modes:

  • capacity_factor sizes the per-expert buffer — the line that tokens drop past in the left panel. Bigger buffers waste memory and compute on padding; smaller buffers drop more tokens. It is a direct memory-vs-accuracy dial.
  • The auxiliary load-balancing loss is a control term that pushes the router toward skew = 0. It is why real MoEs don't live at the right end of that table: the model is actively penalized for building a lopsided topology.

That is the move worth internalizing. A dense model's fabric is plumbing you provision once. An MoE's fabric is a control problem the model is solving as it runs — and the load-balancing loss is the controller. The network is no longer below the model; it is part of what the model is learning to manage.

Code and the simulator: neural-nets/projects/03-moe-routing. Next in the track: elastic membership — re-forming the collective when a node joins or dies mid-run.