A textbook neural network is a static dataflow graph: fixed layers, fixed wiring, data flows forward, gradients flow back. The moment you spread that graph across many devices, the picture stops being static — and the part that moves is the network, in the distributed-systems sense of the word.
This is the first post of a new track, separate from the LLM-engines work. The thesis is simple and, I think, underappreciated:
At scale, a neural network and the distributed system running it are the same object — and that object's communication structure is dynamic. It must continuously reconfigure — morph — in response to the data, the phase of training, and the liveness of the cluster.
Three reasons the network can't hold still
1. The communication pattern depends on the data. In a mixture-of-experts layer, a learned router decides which expert each token goes to — so it decides which device sends to which device, per token, per batch. The all-to-all topology is literally a function of the input. There is no fixed wiring diagram to draw; the diagram is recomputed every forward pass.
2. The communication pattern depends on the phase. Training does a big gradient all-reduce every step. Decode streams tokens and moves KV cache. Prefill is compute-bound; decode is bandwidth-bound. The collective that is optimal in one phase is wrong in another — the fabric is asked to do different jobs at different times.
3. The communication pattern depends on liveness. At thousand-GPU scale, failure is the steady state, not the exception. A collective has to notice a dead peer, eject it, re-form the ring, and keep going — mid-run. Membership is a live variable, not a launch-time constant.
A static interconnect serves none of these well. The network has to morph.
The punchline: a large model rediscovers distributed systems
Here is the part I find genuinely fun. Each thing a large model's runtime does turns out to be a decades-old distributed-systems idea wearing new clothes:
| What the model does | What distributed systems already called it |
|---|---|
| All-reduce / all-gather / all-to-all, ring vs tree | collective communication & topology |
| Sync SGD vs async / Hogwild / bounded-staleness | consistency models (linearizable → eventual) |
| MoE expert routing under a capacity cap | load balancing under constraint |
| Elastic training, fault-tolerant collectives | failure detection & group membership |
| Re-choosing the parallelism split mid-run | adaptive control over your own topology |
None of these are new problems. They are the same problems, and the literature that solved them for databases and key-value stores in the 2000s is the right toolbox for reasoning about a model at scale today.
How I'm going to study this
CPU-first and runnable, same as the rest of this site. You do not need a GPU
cluster to understand why a ring all-reduce needs 2(N−1) rounds while
recursive-doubling needs log₂N, or why relaxing consistency buys throughput and
costs convergence. You need a laptop, a few processes, and the right two metrics:
bytes-on-wire and message rounds — the hardware-independent quantities
that NCCL optimizes and that Megatron-LM's parallelism choices trade against each
other. Everything simulates the cluster locally and names the real primitive it
stands in for. Megatron-LM is the north star: every toy maps onto where the
production stack uses the real thing.
The code lives in a new repo, neural-nets,
and the first two projects already have posts:
- All-reduce, three ways — the same collective realized three ways, and why the cheapest one morphs with the world size and message size. (This is the difference between tensor-parallel and data-parallel all-reduce in one plot.)
- Consistency models for SGD — relax the all-reduce contract from "everyone sees the exact average" to "bounded staleness," and watch throughput and convergence pull against each other.
Still to come: MoE routing as a data-dependent all-to-all, elastic membership that re-forms the collective on node churn, and adaptive parallelism that morphs the DP/TP/PP split as the bottleneck moves.
The one-line takeaway: stop drawing your large model as a fixed graph on a fixed cluster. Draw it as a control loop over a network that rewires itself — because that is what it is.