Every hands-on post in the neural-nets-as-a-distributed-system track so far assumed a fixed set of ranks. At real scale that assumption is false. With thousands of GPUs, something is always failing — failure is the steady state, not the exception — and a synchronous collective handles it about as gracefully as possible, which is to say: not at all.
The hang
Recall the ring all-reduce from post 12: each rank sends to its neighbor and blocks waiting to receive. Now kill one rank. Its neighbor waits for bytes that never arrive; its neighbor waits for the partial sum that now never propagates; within one step the entire cluster is deadlocked at the barrier. A dead rank doesn't slow the collective — it hangs it, completely, until something outside the collective notices and intervenes.
So membership has to become a live variable. The ring is torn down and re-formed on every failure and every join — the same heartbeat, gossip, and view-change machinery distributed systems have used for decades, now wearing the name "elastic training."
Detection latency is stall time
Here is the property that makes this more than bookkeeping, and it's the whole reason the engineering is subtle: from the instant a rank dies until the failure detector fires, the cluster is hung. Detection latency is not a background cost you can hide — it is dead time the entire job pays, on top of the reconfiguration that follows.
That gives the heartbeat timeout a real two-sided cost:
- Too long → every genuine failure hangs the whole cluster for roughly the timeout before it's even noticed. Detection-stall dominates.
- Too short → a transient pause on a healthy rank (a GC stall, a network blip) overshoots the timeout and the rank is wrongly declared dead — a false positive — triggering a needless eject, rebuild, and rejoin.
I simulated a small cluster under churn to make both visible:
Left — the morph. Cluster size steps 8 → 7 → 6 → (join) 7 → 6 as ranks die
and rejoin. Each red band is a hang + reconfigure window; cumulative training
progress (orange) flat-lines through every band and resumes after. Goodput here
is 0.86 — 14% of the wall-clock went to churn.
Right — the U. Sweeping the heartbeat timeout, the detection-stall cost climbs linearly while the false-positive cost falls as fewer transient pauses exceed the threshold. Their sum is U-shaped with an optimum around twice the reconfigure time:
timeout detect lat FP rate overhead
0.50 0.25 0.285 0.3297 <- false positives dominate
2.00 1.75 0.018 0.0926 <- optimum
5.00 4.75 0.001 0.1360 <- detection hangs dominate
This is exactly the Φ-accrual / failure-detector timeout tradeoff from the distributed-systems literature — measured, here, against training goodput.
Goodput collapses with churn
Hold the timeout at its optimum and vary how often failures happen:
MTBF failures/100 goodput
200 0.5 0.964
50 2.0 0.907
12 8.3 0.670
At high churn the reconfigure tax dominates, and this is precisely the pressure behind two ideas from earlier in the track. First, asynchronous / bounded- staleness training (post 13): drop the barrier so a dead or slow rank can't hang everyone — you trade consistency for the ability to make progress through churn. Second, cheaper reconfiguration: rebuild only the broken ring segment instead of re-forming the world.
What this is in Megatron
This is fault-tolerant / elastic training: torchrun --max-restarts, elastic
agents, in-job membership protocols. A dead rank forces a new view and a rebuilt
process group — the ring from post 12,
re-formed on the fly. Membership is the subtlest of the "morphs" this track has
walked through: the data-dependent topology of MoE,
the fabric it needs,
and now the live set of participants itself.
Code and the simulator:
neural-nets/projects/05-elastic.
Next: the split itself becomes dynamic — adaptive parallelism as closed-loop
control over the topology.