This is the final hands-on post in the neural-nets-as-a-distributed-system track, and it closes the loop the whole series has been circling. Every previous post held the parallelization strategy fixed and asked how to run it well. But the strategy itself is not fixed. The optimal way to carve N devices into data × tensor × pipeline parallelism depends on the workload — and the workload moves.

Why the optimum moves

The cleanest driver is sequence length. Activation memory grows like S, so as S increases — long-context training, a curriculum that lengthens sequences — activations eventually stop fitting in device memory. When that happens you have no choice but to shard them harder (more tensor or pipeline parallelism), which leaves fewer devices for data parallelism. The optimum is pushed around by memory, not by preference.

I built a transparent cost model over every dp×tp×pp = N config — per-device memory (over the cap = OOM), the tensor-parallel all-reduce on the critical path (cheap while it stays inside a node, expensive once it must cross to the slow inter-node link), the data-parallel gradient all-reduce, and the pipeline bubble — and let a controller pick the highest-throughput feasible config at each sequence length.

The morph

Left: throughput per phase — the adaptive controller stays on the ceiling, static-small OOMs past the first phase, static-robust ramps up from far below. Right: the chosen (dp, tp, pp) as sequence length grows — dp falls from 64 to 4, tp climbs from 1 to 8, and pp finally engages.

    seq   best (dp,tp,pp)    tokens/s
    512        (64, 1, 1)     4047579     roomy memory -> pure data parallel
   2048        (32, 2, 1)     8068620
   4096        (16, 4, 1)     8016063
   8192         (8, 8, 1)     7912975     tp maxed at the node boundary
  16384         (4, 8, 2)     7422129     now pipeline sharding kicks in

Data parallelism falls 64 → 4; tensor parallelism climbs 1 → 8 and then stops — because going past 8 would push the TP all-reduce across the slow inter-node link — at which point pipeline parallelism takes over the remaining sharding. That ordering isn't hand-coded; it falls out of the cost model preferring cheap intra-node TP until it runs out of room, then paying for pipeline bubbles. The topology reconfigures itself to the workload.

Static loses two different ways

    seq    adaptive  static-small  static-robust
    512     4047579       4047579         253737    static-robust wastes 16x DP
   1024     4048247             0         505941    static-small OOMs
   16384    7422129             0        7422129
adaptive total / static-robust = 2.62x
  • static-small (64,1,1) — optimal at S=512 — OOMs the instant activations grow. It cannot run the later phases at all.
  • static-robust (4,8,2) — the one config that fits every phase — runs everywhere but throws away 16× of its data parallelism at short sequences.

Only re-picking each phase keeps you on the throughput ceiling throughout.

The whole track, in one loop

This reconnects everything. Each config change is a different traffic matrix $M_{ij}$ (post 17) landing in a different point of post 12's latency-vs-bandwidth crossover, and every reshard is a membership/view change (post 18). In Megatron terms this is the parallelism-config search (--tensor-model-parallel-size, --pipeline-model-parallel-size, DP) made dynamic — what autotuners and elastic frameworks reshard toward. Real controllers add switch cost and hysteresis so they don't thrash on noisy signals; the greedy per-phase optimum here is the target they track.

That is the thesis the whole track set out to demonstrate, now complete: a large model is not a fixed graph on a fixed cluster. It is a control loop over a network that rewires itself — realizing a collective (01), relaxing its contract (02), letting the data pick the topology (03), measuring it and sizing the fabric (04), surviving churn (05), and finally morphing the entire split under closed-loop control (06).

Code and the model: neural-nets/projects/06-adaptive-parallelism. The full first-principles write-up tying all six together is in the repo's docs/01-scale-out-challenge.md.