Measuring Training Goodput: What a Killed Rank Actually Costs

The thesis

Goodput at scale is an HPC-systems problem, not an ML-modeling problem. The useful-work-per-dollar of a big training run is decided by failures, restarts, stragglers, jitter, collective-communication tails, and MTBF across thousands of nodes — the exact things HPC has fought for decades. This post measures one slice of it honestly: what does a single killed rank actually cost you?

The companion to this post, why goodput is the only number that pays the bill, makes the case for the metric — why MFU flatters and goodput bills. This one measures it.

Definition used throughout: goodput = useful throughput after accounting for wasted work — re-computation since the last good checkpoint, restart latency, and the stragglers that drag a synchronous step to the speed of its slowest rank.

goodput = total_throughput × (1 − wasted_fraction)

What we measured, and on what (honesty first)

This is an SVAC study, not a production foundation-model run. Numbers are stated as measured on the rig below, at the scale below — not extrapolated.

• Baseline rig (Result 1): 1× NVIDIA A100 80GB (Brev/Hyperstack), NGC PyTorch 25.01 container, Megatron-LM GPT (12 layers / 1024 hidden / 16 heads, seq 1024, bf16), synthetic data, global batch 32.
• Core rig (Result 2): 8× A100-SXM4-40GB in a full NVLink mesh (lambda-labs), NGC PyTorch 25.01, Megatron-LM GPT (24 layers / 2048 hidden / 16 heads, seq 1024, bf16), 8-way data-parallel, global batch 256, real NCCL all-reduce across ranks.
• Guardrail: FP8/tensor-core stays at the throughput/goodput-observation level — we do not claim mixed-precision authoring.

Result 1 (from the baseline): the instrument perturbs the metric — and leaves a residue

Before you can measure how a failure costs you goodput, you have to trust the instrument that measures goodput. It turns out the instrument is expensive, and its cost does not end when you switch it off.

The baseline run profiled steps 10–20 under nsys (--profile with --capture-range=cudaProfilerApi), leaving steps 3–9 and 22–25 unprofiled. Per-iteration timings, straight from baseline_train.log:

The observer tax is ~24%. Under capture, throughput falls from 106.1 to 80.4 TFLOP/s/GPU — a 24.2% throughput loss, equivalently a 32.0% inflation of iteration time (363.9 → 480.2 ms). Anyone quoting a profiled number as the number is quoting one about a quarter too low.

The transitions are not free. Opening the capture range costs a one-off ~723 ms. Closing it costs far more: iteration 21 takes 5.86 seconds, roughly 5.5 s of which is the trace flush at cudaProfilerStop. On a short run that single stall outweighs every other perturbation combined.

Throughput may not fully return. After the profiler detaches, the run settles at 98.5 TFLOP/s/GPU, a ~7% deficit versus the pre-profiling 106.1. With only four post-window samples this is an observation, not a finding — residual profiler state, thermal throttling, and allocator state are all live explanations, and separating them (per-iteration clock/thermal logging + a no-profiler control) is the obvious next step.

Why this matters for the rest of the post. The tool used to measure goodput moves that metric by ~24% while running. So profiling is scoped to a window, and every headline number below is reported from unprofiled steady-state steps. The instrument finds where the time goes, never how much time there is.

Result 2 (from the 8×A100 core run): the cost of a killed rank

Baseline goodput, all eight ranks healthy. Steady-state 151.7 TFLOP/s/GPU (~1,213 TFLOP/s aggregate), 1,722 ms/iter, i.e. ~152,200 tokens/s (global batch 256 × 1024 seq). This is the number every cost below is measured against.

Checkpointing is cheap in-band, but not free. A torch_dist checkpoint takes ~8 s of (largely asynchronous) save time; the step that triggers it inflates from 1,722 ms to ~2,020 ms — a +300 ms in-band stall. Fine every 25 steps; a problem every step.

Killing a rank does not crash the job — it strands it. At iteration 75 we sent kill -9 to one rank (GPU 7). What happened, in order:

Add it up and one killed rank costs roughly six minutes of wall-clock — and the dominant term is not the checkpoint gap, it is the ~270 s survivor hang. Detection is fast; clean teardown is not. Seven healthy GPUs sit burning power waiting on a peer that is never coming back, until the watchdog fires. That hang is almost entirely avoidable with fault-tolerant / elastic config, and eliminating it is the single biggest goodput lever here — bigger than any checkpoint-cadence tuning.

Restart is not push-button. Getting a working resume took three fixes: a torch_dist load API mismatch between Megatron-main and the container's PyTorch; an LR-scheduler total-iterations assertion on resume; and a rejected override flag — resolved only by matching --train-iters between save and load. Checkpoint/restart is exactly as reliable as your framework version hygiene. A silent save/load drift turns "resume from checkpoint" into "start over," which is a goodput cliff, not a goodput cost.

One slow GPU taxes all of them. We throttled GPU 7 to 700 MHz — half its 1,410 MHz max — mid-run, then reset it:

One GPU at half clock slows the entire eight-rank step by 70%. The straggler does not slow itself, it slows everyone — a synchronous step runs at the speed of its slowest rank, which is why seeing it requires per-participant fabric visibility, not aggregate throughput. The clean recovery on reset (unlike Result 1's ambiguous residue) makes this causal, not correlational. A killed rank is just the limiting case: a straggler with infinite latency.

Mitigation: cadence tuned to MTBF, and killing the hang

The checkpoint interval is the classic Young/Daly tradeoff — save overhead paid every N iters against expected lost work (≈ N/2 iters redone) on failure. With the measured inputs (1.72 s/iter, ~0.3 s in-band save, ~59 s cold restart), the optimal interval is N* ≈ sqrt(2 · save_cost · MTBF / iter_time): checkpoint often when failures are frequent, seldom when they are rare. But the arithmetic also says the largest win is not cadence at all — it is removing the ~270 s survivor hang so a dead rank fast-fails into the ~59 s restart instead of stranding seven GPUs. Cadence trims the redo; elasticity removes the hang, and the hang is the bigger number.

Why it matters (the fleet arithmetic)

A CSP goodput engineer owns a product of two things this post measured. Node MTBF divided into job duration sets the expected number of failures per run; each failure costs (survivor hang + restart + work since last checkpoint). Goodput is what survives that product. At single-digit node counts a six-minute failure is an annoyance; at thousands of nodes, where the expected time-between-failures collapses toward the checkpoint interval, it is the difference between a run that finishes and one that never does. The lever with the most leverage is the one hiding in plain sight in Result 2: don't let a dead rank strand the living ones.

Method (reproducible)

• Baseline harness (train_baseline.sh): Megatron under nsys, --log-throughput, profiling scoped to steps 10–20 via --profile + --capture-range=cudaProfilerApi.
• Core harness (goodput_8x.sh): 8-way DP Megatron, checkpoint every 25 iters, all numbers from unprofiled steps. Fault injection = kill -9 one rank (host PID via nvidia-smi compute-apps) and clock throttle via nvidia-smi -lgc.
• Full setup, errors, and gotchas: goodput-exp1-setup-log.md; measured results: RESULTS-8x.md; raw logs: 8x_baseline.log, 8x_killrun.log, 8x_straggler.log, 8x_load2.log.
• This work leans on the SVAC distributed-systems posts 18 (elastic-membership) and 19 (adaptive-parallelism) for the recovery/elasticity framing.