Every large training run has two numbers. One gets quoted in launch posts; the other decides what the run actually cost.

MFU — model FLOP utilization — is the quoted one. It asks: while the machine is computing, how close to peak is it? It is a good number. It is also a local number. It says nothing about the hours you spent recomputing work you already did, or waiting on a rank that was never coming back.

Goodput is the other one: of the wall-clock time you paid for, how much produced work you kept? Not FLOPs issued — FLOPs that survived to the next checkpoint and made it into the final weights.

A run sitting at 45% MFU that loses an hour to a dead rank every six hours is not a 45% run. It is a 37% run wearing a 45% badge. The difference is not rounding — at fleet scale it is the budget.

Where goodput leaks

Four holes, and they are all systems holes.

Hard failure and rollback. A node dies, a link flaps, an ECC error escalates. The job restarts from the last checkpoint, and every device throws away the work since that checkpoint. The loss is not one node-hour; it is (time since checkpoint) × (all N devices). Failure is billed at full width.

Stragglers. Collectives are barriers. An all-reduce finishes when the slowest participant arrives, so one device running 10% slow does not cost 10% on one device — it taxes the entire step, every step. I wrote about seeing this structurally in the traffic matrix: the fabric shows you who is late before the loss curve does.

Jitter and system noise. The quiet killer. An OS timer tick, a daemon waking, a background service stealing a core — each is microseconds, and individually beneath notice. But in a tightly-coupled collective those microseconds do not average out; they amplify. With enough participants, the probability that someone is perturbed on any given step approaches one, so the tail becomes the common case. This is the effect that scales against you hardest, and it is invisible unless you instrument the whole machine at once.

Recovery and re-warm. Restart is not free even after the process is back: re-reading the checkpoint, re-establishing communicators, refilling the pipeline. Goodput counts that time as lost too.

The arithmetic that should scare you

Take the rollback loss seriously for a moment. If checkpoints are spaced Δ apart and a failure lands uniformly within that window, the expected work destroyed per failure is about Δ/2 — times every device. Add the checkpoint write cost C paid every Δ, and the useful fraction looks roughly like

goodput ≈ 1 − C/Δ − (Δ/2)/MTBF

Two terms pulling opposite directions. Checkpoint too often and C/Δ eats you; checkpoint too rarely and rollback does. There is an optimal Δ — the classic Young/Daly result — and it scales like √(2·C·MTBF).

The part that matters: MTBF shrinks as you add nodes. Independent-ish failures across N components mean system MTBF falls roughly like 1/N. So the faster you scale, the shorter the interval between failures, the smaller the optimal checkpoint interval, and the larger the fraction of your machine's life spent either writing checkpoints or redoing lost work. Goodput does not degrade gracefully with scale; it degrades structurally.

This is why "just add more GPUs" stops working long before the physics says it should.

The uncomfortable part

Look at that list again — failure recovery, checkpoint scheduling, straggler mitigation, collective resilience, MTBF across thousands of components, noise on the tail. None of it is machine learning. It is distributed systems and high-performance computing, and it is the same fight HPC has been having since the first machine big enough to break while you were using it.

The training stack has spent its attention on the model — architectures, optimizers, parallelism strategies — because that is where the intellectual excitement is. Meanwhile the thing quietly setting the price of a run is the resilience layer underneath, and on a GPU fleet that layer is systems, not PyTorch.

I say this with some scar tissue. My most recent work at exascale was chasing exactly this class of loss on El Capitan — the tail of collective communication and the operating-system noise feeding it. On critical benchmarks, mitigating that noise recovered more than 40% of performance (the basis of our SC'25 paper). No model changed. No kernel got smarter. The machine simply stopped tripping over itself.

That is a goodput fix. It looks like nothing on a model card and like everything on an invoice.

What to actually measure

If you want to manage goodput you have to instrument it, and most stacks do not by default. The minimum honest set:

  • Useful throughput — tokens (or samples) that survived to a committed checkpoint, per wall-clock hour. Not tokens issued. Tokens kept.
  • Lost work per failure — devices × time-since-checkpoint, measured, not assumed.
  • Time-to-recover — process restart, checkpoint read, communicator rebuild, pipeline refill. All of it.
  • Checkpoint overhead — the real C, including the stall it imposes.
  • Straggler distribution — not the mean step time, the spread. Track the slowest participant per collective; that is who sets your pace.
  • Collective tail — p99, not p50. The mean of an all-reduce is a comforting lie.

Notice how much of this is ordinary HPC hygiene pointed at a new workload.

How the pieces fit

This closes a loop the rest of this track has been circling. Elastic membership argued that at scale, failure is the steady state — that a run should survive losing members rather than treating each loss as an exception. The traffic matrix gave a way to see the fabric well enough to spot the straggler. Adaptive parallelism re-picked the DP×TP×PP split as the workload moved.

Each of those is a lever. Goodput is the objective they were all serving. Elastic membership shrinks rollback loss. Fabric visibility finds the straggler. Adaptive parallelism keeps you on the throughput ceiling as conditions change. They are worth doing because of what they do to the number that pays the bill.

The measurement

Defining a metric is the easy half. The companion to this post measures it — what a killed rank actually costs: a real 8×A100 run, instrumented for goodput, with a rank killed and a GPU throttled on purpose. The headline turns out to be counterintuitive. A dead rank costs about six minutes of wall-clock, but only ~9 seconds of that is detection and ~43 seconds is recomputed work — the dominant ~270 seconds is seven healthy GPUs idle-spinning in the collective, waiting on a peer that is never coming back. The lost compute, the thing you would blame first, is the smallest term. Go read it for the numbers.

One question worth asking about your own runs: do you know your goodput, or only your MFU? In my experience most teams can quote the second to one decimal place and have never computed the first.

Srinath Vadlamani is a GPU/HPC performance engineer and the founder of SV Advanced Computing LLC. He most recently owned GPU application performance on El Capitan, the #1 system on the TOP500.