Our serving experiments keep producing the same shape of result: the dashboards stay green while the service dies. A single-slot config served one user perfectly and collapsed to 8% SLA conformance the moment a second arrived — with raw throughput flat and zero errors the whole time. The number that saw it was goodput: throughput that meets the stated SLA (raw × conformance). Which raises the operational question: what actually meters goodput today?

What exists (it's more than you'd think)

Serving-side, benchmark-time: NVIDIA AIPerf treats goodput as a first-class metric — declare SLOs via --goodput-ttft / --goodput-itl, get requests/s that conform. vLLM's serving benchmark grew an equivalent --goodput argument (its lineage runs to DistServe, the paper that formalized SLO-goodput as the serving objective). MLPerf's "Server" scenario is implicit goodput: results only count while latency bounds hold.

Training-side: Google ships ml-goodput-measurement — a library that instruments a training job and reports the fraction of accelerator time making productive, preserved progress: exactly the training-goodput definition (failures, restarts, and un-checkpointed redo metered out). NVIDIA's resiliency stack and the managed platforms (SageMaker HyperPod, CoreWeave Mission Control) optimize the same quantity, marketed in goodput terms.

The gap

Now look at what operators actually watch. Prometheus, Grafana, DCGM: no goodput concept. The standard serving dashboard shows request rate, token throughput, error rate, GPU "utilization" — all of them lobby metrics that our measurements show staying healthy through a 92% goodput collapse. The tools above are load testers (you run them at a system, occasionally) or training-loop instrumentation. Nobody ships passive, always-on goodput metering of a production endpoint.

That's an odd gap, because the ingredients are trivial: an SLA, a probe, a window.

Closing it in ~100 lines

So we added a goodput meter to our evaluation harness (project 13). Design:

  1. State the SLA — ours: TTFT ≤ 2 s AND E2E ≤ 30 s, printed in every output line. An unstated SLA is how goodput degenerates back into vibes.
  2. Probe synthetically, cheaply, honestly — a small request every N seconds (~32 output tokens), cache-busted with a nonce so TTFT measures prefill+queue, not the prefix cache (the lesson from our phase probe: un-busted TTFT benchmarks measure the cache).
  3. Classify, don't average — each probe either conformed or didn't. Errors and timeouts are non-conforming probes, not a separate chart.
  4. Roll a window, emit three numbers: conformance (goodput fraction), goodput_decode_tps (conforming rate), and budget_burn — how fast the error budget (1 − SLO target) is being consumed. Burn > 1 = alert.
  5. Speak dashboard — JSONL plus a Prometheus textfile (inference_goodput_conformance, inference_error_budget_burn), so the number lands where operators already look.

What it caught in its first minute

We pointed it at our local deployment and injected trouble mid-run:

  • t≈16 s — cold-load caught. First probe TTFT 15.1 s (model loading): non-conforming, burn pegged, alert on. Raw tok/s would have shown nothing wrong — just a slow first request.
  • t≈41 s — load spike caught. Six concurrent heavy requests were injected; the next probe queued behind them: TTFT 13.2 s, and rolling conforming-decode degraded 21.3 → 14 tok/s (batch contention). Again: zero errors, throughput fine — only conformance saw it.
  • Recovery then tracked honestly: conformance climbed back probe by probe as the window flushed, burn falling with it.

One synthetic probe every few seconds, and both real incidents of the demo minute were visible as they happened, in a metric with an alert threshold already attached.

The point

The gap between "we benchmark goodput quarterly" and "goodput is on the pager" is an afternoon of code — the meter is ~100 lines of stdlib. What's actually scarce is the decision: state an SLA, classify against it continuously, alert on conformance and burn instead of raw rate. Benchmark harnesses got there years ago; papers formalized it; the training world has a library. Production serving dashboards are the last place still flying on claimed-and-packed while nobody watches working.

The meter, the harness, the demo data, and the evaluation methodology live in the llm-masters repo (project 13). Related: Claimed, Packed, Working · Why benchmark scores lie · Measuring training goodput.