From MLPerf to Agents per Megawatt: How We Benchmark LLM Systems Now

Three questions wearing one coat

My last two posts measured an inference engine: vLLM on a T4, then a vLLM-vs-SGLang bake-off. Both answered a narrow, precise question — how many tokens per second, at what latency, for one request at a time? That's one kind of "performance," and it's the kind most benchmarks report.

But "how fast is this LLM system?" actually hides three different questions, and teams routinely conflate them:

A model can top a capability leaderboard and be ruinous to serve. A server can post huge tokens/sec on a chatbot trace and collapse under agent traffic. Pick the wrong layer and you optimize the wrong thing.

MLPerf: the part that resists marketing

For raw system performance, MLPerf (run by the MLCommons consortium) is the closest thing to an honest standard — precisely because it constrains what vendors otherwise game. Three things worth knowing before you read any MLPerf headline:

• Divisions. Closed fixes the model and a quality target, so hardware and software are actually comparable. Open allows anything — great for showcasing an optimization, useless for apples-to-apples. The division determines whether a number means anything.
• Scenarios. Offline reports raw throughput. Server reports throughput subject to latency SLOs — time-to-first-token and time-per-output-token, with 99th-percentile thresholds. That SLO-bound framing is the same throughput-versus-latency frontier my bake-off mapped, just standardized and audited.
• The current bar (Inference v5.1, 2025). The suite now spans small (Llama-3.1-8B), long-context (Llama-3.1-405B), and reasoning (DeepSeek-R1) models, each with explicit TTFT/TPOT targets, across 27 submitters and the latest silicon (NVIDIA GB300, AMD MI355X, Intel Arc Pro B60). The SLO targets are the spec: "fast enough" gets defined, and you maximize throughput under it.

MLPerf is the vocabulary. But it was built around the single-request world — and that world is changing.

Why agents broke the benchmark

An agent is not a prompt. A coding agent solving one task can make 200+ sequential model calls, dragging a context that grows past 100,000 tokens as tool outputs and history accumulate. Three consequences:

1. Latency and cost compound. A 50-millisecond per-token difference is noise on one reply and decisive across hundreds of turns.
2. Prefix reuse stops being optional. Successive turns share enormous prefixes; whether the system reuses that cached work instead of recomputing it is the difference between viable and bankrupt.
3. Production tricks dominate. KV-cache reuse, disaggregated prefill/decode, and speculative decoding matter far more on agent traffic than on chat.

A benchmark that fires short, unique, single-turn prompts measures none of this — and will mis-rank systems for agentic use. (My own bake-off is a single-turn case: a follow-up with a long shared prefix showed caching it is worth ~3.5× throughput for both engines, yet SGLang's RadixAttention still didn't beat vLLM's prefix caching. The multi-turn agentic regime — where reuse compounds across requests — is where that ranking could still shift.)

AA-AgentPerf: benchmarking the agent era

Artificial Analysis's AA-AgentPerf is the first hardware benchmark built for agentic inference, and it makes several deliberate breaks from the old model:

• Real trajectories. It replays actual long-context coding-agent runs — up to ~200 turns and >100K-token sequences — not synthetic prompts.
• The right question. Not "tokens/sec" but how many concurrent agents a deployment can serve while meeting per-agent SLOs (first-token and output speed).
• A new headline metric: agents per megawatt. Efficiency and energy become the currency — which is what actually bounds a datacenter. That single choice reframes "performance" from a speed number into a throughput-per-watt number.
• It permits the real optimizations. KV-cache reuse, disaggregated prefill/decode, speculative decoding — because that's what production runs.

The first results cover DeepSeek V4 Pro across NVIDIA Blackwell, Hopper, and AMD.

The bridge: it's the same physics, restacked

Here's the part that ties this back to everything else I've written: AA-AgentPerf is not new physics — it's the compute-layer story applied to a multi-turn unit of work.

• Agent latency is the per-turn primitive, multiplied. It's roughly the sum over turns of (time-to-first-token + time-per-token × tokens). The single-turn frontier my bake-off measured is that primitive; an agent stacks it hundreds of times, so small per-token wins compound into large ones.
• Prefix/KV reuse is the agent lever — the very thing SGLang's RadixAttention is built for. My follow-up bake-off showed a single long shared prefix is worth ~3.5× throughput for both engines (vLLM's prefix caching matched it); agents, which reuse across many turns, are where that capability compounds.
• Disaggregated prefill/decode — a long agent prefill followed by token-by- token decode is the textbook case for splitting the two phases across hardware.
• Memory bandwidth and capacity — a 100K-token KV cache is the memory-bound-decode regime at its most extreme.

A useful way to hold it: a ladder of benchmarks, each rung adding realism.

 microbenchmark   -> one op / one collective
 my sweep harness -> one engine's TTFT/TPOT frontier
 MLPerf (Server)  -> standardized single-request serving, SLO-bound
 AA-AgentPerf     -> multi-turn agent trajectories, agents per megawatt

A number from one rung does not transfer to another. The consultant's job is to run the rung that matches the client's real workload.

How to read a benchmark without being fooled

• Match the benchmark to the workload. Chatbot ≠ agent ≠ batch. For agentic use, look at AA-AgentPerf-style multi-turn numbers, not single-turn tokens/sec.
• Check the division and the config. Closed or open? Which optimizations were allowed? What SLO was enforced? A number without its constraints is an ad.
• Separate capability from performance. SWE-bench tells you whether an agent solves the task; AA-AgentPerf tells you how economically you can run it. Different axes; you need both.
• The benchmark that matters is yours — your model, your hardware, your trajectory distribution. The public suites are the shared language; the decision needs your own measurement.

That last point is the whole job. If your team is choosing hardware or an inference stack for agent workloads — where the wrong benchmark can cost an order of magnitude — that's the work I do. The harness and raw data behind my own measurements are open at codeberg.org/srinathv/llm-masters.

References

1. MLPerf Inference v5.1 (MLCommons). mlcommons.org · datacenter results
2. AA-AgentPerf (Artificial Analysis) — first results · methodology
3. Agent capability benchmarks: SWE-bench · GAIA · τ-bench
4. My companion measurements: vLLM on T4 · vLLM vs SGLang