A frontier model posts 85–90% on the standardized benchmark. You deploy it on the same class of task and it produces consistent output less than a quarter of the time. Nothing broke. The model is the one from the leaderboard. So where did the other sixty points go?
That gap is the most expensive number in applied AI right now, because ROI is computed against the leaderboard and paid against production. Two independent lines of work — one academic, one from the buyer's side — arrive at the same answer from opposite directions, and together they tell you exactly what a benchmark number is hiding.
Diagnosis 1 — the benchmark is often badly built
Stanford's BetterBench (Reuel et al., 2024) did the unglamorous thing: it defined 46 quality criteria across a benchmark's whole lifecycle — design, implementation, documentation, maintenance — and graded 24 widely-used benchmarks against them. The result is a quiet indictment:
- Only 3 of 24 ship an easy-to-run script that reproduces their own headline results.
- 14 of 24 never report repeated runs or any statistical significance — the number is a single point estimate with no error bar.
- Benchmarks of wildly different quality are cited side by side as equals. On BetterBench's own 0–15 scale, MMLU scored ~5.5 and GPQA ~11.0 — yet both appear in the same model cards with no asterisk.
The weakest category across the board was implementation/reproducibility (average 6.2/15). A benchmark you cannot re-run on stated hardware and get the same curve is, in the language this practice runs on, an anecdote wearing a lab coat. Before you trust a leaderboard, ask whether the leaderboard itself would pass review.
Diagnosis 2 — even a good benchmark's score is inflated
Now assume the benchmark is well-built. The score can still lie, and this is the buyer's-side complaint (Bajarin, Forbes, 2026): the near-term problem with AI is not a future safety story, it is present-day unreliability, and the leaderboard actively hides it. Two mechanisms:
- Contamination. Test questions have circulated online for years, so a model often recognizes the answer instead of solving the problem. It knows answers, not the solutions to the problem in front of you. The score measures recall of a public set, and your task is not in that set.
- Construct validity. Even uncontaminated, a benchmark measures a proxy. High accuracy on curated single-shot questions does not predict the thing production actually needs: consistency across runs, no silent errors (confident wrong answers that poison everything downstream), and graceful behavior under retries and load.
The buyer's conclusion is the one that should reorganize how we report: reliability is the metric that counts, not the maximum benchmark percentage. The good news buried in it is that reliability is engineering-solvable — verification and reasoning layers, not a bigger pretrain.
The two diagnoses meet
They come from different worlds and land on the same two words: contamination and construct validity. One says the test is built badly; the other says even a well-built test's score overstates real-world reliability. If you only fix one, the gap persists.
I hit this directly in my own agent bake-off-style harness. Running a strong 32B coder agent against a five-task verified suite, it scored 100% on every scenario — every concurrency, every rep. That is a clean pipeline signal and a useless comparison signal: a suite everyone one-shots has no discriminating power. The point estimate said "perfect"; it actually said "these tasks are too easy to tell two agents apart." The honest number wasn't the accuracy — it was the tail (p99, error rate under soak) and the admission that the suite needed harder, multi-file, private tasks.
What to demand instead
A benchmark result is evidence only if it clears both bars — well-built and validated against real use. Concretely, before you trust a number:
| Demand | Because |
|---|---|
| A one-command replication on stated hardware/software | Reproducibility is the floor; only 1-in-8 benchmarks clears it |
| Confidence intervals, not point estimates | "100% (95% CI 75–100%, n=15)" says what a bare "100%" cannot |
| Reliability + tails, not peak accuracy | Consistency and p99 are the product; the average is marketing |
| Contamination controls — canary IDs, a train-on-test probe, private held-out tasks | A recognized answer is not a solved problem |
| Discriminating difficulty | If everyone scores ~100%, the benchmark has stopped measuring |
| The environment pinned — model revision, dtype, engine version, command | A benchmark without its environment is an anecdote |
None of this makes benchmarks worthless — it makes them legible. A score with a runnable script, an error bar, a contamination story, and a reliability tail is a genuine instrument. A single percentage on a leaderboard is a headline. The gap in the chart above is just the distance between the two, and it closes the moment you stop reporting the headline as if it were the instrument.
Rubric and worked example, if you want to grade your own: the BetterBench checklist and a scored self-assessment of the bake-off harness live in the llm-masters benchmarks repo. Related: Benchmarks, from MLPerf to Agents and the order-of-magnitude sanity check.