A compiler and a language model both take text and return text. You can put one of them inside a build system and not the other. The usual explanation is determinism, and that explanation is half right in a way that hides the more important half.
Start with the determinism, because a served LLM has less of it than most people assume.
The non-determinism stack
| source | mechanism | can you pin it? |
|---|---|---|
| Sampling | temperature > 0 draws from a distribution |
yes — temperature 0 |
| Batch composition | floating-point reduction order depends on the batch's shape and contents, and under continuous batching your request is batched with other people's traffic | no, on a shared endpoint |
| Kernel selection | split-K GEMMs, atomics, attention-kernel variants chosen by shape and occupancy | partly, with deterministic-algorithm flags and a speed cost |
| MoE routing | expert capacity limits make which expert sees a token depend on the batch | no — the topology is data-dependent |
| Adaptive thinking | the model chooses reasoning depth per turn; an effort setting biases that choice rather than fixing it | no |
| Server-side state | prefix-cache hits, compaction triggers, a refusal falling back to a different model, an alias quietly resolving to a new version | no |
| Engine | the same weights served by a different runtime (vLLM, SGLang, TensorRT-LLM) or a different version of one: different kernels, fusions, attention and MoE implementations | yes, if you pin engine, version and build |
| Environment | driver, hardware generation | yes, if you host it |
Three rows deserve emphasis.
Temperature zero is not determinism on a served endpoint. Greedy decoding removes the sampling randomness and leaves the numerical randomness untouched. Reduction order changes with batch shape, so the same prompt can produce different logits depending on who else was in flight. That is the same continuous batching that makes serving economical — on a single H100 we measured 21,550 tok/s at batch 256 against 136 at batch 1 — so this is not a bug to be fixed. Throughput and determinism are in tension by construction.
Adaptive thinking is a newer source and a stranger one. On current models the amount of reasoning is chosen by the model per turn; the fixed thinking-token budget has been retired in favour of it. So two byte-identical requests may not perform the same quantity of computation, never mind produce the same bytes. An effort level shifts the distribution of that choice. It does not pin it.
Engine variance is its own axis, separate from model variance. "Which model" and "which engine" are different questions, and the second is usually answered by default. The same weights served by two engines run different kernels, with different fusions and reduction orders, so their logits differ in the last bits; under greedy decoding a near-tie between two tokens can flip, and every token after it follows a different path. We have not yet measured that output divergence ourselves (the probe that would is written, test 1 below, and has not been run across engines). What we have measured is how large the engine axis is on everything else. In our bake-off on AMD and NVIDIA, with the weights fixed:
- swapping vLLM for SGLang moved throughput by −15 % to +24 %, the sign depending on the model and the workload;
- whether a model served at all depended on the engine build: one model served on vLLM for both vendors and on SGLang's CUDA build, but not on SGLang's ROCm build; another served on none of the engines we tried, on either vendor;
- a vLLM version upgrade moved throughput by at most 5 %, and run-to-run noise was 0.3 % at the median.
So an "engine version" footnote understates it. Name the engine, its version and its build (vendor backend) alongside the model ID, or the result does not say what produced it.
The reframe: reproducibility is a ladder
Because bitwise reproducibility is unavailable, the useful move is to stop treating reproducibility as a boolean and name which rung you are claiming.
| rung | the claim | how you test it |
|---|---|---|
| R0 bitwise | identical bytes out | hash the output. Reachable only self-hosted, batch 1, everything pinned |
| R1 token-identical | same text, numerics may differ | string equality across N runs |
| R2 semantically equivalent | different text, same meaning | a verifier — tests pass, schema validates, the number matches |
| R3 distributionally stable | mean ± confidence interval, stable across runs and across days | N samples, and report the interval |
| R4 rank-stable | A beats B consistently even as absolute numbers drift | paired A/B, sign test |
Here is the part that makes this practical rather than depressing: almost every engineering decision needs only R3 or R4. "Did this change help?" is R4. "Is this model good enough to ship?" is R3. Chasing R0 against a hosted endpoint burns weeks and buys nothing. Chasing R4 is the entire job.
The five tests worth writing
1. A determinism probe — measure your non-determinism instead of assuming it. Send the
identical request N times, hash the outputs, report distinct / N. Then do it again at a
different batch size. If the distinct count rises with batch size you have demonstrated
batch-composition non-determinism on your own stack rather than citing a blog post. This is
thirty lines of code and essentially nobody runs it, which means most teams do not know
their own noise floor.
2. A variance budget, established before any claim. Run the same configuration twice and measure the spread. Any difference smaller than that spread is noise. In our serving bake-off three repeats per point put the spread at 0.3 % at the median and 5.6 % at worst, which is what let a 24 % engine difference count as one. We learned this the expensive way on GPU kernels — a phantom 7% "speedup" that turned out to be thermal drift — and LLM work needs it more, because the noise floor is higher and invisible.
3. Verifier-based assertions, never string equality. Assert that the tests pass, the JSON validates, the invariant holds. A test that string-matches model output is testing the tokenizer's mood.
4. Interleave A and B; never run all of A then all of B. Sequential blocks confound your comparison with cache state, server load and model-version drift. Round-robin the arms — the same discipline that stops you measuring cold-versus-warm on a GPU.
5. Record the non-reproducible surface in the artifact. Hardware, software versions, workload — and for an LLM, four more: the resolved model ID (never an alias), the serving engine with its version and build, the effort level, and whether thinking was enabled. A run without those is not reproducible even in principle, and most published LLM benchmarks omit most of them.
Why it can't be your compiler
A compiler's contract rests on three properties. A language model breaks all three, but not equally.
Determinism. Build systems are content-addressed: hash(input) → hash(output). Caching,
incremental builds, reproducible builds and binary diffing all stand on that equation. A
non-deterministic compiler does not merely inconvenience the cache; it invalidates the whole
incremental model.
Soundness — and this is the disqualifying one. A compiler either produces a correct translation or raises an error. Its failure mode is refusal. A model's failure mode is plausible wrong output: it interpolates, and it has no internal representation of "I cannot translate this." A compiler that silently miscompiled one function in a thousand would be worse than useless, because you would never find the one. You would far rather it crash.
Semantic preservation by construction. A compiler's passes are rule-governed transformations you can reason about and increasingly prove. Nothing in a transformer's architecture preserves program semantics; it is pattern completion over tokens that happen to be code.
Determinism is what gets cited. Soundness is what actually settles it — you could pin a model to bitwise-identical output and it would still be unfit, because deterministic and wrong is still wrong.
Where a model does belong in a toolchain
Behind a verifier. The model proposes, a deterministic checker decides. Superoptimizers work exactly this way, and so does translation validation: generate a candidate by whatever means, then prove or test the candidate equivalent before you accept it.
The composite system then inherits the verifier's soundness rather than the generator's. Trust moves out of the model and into the checker — which is the same structure as verifier-gated routing in an agent loop, where the model's output is a candidate rather than an answer.
That is also the resolution of the ladder. The reproducibility you cannot get from the model, you get from the verifier you wrap around it. Which tells you where to spend your engineering effort: not on making the generator deterministic, but on making the checker cheap, fast and total.
The compression: a served LLM is non-deterministic in ways temperature zero does not touch, and adaptive thinking means even the amount of computation is now the model's choice — so stop asking whether output is reproducible and start naming which rung you are claiming. R4 rank-stability answers almost every real question, R3 answers the rest, and both are cheap to establish once you have measured your own noise floor. The model cannot be your compiler not because it is unreliable but because it is unsound: it has no way to tell you it does not know, and no toolchain can be built on a component like that. Put it behind a verifier and the problem becomes tractable — the checker supplies the guarantee the generator never will. If you are trying to work out which rung your benchmarks are actually on, and whether the improvement you just measured is bigger than your noise, that's the work we do.
Related: Why benchmark scores lie · Claimed, packed, working · Metering goodput · The agent loop, pedantically · The wire between you and the GPU · From MLPerf to agents per megawatt.
Revised 2026-10-10: engine variance split out of the Environment row as its own source, with measured speed and support evidence from post 53; the variance-budget and record-the-surface tests updated to match.