vLLM vs SGLang: An Honest Bake-Off (and Why the Numbers Aren't the Point)

The question everyone asks

My last post measured one inference engine — vLLM — and watched batching unlock a commodity GPU. The immediate follow-up question, from every team standing up self-hosted inference, is: vLLM or SGLang? Both are mature open-source engines; both implement paged KV caches and continuous batching. So I ran the bake-off — properly.

"Properly" is the whole point of this practice: same GPU, same model, same workload, same harness — only the engine changes. Anything less is an anecdote.

First finding: it wouldn't even start on the cheap GPU

The plan was to reuse the T4 from the last post, for a clean comparison against those published vLLM numbers. SGLang refused to run on it. The T4 is Turing (compute capability 7.5), and SGLang's compiled kernels ship no image for it:

RuntimeError: RMSNorm failed with error code no kernel image is available for execution on the device

No flag fixes that — the kernel simply isn't built for the architecture. vLLM, by contrast, does run on the T4 (with --attention-backend TRITON_ATTN, as the last post documented). That asymmetry is the first real result: vLLM has the broader hardware reach. If your fleet includes older or commodity GPUs, that alone may decide the question before throughput enters the conversation.

So the throughput bake-off moved to an NVIDIA L4 (Ada, sm89, 24 GB), where both engines run. Both were configured identically and conservatively — no CUDA graphs, no torch.compile (vLLM --enforce-eager, SGLang --disable-cuda-graph) — so the comparison is controlled. Absolute numbers would be higher with those features on; what matters here is that both engines were handicapped equally.

The throughput result: a dead heat

Model Qwen2.5-1.5B-Instruct, FP16, 128-token prompts, 128 output tokens, concurrency swept 1→64, 128 requests per level. Output throughput, tokens/sec:

 output throughput (tok/s) at concurrency 64
 vLLM    |██████████████████████████████████████ 2068
 SGLang  |███████████████████████████████████ 1920

Across the entire frontier the two engines are within single-digit percent — usually within noise. They are effectively tied through concurrency 32. At the top of the sweep vLLM pulls ahead by 7.7% (2068 vs 1920 tok/s); at the bottom SGLang has slightly lower first-token latency (TTFT p50 54.6 vs 55.9 ms) and end-to-end latency. Neither engine "wins." Both produced zero failed requests at every level.

That is a genuinely useful — if anticlimactic — finding. Two mature engines implementing the same core ideas (paged KV cache, continuous batching) converge on near-identical performance for plain single-shot generation. If someone shows you a benchmark where one of these crushes the other, ask what workload and config produced it — because on the vanilla case, there's almost nothing in it.

So what does separate them?

Two things this bake-off makes concrete, neither of which is a single tok/s number:

1. Hardware portability. vLLM ran on Turing; SGLang required Ampere-or-newer. For a deployment constrained to specific or older GPUs, engine choice is a hardware-compatibility question first and a performance question second.

2. Workload shape — and a correction. SGLang's headline feature is RadixAttention: automatic reuse of shared prompt prefixes across requests.

Correction (2026-06-30). An earlier version of this post said the workload used unique prompts, so RadixAttention "had nothing to reuse." That was wrong. The harness actually sent the identical prompt to every request — a fully shared prefix — with prefix caching on by default in both engines. So both engines could reuse; they simply had little to gain here, because the shared prompt was short (~128 tokens) next to a 128-token decode, and decode (bandwidth-bound, un-cacheable) dominates the wall clock. The neck-and-neck result stands — the reason is "reuse barely mattered at these sizes," not "reuse wasn't possible."

So I ran the controlled version: a long (~1K-token) shared prefix + a tiny unique suffix and short output — so prefill/reuse dominate — versus genuinely unique prompts (distinct from token 0), matched in total length. Same L4, both engines, prefix caching / RadixAttention on.

The follow-up result (2026-06-30)

Reuse is decisive here — and it settles the RadixAttention question. Going unique → shared at concurrency 64:

Two findings. (1) With a long shared prefix, caching it is worth ~3.5× throughput and ~5× lower TTFT for both engines — the effect the short-prompt first run couldn't show. (2) But RadixAttention did not beat prefix caching: head-to-head on the shared workload vLLM is slightly ahead (+16% tok/s, lower TTFT at conc 64). My earlier hedge — "the result could look very different" — is answered with data: on a single shared prefix, it doesn't.

The honest caveat: this shares one prefix across all requests, which both caches handle equally well. RadixAttention's structural edge is a tree over many partially-overlapping prefixes (branching few-shot, tree-of-thought) — untested here, and the one place it might still differentiate.

The takeaway for choosing an engine

Don't pick an inference engine off a leaderboard throughput number. On the common case they're close enough that the decision should turn on:

• Your hardware — what GPUs must this run on?
• Your workload shape — lots of shared prefixes? structured generation? Then benchmark SGLang on that. Plain chat at scale? Either is fine; tune the one your team operates best.
• Operations — support, packaging, update cadence, and the surrounding ecosystem usually matter more than 7% of throughput.

The benchmark that matters is the one run on your model, your hardware, and your request distribution. This practice exists to run exactly those.

Caveats

• Both engines in eager / no-CUDA-graph config; absolute throughput rises with graphs + compile enabled. The relative comparison is what's controlled.
• One small model (1.5B), one GPU (L4), one workload — identical prompts (short, ~128 tok) with a 128-token decode (see the correction above).
• Prefix caching / RadixAttention were on, but the short shared prompt gave them little to bite on; the long-prefix / short-decode follow-up above is where reuse actually shows (~3.5×) — so this section's numbers are a baseline, not either engine's best case.

The harness, both raw result sets, and the reproduction commands are open at codeberg.org/srinathv/llm-masters (projects/02-inference-serving/bakeoff/). If your team is choosing or tuning an inference stack, that's work I do.

References

1. vLLM — PagedAttention. arxiv.org/abs/2309.06180 · docs.vllm.ai
2. SGLang — RadixAttention. arxiv.org/abs/2312.07104 · docs.sglang.ai
3. Companion measurements & harness: codeberg.org/srinathv/llm-masters