Batching Is the Parallelism: Measuring LLM Inference on a Commodity GPU

From DRAM to LLMs, same principle

In our memory-bandwidth investigation we found a counterintuitive result: a single CPU core cannot use fast memory. High-bandwidth DRAM only delivered its advantage when a massively parallel consumer — a flood of outstanding requests — kept it busy. The verdict on the same DDR5 chip flipped from "slower than DDR3" to "2.3x faster" depending entirely on how much memory-level parallelism the workload supplied. That, we noted, is the rationale for pairing HBM with GPUs.

This post is that principle at production scale. LLM inference is one of the most important memory-bandwidth-bound workloads running today, and the entire open-source serving stack — vLLM, SGLang, TensorRT-LLM — exists to supply the parallelism that fills a GPU's bandwidth. We move from a simulator to real silicon and measure it directly.

Why decode is memory-bound

A transformer doing autoregressive decode generates one token at a time. To produce each token it must read the entire weight set and the growing KV cache out of GPU memory, but performs comparatively little arithmetic on them. The ratio of work to bytes moved — the arithmetic intensity — is tiny. By the roofline model, that places single-stream decode far on the memory-bound side: the expensive tensor cores sit mostly idle, waiting on HBM.

This is the LLM-scale echo of the single starved CPU core. One request decoding alone cannot saturate the GPU, exactly as one core could not saturate DDR5.

The fix is the same: parallelism. If many sequences decode together in one step, the weight read is amortized across all of them — the bytes are moved once and reused for the whole batch. This is continuous batching, and it is the central trick of every modern inference engine. Batching is to a GPU what the traffic generator was to DRAM in our last study: the source of the parallelism that turns advertised bandwidth into delivered throughput.

Setup

We serve Qwen2.5-1.5B-Instruct (FP16) with vLLM 0.23 on a single NVIDIA T4 — a deliberately modest, inexpensive GPU (16 GB, Turing) on an AWS g4dn.xlarge. The small model and small GPU are the point: they make the memory-bandwidth ceiling visible and the experiment cheap to reproduce, and the qualitative story is hardware-independent.

The benchmark sweeps concurrency — the number of requests in flight — from 1 to 64, holding the workload fixed: ~128-token prompts, exactly 128 output tokens per request (ignore_eos, so every request does identical work and the throughput math is exact), 128 requests per level. We record, per level:

• TTFT — time to first token (prefill latency; what the user feels first)
• TPOT — time per output token (decode speed)
• throughput — output tokens/sec across all concurrent requests
• end-to-end latency (p50/p99)

The harness and a CPU-runnable roofline companion are open source in the llm-masters repository; every number below is reproducible with one command.

Result — batching unlocks the GPU

 output throughput (tok/s) vs concurrency — the weight read, amortized
 conc 1   |█ 68
 conc 2   |███ 132
 conc 4   |█████ 255
 conc 8   |██████████ 474
 conc 16  |█████████████████ 828
 conc 32  |█████████████████████████ 1220
 conc 64  |██████████████████████████████████████ 1855

From a single stream to 64 concurrent requests, output throughput rises 27x — 68 to 1855 tokens/sec — on the exact same GPU. Nothing about the hardware changed. We simply supplied the parallelism, and the idle tensor cores went to work on bytes that were already being moved.

The catch: a throughput–latency frontier

Throughput is only half the story. As concurrency climbs, latency degrades — and the two trade off on a smooth curve:

Two regimes are visible. Below concurrency ~8, batching is nearly free: TPOT barely moves (14.6 → 16.6 ms) while throughput grows almost linearly — we are filling unused bandwidth at no cost. Past the knee at ~concurrency 32, sequences begin competing for the same bandwidth and compute: TPOT rises to 33 ms and doubling concurrency from 32 to 64 buys only 1.5x throughput, not 2x.

That knee is the engineering decision. There is no single "best" setting — only an operating point chosen against a latency budget:

• Interactive / SLO-bound (e.g. TTFT < 100 ms): concurrency ~32 delivers ~1220 tok/s while keeping first-token latency tolerable.
• Batch / throughput-max: push past 64 — the T4 was not yet saturated at 1855 tok/s — and accept multi-second end-to-end latency.

Mapping this frontier, not quoting a single number, is what makes an inference benchmark honest. A vendor's "1855 tokens/sec" is true and useless without the latency it cost.

A production lesson, free of charge

Getting this measurement was not push-button, and the obstacle is worth recording. On the T4, vLLM auto-selected its FlashInfer attention backend, whose paged-prefill kernel crashed the engine (BatchPrefillWithPagedKVCache ... invalid argument). The cause: FlashInfer's kernel and FlashAttention-2 both require compute capability 8.0+, and the T4 is Turing (7.5). The fix was to force a Turing-compatible backend, --attention-backend TRITON_ATTN.

This is the kind of detail that separates a slide from a deployment. Auto-selected defaults are tuned for current-generation datacenter GPUs; run on the commodity or older hardware that real budgets often dictate, and the defaults can fail silently or loudly. Knowing which layer failed — and that it was an architecture/kernel mismatch, not a model or config error — is the difference between a five-minute fix and a lost afternoon.

Caveats

• One small model, one small GPU. Absolute numbers scale with model size, GPU, and precision; the shape of the curve — memory-bound decode, batching unlocking throughput, a latency knee — is what generalizes.
• The T4 was not saturated. Throughput was still climbing at concurrency 64; the true peak and a sharper knee lie further right.
• Synthetic, uniform workload. Fixed-length prompts and outputs isolate the batching effect cleanly. Real traffic has variable lengths and shared prefixes (where prefix caching changes the picture) — deliberately held for follow-ups.
• FP16, default KV cache. Quantizing weights or the KV cache moves the bandwidth ceiling and is a separate axis we have not yet varied.

Where this goes

This is the first measurement in a broader line of work on LLM systems — how the major training and inference frameworks (PyTorch, Megatron-LM, NeMo; vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, Red Hat Inference Server) behave, and how they stress the compute, memory, network, and storage layers beneath them. The same first-principles, measure-don't-guess approach we have applied to caches, energy, and DRAM bandwidth, now turned on the systems that serve large models.

Planned next: extend concurrency to find the T4's saturation point; enable prefix caching on a shared-prompt workload and attribute the gain; and run the identical sweep through SGLang and TensorRT-LLM for an honest, apples-to-apples bake-off. Update: the SGLang comparison and the shared-prefix caching gain are now measured in vLLM vs SGLang: An Honest Bake-Off (TensorRT-LLM still to come). The open companion code lives at codeberg.org/srinathv/llm-masters.

If your team is choosing or tuning an inference stack — or trying to understand why your tokens-per-dollar isn't what the benchmarks promised — this is work we do.

References

1. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023. arxiv.org/abs/2309.06180
2. S. Williams, A. Waterman, and D. Patterson. "Roofline: an insightful visual performance model for multicore architectures." Communications of the ACM, 52(4):65–76, 2009. doi:10.1145/1498765.1498785
3. vLLM documentation. docs.vllm.ai