What the KV Cache Actually Is — and Why It's the Binding Constraint

The map: what each tool is organized around

Before the deep dive, the lay of the land. Training frameworks are organized around throughput of gradient computation under collective-communication constraints; serving engines are organized around KV-cache memory management and the prefill (compute-bound) vs. decode (bandwidth-bound) asymmetry. Concretely:

• Megatron / NeMo decide how to shard a training step across a cluster.
• vLLM / TensorRT-LLM / SGLang decide how to pack many concurrent sequences into finite HBM.
• NVIDIA Dynamo decides how to place and route those requests across a fleet.
• Red Hat packages vLLM for enterprise.

The serving half of that map — packing sequences into finite HBM — is, in one word, a KV-cache problem. So let's go deep on the KV cache, because almost every serving optimization turns out to be a KV-cache optimization in disguise.

What the KV cache actually is

During autoregressive decoding, attention at step t needs the keys and values of all prior tokens. Recomputing them every step is O(n²) wasted work, so you cache them. Size:

KV bytes = 2 (K,V) × layers × kv_heads × head_dim × seq_len × batch × dtype_bytes

The 2 × layers × kv_heads × head_dim × dtype part is per token, per sequence — a fixed cost you pay for every token you keep alive. That linearity is the whole problem.

KV cache vs. weights

Weights are fixed once loaded. KV cache is dynamic, growing with concurrency × context.

For a 70B model in FP16: weights ≈ 140 GB (static). KV per token might be ~few hundred KB; at 128K context × dozens of concurrent requests, KV can rival or exceed the weights. The shift over the last couple years: at long context / high batch, KV cache, not weights, is what caps your batch size — and batch size is what determines throughput. You can fit the model and still OOM on KV.

KV cache vs. activations

Activations (intermediate tensors in the forward pass) are transient — allocated and freed within a step. In training they're a huge cost because you retain them for the backward pass (hence activation recomputation). In inference there's no backward pass, so activations are small and short-lived. KV cache is the inference analog of "state you must keep around," but it persists across the entire generation, not just one step.

KV cache vs. compute (the prefill/decode split)

This is the asymmetry everything is built around:

Decode is memory-bandwidth-bound precisely because of the KV cache: each token requires reading all of it back. More context → more bytes moved per token → slower decode. This is why Dynamo disaggregates the two (different hardware sweet spots) and why decode throughput tracks HBM bandwidth, not FLOPs.

What the engines do about it

• PagedAttention (vLLM) — fragmentation. Naive contiguous KV allocation wastes 60–80% to internal/external fragmentation and over-reservation. Paging into fixed blocks gets utilization near 100%, so you fit more concurrent sequences → bigger batch → more throughput. Also enables prefix sharing (copy-on-write).
• RadixAttention (SGLang) — redundancy. Shared prefixes (system prompts, few-shot examples, multi-turn history) get deduplicated via a radix tree, so identical KV isn't stored or recomputed N times.
• Quantized / lower-precision KV (FP8, INT8 KV cache) — halve the bytes-per-token, directly relieving both the capacity cap and the decode bandwidth bill.
• GQA/MQA (model-architecture, not engine) — fewer kv_heads shrinks the KV cache at the source; this is why modern models use grouped-query attention.
• KV offload / tiering (Dynamo, others) — spill cold KV to CPU DRAM or NVMe for very long context, trading PCIe/network latency for HBM capacity.
• Disaggregated KV transfer (Dynamo) — when prefill and decode run on different nodes, the computed KV must be shipped over RDMA from the prefill pool to the decode pool; this turns KV into a network concern too.

The one-line mental model

Weights set whether the model fits. KV cache sets how many requests you can run at once and how fast each decodes. On modern long-context, high-concurrency serving, the KV cache — not the weights, not the FLOPs — is usually the binding constraint, and almost every serving optimization is ultimately a KV-cache memory or bandwidth optimization.

See it for yourself

Want the roofline view of decode — arithmetic intensity vs. your GPU's HBM bandwidth — to see exactly where you go memory-bound? That's the lens behind my batching measurements and the vLLM-vs-SGLang bake-off, and the acronyms here (PagedAttention, RadixAttention, GQA/MQA, prefill/decode) are defined in the parallelism glossary. The runnable roofline and serving harness are open at codeberg.org/srinathv/llm-masters. If your serving is OOMing on KV or decoding too slowly, that's the work I do.

References

1. PagedAttention (vLLM): https://arxiv.org/abs/2309.06180
2. RadixAttention (SGLang): https://arxiv.org/abs/2312.07104
3. GQA (grouped-query attention): https://arxiv.org/abs/2305.13245
4. Companion posts: batching on a T4 · vLLM vs SGLang · parallelism glossary