The Salad of Acronyms: A Working Glossary of LLM Inference Parallelism

Why a glossary, and why grouped

Spend a week reading LLM-serving docs and you drown in initialisms — DP, TP, PP, EP, SP, CP, KV, MoE, TTFT, TPOT, ITL. NeuReality's nicely-named guide to the salad of acronyms prompted this entry; I've organized it by the problem each term solves, because the acronyms aren't independent — they're all answers to one question:

What do you do when a model won't fit, or won't serve fast enough, on a single GPU?

Keep that frame and the salad turns into a menu.

1. The parallelism axes — splitting the model across GPUs

When one GPU isn't enough, you split the work. How you split determines what must be communicated — the interconnect is the hidden cost of every axis below.

• DP — Data Parallelism. Replicate the whole model on each GPU; each handles different requests/batches. Communicates: nothing during the forward pass (gradients are sync'd only in training). Cheapest to reason about; doesn't help if the model itself doesn't fit.
• TP — Tensor Parallelism. Shard individual weight matrices across GPUs; each computes a slice of every layer. Communicates: partial activations via collectives every layer — the chattiest axis. Keep it inside one node on NVLink.
• PP — Pipeline Parallelism. Split the model by layers into stages on different GPUs. Communicates: activations at stage boundaries (point-to-point); introduces pipeline "bubbles" (idle stages) you minimize with micro-batching.
• EP — Expert Parallelism. For MoE models: place different experts on different GPUs and route each token's activations to the GPU holding the expert it needs. Communicates: token routing (all-to-all) — bursty and topology- sensitive.
• SP — Sequence Parallelism / CP — Context Parallelism. Split along the sequence dimension; paired with TP to cut activation memory and to handle very long contexts. Communicates: along the sequence at attention boundaries.
• FSDP / ZeRO — sharded data parallelism. A DP variant that shards the parameters, gradients, and optimizer state across GPUs instead of replicating them, trading extra communication (gather/scatter each step) for fitting a bigger model. The bridge between "replicate" (DP) and "shard" (TP/PP).

The rule of thumb: these compose ("3D parallelism" = DP×TP×PP). Map the chattiest axis (TP) onto the fastest link (NVLink); let the least chatty (PP, DP) cross slower node boundaries. Get the mapping wrong and the network, not the GPU, sets your speed.

2. The two phases — and the memory that haunts both

A single request runs in two very different regimes:

• Prefill. The prompt is processed all at once — big parallel matmuls, compute-bound, throughput-focused. Sets the TTFT.
• Decode. Output tokens are generated one at a time, each reading the whole model + cache for little arithmetic — memory-bandwidth-bound, latency-sensitive. Sets the TPOT. (Why decode is memory-bound.)
• KV cache. The per-request memory of all prior tokens' keys/values, so the model needn't recompute them each step. Grows with context length × concurrent requests — the dominant, dynamic memory cost of serving, and the thing every optimization below is fighting.
• PagedAttention. Manages the KV cache in fixed-size blocks (like virtual memory) to stop fragmentation — vLLM's core trick.
• Prefix caching / RadixAttention. Reuse the KV of shared prompt prefixes across requests instead of recomputing them — decisive for system-prompt-heavy and agentic workloads (SGLang's headline feature; my bake-off measured it — a long shared prefix bought ~3.5× throughput for both engines, and vLLM's prefix caching matched RadixAttention).

A slight tangent — architecture & serving patterns built on the above

The next three aren't phases or KV-cache mechanics; they're adjacent techniques that ride on them, grouped here because the acronyms tend to travel together.

• MoE — Mixture of Experts. An architecture where only a fraction of the model's "experts" activate per token — more parameters at roughly constant per-token compute. The reason EP exists.
• Disaggregated prefill/decode. Run the compute-bound prefill and memory-bound decode on different GPU pools, each suited to its phase — a datacenter-scale serving pattern.
• Speculative decoding. A small draft model proposes several tokens; the big model verifies them in one pass — more tokens per expensive forward step.

3. The metrics — what "fast" actually means

You can't compare any of the above without agreeing on the yardstick:

• TTFT — Time To First Token. Prefill latency; what the user feels first.
• TPOT / ITL — Time Per Output Token / Inter-Token Latency. Decode speed; the smoothness of streaming.
• Throughput. Tokens/sec or requests/sec across all concurrent users — the fleet economics.
• Goodput. Throughput that meets a latency SLO — the number that actually matters, and what MLPerf's server scenario measures.
• busbw — bus bandwidth. The interconnect-aware way to report collective speed (accounts for the ring algorithm), so it's comparable across GPU counts.

The throughput↔latency frontier ties them together: raising concurrency lifts throughput and degrades latency. Every knob above moves you along that curve — mapping it is the whole job.

The one-line map

Every acronym here is a response to a shortage of one resource:

That's the salad turned into a meal. The companion measurements and harness behind the links above are open at codeberg.org/srinathv/llm-masters; if your team is sizing or tuning a serving stack, that's the work I do.

References

1. NeuReality — LLM Inference Parallelism: A Guide to the Salad of Acronyms: https://www.neureality.ai/llm-inference-parallelism-a-guide-to-the-salad-of-acronyms/
2. PagedAttention (vLLM): https://arxiv.org/abs/2309.06180 · RadixAttention (SGLang): https://arxiv.org/abs/2312.07104
3. Megatron-LM / 3D parallelism: https://arxiv.org/abs/2104.04473
4. Companion posts: batching on a T4 · vLLM vs SGLang · MLPerf → agents