Start simple, then follow the cost

If you read one good beginner explainer — say GeeksforGeeks' What are Diffusion Models? — you come away with the right mental picture and one nagging caveat. The picture: a diffusion model "creates data like images or audio by starting from random noise and gradually refining it into meaningful output." The caveat, listed plainly under limitations: slow sampling.

That caveat is the whole systems story. This piece starts from the basics that primer covers, then does the part it doesn't: a serious review of how the field turned "slow sampling" from a fatal flaw into a solved-enough problem — and how diffusion is now scaling up on the same infrastructure as large language models.

The thread throughout is the one this practice keeps pulling: an autoregressive model pays per token; a diffusion model pays per denoising step. Generation is a loop of full forward passes, and everything below is about making that loop shorter or each pass cheaper.

Part 1 — The basics

A diffusion model defines two processes over a data point x₀ (an image, or — as we'll see — a compressed latent):

Forward (fixed, not learned). Add a little Gaussian noise at each of T timesteps until the data becomes pure noise. Each step depends only on the last (a Markov chain), governed by a variance schedule (βₜ) — and thanks to the cumulative products (ᾱₜ), you can jump straight to any noise level t in one shot rather than simulating the whole chain.
Reverse (learned). Train a network — classically a U-Net — to undo one step of that corruption. Start from pure noise and apply the denoiser repeatedly to walk back to a clean sample.

The move that made this practical to train (Ho et al., DDPM, 2020) is the objective it reduces to: the network simply predicts the noise that was added, under a mean-squared-error loss —

L(θ) = E[ || ε − ε_θ(xₜ, t) ||² ]

— no adversarial game, no likelihood gymnastics. That regression-style stability (no GAN mode collapse) is a big reason diffusion took over high-fidelity image synthesis.

Three equivalent views are worth holding at once, because different accelerators live in each:

Denoising (DDPM): predict the noise ε each step.
Score-based: learn the score ∇ₓ log p(x), the direction toward more probable data (Song & Ermon).
Continuous ODE/SDE: both are discretizations of a differential equation (Song et al., 2021). This is the view that makes "use a better solver to take fewer steps" a theorem instead of a trick.

Conditioning (text-to-image, class-to-image) rides on classifier-free guidance (Ho & Salimans): train with and without the condition, then at inference push the sample toward the conditional prediction. It's why prompts work — and, quietly, it runs the network twice per step.

The cost, stated once. Vanilla DDPM used ~1000 steps, each a full pass through a network with hundreds of millions to billions of parameters — doubled by guidance. A thousand-plus passes per image is the "slow sampling" the primers warn about. The rest of this review is how that number came down.

Part 2 — Acceleration: paying down the step tax

There are exactly two ways to make the sampling loop cheaper: take fewer steps, or make each step cheaper. Every acceleration technique is one of these, and the best systems stack does both.

Fewer steps — solve the ODE better

The continuous view says sampling is just numerically integrating a trajectory, and you don't need 1000 tiny Euler steps. DDIM (Song et al., 2021) makes the process deterministic and lets you skip steps; DPM-Solver / DPM-Solver++ (Lu et al., 2022) apply a high-order solver tuned to the diffusion ODE. Together they cut typical sampling to 20–50 steps with little visible quality loss — and crucially this is a change of the solver, not the model: no retraining, drop it into an existing checkpoint.

Fewer steps — distill the trajectory

The frontier move: spend training compute to teach a student to jump many steps at once. This is the diffusion analog of speculative decoding — pay at train time to buy inference latency.

Progressive distillation (Salimans & Ho): repeatedly halve the step count.
Consistency models (Song et al., 2023) and Latent Consistency Models (Luo et al., 2023): learn to map any point on the trajectory directly to its endpoint, enabling 1–4 step generation.
Adversarial diffusion distillation (SDXL-Turbo; Sauer et al., 2023) and distribution-matching distillation (DMD; Yin et al., 2024): push single-step quality to near-parity with the multi-step teacher.

This family is what put real-time, in-browser, and on-device image generation on the table.

Cheaper steps — work in a smaller space

Don't denoise a 512×512×3 image directly. Compress it first with an autoencoder to a small latent grid (e.g. 64×64×4), run the entire diffusion loop there, and decode once at the end. This is Latent Diffusion / Stable Diffusion (Rombach et al., 2022) — roughly a ~48× reduction in what each step touches. It's the exact principle from One Line, an Order of Magnitude: the win is in not moving the data, not a faster kernel. Latent space is the single highest-leverage acceleration in the whole stack, and it composes with everything else here.

Cheaper steps — quantization

Diffusion decode is the same memory-bandwidth-bound regime as LLM decode: each step reads the whole network for modest arithmetic. So the same lever applies — quantize the U-Net/DiT weights and activations (FP16 → FP8 → INT8/INT4). Post-training quantization for diffusion (Q-Diffusion and successors) moves the bandwidth ceiling and shrinks the per-step cost, with the usual quality-vs-bits tradeoff to measure, not assume.

Cheaper steps — cache across the loop

A property unique to diffusion: adjacent denoising steps are similar, so much of a big network's internal feature maps barely change from step to step. DeepCache (Ma et al., 2023) and block/feature-caching methods exploit exactly this — compute the expensive high-level features occasionally and reuse them on intervening steps. This is a genuinely diffusion-native acceleration with no LLM analog: it mines temporal redundancy that only exists because generation is a loop.

Cut the guidance tax

Because classifier-free guidance doubles the forward passes, guidance distillation folds the guided behavior into a single network so you pay one pass per step instead of two — often stacked with the step-distillation above. If you profile a diffusion endpoint and the FLOPs are 2× your estimate, guidance is why, and this is the fix.

And the ordinary runtime wins

Everything from the transformer world applies: FlashAttention in the backbone, torch.compile / TensorRT graph optimization, CUDA graphs to kill launch overhead, and batching to amortize the weight read across samples.

Regime	Steps / image	Forward passes*	Where it lands
DDPM (2020)	~1000	~2000	research baseline
DDIM / DPM-Solver++	20–50	40–100	default production sampler
Latent + fast solver	20–30 (in latent)	40–60	Stable-Diffusion-class
+ feature caching	effective ~½ the compute	—	high-res / video
Consistency / Turbo / DMD	1–4	1–8	real-time / on-device

*with classifier-free guidance (×2 per step) unless guidance-distilled.

Part 3 — Scale-up: bigger models, bigger canvases

Acceleration shrinks the loop; scale-up is what you do when one model or one GPU isn't enough — and here diffusion converges hard on the LLM playbook.

The backbone becomes a Transformer

The consequential architecture shift is U-Net → Diffusion Transformer (DiT, Peebles & Xie, 2023): treat the latent as a sequence of patches and denoise it with a plain transformer. Two things follow that a systems person cares about:

It scales like an LLM. DiT shows clean compute-vs-quality scaling curves, so diffusion inherits the entire transformer scaling playbook.
It merges the serving stacks. A DiT is attention over patches — the same kernels and the same tensor-parallelism / batching concerns as an LLM (minus the KV cache; each step attends over the full set fresh). The inference infrastructure largely transfers.

Parallelizing a single sample

LLM inference batches many requests; a diffusion image is often batch-of-one, and the latency of that one sample is what matters. So you parallelize the spatial/sequence dimension instead:

Patch parallelism — DistriFusion (Li et al., 2024) splits a high-res image into patches across GPUs and hides the communication by reusing the prior step's stale activations at the boundaries (the cross-step similarity again).
Sequence / context parallelism — for long video DiTs, split the (huge) token sequence across GPUs with Ulysses/ring-attention-style attention, exactly as long-context LLMs do.
Tensor parallelism on the DiT for the largest backbones — same as serving a big LLM.

Parallelizing across steps

The denoising loop is sequential — step t needs t−1 — which looks unparallelizable. But ParaDiGMS (Shih et al., 2023) reframes sampling as a fixed-point (Picard) iteration and refines all steps in parallel, trading extra compute for lower wall-clock latency. It's the one technique that attacks the loop's sequential dependency head-on.

Video is the scale frontier

Video diffusion (DiT over spacetime latent patches, Sora-class) is where every axis compounds: enormous token sequences (space × time), long-sequence attention, and sequence parallelism as a requirement rather than an option. The jump from image to video was mostly an engineering scale-up of the DiT, not a new model class — which is precisely why the transformer backbone mattered.

Training at scale, and flow matching

On the training side, diffusion scales with the same tools as LLM pretraining — FSDP / tensor parallelism / FP8. And the objective itself is being modernized: flow matching / rectified flow (Lipman et al.; Liu et al.) trains the model to follow straight noise→data paths, giving a simpler objective and trajectories that integrate in fewer steps. It's the recipe behind Stable Diffusion 3 (Esser et al., 2024) and Flux, and it's quietly becoming the default — accelerator and scale-up strategy in one, since straight paths are both easier to train at scale and cheaper to sample.

The verdict, and what to measure

Where it stands. The "slow sampling" limitation is, in 2026, largely paid down: latent space (~48×), a modern solver (1000 → ~25 steps), and few-step distillation (→ 1–4 steps) compose into a handful of passes, with quantization, feature caching, and guidance distillation trimming each pass further. Scale-up rides the LLM stack — DiT backbones, sequence/patch parallelism, FP8 — so the infrastructure you built for transformer inference and training largely carries over.

What to measure — the same discipline as our benchmark ladder: report steps and forward passes, and state whether guidance is on and whether it's distilled; quote latency at a fixed quality bar, not steps at a fixed latency; and when you distill or quantize, report the quality delta you paid for the speed. A "fast diffusion" number without its step count, guidance setting, and quality reference is an ad, not a measurement.

Where it's going. Fewer steps (one, ideally), transformer backbones shared with LLMs, flow-matching objectives, and video as the scaling pressure. Diffusion and large language models are converging on one infrastructure — which means the inference-systems and scale-up questions this practice works on are increasingly the same questions on both sides of the generative-model divide.

If your team is sizing or accelerating a diffusion workload — choosing a sampler, deciding whether to distill or quantize, or parallelizing a video model across GPUs — that's the work I do.

References

On-ramp: GeeksforGeeks, "What are Diffusion Models?" — geeksforgeeks.org
J. Ho, A. Jain, P. Abbeel. "Denoising Diffusion Probabilistic Models." NeurIPS 2020. arXiv:2006.11239
Y. Song, S. Ermon. "Generative Modeling by Estimating Gradients of the Data Distribution." NeurIPS 2019. arXiv:1907.05600
Y. Song et al. "Score-Based Generative Modeling through Stochastic Differential Equations." ICLR 2021. arXiv:2011.13456
J. Song, C. Meng, S. Ermon. "Denoising Diffusion Implicit Models" (DDIM). ICLR 2021. arXiv:2010.02502
C. Lu et al. "DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Models." 2022. arXiv:2211.01095
J. Ho, T. Salimans. "Classifier-Free Diffusion Guidance." 2022. arXiv:2207.12598
R. Rombach et al. "High-Resolution Image Synthesis with Latent Diffusion Models" (Stable Diffusion). CVPR 2022. arXiv:2112.10752
T. Salimans, J. Ho. "Progressive Distillation for Fast Sampling of Diffusion Models." ICLR 2022. arXiv:2202.00512
Y. Song, P. Dhariwal, M. Chen, I. Sutskever. "Consistency Models." ICML 2023. arXiv:2303.01469
S. Luo et al. "Latent Consistency Models." 2023. arXiv:2310.04378
A. Sauer et al. "Adversarial Diffusion Distillation" (SDXL-Turbo). 2023. arXiv:2311.17042
T. Yin et al. "One-step Diffusion with Distribution Matching Distillation" (DMD). CVPR 2024. arXiv:2311.18828
X. Ma, G. Fang, X. Wang. "DeepCache: Accelerating Diffusion Models for Free." CVPR 2024. arXiv:2312.00858
W. Peebles, S. Xie. "Scalable Diffusion Models with Transformers" (DiT). ICCV 2023. arXiv:2212.09748
M. Li et al. "DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models." CVPR 2024. arXiv:2402.19481
A. Shih et al. "Parallel Sampling of Diffusion Models" (ParaDiGMS). NeurIPS 2023. arXiv:2305.16317
Y. Lipman et al. "Flow Matching for Generative Modeling." ICLR 2023. arXiv:2210.02747
P. Esser et al. "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis" (Stable Diffusion 3). 2024. arXiv:2403.03206