decode-step economics · one weight-read, many tokens

The MoE batch valley

Why batching is free money for dense models, and why a mixture-of-experts wants either one user or thousands — never a dozen.

Decoder ring — every term on this page, in one line each

token
a chunk of text (word or word-piece) — the unit the model reads and writes
decode step
one turn of the generation loop: the model reads weights and emits one token per sequence
batch (B)
how many sequences are decoded together in one step
weights (W)
the model’s learned numbers; W = their total size in bytes
feed-forward block
= FFN = MLP
the part of a layer applied to each token alone (Feed-Forward Network / Multi-Layer Perceptron — same thing). An expert is one copy of it
KV cache
each sequence’s saved attention keys/values so the prompt isn’t reprocessed. Private per sequence — it never shares
amortize
spread one fixed cost (a weight read) across many tokens, so each pays less
bytes / token
memory traffic divided by tokens produced — the number this whole page is about
tok/s
tokens per second — throughput
HBM
High-Bandwidth Memory — the GPU’s on-package memory, where weights live
MBU
Memory-Bandwidth Utilization — bytes actually moved ÷ peak bytes/s. The decode metric
MFU
Model-FLOPs Utilization — math actually done ÷ peak FLOP/s. The prefill/big-batch metric
arithmetic
intensity (AI)
FLOPs done per byte moved. Low = memory-bound, high = compute-bound
ridge point
the AI where a chip stops being memory-bound and starts being compute-bound
crossover
batch (B*)
the batch size where decode reaches the ridge — past it, batching buys latency, not throughput
BF16 / Q4
number formats: 2 bytes vs ~0.5 bytes per weight. Fewer bytes = less to stream
MoE / expert /
top-k
Mixture of Experts: many parallel feed-forward blocks (experts); a router runs only k of the E per token
expert
parallelism
splitting experts across GPUs so each holds only some of them
all-to-all
the network step shipping tokens to whichever GPU holds their expert
goodput
throughput that actually met the latency target — not raw tok/s

One idea underneath all of it: a decode step reads some weights once and produces B tokens. Batch is how many tokens share that read; MoE changes which weights the read has to cover.

1 · One trip to the warehouse — the whole trick of batching

BATCH = 1 WEIGHTS W bytes tok this token pays for ALL of W bytes/token = W BATCH = 8 WEIGHTS W bytes the SAME trip serves eight bytes/token = W / 8 …but every token still hauls its OWN KV cache — that part never shares.
bytes per token = W / B + kv_bytes(context) └ shared ┘ └ private, scales WITH B ┘

2 · Three regimes as batch grows — dense models

WEIGHT-BOUND small B 2× batch ≈ 2× tok/s free capacity COMPUTE-BOUND past the crossover B* ≈ 300 (H100 BF16) AI ≈ B → hits the ridge KV-BOUND long context batching stops helping B·kv > W

Metric hand-off: at batch 1 quote MBU (MFU is 1–2%, meaningless); approaching B* quote MFU (MBU pins near 1.0 and stops discriminating). Always state B — our M4 sat at ~0.8 MBU at B=1, i.e. regime 1 with no headroom, which is exactly why one batching flag bought queueing relief (15× goodput) but only 1.3–1.45× raw.

3 · MoE: batch decides which weights get hauled

batch size → (tokens per expert per step ≈ B·k / E) bytes / token B ≈ 1 only k experts read THE VALLEY ~every expert touched… …by ~ONE token each dense-sized bytes, sparse-sized amortization huge B all experts, shared by many example shape: E = 256 experts, top-k = 8 → the valley sits near B ≈ 32
dense: bytes/token = W / B (falls monotonically) MoE: bytes/token = bytes(experts touched by the B tokens) / B (U-shaped)

Which is why MoE serving lives at the extremes — one user on a laptop, or thousands of tokens pooled across wide expert parallelism in a datacenter, where each GPU streams only its own experts and the all-to-all dispatch joins the roofline as an interconnect term (and a hot expert becomes the straggler).

Dense amortizes monotonically. MoE has a valley.
Batch for one user — or for thousands. Never for a dozen.