Companion cartoons: the pickle pipeline (the full trace, eight panels) · what is an expert · the MoE batch valley. Each opens with a glossary defining every term it uses.
Ask a language model "why are pickles wiggly?" and the machinery is small enough to trace end to end: the tokenizer cuts 23 characters into 8 tokens, each becomes a row number, the rows become vectors, ~30 blocks of attention and feed-forward math run, and the model emits one token at a time until it stops. About a second, one visit to the GPU, and it comes back with "pickles wiggle because they are mostly water."
Now ask an agent the same question.
The same question, agent-style
Three GPU visits instead of one, each separated by ordinary CPU work, with the context re-fed and longer every lap. For this question the agent bought nothing: the model already knew the answer. It spent roughly 3× the tokens and 4× the wall-clock to arrive at the same sentence.
That is not an argument against agents. It is the sharpest available argument about when to use one — and, along the way, it shows exactly where an agent sits in the machine.
An agent is not a hardware tier
The tempting mental model is that CPUs run programs, GPUs run models, and agents run on… something new. They don't. An agent is a control loop that lives on the CPU and rents the GPU by the lap. Its parts split cleanly:
| Part of the agent | Lives on | Share of wall-clock |
|---|---|---|
| Assemble context, parse the tool call, enforce permissions, run the loop | CPU | small but continuous |
| Forward passes — the model thinking | GPU, often someone else's over a network | modest |
| Executing the tool — tests, builds, searches, API calls | CPU + I/O | frequently dominant |
| Waiting | nothing at all | usually the biggest block |
| Its "memory" | context — text, re-fed | not registers, not HBM |
Three consequences worth holding onto:
The agent's body spans machines. The harness loops on your laptop's CPU, the model answers from a GPU in another building, the tools hit a third place entirely. An agent is a small distributed system whose most expensive component is idle most of the time — which makes serving agents a scheduling problem more than a hardware one. Providers fill your tool-wait holes with other users' tokens; that arbitrage is why per-token pricing beats renting a dedicated box for bursty agent work, right up until you have enough concurrent agents to fill your own holes.
The agent is the CPU's revenge. After a decade of the GPU absorbing every interesting workload, the agent puts the CPU back in charge: it holds the state, decides when to spend GPU time, executes the real-world effects, and enforces the permission boundary. The GPU is demoted to an oracle you consult — expensive, stateless, and on a leash the CPU holds.
Its memory is text, so the growing context is the real cost. Every lap re-sends a longer prompt. That is why prompt-prefix caching is an agent's best friend: in our own phase measurements, a cached prefix collapsed time-to-first-token to a fixed ~0.16 s at any prompt length — roughly a 100× saving against the same prompt cache-busted. Lose that cache between laps and you pay full prefill again, every lap.
Where it lands on the ladder
Our efficiency ladder already had the slot open. Silicon questions are rungs 3–4 (is the hardware working, is the right hardware working); serving is rung 5 (goodput — are users served within SLA); and agents live at rung 6: are problems actually getting solved?
That placement explains a result that otherwise looks like a bug. In our agent bake-off, throughput climbed from 568 to 1,597 tasks/hour as we went from 1 to 4 concurrent agents with zero errors and wall-time barely moving — the A100 was never the constraint. A half-idle GPU under agent load is not waste; it is structural, and consolidating it away just moves the pain to burst time. At rung 6 the unit is not tokens, it is solved tasks, and the cheapest speedup is usually a faster test suite rather than a faster model.
The rule the pickle gives you
The pickle cartoon shows a machine that knows. The agent loop is the machine that does. And the engineering judgment — the part worth paying for — is deciding which one a task needs:
Don't escalate to an agent when one forward pass suffices. Agents earn their keep when the answer requires doing something: checking a live source, running code, editing a file, verifying against a test. If the model already knows, the loop is pure overhead.
It is the same discipline as routing by the verifier, one level up: spend the expensive machinery only on the residual that the cheap path can't handle. There, the cheap path was a 7B model. Here, it is a single forward pass — and the residual is everything that requires touching the world.
Related: Do agents run on CPU or GPU? · The agent loop, pedantically · Claimed, packed, working · Metering goodput.