Do Agents Run on CPU or GPU? Both — and Mostly Neither

"Do agents run on CPU or GPU?" sounds like it should have a one-word answer. It has a three-part one, and the parts fall exactly along the seven steps of the agent loop.

The harness runs on CPU

Steps 1, 5, 6, 7 of the loop — assembling the context, parsing the emitted tool call, executing the tool (running pytest, applying a diff, hitting an API), appending the result, going around again — are ordinary program logic. When a coding agent runs your build, that is pure CPU work, plus disk and network. In our agent bake-off-style harness, the agent process itself (Aider) lived entirely on the box's 14 vCPUs. The permission layer, the schema validation, the loop's halting caps — all CPU-side code.

The model runs on GPU — typically

Steps 2–4 — tokenize, forward passes, emit — are the FLOP- and bandwidth-heavy part, so they live wherever the LLM is served: a vLLM instance on an A100, or a provider's fleet across the network when the agent calls a hosted model. Note the corollary people miss: the agent and its model are usually not on the same machine. The harness loops on your laptop's CPU while the forward passes happen on a GPU in someone else's building. An "agent" is a distributed system with exactly two tiers by default.

But "GPU" is a roofline choice, not a law

Two of our own results mark the edges of that choice:

The V-cycle experiment served a 7B coder through Ollama on an M4 laptop — Apple's GPU cores on unified memory, effectively the "CPU with fast memory" end of the spectrum — and it solved every routine coding task the experiment threw at it.
Agent workloads are decode-dominated: long contexts re-fed every lap, answers generated token by token. Decode is memory-bandwidth-bound, not compute-bound — the regime where the Dongarra/Matsuoka/Hoefler paper argues a matrix-enhanced CPU with on-package HBM is genuinely competitive. If any LLM workload migrates toward CPU-class hardware, it is the agent's, not the trainer's.

Where one agent lap actually spends its wall-clock: thin CPU slivers, a modest GPU burst, and a long stretch in the third place — the world. Proportions illustrative; the shape is what our bake-off measured.

The wrinkle: mostly neither

An agent's wall-clock is often dominated by step 5 — waiting on the world. A 30-second test suite, a slow CI runner, a rate-limited API idles both the CPU loop and the GPU. Our bake-off measured exactly this shape: throughput climbed 568 → 1,597 tasks/hour from 1 → 4 concurrent agents with zero errors and the wall time barely moving — the A100 was never the constraint. The GPU spent most of its life waiting for small, tool-bound trajectories to come back.

Three consequences:

Serving agents is a scheduling problem as much as a hardware one. The win is batching other users' laps onto the GPU while yours waits on its tool — continuous batching and disaggregated prefill/decode exist precisely because agent traffic is bursty and hole-filled.
Utilization numbers mislead. An agent fleet with a "half-idle" GPU may be perfectly provisioned — the idleness is structural (tool waits), not waste. Naive consolidation fills the holes and then the tail latency drifts-style pain arrives at burst time.
The cheapest agent speedup is usually not a faster model. Making the tools faster — test selection instead of full suites, cached builds, parallel tool calls — attacks the block that actually dominates the timeline.

The one-line answer

The agent's brain runs on the GPU, its hands and its loop run on the CPU, and much of its life is spent waiting on the world — a third place entirely. Provision for the brain, write the loop for the hands, and optimize the wait: that's the whole hardware story of agents.