AI Tokens Are Wasted Re-Reading Context: Cut the Bill

Across 12,002 agent sessions and 1.19 million model requests, the dataset records 209.2 billion cumulative input tokens — and 197.1 billion of those were cached tokens, meaning the model had already processed them. That’s 94.2% of all input tokens served from cache, against just 552.5 million output tokens. The waste isn’t in generation. It’s in the re-reading.

What do the Harvard agent traces actually show?

The traces confirm that agent workloads are dominated by cache reads, not fresh computation. Harvard MaDSys released the FreeInference dataset covering 16 weeks of traffic from May 17 to September 5, 2026, spanning 267 accounts. The 209.2 billion figure counts cumulative input processed across requests, not unique text — a distinction that matters because the same growing context accompanies many successive model calls in a single session.

The math is stark: 197.1 billion cached tokens divided by 209.2 billion total input tokens equals roughly 94.2%. Output tokens barely register at 552.5 million. If you’re running agents, the overwhelming majority of what you pay to read is material the model has already seen. That reframes the cost problem entirely. You’re not paying for intelligence; you’re paying for redundancy.

It also explains why the industry is pivoting. Persistent agents amortize one cache write over millions of cheap reads. One-shot workloads get nothing from the discount and pay full list. Your position on that spectrum determines whether caching is a lifeline or a tax — and it’s the first thing to measure before you touch a pricing page.

Why does re-reading context dominate agent bills?

Agent architecture re-sends the same growing context on every turn, and the bill scales with token count, not novelty. In agent workflows, by the 50th tool call, history alone can exceed 150,000 tokens, re-billed on every turn, according to Compresr’s analysis. A 1,500-token system prompt called 1,000 times per day burns 1.5 million input tokens daily before any user message is included.

DeepSeek makes the same point from the serving side: storing and moving context has become a bigger obstacle to keeping costs down than the computation a model performs to generate output, the company argues. The workload shape shows up in traffic data too. StepFun’s listing shows the top applications by token volume are persistent agents — Hermes Agent at 2.12 trillion tokens, Kilo Code at 1.45 trillion, OpenClaw at 1.07 trillion, and Cline at 808 billion. These aren’t chatbots asking one question. They’re systems that write a large context once and re-read it on every step.

The pattern holds in coding agents too. Our breakdown of repository context strategies for AI coding found that 90% of enterprise AI cost is input tokens from agents re-exploring repos. The waste is also architectural. CIO’s token efficiency analysis identifies recursive agent handoffs, frontier models wasted on routine classification, undefined success criteria causing retry loops, and context inflation from stale earlier steps as primary drivers. None of these look expensive in isolation. Across millions of executions, they compound.

How does prompt caching pricing work across providers?

Caching bills three separate things — the write, the read, and sometimes storage — and the multipliers differ enough to change your math. Here’s how the major providers structure it as of October 2026:

Model	Input	Cache Write	Cache Read
OpenAI GPT-5.6 Sol	$4/M	$5/M (1.25x)	$0.40/M (0.1x)
Anthropic Claude Sonnet 5.5	$2/M	$2.50/M (5-min); $4/M (1-hr)	$0.10/M (0.05x)
Google Gemini 3.8 Flash Standard	$0.75/M	No premium; storage fee	—
DeepSeek V4.1-Flash	$0.30/M peak	No premium	$0.006/M

Prompt caching is now the largest discount on most bills: cached reads cost 2% to 10% of the base input price depending on vendor and model, while writes can cost 1.25x or 2x, Prompt20’s pricing guide explains. The mid-tier flagships cluster tightly: OpenAI’s gpt-6.1-sol, Anthropic’s Claude Sonnet 5.5, and Google’s Gemini 4 Argon all list $2 per million input tokens and $10 per million output, The Frontier Wire reports. But headline prices mislead. Anthropic’s tokenizer produces approximately 30% more tokens for the same text, and reasoning tokens are billed at the output rate even when the API doesn’t return them.

Cache-read pricing is more uniform than you’d expect. AI Signal’s survey found 31 of 49 priced models across eight providers sit at exactly 0.1x, with exceptions like Anthropic Fable 5.1 and Mythos 5.1 at 0.025x and DeepSeek at 0.033x. The write side is where providers diverge. Anthropic requires explicit cache_control breakpoints and charges a premium, needing at least 2 reuses in its 5-minute TTL to break even, while OpenAI’s caching is fully automatic with no code changes, Tokens BD notes. StepFun’s Step 5 Preview prices cache reads at $0.05 per million tokens, exactly one-twentieth of its $1.00/M input price, but publishes no cache-write rate at all, Yahoo Finance reports. For anyone modeling a memory-intensive workload, that missing line is a missing term in the cost model.

When does caching actually save you money?

Only when your hit rate clears the break-even threshold — and for many workloads, it doesn’t. With a 1.25x write and a 0.1x read, prompt caching breaks even when about 22% of requests hit; with Anthropic’s one-hour cache (2x write), it needs about 53%, INFRO calculates. On GPT-5.6 and later, caching a prompt you only send once costs 25% more than not caching it.

Real-world measurements expose the gap between vendor framing and reality. Anthropic pitched the Sonnet 5.5 cache-read cut as making the model about 20% cheaper on agentic work. MxChat’s production tests found savings ranging from 0.9% for new questions to 35.4% when the whole knowledge base was cached. A WordPress chatbot with only a 1,212-token system prompt cached got 0.9% cheaper: $13.29 per 1,000 replies, down from $13.41. A bot with its entire knowledge base in a cached 17,946-token prompt got 35.4% cheaper: $3.27 per 1,000, down from $5.06 — while the same request without caching costs $37.43.

Automatic caching isn’t free on the first message either. It writes the whole first question to cache at 1.25 times the input price, adding $2.81 per 1,000 conversations, and pays back only if roughly one conversation in four gets a follow-up within five minutes. The upside case is real when reuse is high: at a 10-reads-per-write ratio, 1 billion reused tokens costs $900 on GPT-5.6 Sol or $350 on Claude Sonnet 5.5 with a five-minute cache, against $4,400 and $2,200 respectively without caching. Vendor claims run hotter still. Anthropic reports up to 90% cost reduction and 85% latency reduction for long prompts with caching, and a January 2026 evaluation found 41-80% cost reduction and 13-31% TTFT improvement across providers for agentic tasks, the AWS Builder Center playbook notes. Those numbers are real for the right workload — long, stable prefixes read many times. They’re not a default.

Watch the floors, too. Minimum prompt sizes differ: 1,024 tokens on Anthropic Sonnet/Opus, 2,048 on Haiku, 1,024 on OpenAI, and 4,096 on AWS Bedrock and Claude Haiku 4.5. OpenRouter passes through each provider’s prompt caching without a per-token markup, with cache reads from 0.1x to 0.5x and writes from free to 2x — but below a model’s minimum prompt length nothing caches and nothing tells you, TrueFoundry’s testing shows. The failure mode is silent. An OpenRouter test harness initially reported caching worked on Claude Haiku 4.5, then reported it didn’t; the real cause was that Haiku 4.5 doesn’t cache prompts under 4,096 tokens and the test prompt was about 3,600. And because attention makes every token depend on all previous tokens, a single changed token early in the prompt invalidates every cached block after it.

What about compression and harness optimization?

Caching discounts re-reads; compression and harness work reduce the tokens you generate in the first place. LLM context compression typically cuts input token costs by 30% to 70% with minimal accuracy loss, and it’s complementary to prompt caching rather than a substitute, Compresr’s guide explains. If you’re wrestling with bloated agent contexts, our deeper dive on context compression techniques for LLMs covers lossless, semantic, and proactive strategies.

The harness matters as much as the model. Nvidia’s SoL-Pi system cuts coding agent token usage by nearly 50% by optimizing the scaffolding around the agent — what context reaches the model, how tool calls are formatted — rather than retraining or swapping the underlying model. If you’re running AI coding tools, our teardown of where Cursor tokens actually go found 90% are input from re-read context, not generated code. That’s a lever you control without waiting for a vendor price cut.

The biggest wins often come from not calling the model at all. Enterprise AI cost optimization should start with business logic layers that avoid repeated LLM processing of the same context and rules, rather than only choosing cheaper models, DigitalToday reports. Heidi cofounder Yu Liu was spending up to $3 million every month on OpenAI, Anthropic and Google models, and cut costs by moving 95% of the startup’s workload to specialized open-weight models hosted on Fireworks, according to Forbes.

Should you self-host or use managed KV cache platforms?

Self-hosting shifts the problem from per-token billing to GPU memory management, and the infrastructure is harder than the papers suggest. KV cache compression research reports large gains, with CatKV achieving 2-5x faster TTFT and 6-10x higher compression ratios while preserving accuracy, and SlimKV retaining over 96% of uncompressed model score at 16x/32x compression, the ACM publication documents.

Production reality is less forgiving. KV cache compression methods collide with existing infrastructure: FlashAttention never writes historical attention scores into GPU memory, and paged attention cannot reclaim partially-filled blocks after token eviction, NVIDIA’s research lab explains. A Qwen3-32B model with 4-bit quantized weights crashes with an out-of-memory error on a 24 GB GPU after roughly 24,000 generated tokens — short of the 32K-token traces reasoning models routinely produce.

Vendors are racing to close that gap. Iterate.ai’s Lifeboat inference engine fits two to six times as many concurrent AI agent sessions per GPU, holding 2,048 concurrent sessions on a single RTX PRO 6000 Blackwell GPU with a Qwen 30B-A3B model. Tensormesh launched an enterprise KV-cache management platform on October 14, 2026, built by the team behind the open-source LMCache project. But remember the limit: prefix caching accelerates prompt processing, not the generation of new output tokens, so workloads dominated by long answers or largely unique inputs may see less benefit.

What’s the actual decision framework?

Treat your context pipeline as a capital asset, not a utility bill. What I call Reuse Ratio Economics comes down to this: the dominant cost driver has shifted from raw token price to reuse architecture. The market is bifurcating between workloads that amortize a single large context over millions of cheap reads and one-shot workloads that pay full list price.

Start by measuring the cached share of input tokens per model and route, not just the cost. A failover to another provider account starts a cold cache, so your routing logic can silently erase your savings. Structure prompts for prefix stability — static content first, variable content last. Use explicit caching for persistent agent state where the write premium pays for itself. And ruthlessly eliminate token generation through business logic layers and harness optimization before you pay for model inference.

One honest gap: a 50-agent support deployment cannot be fully projected from current pricing data. The research provides per-million-token API rates and cache multipliers, but no per-agent token volume, request counts, or cache hit rates needed to compute a total bill. You’ll need to instrument your own traffic to get that number.

The teams winning on cost aren’t the ones who found the cheapest model. They’re the ones who stopped paying to re-read the same tokens.