Prompt Caching Explained: The Cost Lever Teams Get Wrong

A research evaluation across OpenAI, Anthropic, and Google on 500+ agent sessions with 10,000-token system prompts found that prompt caching reduces API costs by 41–80% and improves time-to-first-token by 13–31%. Yet most teams I’ve talked to treat caching as a checkbox — flip it on, walk away, assume the savings flow. They don’t. The discount you see on a pricing page is the ceiling, not the floor, and the gap between those two numbers is where your engineering discipline lives.

Here’s the pattern I’ve observed — what I call Cache Window Engineering. The dominant cost variable isn’t the provider’s headline discount. It’s cache TTL expiry (5 minutes by default on most tiers) and write premiums. Realized savings are a function of request frequency and prefix stability, not provider pricing headlines. Naive full-context caching can actually increase latency, and low-traffic workloads may pay more in write premiums than they save on reads. Caching is not universally beneficial. It can hurt unless you engineer for high-frequency reuse within tight TTL windows.

If you’re building production AI applications, understanding prompt caching mechanics is the single highest-leverage cost optimization available — bigger than model selection, bigger than batch discounts. Let’s break down why.

What Prompt Caching Actually Stores

Prompt caching stores the model’s computed key-value (KV) tensors for a repeated prefix, not the raw input text. Cache hits skip recomputation entirely, delivering identical output at reduced input cost. The model still generates fresh output tokens — caching never discounts output. It only touches the input side.

Think of it this way: LLM inference has two phases. Prefill is the expensive part, where the model reads every input token and computes attention KV tensors. Decode is where it generates output one token at a time. Caching short-circuits prefill. On a cache hit, the server loads pre-computed tensors from fast storage instead of recomputing them. The output is byte-identical. The work to produce it is skipped.

This has implications that trip people up:

• The cache is sensitive to byte-for-byte changes. Any whitespace, ordering, or content variation rewrites the KV tensors and busts the cache.
• Caches are tied to specific model versions. Switching from one model tier to another invalidates everything.
• The cache matches an exact prefix, not a semantic similarity. “Similar” prompts don’t help. Identical prefixes do.

The most common mistake is a system prompt that includes a timestamp, request ID, or user-specific greeting at the top. A single variable string at position one means 0% cache hits. Move all dynamic content to the end, after everything static, and your hit rate transforms.

Provider Pricing: Where the Numbers Actually Land

Cached read discounts have converged near 90% off list across major providers, but the implementation details create very different economics depending on your traffic patterns. Here’s the current landscape:

Provider	Cache Read Rate	Cache Write Fee	TTL	Target Audience
OpenAI (GPT-5.5/5.6)	$0.50/M vs $5.00 base (90% off) per LLM Data Hub	Free (automatic)	5–10 min sliding window	High-traffic apps with shared system prompts
Anthropic (Claude Sonnet 5)	$0.20/M vs $2.00 base (90% off) per LLM Data Hub	1.25× base (5-min) / 2× base (1-hr)	5 min default, 1 hr optional	Explicit cache control, large static context blocks
Google Gemini 3.1 Pro	$0.20/M vs $2.00 base (90% off) per AI//COST	No write fee, $4.50/M/hr storage	Configurable	Long-lived explicit context caches

There’s a contradiction worth flagging in the data. Some sources report OpenAI’s automatic caching at roughly 50% discount on cached input for models like GPT-4o and GPT-4o-mini, with cache hits persisting for a sliding window of 5 to 10 minutes. But verified July 2026 pricing for newer models tells a different story: GPT-5.5 and GPT-5.6 family cached input rates sit at $0.50 per million tokens against a $5.00 base — a 90% discount, not 50%. The discount deepened as OpenAI updated their model lineup. If you’re looking at older pricing guides, you’re understating the savings.

A similar discrepancy exists for Google. Some reports show Gemini cache reads at ~25% of base input (75% off) with per-hour storage fees. But current verified data shows Gemini 3.1 Pro Preview at 10% of base ($0.20 per million vs $2.00 base), indicating the discount rose from 75% to 90%. The storage fee of $4.50 per million tokens per hour for explicit caching is the tradeoff — you’re renting the cache by the hour.

DeepSeek sits in its own category. DeepSeek V4 Flash offers cached input at $0.0028 per million tokens versus $0.14 base — a 98% discount — with automatic caching and no write fee, per AI//COST. That’s the deepest discount in the market, and it’s automatic.

The Write Premium Problem Nobody Talks About

Anthropic’s 5-minute TTL cache write premium recovers after about 1.3–1.4 reads at the cache read discount rate. That sounds trivial — two requests hitting the same cache entry and you’re already ahead. But it’s the detail that determines whether caching saves you money or costs you extra.

Here’s the math that matters. Anthropic charges 1.25× base input to write a 5-minute cache entry and 2× base input to write a 1-hour cache entry. OpenAI and DeepSeek charge nothing for the write. Google charges no write fee on implicit caching but bills $4.50/M/hr storage on explicit caching for Gemini 3.1 Pro Preview.

What this means in practice:

• High-frequency reuse within 5 minutes: Anthropic’s write premium amortizes instantly. You pay 25% more once, then 90% less on every subsequent read. The economics are excellent.
• Sporadic traffic with gaps longer than 5 minutes: You pay the write premium on nearly every invocation. The cache expires between bursts. You’re paying more than you would without caching at all.
• Batch processing with long pauses: Same problem. Every burst starts cold.

The 5-minute default TTL is the hidden cost variable. A background worker processing documents every few minutes might get 20 reads per cache write under a 60-minute TTL. Under the current 5-minute TTL, that same worker gets maybe 2 reads per write. The savings profile shifts dramatically.

Minimum Prefix Lengths and Platform Constraints

You can’t cache everything. Every provider enforces minimum prefix lengths, and if you’re routing through a platform like Amazon Bedrock, the requirements vary by model family.

Per AWS Bedrock’s prompt caching documentation, Claude 3.7 Sonnet requires at least 1,024 tokens per cache checkpoint. Claude Opus 4.5/4.6, Haiku 4.5, and Sonnet 4.5 require at least 4,096 tokens. If you try to add a cache checkpoint before meeting the minimum, your inference still succeeds — but your prefix won’t be cached. You pay full price and get none of the benefit.

OpenAI requires 1,024 tokens uniformly. Google requires 4,096 on Gemini 3.1 Pro Preview, with lower thresholds on Flash tiers.

The practical implication: if your system prompt is below the minimum on every provider, you’re below the threshold for caching to fire. You either need to pad your prompt with additional static context (tool definitions, few-shot examples, reference material) or accept that caching won’t fire. This is where prompt structure becomes architecture — you’re designing your prompt to clear the caching threshold, not just to communicate with the model.

When Caching Hurts: The Contrarian Case

A research evaluation across OpenAI, Anthropic, and Google found that naive full-context caching can paradoxically increase latency. The same study, covering 500+ agent sessions with 10,000-token system prompts, found that strategic prompt cache block control — placing dynamic content at the end, avoiding dynamic function calling in the cached prefix, and excluding dynamic tool results — provides more consistent benefits than caching everything.

This is the part most guides skip. Caching isn’t free even when the write fee is zero. There’s overhead in cache lookup, prefix matching, and tensor loading. For low-traffic workloads where each request has a unique prefix, you’re paying that overhead with no offsetting savings. For workloads where the cached prefix is so large it dominates the request, the cache lookup itself can add latency if the prefix doesn’t match cleanly.

The patterns that actually pay off:

1. Long system prompt: Instructions, persona, tools list cached once, reused across every user message. Single biggest win for chat apps and agents.
2. Retrieval context: A stable corpus (book, codebase, PDF) cached as the prefix; the user question is the only changing part.
3. Few-shot examples: A long block of in-context examples for classification or extraction tasks that don’t change between calls.
4. Tool definitions: A long list of function schemas (10+ tools) cached as part of the system prefix.
5. Conversational history: For long sessions, cache the growing transcript so the model doesn’t re-process hundreds of prior turns on each call.
6. Multi-document RAG: Cache the reference set (regulatory text, product manuals) as the prefix for domain-grounded agents.

Each of these shares the same structural requirement: stable content first, variable content last. If you’re working with AI coding agents, this same principle applies to context files — though as we’ve seen in our analysis of AGENTS.md vs Claude Code Memory, context file design has its own tradeoffs that interact with caching in non-obvious ways.

Cache Hit Rate as a Primary SLO

Healthy production prompt cache hit rates are typically 60–90%. Below that range, you’re either not structuring prompts for cache reuse or your traffic patterns don’t support the TTL window. Either way, you’re leaving money on the table.

In 2026, optimizing prompt structure for cache hit rate within provider TTL constraints delivers larger and more reliable cost reductions than model selection or batch discounts. Teams should treat cache hit rate as a primary SLO — not a nice-to-have metric, but a number you monitor, alert on, and optimize for.

Here’s why that matters. If you’re running an agent loop with a 5,000-token stable prefix reused 10 times, caching cuts per-call input cost on the cached portion by ~90% across providers. Stack that with batch processing discounts (50% off input and output on all three major providers where supported) and your effective price runs 5–50% of list. But none of that stacks if your cache hit rate is 15%.

The monitoring approach I’d recommend:

• Track cached_tokens in your API responses (exact field names differ per vendor — cache_read_input_tokens on Anthropic, cached_tokens on OpenAI).
• Alert when hit rate drops below 60% on stable-prompt workloads.
• Treat a sudden hit rate drop as a production incident. The Claude Code team reportedly declares SEV incidents when cache hit rates drop. That’s the right instinct.
• Log prefix hashes to identify which prompts are busting the cache and why.

If you’re dealing with systems that lack native persistent memory — like Cursor’s memory gap — prompt caching becomes even more critical because you’re re-sending context that other tools would persist automatically.

The Decision Framework

The right caching strategy depends on three variables: your traffic frequency, your prefix stability, and your tolerance for explicit configuration.

Choose OpenAI-style automatic caching when: You want zero code changes, your traffic is high-frequency with shared system prompts, and you’re on newer GPT-5.x models where the discount is 90% rather than 50%. The tradeoff is no explicit control — you can’t mark specific breakpoints or extend TTL.

Choose Anthropic-style explicit caching when: You have large static context blocks (tool definitions, reference docs, system instructions), you’re willing to add cache_control markers, and your traffic reuses the same prefix within the 5-minute TTL window. The tradeoff is the write premium — you need at least 1.4 reads per write to break even. The upside is the deepest read discount and surgical control over what gets cached.

Choose Google-style explicit caching when: You need long-lived caches (hours, not minutes), you’re willing to pay per-hour storage fees, and your workload involves large reference documents queried repeatedly over time. The tradeoff is the storage cost — $4.50/M/hr for Gemini 3.1 Pro Preview — which adds up if your cache sits idle.

Avoid caching entirely when: Your traffic is sporadic with gaps longer than the TTL, your prompts have no stable prefix, or your prefix is below the minimum token threshold. In these cases, caching adds overhead with no offsetting savings.

The question I’d leave you with: if your cache hit rate dropped to 30% tomorrow, would your monitoring catch it — or would you find out when the monthly bill arrives?