Prompt Caching: Economics, Instrumentation, and Failure Modes

Prompt caching pays when many requests reuse a long, byte-stable prefix within the provider's retention window. It does not make changing retrieval results, user messages, or policy state safe to reuse. Put stable instructions and tool definitions first, dynamic content last, then measure writes, reads, misses, latency, and task quality by prompt version.

Calculate break-even reuse from current provider prices. Do not optimize around a quoted discount that may change next month.

Prompt caching is not response caching

Prompt caching reuses provider-side computation for repeated input prefixes; the model still generates a new response. Response caching returns a stored answer and may skip generation. They have different correctness and privacy risks.

OpenAI's current guidance describes automatic and explicit prompt caching, cache-write and cached-token usage, and stable content first. Anthropic documents five-minute and one-hour durations with distinct write and read pricing. Check OpenAI's model guidance and Anthropic pricing before implementing a calculator.

Calculate break-even reuse

Let U be uncached input price, W cache-write price, R cache-read price, and n requests sharing the prefix:

cost_uncached = n × U
cost_cached = W + (n - 1) × R

caching saves when:
n > (W - R) / (U - R)

Multiply prices by eligible prefix tokens for absolute cost. Add miss rate, expiry, and premature invalidation for a realistic estimate.

Do not count output-token savings: prompt caching changes input processing, not the amount generated.

Build a stable prefix deliberately

Good candidates are system instructions with a version ID, tool definitions in deterministic order, long static examples, and a stable reference document.

Put per-user state, retrieval results, timestamps, nonces, and current policy context after the stable prefix. A timestamp near the beginning can destroy reuse. Canonicalize generated schemas and examples; stable meaning is not enough when matching depends on an exact prefix.

Instrument hits as a quality feature

provider and model
prompt and tool-set version
cache key or breakpoint version
eligible prefix tokens
cache write and cached read tokens
miss reason when known
latency, total cost, and task outcome

Monitor hit rate by prompt version and deployment. A global rate can hide one tenant or region with no reuse. Join these fields to outcomes using the production LLM trace schema.

Treat invalidation as a product decision

prompt:v17 | tools:v8 | policy:v12 | corpus_snapshot:2026-08-25

Force a new prefix when a security rule, tool contract, or critical instruction changes. Do not preserve a hit at the cost of stale policy.

OpenAI notes that extended caching can involve stored application state and may be incompatible with Zero Data Retention. Behavior varies by endpoint and mode, so include it in the architecture review. See OpenAI platform data controls.

Test five failure modes

Run an A/B rollout with the same semantics. Compare task success, refusals, latency percentiles, and total cost. A higher hit rate is not a win if reordered context changes behavior or writes exceed savings.

Prompt caching is one lever. Use it after measuring the pipeline with the LLM cost optimization guide, and protect prompt rearrangements with an evaluation set.

Sources and further reading

• OpenAI model and prompt-caching guidance
• OpenAI Responses API cache fields
• OpenAI platform data controls
• Anthropic pricing and prompt-cache durations