Prompt Caching Strategy at Scale

Prompt caching is the highest-leverage cost and latency optimisation available on the Claude platform, and the one most often left underperforming. Caching is simple in principle (reuse a processed prefix) and unforgiving in practice (one changed byte and the reuse is gone). At scale, cache behaviour is a property of your architecture, not of any one prompt.

Mechanics

• Caching works on the prefix of the request as rendered: tools first, then the system prompt, then messages in order.
• You mark cache breakpoints with cache_control of type ephemeral on content blocks; up to four per request. A top-level cache_control on the request can also cache the last cacheable block automatically.
• A cached prefix is matched exactly. Any change anywhere before a breakpoint invalidates that breakpoint and everything after it.
• Minimum cacheable length depends on the model; shorter prefixes silently do not cache.
• The default time to live is short (minutes) and refreshed on each hit; a longer TTL option exists at a higher write cost for prefixes reused less frequently.
• Writes cost more than normal input tokens; reads cost a fraction. The usage object reports cache_creation_input_tokens and cache_read_input_tokens.
• Caches are scoped to your organisation and to the model. Switching model forfeits the cache.

Current multipliers, minimums and TTL options are in the Anthropic docs; design to the principles.

Layered breakpoints

Think of the request as layers from most to least stable, and place a breakpoint at the end of each layer that is reused:

[ tools + system instructions ]        <- breakpoint 1: shared by all traffic on the route
[ reference corpus / policy docs ]     <- breakpoint 2: shared by a tenant or a product
[ conversation history so far ]        <- breakpoint 3: moves forward each turn
[ current user turn ]                  <- never cached

In a multi-turn conversation, move the history breakpoint to the latest assistant turn on each request. The previous breakpoint's cache serves the read; the new one is written. This pattern turns an agent loop's quadratic token growth into something close to linear cost.

Designing for a fleet

Hit rates collapse when several services or instances assemble "the same" prompt slightly differently. Centralise prompt assembly in the gateway and enforce:

Rule	Why
Identical tool definitions in identical order	Tools are the first thing rendered; any difference invalidates everything
Byte-identical system prompt per route and version	Whitespace, a reordered sentence or a rendering difference breaks the match
No volatile values in the stable prefix	Timestamps, request IDs, user names, session IDs, random example selection
Deterministic JSON serialisation	Unsorted keys render differently between calls
Same model and same thinking configuration across the route	Both are part of the cache identity or affect rendering
Volatile content after the last breakpoint	Keep the current turn, user context and dynamic data at the end

A frequent and expensive bug is a "last updated" or "today's date" line in the system prompt. If the model needs the date, pass it in the user turn.

Multi-tenant designs

Tenants should share as much prefix as possible. Put tenant-independent instructions and tools first with a breakpoint, then tenant-specific documents with a second breakpoint, then the conversation. The shared layer is written once for everyone; the tenant layer is written once per tenant and read many times. Never put tenant identifiers into the shared layer.

Operator changes without invalidation

Changing the system prompt mid-conversation to inject an instruction invalidates the cached conversation. On models that support system-role messages inside the messages array, append the instruction there: the prefix is preserved and the instruction carries operator authority. Where unsupported, append it to the next user turn rather than editing earlier content.

Measuring and diagnosing

Instrument every route with cache hit rate (cache read tokens divided by total input tokens) and alert when it drops. A zero read count on a route with repeated traffic always means an invalidator, never "caching is off for this model". Diagnosis checklist:

1. Diff two consecutive requests byte for byte up to the breakpoint.
2. Check tool order and tool description text between service instances.
3. Look for interpolated time, IDs or randomness.
4. Confirm the prefix exceeds the model's minimum cacheable length.
5. Confirm the TTL has not expired between calls; infrequent traffic may need the longer TTL.
6. Confirm model and thinking settings have not changed between calls.

Where available, cache diagnostics features can report why a request missed; check the docs for current support.

Warming and TTL strategy

For routes with bursty traffic, a scheduled request that touches the prefix keeps it warm; some designs do this with a minimal-output request. For prefixes reused every few minutes the default TTL is enough; for hourly reuse the longer TTL may be cheaper despite the higher write cost. Model it: write cost multiplied by writes per hour versus the savings on reads.

Exam focus

• Caching is a prefix match over tools, system and messages in that order; a change before a breakpoint invalidates it and everything after.
• Up to four breakpoints; layer them from most to least stable and move the history breakpoint forward each turn.
• Volatile content (dates, IDs, user data) belongs after the last breakpoint; tool order and serialisation must be deterministic.
• Verify with cache_read_input_tokens; zero reads on repeated traffic means a silent invalidator.
• Caches are scoped by organisation and model; multi-model designs and mid-conversation system prompt edits forfeit reuse.
• TTL choice is a cost model: default for frequent reuse, longer TTL for infrequent reuse with higher write cost.