Prompt caching can reduce the repeated processing of large, identical context blocks. It works particularly well for stable system instructions, tool definitions, policies, or extensive documents that remain the same across many requests.
The difficult part is not enabling it. The difficult part is structuring prompts so that stable and variable content are clearly separated.
Which content is suitable?
A good cache candidate is reused frequently and changes rarely. Examples include:
- system instructions and output formats,
- fixed tool schemas,
- approved policies,
- large reference documents,
- recurring examples.
User inputs, current data, and task-specific retrieval results belong in the variable section. The earlier a prompt diverges, the smaller the reusable prefix may be.
Prompt structure for cache hits
Place stable content first and keep its order consistent. Small changes to an early block can cause subsequent content to be processed again.
Version system prompts and tool schemas. This lets you associate a cache hit with a known configuration and invalidate it selectively when changes occur.
Invalidation is a business rule
A cache must not expire based on time alone. Policies, prices, permissions, or product information can change. Define events that force a new version.
Permissions are particularly critical. Cached context that was permitted for role A must not automatically be reused for role B. The cache key must represent the relevant access controls and data classes.
Separate provider caching from semantic caching
Provider prompt caching reduces the repeated processing of identical prefixes. A semantic cache, by contrast, may return an earlier response for a similar request. These approaches carry different risks.
Semantic caching needs stricter rules for freshness, identity, and response quality. It is often unsuitable for time-critical, personal, or decision-relevant responses.
What should you measure?
Cache hit rate alone is not enough. Measure:
- input tokens or provider costs saved,
- latency with and without a hit,
- the proportion of stale results or results returned under incorrect permissions,
- cost per successful task,
- the impact of prompt or model updates.
A high hit rate is worthless if users have to make more corrections or receive outdated answers.
Caching remains provider-specific
OpenAI, Anthropic, and Google offer different mechanisms, minimum sizes, retention periods, and billing models. Check the current documentation for each model and route. A gateway can standardise usage and measurement, but the technical differences remain.
Sources
- OpenAI Prompt Caching: https://platform.openai.com/docs/guides/prompt-caching
- Anthropic Prompt Caching: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
- Google Gemini Context Caching: https://ai.google.dev/gemini-api/docs/caching
- AWS, Caching best practices: https://docs.aws.amazon.com/whitepapers/latest/database-caching-strategies-using-redis/cache-validity.html
Next step
Analyse the ten most expensive prompt paths. Mark stable prefixes, variable user data, and content where freshness is critical. Test caching on one route first and measure hits, cost, latency, and errors after every content change.