AI Prompt Caching Explained: How It Reduces Cost and Latency

Prompt caching lets an AI provider avoid recomputing the same long input prefix every time your application sends a similar request. When it works, repeated context can cost less to process and can reach generation sooner.

The important word is prefix.

Prompt caching is usually not a general-purpose key-value cache where you say “cache this answer.” It is primarily an optimization for reusing already processed input context: system instructions, tool definitions, examples, documents, or conversation history that remain identical across requests.

That distinction explains most cache hits—and most cache misses.

The basic idea

Suppose an application repeatedly sends this shape:

[12,000 tokens of stable instructions and reference material]
[200 tokens of user-specific context]
[current user question]

Without caching, the provider may need to process the stable 12,000-token prefix on every request.

With a reusable prompt cache, the serving system can recognize that prefix and reuse its previously computed representation. The new request still includes or references the changing suffix, but the expensive stable prefix gets cheaper or faster treatment according to the provider’s caching model.

Conceptually:

Caching does not mean the model remembers arbitrary facts forever. It is an inference optimization with provider-defined matching and lifetime rules.

Prompt caching is different from application caching

These are separate layers:

Cache	What it reuses	Example
HTTP/CDN cache	A previous network response	Static image or public GET result
Application result cache	A previous application answer	Same deterministic database lookup
Retrieval cache	Search/retrieval work	Reusing document ranking results
Prompt cache	Previously processed model input	Stable prompt prefix
Conversation storage	Durable user/app state	Saved messages and attachments

A prompt cache normally does not let you skip model generation entirely. It reduces work on repeated input before generation.

Why prefix order matters

If cache matching works on a reusable prefix, request ordering becomes an architectural concern.

Consider two prompt layouts.

Cache-friendly

system instructions
stable tool definitions
stable reference document
conversation history
current timestamp
current user message

Cache-hostile

current timestamp
random request UUID
system instructions
stable tool definitions
stable reference document
conversation history
current user message

The second request changes almost immediately, so a prefix cache may stop matching before it reaches the expensive stable content.

That leads to a durable rule:

Put stable, widely reused context before volatile request-specific context whenever the provider’s prompt format allows it.

Do not blindly reorder semantic conversation content just for caching, but avoid injecting unnecessary volatility near the beginning of a request.

A cache hit still consumes context

Prompt caching reduces processing cost or latency. It does not magically remove cached tokens from the model’s context window.

If a model has a context limit of $C$, and your request contains:

• $I_c$ cached input tokens;
• $I_u$ uncached input tokens;
• $O$ output tokens;

then the context constraint is still conceptually related to the total input and output budget:

$$I_c + I_u + O \le C$$

The provider may charge or process $I_c$ differently, but those tokens still represent context the model attends to.

This matters for long chats. Prompt caching can make a 60,000-token conversation cheaper to continue, but it does not make that conversation fit inside a 32,000-token model.

See What Is an AI Context Window? for the separate context-budget problem.

The cost model is read, write, and uncached input

Provider pricing changes over time, so it is better to understand the variables than memorize one table.

Let:

• $U$ = uncached input tokens;
• $W$ = cache-write tokens;
• $R$ = cache-read tokens;
• $O$ = output tokens;
• $p_u$, $p_w$, $p_r$, $p_o$ = their respective token prices.

Then a simplified request cost is:

$$\text{cost} = U p_u + W p_w + R p_r + O p_o$$

A first request may contain a large $W$ because the reusable prefix has to be written. Follow-up requests can become attractive when much of that prefix becomes $R$.

This is why caching is not automatically cheaper for a prefix used once.

If a provider charges more for a cache write than ordinary input, the cache only pays for itself after enough reads.

Break-even thinking

Suppose a stable prefix contains $N$ tokens.

Without caching, processing it $k$ times costs approximately:

$$kN p_u$$

With one cache write and $k-1$ reads:

$$N p_w + (k-1)N p_r$$

Caching is economically useful when:

$$p_w + (k-1)p_r < k p_u$$

You do not need to calculate this manually for every chat, but the equation explains the design tradeoff:

• a huge prefix reused many times is a strong candidate;
• a tiny prefix used twice may not matter;
• a costly long-lived cache may not make sense for traffic that never returns.

OpenAI: prefix caching is increasingly explicit

OpenAI has supported automatic prompt caching for repeated prefixes for years. Current OpenAI APIs also expose cache usage in token accounting, and newer models add more explicit cache controls and breakpoints.

The exact controls depend on the model/API generation, so avoid hard-coding assumptions that every OpenAI-compatible endpoint supports the same cache fields.

The durable OpenAI-side principles are:

• keep reusable content at the beginning;
• keep volatile content later;
• inspect reported cached-token and cache-write usage instead of assuming hits;
• use stable cache keys/breakpoints only where the current model supports them;
• do not assume an OpenAI-compatible server implements OpenAI’s caching semantics.

For OpenAI-specific state and API choices, see OpenAI Responses API vs Chat Completions.

Anthropic: cache control is part of the request model

Anthropic exposes prompt caching more explicitly.

Current Claude API documentation supports both:

• automatic caching, enabled with a top-level cache_control; and
• explicit cache breakpoints attached to cacheable content blocks.

The cache applies to the prompt prefix through the selected breakpoint, including relevant tool definitions, system content, and messages in their protocol order.

A simplified explicit example looks like:

{
  "system": [
    {
      "type": "text",
      "text": "Long stable instructions...",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [
    { "role": "user", "content": "Current question" }
  ]
}

Anthropic also exposes different TTL choices, including a short default lifetime and an optional longer lifetime. Longer retention can carry a higher write cost, so it should match actual request cadence rather than being selected automatically.

A useful implementation detail is that the response reports separate cache-read and cache-creation token counts. Those fields should feed your analytics rather than being collapsed into ordinary input tokens.

Gemini: implicit and explicit context caching are distinct

Gemini currently exposes two caching ideas, but they are not available through every API surface.

Implicit caching

On supported newer Gemini models, implicit caching can happen automatically when requests share sufficiently large common prefixes. Developers do not create a cache object; the service identifies reuse and reports cached usage.

The newer Interactions API currently supports implicit caching only.

Explicit caching

Gemini’s Generate Content API can also create a named cached-content resource. You cache a large body of input once and refer to that cache from later Generate Content requests. Explicit cached-content resources are not currently supported by the Interactions API.

This is useful for workloads such as:

• asking many questions about the same long document;
• repeated analysis of the same media;
• a large stable instruction corpus;
• recurring work against the same repository snapshot.

Explicit caching introduces lifecycle state: cache name, model association, creation/expiration, and TTL. That is different from an automatic prefix cache and should be represented separately in application code.

Caching and long chat history

Long multi-turn chats often have an attractive caching shape because each request begins with much of the previous request.

Request 10 may look like:

system
messages 1...9
new message 10

Request 11 becomes:

system
messages 1...9
message 10
assistant 10
new message 11

Most of the beginning is unchanged, so a prefix cache can reuse substantial work.

But several client behaviors can destroy that stability.

Rewriting old messages

If your client continuously reformats, normalizes, or reserializes earlier messages differently, the byte/token sequence may change and reduce cache reuse.

Moving instructions around

Changing the system/developer prefix every turn can invalidate a large reusable region.

Reordering tools

A tool list generated from an unordered map may serialize in a different order between calls even if the tool set is logically identical.

Injecting volatile metadata early

Timestamps, request IDs, analytics metadata, or temporary status text placed near the front can break prefix matching.

Deterministic serialization is a performance feature.

Tool definitions can dominate the prefix

Agentic applications may send dozens or hundreds of tool schemas with every request. Those schemas can consume a surprising number of tokens.

If the provider supports caching tool definitions, a stable tool catalog can be an excellent cache target.

But caching does not fix an unnecessarily huge tool universe. Sending 150 tools to a model that only needs 8 wastes context and can make tool selection harder.

The better order of operations is:

1. expose only relevant tools;
2. keep their definitions deterministic;
3. cache the stable tool prefix where supported.

Prompt caching should optimize a good prompt architecture, not excuse a bloated one.

Cache invalidation is semantic, not only textual

Changing any cache-relevant prefix can cause a miss, but some changes deserve special attention because they alter behavior:

• system instructions;
• tool schemas;
• enabled built-in tools;
• model ID;
• provider endpoint;
• safety or generation settings that participate in provider cache identity;
• attached documents or files;
• message content/order.

A client should not try to “force” a cache hit by pretending two semantically different prompts are equivalent.

Correctness comes before cache reuse.

Provider fallback complicates caching

Imagine a conversation has a warm prompt cache at Provider A. Provider A becomes unavailable and the router sends the next request to Provider B.

Provider B may have:

• no cache entry;
• different caching capabilities;
• different cache pricing;
• a different tokenizer;
• a different model entirely.

The fallback request can still succeed, but latency and cost may jump.

That means routing decisions and cache locality are connected. Some gateways use sticky routing to keep a conversation on the same healthy provider when doing so improves cache reuse.

Do not assume a cache hit survives a provider or model switch.

See How to Build Reliable AI Provider Fallback and Model Routing for the broader routing problem.

Privacy: a prompt cache is still provider-side processing state

Caching is a performance feature, not a privacy boundary.

Questions to ask include:

• Is cached content stored only in memory or in another retention tier?
• How long can it remain available?
• Is it isolated by organization/project/account?
• How does Zero Data Retention interact with caching?
• Can the application explicitly delete an explicit cache object?
• Does the provider document whether prompt caches are used for training? Do not infer this from the word “cache.”

The answers vary by provider and product configuration. Check the current provider documentation for sensitive deployments.

What to measure

A cache-aware analytics model should preserve more than input_tokens and output_tokens.

Useful fields include:

type CacheUsage = {
  uncachedInputTokens?: number;
  cacheWriteTokens?: number;
  cacheReadTokens?: number;
  outputTokens?: number;
  providerReportedCost?: number;
};

Then monitor:

• cache-hit ratio;
• cache-read tokens / total input tokens;
• cache-write volume;
• cost per completed request;
• first-visible-output latency;
• model/provider switches;
• cache performance by conversation type.

A high hit percentage is not a goal by itself. If you doubled the prompt size to improve the hit ratio, you may have made the system worse.

Common prompt-caching mistakes

“Caching means I can send unlimited context”

No. Cached tokens still occupy context.

“The second request will definitely hit”

No. Matching rules, minimum sizes, TTLs, model selection, routing, and provider implementation all matter.

“I should cache everything”

No. Cache writes can have cost, and short-lived or unique content may never be reused.

“A cache hit means the answer is cached”

No. The model normally still generates a new output. The reused object is processed input context, not the previous answer.

“All OpenAI-compatible providers cache the same way”

No. Compatibility of /v1/chat/completions does not imply compatibility of provider-side caching semantics.

“Caching is only about money”

No. For long prompts it can improve time to first output and reduce repeated preprocessing work, even when cost is not the main concern.

A cache-friendly request checklist

For a long-running AI client:

• keep stable instructions early;
• keep volatile per-request metadata late;
• serialize tools deterministically;
• avoid rewriting old transcript content unnecessarily;
• preserve provider-reported cache metrics;
• distinguish cache reads from cache writes;
• understand TTL before paying for longer retention;
• do not treat cached tokens as free context-window space;
• expect cache loss after provider/model switches;
• benchmark actual cost and latency rather than assuming caching helps.

Where BYOKchat fits

A BYOK client cannot assume one cache model because the user’s chosen provider controls caching behavior. The useful client-side job is to keep conversation serialization stable, preserve provider usage fields, and avoid unnecessary prefix churn while still respecting each provider’s native protocol.

That lets prompt caching remain an optimization rather than becoming a hidden dependency in the conversation model.

Further reading

• OpenAI model guidance
• Anthropic prompt caching
• Gemini context caching
• Understanding AI API Costs and Token Usage
• What Is an AI Context Window?