LLM caching is the practice of storing processed prompt representations or full model responses so subsequent requests can reuse them instead of re-running inference from scratch. It reduces both cost and latency with no change to model quality.

Two types matter for enterprise deployments: prompt caching (provider-level, discounts stable prefix tokens) and semantic caching (application-level, returns stored responses for semantically similar queries without hitting the model at all).

Every time your application sends the same 8,000-token system prompt to the model and pays full input price, it is paying for something it already paid for yesterday. And the day before. Caching is the fix. It is not complicated. Most teams are just not using it.

TL;DR - Key Takeaways

Prompt caching reduces the cost of repeated stable input tokens by up to 90% on Anthropic (Claude Sonnet 4.6 input drops from $3.00/M to $0.30/M on cache hits) and 50% on OpenAI. No quality change. No model change. One implementation step.
Semantic caching goes further: it returns a stored response for any query semantically similar to a previously answered one, skipping the model entirely. Production deployments hit 20-45% of traffic without touching the model, per benchmarks from Technion (2026).
The average prompt token count grew nearly 4x between early 2024 and late 2025, from roughly 1,500 tokens to 6,000 per request. Longer prompts make caching more valuable, not less.
ProjectDiscovery raised their cache hit rate from 7% to 84%, cutting total LLM spend by 59-70% with a single architectural change. That is a documented production outcome, not a benchmark projection.
31% of LLM queries in production exhibit semantic similarity sufficient to return a cached response, per analysis from Introl (2025). That is a significant amount of spend going toward identical work.
NeuralTrust TrustGate enforces caching policies at the gateway layer so they apply consistently across every connected application.

What is LLM caching?

The first time a startup I know switched on prompt caching, their monthly Anthropic bill dropped $14,000 in thirty days. They had not changed the model. They had not rewritten a single prompt. They had not touched the application logic. They added one JSON field to their API calls.

The bill had been high for a simple reason: their agents ran a 12,000-token system prompt containing tool definitions, company policies, and a long document context. Every request re-sent every token. Every token got billed at full input price. The model processed something it had already processed ten thousand times before.

Caching fixed that. On a cache hit, the model does not reprocess the cached prefix. It reads from stored state. That is why the discount is so large. You are not buying a cheaper computation. You are skipping a computation you already paid for.

LLM caching consists in storing the processed representations of prompt prefixes (prompt caching) or full model responses (semantic caching) so subsequent requests can reuse them at a fraction of the cost of re-running inference from scratch.

Two types. Different mechanics. Different use cases. Both worth understanding before you pick one.

Prompt caching: how it works, what it costs, and what it saves

Prompt caching operates at the provider level. When your request arrives, the provider checks whether the opening portion of your prompt has been seen recently and stored. If it has, the model skips reprocessing that portion and reads the cached key-value tensors instead. You pay the cache-read rate, not the standard input rate.

The economics are straightforward:

On Anthropic (Claude Sonnet 4.6, July 2026):

Standard input: $3.00 per million tokens
Cache write (5-minute TTL): $3.75 per million tokens (1.25x the standard rate)
Cache write (1-hour TTL): $6.00 per million tokens (2.0x the standard rate)
Cache read: $0.30 per million tokens (0.10x the standard rate, a 90% discount)

On OpenAI (GPT-4.1 and newer, July 2026):

Cache reads: up to 90% off standard input on newer models
No separate write fee: caching is automatic, no code changes required

You break even on Anthropic's 5-minute TTL after roughly 1.4 cache reads per cache write. A hit rate above 30% on stable prompts means caching saves money. A hit rate above 60% means caching saves a significant amount of money.

The image shows a chart comparing the breakeven points for cached vs. non-cached prompts

Latency also improves

Cached reads skip the prefill computation for the cached portion. OpenAI reports up to 80% faster time-to-first-token on cached reads compared to fresh prompts. (Source: OpenAI Prompt Caching documentation)

What qualifies for caching?

Prompts must be at least 1,024 tokens on both Anthropic and OpenAI for caching to apply. Shorter blocks are not cached even if you add the cache marker. The cached content must be a contiguous block at the start of your prompt, identical across requests.

This means stable content (system prompts, tool definitions, long document context, example libraries) gets cached. On the other hand, dynamic content (the user's specific question, real-time context) goes at the end and does not get cached. Prompt structure matters enormously, and the wrong structure silently kills your hit rate.

Semantic caching: how it works and when it beats prompt caching

Prompt caching handles the stable parts of your prompts. Semantic caching handles the queries themselves.

The idea: when a user asks "what is your refund policy?", you compute an embedding of that query and check your cache. If a semantically similar query ("how do I get a refund?", "can I return my order?") was recently answered, you return the cached response without calling the model at all. The similarity threshold (typically a cosine similarity score above 0.8-0.85) determines whether a query counts as a cache hit.

Production deployments typically hit 20-45% of traffic with semantic caching, per benchmarks published from Technion's Computer Science department (2026). Not the 95% that vendor materials sometimes suggest, but 20-45% of your total LLM calls never reaching the model is still material cost reduction, especially for high-volume workloads where 31% of queries exhibit meaningful semantic similarity. (Source: Introl, 2025)

The open-source toolkit is GPTCache (github.com/zilliztech/GPTCache). GPTSemCache, a research implementation, achieved cache hit rates of 61.6-68.8% across query categories with 97%+ accuracy. (Source: GPTSemCache, Regmi and Pun, 2024, cited in arxiv.org/html/2603.03301v1)

The tradeoff: semantic caching introduces a new failure mode. A poor embedding model maps semantically distinct queries to nearby vectors and returns a wrong cached response. This is acceptable in search, near-misses are fine. It is not acceptable in LLM caching: a hallucinated response served from cache is worse than a fresh model call. The embedding model quality and similarity threshold are the two most important implementation decisions.

Prompt caching vs. semantic caching: which one to use

These are not alternatives. They work at different layers and solve different problems. The question is not which to choose, it is which to implement first.

	Prompt caching	Semantic caching
What it caches	Stable prompt prefixes (system prompt, tool defs, documents)	Full model responses for semantically similar queries
Where it operates	Provider-level (Anthropic, OpenAI, Google)	Application-level (your infrastructure)
Implementation	One JSON field (`cache_control`) or automatic	Embedding model + vector store + similarity threshold
Minimum token requirement	1,024 tokens minimum	No minimum; works on any query
Cache hit discount	90% off on Anthropic; up to 90% on OpenAI newer models	100%: no model call at all
Quality risk	None: identical model output	Low to moderate: depends on embedding model quality and threshold
Best for	Large stable system prompts, tool catalogs, RAG document context	FAQ-type queries, support tickets, repetitive search, classification
Production hit rates	80-95% on stable-prompt workloads	20-45% in practice

Start with prompt caching if you have a system prompt over 2,000 tokens and more than a few hundred daily API calls. It is the easiest implementation and the fastest payback. The Agentbrisk analysis (April 2026) shows a customer support application with a 4,000-token system prompt and 6,000-token RAG context achieving a 76% total API cost reduction at a 95% cache hit rate, from $10,000/month to $2,361/month.

Add semantic caching for query types that are genuinely repetitive. FAQ chatbots, support ticket classifiers, product catalog search, document summarization on the same documents. These are the use cases where the same user intent appears in different phrasings across different sessions. Semantic caching captures that reuse. Prompt caching does not.

How to structure your prompts to maximize cache hit rate

The structure of your prompt determines whether caching works. The rule is simple: put the most stable content first, the most dynamic content last. Any change to a cached block invalidates that block and everything that follows it.

The correct order is:

Tool definitions (most stable: change only when you update your tools)
System prompt instructions (stable: change when you update application behavior)
Reference documents (moderately stable: change when your knowledge base updates)
Conversation history (changes every turn)
User query (changes every request)

The common mistake is mixing stable and dynamic content. If your system prompt contains a timestamp, a user name, or any session-specific value, the cache block invalidates on every request. The entire prefix must be byte-identical for a cache hit.

A 10,000-token tool catalog cached at 80% hit rate saves more per month than a perfectly compressed 500-token prompt at 100% hit rate. The absolute token count matters more than the hit rate percentage. (Source: Tokonomics, May 2026)

Common mistakes that kill your cache hit rate

Mistake 1: Dynamic content inside the cached prefix

A timestamp, user ID, or any session-specific value in your system prompt breaks the cache on every single request. Audit your system prompt for anything that changes per-session or per-user before enabling caching.

Mistake 2: Wrong prompt order

Conversation history before system prompt. RAG context before tool definitions. Any ordering that puts unstable content early in the prompt means the stable content that follows it never gets a stable prefix to build on.

Mistake 3: Prompts under 1,024 tokens.

Caching does not activate below the minimum threshold. If your system prompt is 800 tokens, caching does nothing. Consolidating scattered instructions into a single coherent system prompt both improves the model's instruction-following and crosses the caching threshold.

Mistake 4: Not monitoring hit rate.

You cannot optimize what you cannot measure. Log cache hits and misses explicitly. A hit rate below 40% on a stable-prompt workload signals a structural problem, usually dynamic content inside the cached prefix or prompt ordering issues. See our Token Usage Monitoring guide for the full attribution and alerting framework.

Mistake 5: Using the wrong TTL

Anthropic offers 5-minute and 1-hour TTLs. The 1-hour TTL costs 2x the write premium versus 1.25x for 5-minute, but you break even after two cache reads instead of one. If your requests are spaced more than 5 minutes apart but arrive within the hour, the 1-hour TTL is cheaper.

The image shows the right order to structure your prompts for cache hits

FAQs about LLM caching strategies

1. What is prompt caching in LLMs?

Prompt caching is a provider-level feature that stores the processed key-value tensors from a stable prompt prefix on the provider's servers. When a subsequent request starts with the same prefix, the provider reads from the stored cache instead of reprocessing those tokens, and charges a discounted rate. Anthropic's cache read rate is 90% lower than the standard input rate. OpenAI's automatic caching delivers up to 90% off on newer models with no code changes required. Minimum 1,024 tokens for the cached block.

2. What is semantic caching and how does it differ from prompt caching?

Semantic caching operates at the application level and stores full model responses indexed by the embedding of the query. When a new query arrives with a cosine similarity above a configured threshold (typically 0.8-0.85) to a previously cached query, the stored response is returned without calling the model at all. This is fundamentally different from prompt caching, which reduces the cost of model calls that still happen. Semantic caching eliminates some model calls entirely. The tradeoff is implementation complexity and the risk of returning semantically similar but factually different responses.

3. How much can LLM caching save in production?

It depends on workload type and hit rate. Prompt caching at an 80% hit rate on a 10,000-token stable prefix saves roughly 72% of that prefix's input token cost. A customer support application with a 4,000-token system prompt and 6,000-token RAG context achieved a 76% total API cost reduction at a 95% cache hit rate, per Agentbrisk's analysis (April 2026). ProjectDiscovery raised their cache hit rate from 7% to 84% and cut total LLM spend by 59-70%. Semantic caching in production typically hits 20-45% of traffic, per Technion (2026) benchmarks.

4. Do I need both prompt caching and semantic caching?

For most production applications, start with prompt caching: it is one implementation step and delivers the highest immediate ROI. Add semantic caching for query types that are genuinely repetitive across different phrasings: FAQ chatbots, support ticket classifiers, document summarization on fixed corpora. The two techniques operate at different layers and compound. Prompt caching reduces the cost of every model call. Semantic caching eliminates some model calls entirely.

5. Does caching affect output quality?

Prompt caching produces byte-identical outputs to uncached calls: the model processes exactly the same context, just starting from stored state rather than recomputing it. There is no quality difference. Semantic caching can affect quality if the similarity threshold is set too low and semantically different queries receive cached responses. A well-tuned threshold (0.85+ cosine similarity) and a high-quality embedding model minimize this risk. GPTSemCache achieved 97%+ accuracy at production hit rates of 61.6-68.8%.

Key Takeaways

Prompt caching is the single highest-ROI optimization available to most production LLM applications right now. A 90% discount on cached tokens with no quality change and one implementation step, most teams are not using it.
Semantic caching eliminates model calls entirely for repetitive query types. Production hit rates of 20-45% are realistic for FAQ, support, and search workloads.
The two techniques are not alternatives. Stack them: prompt caching reduces the cost of every model call, semantic caching eliminates some model calls entirely.
Prompt structure determines cache hit rate. Most-stable content first, most-dynamic last. Dynamic content inside the cached prefix silently kills your hit rate.
Monitor hit rate explicitly. A rate below 40% on stable-prompt workloads signals a structural problem, not a traffic problem.
NeuralTrust TrustGate enforces caching policies and token budgets at the gateway layer across all connected applications.

AI Token Optimization: The Complete Enterprise Guide to Reducing LLM Costs (2026)
Prompt Compression: How to Cut Token Costs Without Losing Output Quality
LLM Cost Reduction: 12 Proven Strategies to Cut Your AI Inference Bill
Context Window Optimization: How to Manage Long Contexts Without Blowing Your Budget
LLM Model Routing: How to Automatically Send Queries to the Right Model
Token Usage Monitoring: How to Track, Attribute, and Optimise AI Spend in Production
Output Length Control: How to Stop LLMs from Over-Generating and Wasting Tokens
LLM Batching and Async Inference: How to Cut Costs on High-Volume AI Workloads
Fine-Tuning vs Prompting: A Cost Comparison for Enterprise AI Teams

About the Author

Roger Howroyd is Head of Global SEO and AI at NeuralTrust, where he leads the company's search strategy across SEO, AEO, GEO, and LLM optimization. He specializes in AI-powered search, content strategy, backlink development, and SEM. Connect on LinkedIn

NeuralTrust is an AI agent security platform, recognized in the Gartner 2025 Market Guide for AI Gateways and Guardian Agents, and the KuppingerCole 2025 Leadership Compass for Generative AI Defense. Headquartered in Barcelona with ISO 27001 certification.

LLM Caching Strategies: Prompt Caching, Semantic Caching, and When to Use Each