Why semantic caching can remove entire LLM calls, where prefix caching stops, and what Redis LangCache is actually doing

Imagine paying an engineer every time someone asks:

“Can I get a refund on my monthly plan?”

Then paying again for:

“Is my monthly subscription refundable?”

And again for:

“Can I cancel and get my money back?”

A human support agent immediately recognizes these as the same question.

Most LLM applications do not.

Unless we deliberately add another layer, all three requests travel through the complete generation pipeline: retrieve context, construct a prompt, process thousands of input tokens, generate an answer, and pay for all of it again.

That is a strange property of modern AI systems.

We have spent enormous engineering effort making inference faster, batching requests, quantizing models and optimizing KV caches, while sometimes ignoring the cheapest inference call of all:

the one you never make.

That is the idea behind semantic caching.

And it is also why Redis's LangCache is more interesting than the headline “90% cheaper LLM calls” suggests.

First, understand what we are actually caching

There are several things people call “LLM caching,” and mixing them together makes the architecture confusing.

Consider this request:

System:
You are a helpful support assistant for Acme.

Context:
Refunds are allowed within 14 days...

User:
Can I get my money back for the monthly plan?

There are at least three places we can reuse previous work.

1. Exact response caching

If exactly the same request appears again, hash the request and return the previous response.

This is ordinary caching.

It is extremely cheap and extremely safe when the request is deterministic enough.

Unfortunately:

Can I get a refund?

and

Can I get my money back?

produce different hashes.

Humans see the same intent. The cache sees different bytes.

2. Prefix caching

Modern LLM inference engines often cache the KV states generated while processing repeated prefixes.

If thousands of requests share:

System prompt
+ company policy
+ tool instructions
+ common context

the model does not need to recompute all of that attention state every time.

This can save substantial prefill compute.

But notice what still happens.

The new user tokens must be processed.

The request still reaches the model.

And the answer still has to be generated token by token.

Prefix caching makes an LLM call cheaper.

It does not eliminate the LLM call.

3. Semantic response caching

Semantic caching moves one level higher.

Instead of asking:

“Have I seen these exact tokens before?”

it asks:

“Have I already answered something that means essentially the same thing?”

The application embeds the new question into a vector, searches previously cached questions for a sufficiently similar vector, and, if confidence is high enough, returns the stored response.

No generation call.

No output decoding.

No model queue.

Potentially no expensive RAG pipeline either.

Redis LangCache packages this pattern as a managed service. It generates embeddings, performs similarity search, stores prompt-response pairs and returns a cached response when the similarity threshold is satisfied.

What happens on a semantic-cache request?

The core loop is almost disappointingly simple.

Redis describes LangCache integration as essentially two operations around your existing LLM call:

1. search before calling the LLM;
2. store the fresh response after a miss.

The managed service can generate embeddings itself, and Redis currently supports Redis-provided embeddings, OpenAI embeddings, or compatible bring-your-own embedding endpoints.

This sounds almost trivial.

The difficult part is deciding what “similar enough” means.

And that is where semantic caching stops being a caching problem and becomes a correctness problem.

Similarity is not equivalence

Imagine these two questions:

“Can I cancel my subscription?”

“Can I cancel my subscription after 30 days?”

Their embeddings will probably be close.

Their answers may not be.

Or:

“Can employees access customer financial records?”

and

“Can administrators access customer financial records?”

Only one word changed.

That one word may completely change the authorization answer.

This is the fundamental failure mode of semantic caching:

false-positive cache hits are worse than cache misses.

A miss costs money.

A wrong hit confidently returns the wrong answer without even giving the LLM an opportunity to reason again.

That means your similarity threshold is not merely a performance knob.

It is part of your application's correctness policy.

Redis Cloud's LangCache currently allows similarity thresholds from 0.5 to 1.0, with 0.85 as the documented default and Redis recommending roughly 0.8–0.9 as a starting region before tuning against your workload.

But there is no universal “correct” threshold.

Your embedding model matters.

Your language matters.

Your domain matters.

And the cost of returning the wrong result matters enormously.

A movie recommendation system can tolerate much looser semantic matching than an insurance eligibility assistant.

The production architecture We would actually build

I would not put semantic similarity directly in front of every LLM call.

I would use multiple gates.

There are several important pieces hiding in this diagram.

Gate 1: Is this request cacheable?

Some questions should never be semantically cached.

For example:

What is my current account balance?

Where is my package right now?

Summarize today's incidents.

The answer depends on current state.

But:

What is your refund policy?

may be an excellent candidate.

Classify requests before trying to reuse their answers.

Gate 2: Scope before similarity

Suppose two customers ask semantically identical questions but have different contracts.

A globally shared semantic cache could accidentally return Tenant A's response to Tenant B.

The cache search therefore needs more than embeddings.

It needs metadata boundaries.

Redis's underlying semantic-cache pattern supports storing fields such as tenant, locale, model version and other metadata beside the prompt/response, then applying those filters during vector search. LangCache also exposes custom attributes and scopes for this purpose.

Think of the lookup as:

Find something semantically similar
AND tenant = "acme"
AND locale = "en-IN"
AND policy_version = "2026-09"
AND response_type = "refund-policy"

Now vector similarity answers the final question, not every question.

Then comes invalidation, the oldest problem in caching

Semantic caching does not escape the classic cache problem:

When is the stored answer no longer true?

Suppose your refund policy changes tomorrow.

You now have thousands of beautifully embedded, semantically indexed, completely wrong answers.

You need some combination of:

• TTL;
• manual invalidation;
• policy/version attributes;
• model-version boundaries;
• source-document versions;
• event-driven eviction.

LangCache supports configurable TTL and cache-entry deletion, while Redis Cloud exposes cache-management and monitoring controls.

For knowledge-heavy applications, I particularly like versioned cache scopes.

Instead of asking:

semantic_key = embedding(question)

think closer to:

semantic_key_space =
    tenant
  + knowledge_version
  + model_version
  + response_policy
  + locale

Then similarity happens inside the correct world.

Where the economics become interesting

Consider the example that motivated this article.

A direct LLM call used:

• 514 input tokens
• 250 output tokens
• 2.232 seconds end-to-end

A paraphrased request served from Redis LangCache returned in:

• 0 LLM input tokens
• 0 LLM output tokens
• 0.37 seconds

That particular run was about 6× faster.

The exact number is not important because it will change with model, region, prompt length and network latency.

What matters is what disappeared from the request path.

The cache hit did not make generation faster.

There was no generation.

Redis currently markets LangCache as saving up to 90% on API costs, and other Redis material cites up to 15× faster cache-hit responses. Treat those as vendor-reported upper-bound figures rather than expected production averages. Redis's own documentation gives the more conservative model: actual savings depend heavily on cache-hit rate and token economics.

A useful simplified model is:

Expected LLM spend
≈
requests
× (1 - semantic_cache_hit_rate)
× average_generation_cost

Then add:

+ embedding cost
+ cache infrastructure
+ storage
+ false-hit remediation cost

That last term rarely appears in marketing calculators.

It should.

Prefix caching or semantic caching?

Both.

They solve different levels of repetition.

Imagine 100 requests.

Twenty can reuse complete answers because they mean the same thing as something already answered.

Those 20 should ideally stop at the semantic cache.

The other 80 still reach the model.

If all 80 share the same large system prompt and RAG prefix, prefix caching can make those model calls cheaper.

So the architecture becomes:

Semantic caching reduces number of calls.

Prefix caching reduces cost of remaining calls.

Context pruning reduces tokens inside remaining calls.

Quantization and optimized inference reduce compute per remaining token.

These mechanisms are complementary, not competing.

What I would monitor in production

A semantic cache deserves its own dashboard.

At minimum:

Hit rate

What percentage of eligible requests are actually served from cache?

Redis Cloud exposes cache-hit ratio and related service metrics.

But a high hit rate by itself can be dangerous.

Lowering the similarity threshold can make hit rate look fantastic while quietly increasing wrong answers.

So also measure:

False-hit rate

Sample cached responses and verify that the matched query really deserved the same answer.

This is arguably the most important semantic-cache metric.

Nearest-neighbor score distribution

Do not observe only hits and misses.

Watch the similarity scores around the threshold.

If thousands of requests cluster at 0.84 while your cutoff is 0.85, the cache may be missing obvious reuse.

If accepted matches cluster barely above the boundary, you may be too aggressive.

Cost avoided

Measure actual tokens that would have been generated, not simply number of cache hits.

A cache hit avoiding a 20-token answer is very different from avoiding a 3,000-token analysis.

Staleness

How often are cached answers invalidated?

Which entries are repeatedly served after their underlying source changed?

Hit latency vs generation latency

The entire reason this layer exists is to remove expensive work.

Prove that it does.

One architecture improvement: use confidence bands

I would not necessarily make semantic caching binary.

Instead of:

similarity >= 0.85 → HIT
otherwise → MISS

consider:

>= 0.93
high-confidence hit
return immediately

0.84 - 0.93
borderline
run additional validation

< 0.84
miss
call normal pipeline

The middle path could use:

• intent classification;
• structured metadata;
• business rules;
• a small reranker;
• or even a cheap model acting as a semantic-equivalence judge.

This increases complexity, so I would only add it where false hits are costly.

But it exposes an important principle:

semantic caching is really a routing system with memory.

Once you see it that way, the production design becomes much clearer

When semantic caching is a bad idea

There are workloads where I would barely use it.

Highly personalized outputs.

Real-time financial or operational queries.

Creative generation where variation is intentional.

Tasks where the answer depends heavily on conversation history.

Prompts whose meaning changes based on hidden authorization context.

High-risk medical, legal or financial outputs where reuse deserves additional validation.

The mistake is assuming that because two prompts are semantically similar, their correct responses are interchangeable.

Those are not the same claim.

Semantic caching works best where answers are repeatable, stable and expensive enough to be worth reusing.

Customer-support policy questions are almost the textbook case.

Why Redis is a sensible place for this

You do not need Redis LangCache to implement semantic caching.

You can build the pattern yourself using:

• an embedding model;
• a vector index;
• metadata filtering;
• TTL;
• cache storage;
• metrics;
• invalidation logic.

Redis itself documents this lower-level pattern using Hash or JSON entries plus vector search and metadata prefilters.

LangCache is essentially the managed version of that stack.

It currently handles embedding generation, semantic lookup, TTL, attributes and filtering, APIs/SDKs, and Redis Cloud monitoring. It is available in public preview.

That means the build-vs-buy question is straightforward.

If semantic caching is a core differentiator or you need unusual ranking logic, build it.

If it is infrastructure you simply want working reliably in front of an existing LLM/RAG application, a managed semantic cache becomes attractive.

The bigger lesson

LLM infrastructure has spent several years optimizing how efficiently we perform inference. Semantic caching asks an almost embarrassingly simple question first:

Do we need inference at all?

That question becomes increasingly valuable as AI applications mature.

Real production traffic is repetitive.

Users paraphrase the same questions.

Agents execute similar sub-tasks.

Support systems repeatedly explain the same policies.

RAG systems regenerate the same conclusions from unchanged documents.

Once that repetition becomes visible, the architecture changes.

Exact caching removes identical work.

Semantic caching removes equivalent work.

Prefix caching removes repeated model computation.

Inference optimization makes whatever remains cheaper.

The mature LLM stack will probably use all four.

Because the most impressive inference optimization is still not a faster GPU.

It is recognizing that you already know the answer.

SPONSORED

Need to stop burning your engineering budget on repetitive LLM calls?

Lert's Connect

References & Further Reading

• Redis Engineering: Redis LangCache Documentation & Architecture Overview— Official specification for managed semantic caching, integration patterns, and vector indexing.
• OpenAI Developer Docs: Optimizing Latency & Costs with Prompt Caching— Technical guide on model-side prefix caching mechanisms and token-level optimization.
• AWS Machine Learning Blog: Building a Read-Through Semantic Cache with Amazon OpenSearch Serverless and Amazon Bedrock— Enterprise reference architecture for vector-based LLM response caching.
• Redis Vector Library: Vector Search & Hybrid Metadata Prefiltering— Detailed guide on implementing cosine distance metrics, indexing strategies, and multi-tenant scoping.