Skip to content
←Back to Blog
KV, Prefix, Prompt and Semantic Caching in LLMs
LLM InferenceKV CachePrefix Caching

KV, Prefix, Prompt and Semantic Caching in LLMs

A first-principles guide to KV, prefix, prompt, semantic, and exact-response caching, including RAG stitching failures, token-level invalidation, and production trade-offs.

Avi ChawlaAugust 28, 202630 min read71 views
ZH

Four different objects in an LLM stack are routinely called “caching.” They live at different scopes, survive for different amounts of time, and fail in different ways. A KV cache holds attention state for one request. Prefix caching keeps that state on a serving engine so another request can reuse it. Prompt caching is a hosted provider selling access to the same idea. A semantic cache stores complete answer strings and uses embedding similarity to decide whether an old answer is close enough to return.

The distinction matters because the first three layers are exact-match mechanisms. A miss costs latency and money, but it does not change the model's arithmetic. The fourth is a fuzzy-match mechanism: a hit can be factually wrong, return with HTTP 200, and never execute the model at all.

This guide follows Avi Chawla's X longform explanation from first principles. It keeps the original's executable examples, provider economics, vLLM block mechanics, RAG failure case, and production checklist. It also frames where the same machinery matters for physical-AI systems: hosted vision-language-action agents, robot-fleet assistants, retrieval over operating manuals, and long embodied-agent loops that repeatedly send the same tool schema and safety instructions.

Diagram separating KV cache, prefix cache, prompt cache, and semantic cache by the object each stores
Figure 1 | Four cache layers, four stored objects. KV cache stores attention tensors for one request, prefix cache keeps the same tensors as reusable server-side blocks, prompt cache exposes provider-side prefix reuse as a billing feature, and semantic cache stores finished response strings. The first three preserve exact computation; the fourth trades exactness for similarity.

The useful question is not “is caching faster?” It is “which object is being reused, under what exact identity, and which part of the request still has to run?”

1. The KV cache: reuse attention state inside one request

An autoregressive language model processes a request in two phases. During prefill, it consumes the prompt in parallel, computes a key vector and a value vector for every prompt token at every layer, and produces the first output token. During decode, it generates one token at a time. Instead of recomputing the entire sequence after every new token, the model retains the earlier keys and values and appends one new pair for each generated token.

Queries are not cached. Causal masking means a token's query is used when that token is processed and is not needed again. Its key and value, however, are read by every later token through attention. Saving K and V therefore removes the repeated matrix-matrix multiplication over the whole prefix. A decode step becomes a much cheaper matrix-vector operation over the new token, with attention reading the stored history.

Causal attention diagram showing each query used once while keys and values are reused by later tokens
Figure 2 | Why only K and V survive. A causal mask gives each query one job at the position where it is processed. Keys and values are read by all later tokens, so retaining them removes work from every decode step.

The optimization changes the bottleneck. Without a KV cache, each decode step performs attention over the full generated sequence. With it, the engine performs less arithmetic but must stream the whole cache from high-bandwidth memory on every step. The attention kernels can finish before the data has arrived, so decoding often becomes memory-bandwidth-bound rather than compute-bound.

Video 1 | The same generation with and without a KV cache. The side-by-side demonstration shows how the uncached path keeps repeating sequence work while the cached path advances from a retained state.

KV cache size is a serving constraint

For a simplified multi-head attention model, the cache is composed of two tensors per layer. A useful size estimate is:

$$S_{\mathrm{KV}}=2\,L\,H_{\mathrm{kv}}\,d_h\,T\,b$$

Here $L$ is the number of layers, $H_{\mathrm{kv}}$ is the number of key/value heads, $d_h$ is the head dimension, $T$ is the cached sequence length, and $b$ is bytes per element. The leading 2 is for K and V. The cache grows linearly with both context length and concurrency.

At long context, this is no longer a small side allocation. The source uses a 70B model at BF16 and a 128K context as an example: the KV cache can approach 40 GB, comparable to the model itself when its weights are stored at 4-bit precision. The result is a direct capacity trade: every cached token is useful because it avoids recomputation, but it also occupies memory that could hold another active request.

Comparison showing arithmetic removed by a KV cache while cache reads add memory traffic
Figure 3 | A KV cache removes arithmetic and adds a memory stream. The model no longer multiplies over the full history on every token, but it must load the stored K/V state. This is why a compute optimization can produce a bandwidth bottleneck.

Dynamic allocation versus fixed allocation

Hugging Face Transformers exposes the cache as a first-class object. The default DynamicCache grows as generation proceeds. It does not reserve space for tokens that may never be produced, which is a good fit for short or unpredictable requests. A StaticCache reserves a fixed maximum length up front. It is useful when the caller wants a stable memory region that can be copied or reused, but short requests pay for unused capacity.

Dynamic and static cache allocation diagrams comparing growth per token with a pre-reserved memory block
Figure 4 | Growth versus reservation. Dynamic cache allocates as tokens arrive; static cache reserves the room needed by the longest permitted generation. The right choice depends on whether memory predictability or short-request efficiency matters more.

A minimal Transformers v5 example makes the object visible. Normally generate() creates and destroys the cache internally. Passing a cache explicitly keeps a reference after generation returns.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache

model_id = "HuggingFaceTB/SmolLM2-360M-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("The capital of France is", return_tensors="pt")
inputs = inputs.to(model.device)

past_key_values = DynamicCache(config=model.config)
out = model.generate(
    **inputs,
    do_sample=False,
    max_new_tokens=20,
    past_key_values=past_key_values,
)

print(tokenizer.decode(out[0], skip_special_tokens=True))
print("prompt tokens:", inputs["input_ids"].shape[1])
print("total tokens:", out.shape[1])
print("cache length:", past_key_values.get_seq_length())

With a five-token prompt and twenty generated tokens, the final sequence contains 25 IDs. The reported cache length is 24: the final token's key and value are computed, but no later token attends to them during that call. The example confirms the important operational fact: the cache grows by exactly one position per decode step and belongs to the lifetime of this generation.

Four ways to shrink the cache

The article highlights attention-head sharing and cache quantization as practical capacity levers.

  • Grouped-query attention (GQA) shares one K/V head across a group of query heads. The cache becomes smaller, and the model performs more arithmetic per byte of cache loaded.
  • Multi-query attention (MQA) goes further by sharing one K/V head across all query heads. It produces the smallest cache and the largest reduction in bytes loaded per decode step.
  • Multi-head latent attention (MLA), used in the DeepSeek model family, compresses the key/value representation into a latent vector rather than storing a full independent head for every group.
  • KV-cache quantization stores the cache at lower numerical precision. It can nearly double capacity, but every read and write pays to quantize or dequantize.
Multi-head, grouped-query, and multi-query attention diagrams comparing key and value cache sizes
Figure 5 | Attention mapping determines cache width. Multi-head attention stores a separate K/V head for every query head. GQA shares one pair across a group, while MQA uses a single pair. Fewer K/V heads reduce memory traffic at the cost of less per-head representational freedom.
# Requires: pip install optimum-quanto
out = model.generate(
    **inputs,
    do_sample=False,
    max_new_tokens=20,
    cache_implementation="quantized",
    cache_config={"nbits": 4, "backend": "quanto"},
)

Quantization is not a universal speed switch. The backend may require the group size to divide the head dimension evenly, and short contexts may spend more time converting values than they save in memory traffic. It is most attractive when the alternative is running out of cache capacity or reducing concurrency.

A KV cache does not outlive the request by default

In ordinary serving, the engine frees the request's cache when generation finishes. A twenty-turn chat therefore does not automatically remember the first nineteen turns as GPU state. On turn twenty, the rendered conversation may contain all earlier messages, but the server can still prefill them again at full cost.

A local Python process can keep the cache object alive across turns. The key requirement is that the new token sequence must begin with the old sequence exactly.

from transformers import DynamicCache

past_key_values = DynamicCache(config=model.config)
messages = []

questions = [
    "What is the capital of France?",
    "And its population?",
]

for prompt in questions:
    messages.append({"role": "user", "content": prompt})

    inputs = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        return_tensors="pt",
        return_dict=True,
    ).to(model.device)

    input_length = inputs["input_ids"].shape[1]
    outputs = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=64,
        past_key_values=past_key_values,
    )

    completion = tokenizer.decode(
        outputs[0, input_length:],
        skip_special_tokens=True,
    )
    messages.append({"role": "assistant", "content": completion})
    print(input_length, past_key_values.get_seq_length())

The full chat template is rebuilt on every turn, and its prefix grows as prior messages are appended. If the rendered prefix remains byte-for-byte and token-for-token identical, only the new suffix needs prefill. If an earlier message is edited, a system prompt changes, or a template inserts a different control token, the old state can no longer be treated as the prefix of the new request.

Three conversation turns contrasting per-request cache release with a retained prefix cache
Figure 6 | Request-local state versus cross-turn reuse. A normal generation frees its KV blocks at completion, so a later turn repeats the accumulated history. Keeping the state alive, or moving it into a shared prefix cache, lets the next turn prefill only the extension.

2. Prefix caching: keep completed K/V blocks for later requests

Prefix caching begins with a simple change in serving policy: when a request ends, the engine keeps its KV blocks in memory instead of discarding them, and indexes them so a later request can search for an exact token prefix. The stored object is still attention state, but its scope is now the whole server rather than one generation call.

vLLM's default block size is 16 tokens. Each complete block receives a key built from the parent block's hash and the token IDs in the current block:

$$h_i=H(h_{i-1},\,\mathrm{tokens}_{16i:16i+15})$$

Because the parent hash is folded into the child, matching a block means that every earlier block matched as well. A flat cache keyed only by sixteen-token contents could confuse two blocks that look alike but occur after different histories. The hash chain turns block lookup into a prefix lookup.

Chained block hashes showing lookup stopping at the first missing block in a prefix
Figure 7 | The hash chain enforces prefix identity. A child block includes its parent's hash, so later matches are meaningful only after every earlier block has matched. The scheduler walks the chain from the front and stops at the first miss.

The scheduler's reuse rule can be reduced to a small pseudocode example:

BLOCK_SIZE = 16

def block_hashes(token_ids, salt=None):
    hashes, parent = [], hash(salt)

    # Only complete blocks are indexed.
    for start in range(0, len(token_ids) - BLOCK_SIZE + 1, BLOCK_SIZE):
        block = tuple(token_ids[start:start + BLOCK_SIZE])
        parent = hash((parent, block))
        hashes.append(parent)

    return hashes


def schedule(token_ids, cache):
    matched_blocks = 0

    for h in block_hashes(token_ids):
        if h not in cache:
            break
        cache[h].ref_count += 1
        matched_blocks += 1

    reused_tokens = matched_blocks * BLOCK_SIZE
    return reused_tokens, token_ids[reused_tokens:]

Two details in this short loop determine real behavior. First, only a complete sixteen-token block is hashed, so an incomplete trailing block is always prefetched again. Second, lookup stops at the first missing block; there is no attempt to resume at block five after block four misses, because the chain hash says the later state no longer has the same ancestry.

Cached full token blocks followed by a partial tail that is recomputed
Figure 8 | The tail below one block is not invisible. Only whole blocks receive reusable keys. If a prompt ends with a partial block, those remaining tokens are recomputed on the next request even though the earlier blocks hit.

Block size is therefore a trade-off. Larger blocks reduce table lookups and can improve memory locality, but they waste more work at the tail and share less finely across requests. Smaller blocks improve exact sharing and reduce tail waste, but increase metadata and lookup overhead.

Tenant salts prevent accidental sharing

Identical token sequences produce identical block keys. For requests from one application, that is the point: two users sending the same system prompt can read one physical copy of its K/V tensors. For requests from different customers, shared physical state may violate the desired isolation boundary even when no plaintext is returned.

A per-tenant salt changes the root of the hash chain. Identical text now maps to different keys and therefore different blocks. The separation costs memory and lowers the hit rate because each tenant maintains a private copy, but it makes cache ownership explicit rather than relying on the hope that identical text never crosses a boundary.

A local example with StaticCache

Transformers can prefill one shared prefix and reuse the resulting cache for several continuations. The local demonstration copies the cache because each generate() call mutates its past state in place. A production engine does not deep-copy the tensors; it shares physical blocks and increments reference counts.

import copy
from transformers import StaticCache

SHARED_PREFIX = """You are a careful assistant.
Answer in one short sentence."""

prompt_cache = StaticCache(config=model.config, max_cache_len=1024)
prefix_inputs = tokenizer(SHARED_PREFIX, return_tensors="pt").to(model.device)

with torch.no_grad():
    result = model(**prefix_inputs, past_key_values=prompt_cache)
    prompt_cache = result.past_key_values

for question in ["What is the capital of France?", "Name one ocean."]:
    full_prompt = SHARED_PREFIX + question
    inputs = tokenizer(full_prompt, return_tensors="pt").to(model.device)

    # Every request mutates its cache, so the demo gives each one a copy.
    past_key_values = copy.deepcopy(prompt_cache)
    outputs = model.generate(
        **inputs,
        past_key_values=past_key_values,
        do_sample=False,
    )

The prefill call samples no token. It exists only to populate and return a cache. The subsequent full prompts are tokenized together with the shared prefix, and reuse is valid only because the prefix IDs are identical. In vLLM, the physical blocks can be shared directly while each request holds references that prevent eviction.

Screenshot of the LMCache open-source repository page and project description
Figure 9 | Prefix reuse is also being pushed across storage tiers. LMCache is an open-source KV-cache layer for scalable LLM inference. It becomes relevant when the reuse problem extends beyond a single GPU's resident block table. Source: LMCache repository screenshot in the original article.

Prefix caching is not free

Three deployment constraints are easy to miss.

  • It saves prefill, not decode. Every generated token still attends over the cached state and requires the same decode work. Reporting a whole end-to-end speedup as a “cache win” overstates the mechanism.
  • Hashing and lookup consume work. On a workload where most prompts are unique, the saved prefill may be smaller than the cost of building and maintaining the index. Benchmarks have observed throughput regressions in that regime.
  • Cache and running sequences share GPU memory. Keeping more history can reduce the number of concurrent requests. Under pressure, eviction removes unreferenced blocks by recency, and mixed traffic can preferentially lose long prefixes whose loss is expensive.

RAG exposes the weakness of a single prefix chain

A retrieval-augmented prompt usually contains a stable instruction, several retrieved chunks, and a query. If the same documents return in a different order, the token sequence diverges early. The old chain hash does not match a later permutation even though the set of documents is identical.

RAG pipeline showing documents being embedded, retrieved, and reused as cached chunks by an LLM
Figure 10 | Reuse becomes harder when retrieval changes order. Each request still begins with stable instructions, but the retrieved chunk sequence can change. A naive chained prefix then sees a miss even when most of the documents have been processed before.

Prefilling each retrieved chunk independently and concatenating the resulting KV tensors does not repair the problem. Three things break:

  1. The stitched tensors carry positions from their original contexts. Moving a chunk later in the prompt requires different positional information.
  2. No chunk attended to any other chunk while its cache was built, so cross-chunk attention is missing.
  3. Each separately prefilled chunk can create its own attention sink at what the model interpreted as position zero.
Three failure modes of stitching independently prefilled chunk caches into one prompt
Figure 11 | Plain cache concatenation gives the wrong attention history. Positional state is not the only problem: independently built chunks never attended to one another, and each contributes an attention sink. The patches must recompute selected boundary tokens, not merely concatenate tensors.

LMCache's CacheBlend takes the more defensible route. It reuses chunk caches at new positions, then runs a cheap scoring pass to identify tokens whose cached values deviate most from what full attention would require. It recomputes roughly the top fifteen percent of those high-deviation tokens. That partial pass restores cross-chunk attention and repairs positional effects without falling back to a complete prefill.

CacheBlend scoring layers and recomputing selected high-deviation tokens after reusing chunk caches
Figure 12 | CacheBlend repairs only the tokens that need repair. Reused chunks are brought to their new positions, a scoring pass ranks deviations, and selected high-deviation tokens are recomputed. The original reports roughly two to three times better time to first token than full recomputation.
Video 2 | Prefix matching is a chain, not a bag of chunks. The animation shows reuse continuing only through the unchanged prefix and stopping where the suffix diverges.

CacheBlend plugs into vLLM, reads chunk boundaries from the prompt, and targets retrieval traffic whose document order changes. The important architectural lesson is broader: a shared prefix cache is exact and cheap to reason about; arbitrary cached chunks require a correctness repair step.

3. Prompt caching: the provider's billed prefix cache

With a hosted API, the caller does not control the block table, eviction policy, or physical placement of KV tensors. The provider exposes only a smaller interface: a marker that says which request prefix is worth retaining, a time-to-live, and usage counters that reveal whether the previous request wrote and the current request read.

The cached object is still model state, not a reusable text file. It is keyed by the fully rendered prompt. That includes chat templates, system defaults, tool descriptions, and other provider-side representation details. Two strings that look identical in application logs can render to different token sequences and miss.

Anthropic cache control in practice

import anthropic

client = anthropic.Anthropic()  # ANTHROPIC_API_KEY

LONG_INSTRUCTIONS = "You are a precise technical editor. " * 400

def ask(question):
    return client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=512,
        system=[{
            "type": "text",
            "text": LONG_INSTRUCTIONS,
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[{"role": "user", "content": question}],
    )

for question in ["Summarize section 3.", "Now rewrite it for a beginner."]:
    response = ask(question)
    usage = response.usage
    print(
        usage.cache_creation_input_tokens,
        usage.cache_read_input_tokens,
        usage.input_tokens,
    )

Only the cache_control marker touches caching. It belongs on the last block that should be included, not on a range. Everything from the start of the request through that breakpoint becomes one cacheable prefix. The variable user question remains after the breakpoint, so changing it does not invalidate the instruction block.

The first call writes a nonzero cache_creation_input_tokens count and reads zero. The next call writes zero, reads roughly the same prefix, and reports the remaining variable question as uncached input. If both write and read counters are zero, the prefix may have been below the model's minimum cacheable length. The request still succeeds; it simply receives no caching and no error.

If the marker is moved below the user question, the marked prefix changes on every call. The first call may still write a cache entry, but later calls have no reason to hit it.

Prompt-cache economics

Anthropic charges 1.25x the ordinary input rate to write a cache entry and 0.1x to read it. Longer retention can carry a higher write multiplier. The source notes that OpenAI applies the same 1.25x/0.1x pair on its current models.

The write premium is recovered only by later reads inside the TTL. For a prefix of $T_{\mathrm{cache}}$ tokens, a highly simplified comparison is:

$$C_{\mathrm{cached}}=1.25T_{\mathrm{cache}}+N_{\mathrm{read}}(0.1T_{\mathrm{cache}})+\text{uncached tokens}$$

That equation is not enough to decide whether caching is worthwhile. Recompute time, request arrival rate, eviction, and the probability that the route reaches the same provider-side cache all matter. A write with no later read is more expensive than doing nothing.

Diagram showing provider system content, tool definitions, and system prompt billed at uncached, cache-write, and cache-read rates
Figure 13 | One request can contain three different input prices. Stable provider content can be read from cache, a new prefix may be written at a premium, and the variable request suffix remains uncached. The source summarizes the common multipliers as 1.25x for writes and 0.1x for reads.

Breakpoints have a search horizon

Anthropic checks the declared breakpoint. On a miss, it can walk backward through a limited number of earlier blocks looking for a previous write. The source cites a cap of 20 blocks. If more than twenty blocks of conversation are added between calls, the previous breakpoint can fall outside that lookback window and later calls stop hitting even though the old data still exists somewhere.

Prompt caching diagram showing breakpoint placement above changing user content and a twenty-block lookback
Figure 14 | Place the boundary at the last stable block. Stable instructions and tool descriptions belong above the marker; query-specific content belongs below it. The provider can search backward only within a bounded window, so reusing an old breakpoint depends on how much new context has accumulated.

Conversation compaction rewrites the front

Prompt caching rewards append-only history. A summarization or compaction step replaces older turns near the beginning with a shorter summary. That may be the right context-management policy, but it creates a new prefix. The previous cache was keyed to the old history and cannot be used for the rewritten front.

Before-and-after conversation history showing compaction replacing the cached prefix with a summary
Figure 15 | Cache growth happens at the tail; compaction edits the head. Normal conversation extension preserves the earlier tokens. Replacing early history with a summary changes the prefix itself, so the next request starts cold from the cache's perspective.

When a context must be shortened, truncating tool output in place can be less disruptive than rewriting the full conversation. The stable instructions, tool schemas, and earlier turns remain byte-identical, while one later block changes. Whether that is acceptable is a product decision, not only an inference optimization.

4. Semantic caching: skip the model, including its correctness check

The first three layers all preserve exact model state. A semantic cache does something different. It embeds the incoming prompt, compares that vector with stored prompt vectors, and returns a stored response when cosine similarity crosses a threshold. On a hit, neither prefill nor decode runs.

That makes semantic caching attractive: it can save output tokens as well as input tokens. It also makes it dangerous: there is no model call in which the current request can contradict the stale answer.

A very small implementation looks like this:

import numpy as np
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("all-MiniLM-L6-v2")

class SemanticCache:
    def __init__(self, threshold=0.95):
        self.threshold = threshold
        self.vectors = np.empty(
            (0, encoder.get_sentence_embedding_dimension())
        )
        self.prompts = []
        self.responses = []

    def _embed(self, text):
        return encoder.encode([text], normalize_embeddings=True)[0]

    def lookup(self, prompt):
        vector = self._embed(prompt)
        if not self.prompts:
            return None, 0.0, vector

        scores = self.vectors @ vector
        best = int(np.argmax(scores))
        if scores[best] >= self.threshold:
            return self.responses[best], float(scores[best]), vector
        return None, float(scores[best]), vector

    def store(self, prompt, response, vector):
        self.vectors = np.vstack([self.vectors, vector])
        self.prompts.append(prompt)
        self.responses.append(response)

Each implementation choice maps to a production decision.

  • normalize_embeddings=True makes every vector unit length, so a dot product is cosine similarity. Without normalization, scores are not comparable across prompts of different lengths.
  • The lookup returns its vector so the caller can store it after a miss without embedding the same request twice. Every request pays for one embedding round trip, hit or miss.
  • A linear scan is only a teaching implementation. A real deployment needs an approximate nearest-neighbor index, which adds its own recall parameter above the similarity threshold.
  • The response is stored only after a miss. Nothing checks whether that answer is correct, scoped to the right user, current in time, or safe to reuse.
Semantic cache pipeline embedding a prompt, searching similar prompts, and returning a stored answer above threshold
Figure 16 | Semantic caching returns text, not model state. The lookup can bypass the LLM after an embedding search. A local hit-rate chart can look excellent while the cache is serving answers to questions whose meaning differs in one critical detail.

A high similarity score is not semantic equivalence

pairs = [
    ("How do I reset my password?",
     "How can I reset my password?"),
    ("Is the API rate limited?",
     "Is the API not rate limited?"),
    ("Refund policy for annual plans",
     "Refund policy for monthly plans"),
]

for a, b in pairs:
    va, vb = encoder.encode([a, b], normalize_embeddings=True)
    print(f"{float(va @ vb):.3f}", a, "vs", b)

The source's demonstration produces:

Embedding similarity can hide a changed answer
Similarity Prompt pair Why reuse is unsafe
0.961 “How do I reset my password?” / “How can I reset my password?” Genuine paraphrase; the same answer is usually valid.
0.952 “Is the API rate limited?” / “Is the API not rate limited?” Negation reverses the required answer.
0.887 Annual plan refund policy / monthly plan refund policy One operational value changes the policy.

The paraphrase and the negation are separated by less than one hundredth of a point. Raising the threshold to exclude the negation can collapse the hit rate; lowering it to recover hits increases the chance of returning a confident wrong answer. Published defaults range from roughly 0.75 to 0.97, which is evidence that the threshold is workload-specific rather than a universal constant.

Some failures cannot be solved by any threshold because the embedding itself does not encode the distinction that matters. Negation, units, dates, tenant identity, permission scope, model version, or a changed policy can be tiny in embedding distance but decisive in meaning.

The fifth layer: exact response caching

A less visible fifth layer returns a stored response only when the request is byte-identical. It saves input and output tokens like a semantic cache, but it performs no similarity matching and therefore carries no false-positive risk. The first measurement to collect is the byte-identical repeat rate. If that rate is low, embeddings are not a solution to a cache problem that does not exist.

Exact response cache still has ordinary distributed-systems problems: key canonicalization, tenant isolation, authorization, TTL, invalidation when source data changes, and storage growth. Its correctness boundary is simply much easier to explain than a cosine threshold.

5. How the layers interact

The first three layers are one idea at different scopes. The KV cache retains attention state for a request. Prefix caching keeps that state after a request so another request can find it. Prompt caching packages provider-managed prefix reuse with a marker, a TTL, and a price. Semantic caching leaves exact state behind and returns a completed answer based on similarity.

Comparison table showing what each cache stores, how it invalidates, and whether a hit can be wrong
Figure 17 | The caches differ in object and failure model. KV and prefix caches key exact attention state. Prompt cache keys an exact provider-rendered prefix. Semantic cache keys an embedding neighborhood, and only that layer can return a wrong answer on a nominal hit.
Five cache layers and their correctness boundary
Layer Stored object Lifetime What invalidation means Can a hit be wrong?
KV cache Attention K/V tensors One generation request, unless explicitly retained Sequence or model state changes No, assuming the cache implementation is correct
Prefix cache Reusable K/V blocks Server memory, until eviction Any earlier token, hash-chain, tenant salt, or model change No, under exact prefix and model identity
Prompt cache Provider-side prefix state Provider TTL and routing policy Rendered prompt changes, breakpoint falls out of lookback, route changes No, under exact provider prefix identity
Semantic cache Complete response string Application policy Threshold, source freshness, scope, or policy changes Yes; similarity is not equivalence
Exact response cache Complete response string Application policy Canonical request or source data changes No false positive from similarity, but stale data remains possible

A production system can use several layers at once. A hosted VLA agent may receive a direct prompt-cache hit for a stable safety policy and tool catalog, then create a fresh KV cache for the current camera observations and action query. A separate exact response cache can answer a byte-identical fleet status request before the model is called. A semantic cache should be treated as a separate product decision, not as a faster variant of the first three.

6. What silently breaks prefix reuse

Most cache incidents are not caused by the cache implementation. They are caused by a token sequence that the application believed was stable and the model represented differently. The source identifies five common patterns, plus model routing and tokenizer/template effects.

Five prefix-invalidation causes: tool definitions, system prompt, conversation history, current message, and variable content
Figure 18 | Five ways a prefix silently changes. Tool definitions, the system prompt, conversation history, the current request, and any variable content near the front can all alter the rendered token prefix. The request still succeeds; reuse simply disappears.
GPU memory pool shared by cached prefix blocks and live decode sequences under LRU eviction
Figure 19 | The cache competes with the running batch. Cached prefix blocks and active sequences draw from the same GPU memory pool. A larger cache, a larger batch, or a longer context can each leave less room for the others; unreferenced blocks are evicted under least-recently-used pressure.

1. Variable content at the front

A timestamp, request ID, user name, trace ID, or changing camera-frame count near the beginning of the system prompt invalidates every block after it. Put stable instructions first, variable values last, and place the cache breakpoint at the last stable boundary.

Prompt with a timestamp at the front crossing out all later cached blocks compared with a stable prefix
Figure 20 | One changing value at the front poisons the whole chain. Stable tool definitions, system policy, and examples can all be invalidated when a date or request-specific value is rendered before them. Moving the variable suffix after the marker preserves the reusable region.

2. Reordered tool schemas

Tool definitions are often serialized ahead of the system prompt. If a framework returns a dictionary in a different order, or an A/B test assigns tools in a different sequence, the rendered prefix changes. Sorting tool schemas deterministically is part of inference correctness, not merely code style.

3. Settings that are rendered into the prompt

On Anthropic, changing web search, citations, thinking configuration, or tool_choice can rewrite the prompt representation. Running an A/B test between two reasoning efforts can intentionally split the cache population into two prefixes. That may be a valid experiment, but the latency and cost results no longer share the same cache baseline.

4. History summarization or compaction

Replacing old turns with a summary edits the front of the prompt. It can improve context quality and reduce token count while simultaneously throwing away the previous prefix. Cache-aware context management favors append-only growth and local truncation over full rewrites when the task permits it.

5. Model routing

Cache entries are tied to the model. Routing a request to a cheaper or differently configured model still requires cold prefill for the accumulated history even if the text is identical. A router that considers price without considering cache residency can move a request to the wrong place.

6. Tokenizer and chat-template differences

Two prompts can look identical in logs and render differently because one has a BOS token, a trailing newline, a changed tool schema serialization, or a default system message inserted by the chat template. The safest diagnostic is to compare token IDs, not logged text.

messages_turn_1 = [
    {"role": "user", "content": "What is the capital of France?"},
]
messages_turn_2 = [
    {"role": "system", "content": "Today is Tuesday."},
    {"role": "user", "content": "What is the capital of France?"},
]

a = tokenizer.apply_chat_template(messages_turn_1)
b = tokenizer.apply_chat_template(messages_turn_2)

shared = 0
for x, y in zip(a, b):
    if x != y:
        break
    shared += 1

print(f"shared prefix: {shared} tokens of {len(a)} and {len(b)}")
print(a[shared:shared + 8])
print(b[shared:shared + 8])

In the source's example, the first prompt has no explicit system message, so the template inserts its default assistant header. Once a timestamp system message is supplied, the rendered prefix diverges after only three shared tokens. The two requests are not cache-compatible even if the visible user question is identical.

7. Where this matters for physical AI

The source is general LLM infrastructure, not a robotics study. Its mechanics still map directly to deployed physical-AI software because robot agents are unusually prompt-heavy and repetitive.

  • Hosted VLA orchestration. A planner may send the same robot safety policy, action vocabulary, tool definitions, and scene-description schema before every observation or task. Stable ordering lets prefill work be reused; changing robot IDs or timestamps belong after the breakpoint.
  • Embodied-agent loops. Perception summaries, retrieved manual sections, and tool results can accumulate for many turns. Exact prefix reuse is compatible with append-only logs, while semantic caching is not safe for current joint state, battery state, or collision status.
  • Robot-fleet assistants. Operators ask repeated questions about a known platform and skill set. Exact response caching can serve byte-identical status queries; prompt caching can retain stable documentation context; semantic reuse requires tenant and authorization boundaries plus strict invalidation.
  • Retrieval over operating procedures. The same maintenance documents may be retrieved in different orders. A plain prefix cache will miss after a reorder, while CacheBlend-style reuse can help only if the cached attention state is repaired at chunk boundaries.

None of these systems should cite the source's two-to-three-times TTFT improvement as a robotics benchmark. The transferable claim is narrower: if a hosted model receives a long, stable, exactly repeated token prefix, caching can remove repeated prefill. The robotics workload still has to measure its own hit rate, safety constraints, model routing, and time budget.

8. A production checklist

  1. Name the layer. Decide whether the problem is one-request KV state, cross-request prefix reuse, provider prompt billing, semantic response reuse, or exact response reuse.
  2. Measure the reusable unit. For prefix caching, report the proportion of prompts with an exact shared prefix and the average length of that prefix, not just a global cache-hit percentage.
  3. Put stable content first. Canonicalize tool schemas, freeze system-prompt order, move timestamps, request IDs, user state, observations, and queries to the suffix.
  4. Inspect rendered tokens. Compare token IDs around the first divergence when a supposed hit misses. Logging the human-readable prompt is not enough.
  5. Separate prefill and decode metrics. A prefix hit should reduce time to first token or prefill work; it should not be credited with a change in decode throughput.
  6. Budget cache memory against concurrency. More retained KV state can mean fewer active requests. Watch eviction, queueing, and tail latency rather than only hit rate.
  7. Track write and read events separately for prompt caching. A cache write without later reads is a cost increase. Monitor TTL, breakpoint distance, model routing, and tenant identity.
  8. Do not use semantic similarity as an authorization boundary. Scope entries by tenant, permissions, model, prompt version, and data freshness. Keep a human-readable reason when a semantic hit is served.
  9. Prefer exact-response caching before embeddings when the traffic is repetitive. It has a much cleaner correctness argument and is usually easier to invalidate.

A cache hit is a performance event. Whether it is a correct reuse event depends on the identity rule around the cached object.

Conclusion

KV caching is the foundation: keep the keys and values that later tokens will read. Prefix caching makes the same state shareable across requests by indexing exact token prefixes. Prompt caching exposes that mechanism through a provider API and a price sheet. Semantic caching abandons exact state and returns a stored answer when an embedding says two questions are close.

The first three remain attractive because they preserve the model's computation under a precise identity contract. The fourth can save the entire model call, which is exactly why its hit rate is not a sufficient quality metric. In all four cases, the operational work is to define the reusable object, prove its identity, and measure what remains after the hit.

Source: Avi Chawla, “KV, Prefix, Prompt and Semantic Caching in LLMs, clearly explained,” X longform, August 28, 2026. The original X article is available at x.com/i/article/2093210840468189184.

Related Articles

As an Amazon Associate, we earn from qualifying purchases.