Skip to content

LLM prompt caching vs response caching: how each works and when to use it

Published 9 min read

Prompt caching (also called prefix caching) happens inside the model provider: when a new request starts with the same tokens as a recent one, the provider reuses its already-computed state for that prefix and bills those input tokens at a reduced cached rate, but the model still generates a fresh answer. Response caching happens in front of the model: if an equivalent request was answered before, the stored answer is returned and the model is not called at all. Prompt caching is safe almost everywhere; response caching is only safe when the same input should always produce the same output.

What is the difference between prompt caching and response caching?

Both reduce cost and latency on repeated work, but they cache different things at different layers. The table is the short version; the sections below explain each row.

Prompt caching discounts input. Response caching skips the call.
AspectPrompt / prefix cachingExact response cachingSemantic response caching
Where it runsInside the model providerIn front of the model (gateway or your app)In front of the model
What is reusedComputed attention state (KV cache) for a shared prefixA full stored responseA stored response for a similar prompt
Does the model run?Yes, on the new suffix and the outputNoNo
Match ruleIdentical leading tokensIdentical requestEmbedding similarity above a threshold
What you savePart of the input cost, plus time to first tokenMost or all of the callMost or all of the call
Answer freshnessAlways freshly generatedIdentical to the stored answerAnswer to a different, similar question
Main riskVery little; savings depend on prompt layoutServing a stale or wrong-scope answerServing an answer to a question that was not asked

How does prompt (prefix) caching work?

When a transformer reads your prompt, it computes key and value tensors for every token at every layer: the KV cache. That computation is the "prefill" phase, and on long prompts it dominates time to first token. If the next request begins with exactly the same tokens, the provider can keep those tensors around for a short while and skip recomputing them. Only the part after the shared prefix needs fresh prefill.

Providers pass some of that saving on as a lower cached input price for the reused tokens. How it is exposed differs by provider: some cache automatically once a prompt passes a minimum length, others require you to mark cache breakpoints explicitly, and some charge a premium to write the cache and a discount to read it. Cached entries typically live for minutes rather than hours and are evicted when idle. The LLM API pricing guide covers how cached-input rates appear on a price sheet.

How to structure prompts so prefix caching hits

The match is on leading tokens, so a single changed character early in the prompt invalidates everything after it. The rule is static first, dynamic last:

  1. System prompt and instructions, byte-for-byte stable. No timestamps, request ids or user names at the top.
  2. Tool and function definitions, in a fixed order with stable serialization.
  3. Large shared context: documents, few-shot examples, a codebase snapshot.
  4. Conversation history, appended rather than rewritten.
  5. The new user message last.

Coding agents and chat sessions benefit the most, because every turn resends the same system prompt, tool list and growing history. If you want to know how large that repeated prefix is, the tokens guide shows how to read it off the usage object.

How does response caching work?

Response caching stores the complete output of a call under a key derived from the request. When an equivalent request arrives, the stored output is returned without calling the model. It can remove nearly all of the latency and most of the cost of the call, which is why it is attractive for repeated questions, evaluation reruns, development loops and pipelines that reprocess the same inputs.

Exact caching

An exact cache hits only when the request is identical in every way that affects the output: model, messages, tools, sampling parameters and response format. The cache key is typically a hash of a normalized form of those fields. Fields that do not change the output, such as your own request labels, should be excluded from the key so they do not fragment the cache.

Semantic caching

A semantic cache embeds the prompt and returns a stored answer when a previous prompt is close enough in embedding space. "How do I reset my password?" and "how can I reset my password" can share an answer. The trade-off is that close is not the same as identical: "how do I reset my password" and "how do I reset my admin password" may also be close. Semantic caching suits FAQ-style and support traffic where near-duplicates are common and a slightly generic answer is acceptable. It is a poor fit for anything where small wording differences change the correct answer.

When is response caching safe to use?

The core requirement is determinism: caching a response is only correct if you would be happy getting that exact response again for that exact request. That rules out more than people expect.

  • Sampled output. With a non-zero temperature, the same request is supposed to produce different answers. Caching it silently turns a creative or diverse output into a fixed one. Exact caching is safest when sampling is deterministic, with temperature: 0 or a pinned seed.
  • Time-sensitive or external facts. If the correct answer depends on today’s data, a TTL is your only protection against serving yesterday’s answer.
  • Tool calls and agents. A tool-calling step depends on state outside the prompt, so a cached decision may be wrong even if the messages match.
  • Per-user data. A cache shared across tenants can leak one user’s answer to another. Cache scope must match your data boundaries.

Cache keys, namespaces and TTLs

Three controls decide whether a response cache is correct rather than just fast. The key decides what counts as the same request. A namespace partitions entries so that, for example, each end user or each tenant has its own cache even when prompts are identical. A TTL bounds how long an answer can be reused, which caps how stale it can get. A good default is a short TTL and a namespace per data boundary, widened only once you have measured hit rates.

How does caching work on Smouter?

Smouter handles both kinds of caching for requests on the value lane, the default. The behavior is documented in the caching section of the docs and summarized in how response caching works:

  • Exact response caching is on by default and account-isolated: one account’s entries are never served to another. It applies only to deterministic requests, meaning an explicit temperature: 0 or a pinned seed. A cached answer to a streaming request is replayed in the same stream shape.
  • Prefix savings use the upstream provider’s warm-prefix accounting. When they apply, the response can carry an x-smouter-prefix-saved header. Provider-specific cache-control extensions in the request body are stripped; use Smouter’s cache headers instead.
  • Semantic caching is opt-in with x-smouter-cache-semantic: on, and it excludes tool calls, structured-output calls and streaming calls.
  • Request metadata is not forwarded to the provider and does not affect cache matching, so labeling requests does not fragment the cache.
  • Every response carries x-smouter-cache with the cache outcome, and the Usage page shows the cache tier of each request.

Per-request controls are all headers:

HeaderEffect
x-smouter-cache: no-storeSkip both the cache lookup and storing the answer
x-smouter-cache: no-cacheSkip the lookup but store the fresh answer
x-smouter-cache-semantic: onOpt this request into semantic caching
x-smouter-cache-ttl: <seconds>Set the reuse window for this entry
x-smouter-cache-namespace: <name>Partition cache entries inside your account
bash
# Deterministic request: eligible for exact response caching.
# Entries are partitioned per end user and reusable for five minutes.
curl -i https://api.smouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $SMOUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-smouter-cache-namespace: user-8f2c" \
  -H "x-smouter-cache-ttl: 300" \
  -d '{
    "model": "<model-id>",
    "temperature": 0,
    "messages": [
      {"role": "system", "content": "Answer in one sentence."},
      {"role": "user", "content": "What is a KV cache?"}
    ]
  }'

# Force a fresh answer but refresh the cache with it
#   -H "x-smouter-cache: no-cache"
# Never read or write the cache for this request
#   -H "x-smouter-cache: no-store"

With -i you can see the receipt headers, including x-smouter-cache and x-smouter-charge, on the response.

When Smouter does not cache

  • The verified lane (x-smouter-lane: verified) routes to the model maker’s own endpoint under a zero-retention contract. Nothing is stored or cached, and responses carry cache-control: no-store. See value vs verified lanes.
  • Zero retention, set on the key or with x-smouter-zero-retention: on, means prompts and responses are not retained for caching or replay. The zero data retention guide covers when you need it.
  • model: "auto" is not cacheable, and embeddings do not use response caching.

Caching is not idempotency

A related but separate mechanism is the Idempotency-Key header. It protects against duplicate charges when a client retries the same logical request, for example after a network timeout: Smouter replays the completed response without charging again and marks it with x-smouter-idempotent-replay: true. Use idempotency keys for retries, and caching for genuinely repeated work. The fallbacks guide covers retry design more broadly.

Which caching should you use?

  • Always: structure prompts for prefix caching. It never changes answers and only requires discipline about prompt layout.
  • Usually: exact response caching for deterministic calls such as classification, extraction, eval reruns and repeated pipeline steps, with a namespace per data boundary and a TTL matched to how fast the underlying facts change.
  • Selectively: semantic caching for high-volume, FAQ-like traffic where near-duplicate questions should get the same answer, after reviewing what it actually matches.
  • Never: response caching for sampled creative output, per-user data without a namespace, or anything that must be computed fresh. Send x-smouter-cache: no-store there.

Caching is one lever among several. Combined with batching non-urgent work through the batch API and buying each model at a lower per-token rate, it is part of the broader playbook in how to reduce LLM API costs. Live per-model prices for both lanes are on the models page.

Frequently asked questions

What is the difference between prompt caching and response caching?

Prompt caching reuses the provider’s computed state for a repeated prompt prefix, so those input tokens are cheaper and faster, but the model still generates a new answer. Response caching stores a full answer and returns it for an equivalent request without calling the model. Prompt caching discounts input; response caching skips the call.

Does prompt caching change the model’s output?

No. Prefix caching only avoids recomputing the attention state for tokens the model has already processed. The model still reads the full prompt and generates a fresh response, so output quality and behavior are the same as without caching. What changes is the input cost of the cached tokens and the time to first token.

Is it safe to cache LLM responses?

It is safe when the same request should always produce the same answer: deterministic sampling with temperature: 0 or a fixed seed, no time-sensitive facts, and cache scope that matches your data boundaries. Use namespaces to separate users or tenants and a TTL to bound staleness. Avoid it for creative, sampled or agentic tool-calling steps.

What is semantic caching for LLMs?

Semantic caching embeds each prompt and returns a stored answer when a new prompt is similar enough to a previous one, even if the wording differs. It raises hit rates on FAQ-style traffic but can return an answer to a subtly different question. On Smouter it is opt-in with x-smouter-cache-semantic: on and excludes tool, structured-output and streaming calls.

How do I disable caching for a single request on Smouter?

Send x-smouter-cache: no-store to skip both the cache lookup and storing the answer, or x-smouter-cache: no-cache to force a fresh answer while still storing it for later requests. Requests on the verified lane and requests under zero retention are never cached, and model: "auto" is not cacheable.

How do I get more prompt cache hits?

Put everything static first and everything dynamic last: a byte-stable system prompt, tool definitions in a fixed order, shared documents, then conversation history, then the new message. Remove timestamps, request ids and per-user text from the top of the prompt. A single changed token early in the prompt invalidates the cache for everything after it.