Skip to content

Zero data retention LLM APIs: what ZDR covers, what it doesn’t, and how to verify it

Published 9 min read

A zero data retention (ZDR) LLM API processes your prompt and completion in memory to serve the request and does not store that content afterwards: no logs of bodies, no training, no cache, no replay. It does not mean nothing is recorded; usage metadata such as model, token counts, cost and timestamps is still kept for billing. And it only holds end to end if every hop, gateway and model provider, commits to it.

What does zero data retention mean for an LLM API?

Every LLM request carries content (your prompt, attached files, tool results, the model’s answer) and metadata (who sent it, which model, how many tokens, what it cost, when, and whether it succeeded). ZDR is a promise about the content. The request is processed, the response is returned, and the content is not written anywhere durable once the request is done.

In practice a ZDR commitment usually rules out four things:

  • Request and response logging of bodies, including debug logs and log pipelines that sample traffic.
  • Training or evaluation on your data.
  • Caching of responses for reuse, because a cache is a store.
  • Replay storage, where a completed response is kept so a retry can be answered without regenerating it.

What is still kept under zero data retention?

Metadata. A provider cannot bill you, enforce rate limits or investigate an outage without recording that a request happened. Expect model id, token or unit counts, charge, status, latency, API key or account attribution, timestamps, and any labels you attach yourself (tags, request ids, metadata fields). Treat anything you put in a label as stored: do not put customer names or ticket text into a metadata field and assume ZDR covers it.

Safety systems are the other gray area. Many model providers describe some form of abuse or misuse monitoring in their terms, and in general that can involve retaining flagged content for a limited period unless a specific zero-retention arrangement applies to your account. The details vary by provider and change over time, so read each provider’s current terms rather than relying on a summary, including this one.

Why do caching, batch and replay conflict with ZDR?

Several cost and reliability features work precisely because content is stored. Turning on zero retention turns them off or makes them unavailable, and that is the right outcome, not a bug:

Features that depend on stored content, and what happens when you turn retention off.
FeatureWhat it storesUnder zero retention
Response cachingThe full response, keyed by the request, so an identical request can be answered without a model call.Off. Every request is generated fresh and billed in full.
Semantic cachingResponses plus a representation of prompts, to match similar requests.Off, for the same reason.
Idempotent replayThe completed response, so a retry with the same idempotency key returns it without a second charge.No body to replay. A retry has to regenerate, or fails rather than replaying.
Batch processingInputs and results, persisted until you download them hours later.Incompatible by design. Run the work synchronously instead.
Multi-turn reasoning continuityProvider-side reasoning context reused across turns.Not available, or not recoverable later in the conversation.

This has a cost implication. Caching and batch are two of the larger levers in reducing LLM API costs, so a ZDR workload will usually pay more per answer than the same workload with retention allowed. For the mechanics of each, see prompt caching vs response caching and how batch APIs work. Upstream prompt-prefix caching is a separate case: whether a provider’s in-memory prefix reuse counts as retention is a question to put to that provider.

Does ZDR at the gateway cover the model provider too?

No, and this is the part most evaluations miss. A request through a gateway touches at least two parties: the gateway, and whichever provider actually runs the model. Each has its own retention policy. A gateway can promise it stores nothing and still forward your prompt to a provider that logs it, and a provider can offer ZDR while the gateway in front of it caches responses. If you use an LLM gateway, you need an answer for each hop.

Routing makes this harder. If a gateway picks among many providers per request, or fails over to another provider when one degrades, the set of parties that may see your prompt is the whole candidate pool, not one vendor. For a ZDR workload you want the path pinned to providers whose terms you have actually reviewed.

How does zero retention work on Smouter?

Smouter separates the two hops into two controls: one for what Smouter keeps, and one for where the request is served.

What Smouter keeps: the zero-retention setting

Enable zero retention on an API key, or per request with x-smouter-zero-retention: on. When it is effective, the response echoes x-smouter-zero-retention: on, and Smouter does not retain full prompts or responses for replay or caching. Per the docs, it also:

  • replaces customer-content evidence in safety events with non-content redaction metadata;
  • disables replay-body storage, so a retry with a reused Idempotency-Key returns 502 idempotency_error instead of a replay;
  • disables Anthropic signed-thinking continuity, so an extended-thinking tool conversation started under zero retention cannot later recover its discarded reasoning context;
  • rejects Batch API jobs from a zero-retention key, because batch inputs and results are persisted for retrieval.

What it does not change: usage and billing are still metered, and compliance events can retain non-content metadata. That matches the general split above. Independently of zero retention, Smouter’s usage records store model, outcome, token or unit counts, charge, attribution metadata and timestamps rather than prompt or completion bodies, and request metadata, fallback lists and Smouter credentials are never forwarded to the provider.

bash
curl -i https://api.smouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $SMOUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-smouter-zero-retention: on" \
  -H "x-smouter-lane: verified" \
  -d '{
    "model": "<model-id>",
    "messages": [{"role": "user", "content": "Summarize this contract clause."}]
  }'

# Check the response headers:
#   x-smouter-zero-retention: on
#   x-smouter-lane: verified
#   cache-control: no-store

Where the request is served: the verified lane

Smouter has two lanes, chosen per request with x-smouter-lane: value|verified. The value lane (the default) shops the request across a benchmarked pool of providers serving the same model weights; it is where the discounts are, and responses are eligible for caching. The verified lane sends the request to the model maker’s own endpoint under a zero-retention contract: nothing is stored or cached, and every response carries cache-control: no-store. The lane actually used is echoed in x-smouter-lane.

So the zero-retention header governs Smouter’s side, and the verified lane is the option for a contractual zero-retention commitment from the model maker as well. Not every model is available on both lanes, and verified is priced separately; check availability and live prices per lane on the models page, and read value vs verified lanes for when each makes sense. One practical difference: on the verified lane, images must be sent as inline base64 data: URLs, and remote image URLs or audio content parts return a 400.

How do I evaluate a vendor’s data retention claims?

Marketing pages say "we don’t train on your data" far more often than they say "we don’t store your data". Those are different promises. Use this checklist on every party in the request path, including any gateway, observability proxy or logging SDK you add yourself:

  1. Is retention zero, or just short? "Deleted after 30 days" is a retention policy, not ZDR. Ask for the retention period of request bodies specifically.
  2. Is it the default, or does it need to be enabled? Find out whether ZDR is per account, per key, per request, or by contract only, and what the default is for a new key.
  3. Can you verify it per request? A response header or log field that reflects the effective setting lets you test it in CI instead of trusting a dashboard toggle.
  4. Which features turn off? If caching, batch or replay still work under "zero retention", ask where the stored content lives.
  5. What metadata is kept, and for how long? Model, tokens, cost and timestamps are expected. Labels, user ids and error payloads may include content; ask.
  6. What do safety and abuse systems retain? Ask whether flagged content is kept, for how long, and who can see it.
  7. Who are the downstream processors? For a gateway, ask which providers can serve your request and whether you can pin the path to ones with their own ZDR terms.
  8. Does it apply to failover? If a request is rerouted, confirm the fallback target is covered by the same commitment.
  9. Is it in the contract? A setting is a product behavior; a data processing agreement is an obligation. For regulated data you want both.

Finally, look at your own side. Application logs, tracing tools and error reporters are the most common place prompts end up stored, and no vendor setting reaches them. Strip bodies from your own logs before you worry about anyone else’s.

Frequently asked questions

What is zero data retention in an LLM API?

Zero data retention means the provider processes your prompt and completion to serve the request and does not store that content afterwards: no body logging, no training, no response cache, no replay storage. Usage metadata such as model, token counts, cost and timestamps is still recorded, because billing and rate limiting depend on it.

Does zero data retention mean no logs at all?

No. ZDR covers content, not the fact that a request happened. Expect the provider to keep metadata like model id, token counts, charge, status, key attribution and timestamps. Labels you attach, such as tags or metadata fields, are usually stored too, so keep customer content out of them.

Can I use prompt caching with zero data retention?

Response caching needs a stored copy of the answer, so it is off under zero retention, and every request is generated and billed fresh. Upstream prompt-prefix caching that happens in a provider’s memory is a separate question; whether it counts as retention depends on that provider’s terms, so ask them directly.

Why is batch processing incompatible with zero data retention?

A batch job stores your inputs until they are processed and stores the results until you download them, often hours later. That persistence is the opposite of zero retention. On Smouter, a zero-retention key submitting a batch is rejected with an invalid_request_error; run that work as synchronous requests instead.

Is a gateway’s zero retention enough on its own?

Not necessarily. The gateway’s setting controls what the gateway stores; the model provider that actually serves the request has its own retention terms. For end-to-end coverage you need both hops covered. On Smouter that means zero retention for Smouter’s side and the verified lane, which uses the model maker’s endpoint under a zero-retention contract.

Does zero data retention make an LLM API compliant with GDPR or HIPAA?

Not by itself. Retention is one part of compliance; lawful basis, processor agreements, data location, access controls and your own logging matter too. Treat ZDR as a technical control to verify, and have counsel review the contracts and data processing terms of every vendor in the path. This guide is not legal advice.