What is an LLM gateway (AI gateway)? What it does and when you need one
An LLM gateway (also called an AI gateway) is a proxy that sits between your application and one or more model providers and exposes them all through a single API. Instead of every service holding provider keys and its own retry logic, requests go through the gateway, which handles routing, fallbacks, caching, rate limits, key management, spend caps and logging in one place.
What does an LLM gateway actually do?
Think of it as the API gateway pattern applied to model calls, with a few problems that are specific to LLMs: responses are slow and streamed, cost is per token rather than per request, providers rate-limit by tokens per minute, and the same model can be served by several providers at different prices. A gateway takes a request in one format, decides where it should go, sends it, and returns the answer in the same format, while recording what happened and what it cost.
The request path is simple. Your code calls the gateway with a gateway key. The gateway authenticates the key, checks its limits and allowlists, looks in the cache, picks an upstream, calls it with the gateway’s own provider credentials, streams the response back, and settles usage. Your application never holds provider keys and never needs to know which upstream served the call.
What are the core features of an AI gateway?
| Capability | What it does | Without a gateway |
|---|---|---|
| Unified API | One request format (usually OpenAI-compatible) for many providers and models | An SDK and a request shape per provider |
| Routing | Picks the provider or model per request by price, health, capability or rules | Hard-coded model and provider per call site |
| Fallbacks | Tries another candidate when the first is unavailable or rate-limited | Retry and failover code in every service |
| Caching | Returns stored answers for repeated requests | Paying again for identical calls |
| Rate limits | Caps tokens per minute per key or team before upstream does | Discovering limits through 429s in production |
| Keys and spend caps | Issues scoped keys with budgets and model allowlists | Shared provider keys and surprise invoices |
| Observability | Logs tokens, cost, latency and status per request, key and tag | Piecing usage together from several provider dashboards |
| Data controls | Applies retention, caching and safety policy per key or request | Policy enforced by convention, if at all |
Unified API
Most gateways expose an OpenAI-compatible endpoint, so any client that lets you set a base URL works unchanged. Some also accept the Anthropic Messages format, which matters for tools such as Claude Code. Compatibility is about the wire format, not identical behaviour: tool calls, structured output and reasoning fields still vary by model. What an OpenAI-compatible API is covers where those edges are.
Routing
Routing comes in two flavours. Provider routing sends a request for a specific model to one of several upstreams serving it, chosen on price, latency or health. Model routing chooses which model should answer at all, for example sending simple prompts to a cheaper model. They solve different problems and are worth evaluating separately; LLM model routing goes deeper on the second.
Fallbacks
A fallback list says “if this model can’t take the request, use that one.” The important detail is when the switch happens. Choosing another candidate before generation starts is safe and invisible. Retrying after a stream has already started sending tokens is not, because the client has already received part of an answer. Read LLM API fallbacks before you rely on any gateway’s failover semantics.
Caching
Gateways cache in up to three ways: exact response caching for identical deterministic requests, prefix caching where the upstream bills a repeated prompt prefix at a lower rate, and semantic caching that matches near-duplicate prompts. Each has a correctness trade-off, which is why good gateways scope caches per account and let you opt out per request. Prompt caching vs response caching explains the difference.
Rate limits, keys and spend caps
This is often the feature that gets a gateway approved. Instead of one shared provider key, each service, environment or developer gets its own key with a tokens-per-minute cap, a daily or monthly budget, and an allowlist of models it may call. A leaked or runaway key then fails at its own cap instead of draining the account. A well-behaved gateway returns a 429 with a retry-after header when a cap is hit, so clients can back off cleanly.
Observability and data controls
Every call through the gateway can be logged with model, tokens, cost, latency, status and your own tags, which turns “why is the bill up?” into a filter rather than an investigation. The same choke point is where data policy is enforced: whether prompts are retained, whether a request may be cached, and whether content passes a safety check before it reaches a model.
What does using a gateway look like in code?
Usually one line changes: the base URL. Gateway-specific behaviour is then controlled with headers, so the request body stays a normal OpenAI-style request. Here is a call to Smouter with a fallback list, a tag for cost attribution and a cache opt-out:
curl https://api.smouter.ai/v1/chat/completions \
-H "Authorization: Bearer $SMOUTER_API_KEY" \
-H "Content-Type: application/json" \
-H "x-smouter-models: <fallback-model-id>" \
-H "x-smouter-tags: production,checkout" \
-H "x-smouter-cache: no-store" \
-d '{
"model": "<model-id>",
"messages": [{"role": "user", "content": "Summarise this ticket."}]
}'The response comes back in the standard format, with receipt headers such as x-smouter-charge, x-smouter-lane and x-smouter-cache when those values are known. The docs list every header, and the support article on rate limits and TPM caps covers how per-key and organization limits combine.
When do you need an LLM gateway?
A gateway earns its place once you have more than one of the following:
- Two or more model providers in production, or plans to add one.
- More than one service, team or customer sharing the same provider account.
- A need for per-key budgets, model allowlists or tokens-per-minute caps.
- Outages or 429s from a single provider that you currently handle with ad hoc retries.
- A token bill large enough that you want to pay less for the same model, or attribute spend by feature.
- Coding agents or internal tools used by several developers, each of whom should not hold a raw provider key.
When don’t you need one?
If you call one model from one service, traffic is modest, and your provider’s dashboard tells you everything you need, a gateway adds a hop and a vendor without solving a problem you have. The same goes for prototypes: call the provider directly and add a gateway when the second provider or the second team shows up. Because gateways are mostly OpenAI-compatible, adding one later is a configuration change rather than a rewrite.
Should you build or buy an LLM gateway?
A minimal internal gateway (one endpoint, a provider switch, a retry) is a few days of work and a reasonable thing to build. The cost is in everything that follows: streaming edge cases, token accounting per provider, per-key budgets, cache correctness, failover that doesn’t double-charge, logs you can query, and keeping up with every provider’s API changes. That becomes a product someone on your team maintains.
| Approach | Choose it when | You take on |
|---|---|---|
| Build your own | Requirements are narrow and unusual, and you have platform engineers to own it | All maintenance, provider contracts and on-call |
| Self-host open source | The gateway must run in your network and you want a head start | Deployment, scaling, upgrades and provider contracts |
| Hosted gateway | You want the features without running infrastructure | A third party in the request path; review its data terms |
Hosted gateways also differ in what they optimise. Some focus on observability and prompt management, some on platform integration, and some, like Smouter, on price: each request is shopped across a benchmarked pool of providers serving the same model weights, with the site stating prices up to 50% under OpenRouter on the same model. If you are comparing hosted options, OpenRouter alternatives lays out the categories side by side, and how to reduce LLM API costs covers the levers a gateway can and can’t pull.
One more decision is worth making up front: which traffic may leave the first-party path. Smouter handles this with two lanes chosen per request. The value lane uses the sourcing pool and caching; the verified lane sends the request to the model maker’s own endpoint under a zero-retention contract. Value vs verified lanes explains the split, and the models page shows which models are on each lane with live prices.
Frequently asked questions
What is the difference between an LLM gateway and an API gateway?
A general API gateway routes and secures HTTP requests. An LLM gateway does that plus model-specific work: it translates between provider formats, meters cost per token, enforces tokens-per-minute limits, handles streamed responses, caches answers, and can route the same model across several providers. Some teams run an LLM gateway behind their existing API gateway.
Is an AI gateway the same as an LLM router?
They overlap. Router usually means the part that decides which model or provider handles a request. Gateway is the broader term and includes routing plus keys, spend caps, rate limits, caching, logging and data policy. Most hosted products marketed as routers include gateway features, and most gateways include some routing.
Does an LLM gateway add latency?
Yes, every extra hop adds some time. How much depends on the gateway and where it runs relative to you and the provider; Smouter states sub-50ms gateway overhead. A cache hit can make a call much faster than going to the model at all. Measure time to first token on your own traffic to see the real effect.
Do I need an LLM gateway for a single model?
Often not. With one model, one service and modest traffic, calling the provider directly is simpler. A gateway starts to pay off when several services or people share the account, when you need per-key budgets and allowlists, when you want fallbacks across providers, or when you want to pay less for the same model.
Can an LLM gateway see my prompts?
A gateway is in the request path, so it processes your prompts and responses. What matters is what it keeps. Review its retention terms and whether you can disable storage. On Smouter, zero retention can be enabled per key or per request with x-smouter-zero-retention: on, which stops prompts and responses being retained for replay or caching.
What is an open-source LLM gateway?
An open-source gateway is one you can download and run on your own infrastructure, such as LiteLLM. You get full control over configuration, logs and network placement, and you can modify it. In return you deploy, scale and patch it yourself, and you still need accounts with each model provider behind it.