Skip to content

What is an LLM gateway (AI gateway)? What it does and when you need one

Published 9 min read

An LLM gateway (also called an AI gateway) is a proxy that sits between your application and one or more model providers and exposes them all through a single API. Instead of every service holding provider keys and its own retry logic, requests go through the gateway, which handles routing, fallbacks, caching, rate limits, key management, spend caps and logging in one place.

What does an LLM gateway actually do?

Think of it as the API gateway pattern applied to model calls, with a few problems that are specific to LLMs: responses are slow and streamed, cost is per token rather than per request, providers rate-limit by tokens per minute, and the same model can be served by several providers at different prices. A gateway takes a request in one format, decides where it should go, sends it, and returns the answer in the same format, while recording what happened and what it cost.

The request path is simple. Your code calls the gateway with a gateway key. The gateway authenticates the key, checks its limits and allowlists, looks in the cache, picks an upstream, calls it with the gateway’s own provider credentials, streams the response back, and settles usage. Your application never holds provider keys and never needs to know which upstream served the call.

What are the core features of an AI gateway?

Not every gateway does all of these. Check which ones you actually need.
CapabilityWhat it doesWithout a gateway
Unified APIOne request format (usually OpenAI-compatible) for many providers and modelsAn SDK and a request shape per provider
RoutingPicks the provider or model per request by price, health, capability or rulesHard-coded model and provider per call site
FallbacksTries another candidate when the first is unavailable or rate-limitedRetry and failover code in every service
CachingReturns stored answers for repeated requestsPaying again for identical calls
Rate limitsCaps tokens per minute per key or team before upstream doesDiscovering limits through 429s in production
Keys and spend capsIssues scoped keys with budgets and model allowlistsShared provider keys and surprise invoices
ObservabilityLogs tokens, cost, latency and status per request, key and tagPiecing usage together from several provider dashboards
Data controlsApplies retention, caching and safety policy per key or requestPolicy enforced by convention, if at all

Unified API

Most gateways expose an OpenAI-compatible endpoint, so any client that lets you set a base URL works unchanged. Some also accept the Anthropic Messages format, which matters for tools such as Claude Code. Compatibility is about the wire format, not identical behaviour: tool calls, structured output and reasoning fields still vary by model. What an OpenAI-compatible API is covers where those edges are.

Routing

Routing comes in two flavours. Provider routing sends a request for a specific model to one of several upstreams serving it, chosen on price, latency or health. Model routing chooses which model should answer at all, for example sending simple prompts to a cheaper model. They solve different problems and are worth evaluating separately; LLM model routing goes deeper on the second.

Fallbacks

A fallback list says “if this model can’t take the request, use that one.” The important detail is when the switch happens. Choosing another candidate before generation starts is safe and invisible. Retrying after a stream has already started sending tokens is not, because the client has already received part of an answer. Read LLM API fallbacks before you rely on any gateway’s failover semantics.

Caching

Gateways cache in up to three ways: exact response caching for identical deterministic requests, prefix caching where the upstream bills a repeated prompt prefix at a lower rate, and semantic caching that matches near-duplicate prompts. Each has a correctness trade-off, which is why good gateways scope caches per account and let you opt out per request. Prompt caching vs response caching explains the difference.

Rate limits, keys and spend caps

This is often the feature that gets a gateway approved. Instead of one shared provider key, each service, environment or developer gets its own key with a tokens-per-minute cap, a daily or monthly budget, and an allowlist of models it may call. A leaked or runaway key then fails at its own cap instead of draining the account. A well-behaved gateway returns a 429 with a retry-after header when a cap is hit, so clients can back off cleanly.

Observability and data controls

Every call through the gateway can be logged with model, tokens, cost, latency, status and your own tags, which turns “why is the bill up?” into a filter rather than an investigation. The same choke point is where data policy is enforced: whether prompts are retained, whether a request may be cached, and whether content passes a safety check before it reaches a model.

What does using a gateway look like in code?

Usually one line changes: the base URL. Gateway-specific behaviour is then controlled with headers, so the request body stays a normal OpenAI-style request. Here is a call to Smouter with a fallback list, a tag for cost attribution and a cache opt-out:

bash
curl https://api.smouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $SMOUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-smouter-models: <fallback-model-id>" \
  -H "x-smouter-tags: production,checkout" \
  -H "x-smouter-cache: no-store" \
  -d '{
    "model": "<model-id>",
    "messages": [{"role": "user", "content": "Summarise this ticket."}]
  }'

The response comes back in the standard format, with receipt headers such as x-smouter-charge, x-smouter-lane and x-smouter-cache when those values are known. The docs list every header, and the support article on rate limits and TPM caps covers how per-key and organization limits combine.

When do you need an LLM gateway?

A gateway earns its place once you have more than one of the following:

  • Two or more model providers in production, or plans to add one.
  • More than one service, team or customer sharing the same provider account.
  • A need for per-key budgets, model allowlists or tokens-per-minute caps.
  • Outages or 429s from a single provider that you currently handle with ad hoc retries.
  • A token bill large enough that you want to pay less for the same model, or attribute spend by feature.
  • Coding agents or internal tools used by several developers, each of whom should not hold a raw provider key.

When don’t you need one?

If you call one model from one service, traffic is modest, and your provider’s dashboard tells you everything you need, a gateway adds a hop and a vendor without solving a problem you have. The same goes for prototypes: call the provider directly and add a gateway when the second provider or the second team shows up. Because gateways are mostly OpenAI-compatible, adding one later is a configuration change rather than a rewrite.

Should you build or buy an LLM gateway?

A minimal internal gateway (one endpoint, a provider switch, a retry) is a few days of work and a reasonable thing to build. The cost is in everything that follows: streaming edge cases, token accounting per provider, per-key budgets, cache correctness, failover that doesn’t double-charge, logs you can query, and keeping up with every provider’s API changes. That becomes a product someone on your team maintains.

ApproachChoose it whenYou take on
Build your ownRequirements are narrow and unusual, and you have platform engineers to own itAll maintenance, provider contracts and on-call
Self-host open sourceThe gateway must run in your network and you want a head startDeployment, scaling, upgrades and provider contracts
Hosted gatewayYou want the features without running infrastructureA third party in the request path; review its data terms

Hosted gateways also differ in what they optimise. Some focus on observability and prompt management, some on platform integration, and some, like Smouter, on price: each request is shopped across a benchmarked pool of providers serving the same model weights, with the site stating prices up to 50% under OpenRouter on the same model. If you are comparing hosted options, OpenRouter alternatives lays out the categories side by side, and how to reduce LLM API costs covers the levers a gateway can and can’t pull.

One more decision is worth making up front: which traffic may leave the first-party path. Smouter handles this with two lanes chosen per request. The value lane uses the sourcing pool and caching; the verified lane sends the request to the model maker’s own endpoint under a zero-retention contract. Value vs verified lanes explains the split, and the models page shows which models are on each lane with live prices.

Frequently asked questions

What is the difference between an LLM gateway and an API gateway?

A general API gateway routes and secures HTTP requests. An LLM gateway does that plus model-specific work: it translates between provider formats, meters cost per token, enforces tokens-per-minute limits, handles streamed responses, caches answers, and can route the same model across several providers. Some teams run an LLM gateway behind their existing API gateway.

Is an AI gateway the same as an LLM router?

They overlap. Router usually means the part that decides which model or provider handles a request. Gateway is the broader term and includes routing plus keys, spend caps, rate limits, caching, logging and data policy. Most hosted products marketed as routers include gateway features, and most gateways include some routing.

Does an LLM gateway add latency?

Yes, every extra hop adds some time. How much depends on the gateway and where it runs relative to you and the provider; Smouter states sub-50ms gateway overhead. A cache hit can make a call much faster than going to the model at all. Measure time to first token on your own traffic to see the real effect.

Do I need an LLM gateway for a single model?

Often not. With one model, one service and modest traffic, calling the provider directly is simpler. A gateway starts to pay off when several services or people share the account, when you need per-key budgets and allowlists, when you want fallbacks across providers, or when you want to pay less for the same model.

Can an LLM gateway see my prompts?

A gateway is in the request path, so it processes your prompts and responses. What matters is what it keeps. Review its retention terms and whether you can disable storage. On Smouter, zero retention can be enabled per key or per request with x-smouter-zero-retention: on, which stops prompts and responses being retained for replay or caching.

What is an open-source LLM gateway?

An open-source gateway is one you can download and run on your own infrastructure, such as LiteLLM. You get full control over configuration, logs and network placement, and you can modify it. In return you deploy, scale and patch it yourself, and you still need accounts with each model provider behind it.