LLM API fallbacks: handling 429s, 5xx errors and provider outages
A good LLM API fallback strategy retries only transient failures (429, 5xx, timeouts) with exponential backoff, jitter and the server’s retry-after, makes every retry safe with an idempotency key, and then fails over: first to another provider serving the same model, then to a different model from an ordered list. Never retry 4xx validation errors, and accept that a stream that has already started cannot be transparently moved to another backend.
Every team that ships on top of a hosted model eventually learns the same lesson: the model is a remote dependency with its own capacity limits, maintenance windows and bad days. Popular models get congested at peak hours, rate limits change without notice, and a provider that served you fine for a month can return a burst of 503s on a Tuesday afternoon. The fix is not one clever library call. It is a small stack of habits, each covering a different failure mode. This guide walks through that stack in the order you should build it.
Which LLM API errors should you retry?
Start by classifying errors, because retrying the wrong one burns money and hides bugs. The rule of thumb: retry what the server says is temporary, never retry what says your request is wrong. A 400 for a malformed tool schema will fail identically on the fifth attempt, and on some APIs each attempt can still cost you.
| Status | Typical cause | Retry? | What to do |
|---|---|---|---|
| 400, 422 | Malformed body, unsupported parameter, context too long, conflicting options | No | Fix the request. Log the body shape and error type. |
| 401, 403 | Bad or revoked key, scope or allowlist forbids the model | No | Alert a human. Retrying a revoked key is noise. |
| 402 | Out of credit or a spend cap hit | No | Top up or raise the cap; do not loop. |
| 404 | Unknown model id or resource | No | Usually a typo or a retired model; fall back to a different model id. |
| 408, timeouts | Slow first token, network stall | Yes, once or twice | Retry with a fresh connection and an idempotency key. |
| 409 | A request with the same idempotency key is still running | Yes, after a wait | Back off, then retry to receive the replay. |
| 429 | Rate limit or tokens-per-minute window exhausted | Yes | Wait at least retry-after, then retry with backoff. |
| 500, 502 | Unexpected failure upstream or in the gateway | Yes, bounded | Backoff with jitter; count toward a circuit breaker. |
| 503, 504 | No healthy capacity, overloaded, upstream timeout | Yes | Backoff, honor retry-after if present, then fail over. |
Two refinements matter in practice. First, branch on the error type as well as the status: a 429 from a per-minute token cap and a 429 from a monthly subscription quota look the same at the status line but need very different responses. Second, treat a 400 that mentions context length as a routing signal, not a retry signal. The right move is to trim the prompt or switch to a long-context model, which is a fallback decision rather than a retry.
How do you implement retries with exponential backoff and jitter?
Exponential backoff doubles the wait after each failure so a struggling backend gets breathing room. Jitter randomizes that wait so a thousand clients that failed at the same instant do not all come back at the same instant and knock the backend over again. “Full jitter” (sleep a random amount between zero and the current cap) is the simplest variant that works well. On top of that, if the server sends retry-after, treat it as a floor: never come back earlier than the server asked.
Set a total time budget, not just an attempt count. A chat UI might give up after 20 seconds and show an error; a nightly job can afford minutes. And turn off the automatic retries inside your SDK if you implement your own, otherwise the two layers multiply (three SDK retries times four of yours is twelve calls).
import os
import random
import time
import uuid
import requests
URL = "https://api.smouter.ai/v1/chat/completions"
HEADERS = {"Authorization": f"Bearer {os.environ['SMOUTER_API_KEY']}"}
RETRIABLE = {408, 409, 429, 500, 502, 503, 504}
def retry_after_seconds(resp):
value = resp.headers.get("retry-after")
try:
return float(value) if value else 0.0
except ValueError:
return 0.0 # HTTP-date form; treat as no hint
def chat(body, max_attempts=5, base=0.5, cap=20.0, budget=60.0):
# One idempotency key per logical request, reused on every attempt.
headers = {**HEADERS, "Idempotency-Key": str(uuid.uuid4())}
deadline = time.monotonic() + budget
for attempt in range(max_attempts):
try:
resp = requests.post(URL, json=body, headers=headers, timeout=(5, 120))
except (requests.ConnectionError, requests.Timeout):
resp = None
if resp is not None and resp.status_code < 400:
return resp.json()
if resp is not None and resp.status_code not in RETRIABLE:
resp.raise_for_status() # 4xx: fix the request, do not retry
# Full jitter, with the server's retry-after as a floor.
backoff = random.uniform(0, min(cap, base * 2 ** attempt))
wait = max(backoff, retry_after_seconds(resp) if resp is not None else 0.0)
if time.monotonic() + wait > deadline:
break
time.sleep(wait)
raise RuntimeError("LLM call failed after retries")The connect timeout is short (5 seconds) because a backend that cannot accept a TCP connection is not going to recover inside your request. The read timeout is long because a large non-streaming completion can legitimately take a minute or more. If you stream, use an idle timeout between chunks instead of a total read timeout; more on that below.
Why do LLM retries need idempotency keys?
A timeout does not tell you whether the request failed. The model may have finished generating, and you may have been billed, while the response was lost on the way back. Retrying blindly means paying twice and, for agent workflows with side effects downstream, possibly acting twice. An idempotency key fixes this: you generate one unique key per logical request, send it on every attempt, and the server returns the stored result of the first completed attempt instead of running the work again.
On Smouter, sending an Idempotency-Key header on the chat-shaped surfaces replays a completed response without charging again, and the replay carries x-smouter-idempotent-replay: true. Reusing the key while the original is still running returns 409, and reusing it with a different body returns 422, which is why the code above treats 409 as “wait and try again” and generates exactly one key per logical call. Two caveats from the API reference: image endpoints do not honor the key, so an image retry generates and bills a new image, and zero-retention requests do not store a replay body, so a retry after a lost response returns an error instead of a replay.
Provider failover vs model fallback: what is the difference?
These get lumped together, but they solve different problems and have different costs.
- Provider failover keeps the same model and moves the request to another backend serving the same weights. Your prompts, output format and evals stay valid. This is the first thing to try, and for open-weight models there are often many hosts to choose from (see our guide to open-weight model APIs).
- Model fallback switches to a different model when the first one is unavailable. It keeps you online, but the output changes: different tone, different tool-calling quirks, different token counts and price. Order your fallback list by “closest behavior first,” not “cheapest first,” and run your evals against every model on the list.
With direct provider APIs you build both yourself: keys for several providers, a translation layer for their request shapes, health tracking per provider. That is a large part of why teams adopt a gateway; what an LLM gateway does covers the trade-offs, and OpenAI-compatible APIs explains why a single request shape makes failover cheap to build.
On Smouter the two layers map to two mechanisms. Provider failover is automatic: each request is routed across a pool of benchmarked nodes serving the same model, and the gateway fails over when a node degrades, so you do not manage per-provider keys. Model fallback is explicit: pass models: [...] in the body, or the x-smouter-models header when the body field is absent, with up to 16 model ids in priority order. Smouter selects the first viable candidate at dispatch. That wording is precise on purpose: it is a pre-dispatch choice, not a retry after a model has begun generating. If the chosen model fails mid-response, your client-side retry logic still applies.
curl https://api.smouter.ai/v1/chat/completions \
-H "Authorization: Bearer $SMOUTER_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: job-4821-step-3" \
-d '{
"model": "<primary-model-id>",
"models": ["<primary-model-id>", "<fallback-1>", "<fallback-2>"],
"messages": [{"role": "user", "content": "Summarize this incident report."}]
}'Every model in the list is still checked against the key’s model and provider allowlists, auto and @route ids are not allowed inside the list, and the list itself is never forwarded upstream. If you want the gateway to pick the model for you instead of following a fixed order, that is routing rather than fallback; see LLM model routing.
Can you fail over in the middle of a streaming response?
Not transparently. Once the first tokens have reached your user, any failover means starting a new generation, and a new generation will not continue the old one word for word. You have three honest options: restart the stream and replace the partial text in the UI, send the partial output back as context and ask a model to continue (which works for prose and fails often for code and JSON), or surface an error with a retry button.
What you can do is fail fast before the first token. Put a tight time-to-first-token timeout on streaming calls (a few seconds for interactive use), retry or fall back if it trips, and only then commit to the stream. After that, switch to an idle timeout: if no chunk arrives for N seconds, abort and treat it as a failure. Watch for end-of-stream signals too. A stream that closes without a finish reason, or the Responses API emitting response.failed, is a failure even though the HTTP status was 200. On reasoning models, finish_reason: "length" with empty content usually means the reasoning consumed the max_tokens budget; raise the budget rather than retrying with the same one.
How should you handle 429 rate limits specifically?
A 429 is the one error where the server usually tells you exactly how long to wait, so read the headers before you invent a backoff. On Smouter, a tokens-per-minute rejection returns 429 tokens_per_minute_exceeded with retry-after, x-ratelimit-reset, x-ratelimit-remaining-tokens and x-ratelimit-limit-scope (whether the key cap or the organization cap was the binding one). Request-rate 429s carry retry-after too. The rate limits and TPM support article explains how per-key caps are set.
If you are hitting 429s in steady state, retries are the wrong tool: you are simply sending more than your budget. Add a client-side token bucket that paces requests to your known limit, move bulk work that is not latency-sensitive to an async queue such as a batch API, and cache deterministic repeats so they never count against the limit at all (LLM caching explained).
Do you need a circuit breaker for LLM calls?
If you call more than one backend yourself, yes. Retries with backoff protect a single request; a circuit breaker protects the whole system. It counts recent failures per backend (per provider, or per provider and model pair) and, past a threshold, stops sending traffic there for a cool-down period, routing straight to the next option instead of paying a timeout on every request first. After the cool-down it lets a few probe requests through and closes again if they succeed.
- Trip on a failure rate over a sliding window (for example, more than 50% of the last 20 calls), not on a single error.
- Count timeouts and 5xx toward the breaker; do not count 4xx, which are your bugs, or 429, which is pacing.
- Keep breaker state per model, not just per provider: one provider can serve one model well and another badly.
- Expose breaker state in metrics. A breaker that is quietly open for a day is an outage you did not notice.
A gateway that tracks node health centrally does a version of this for you across every customer’s traffic, which gives it far more signal than your own process sees. You still want a breaker around the gateway itself, as you would around any single dependency. Check the status page when you see a sustained error rate rather than guessing.
What timeouts should you set for LLM API calls?
- Connect timeout: 3 to 5 seconds. Failing to connect is not something a longer wait fixes.
- Time to first token (streaming): 5 to 15 seconds for chat, longer for reasoning models that think before they write.
- Idle timeout between stream chunks: 30 to 60 seconds.
- Total timeout (non-streaming): sized to your largest expected completion, often 60 to 180 seconds.
- Overall budget including retries: set by the caller’s tolerance, for example 20 seconds behind a UI, several minutes for background jobs.
Long reasoning runs blur these numbers, so measure your own p99 latencies per model and set timeouts a comfortable margin above them. A timeout below your normal p99 is a self-inflicted outage that also doubles your bill.
A fallback checklist you can ship this week
- Classify errors by status and type; retry only 408, 409, 429 and 5xx.
- Use exponential backoff with full jitter, with
retry-afteras a floor, under a total time budget. - Disable SDK-level retries if you run your own loop.
- Send one idempotency key per logical request and reuse it across attempts.
- Fail over to another provider for the same model before switching models.
- Keep an ordered fallback model list, closest behavior first, and eval every entry.
- Put a time-to-first-token timeout on streams; never pretend a mid-stream failover is seamless.
- Wrap each backend in a circuit breaker and alert on its state.
Reliability and cost are linked: every wasted retry is a paid call, and fallback lists are a natural place to put a cheaper model. If cost is your next problem, how to reduce LLM API costs picks up where this guide ends.
Frequently asked questions
Should I retry a 400 error from an LLM API?
No. A 400 means the request itself is invalid: a malformed tool schema, an unsupported parameter, or a prompt longer than the model’s context. It will fail the same way every time. Log the error type, fix the request, or treat a context-length 400 as a signal to trim the prompt or fall back to a model with a longer context window.
How long should I wait after a 429 Too Many Requests?
At least as long as the retry-after header says, and add jitter on top so many clients do not return at once. If the response includes reset or remaining-token headers, use them to pace the next requests. If you see 429s constantly, slow your send rate with a client-side token bucket rather than leaning on retries.
What is the difference between a fallback model and provider failover?
Provider failover keeps the same model and moves the request to another backend serving the same weights, so outputs stay consistent. A fallback model is a different model used when the first is unavailable; it keeps you online but changes behavior, price and token counts. Try provider failover first, then fall back to the closest-behaving model.
Does Smouter retry my request if the model fails mid-generation?
No. The models list and the x-smouter-models header choose the first viable candidate at dispatch; they are not a retry after a model has started generating. Separately, the gateway fails over automatically between nodes serving the same model when one degrades. Keep client-side retries with an Idempotency-Key for failures that happen after dispatch.
Will retrying an LLM request charge me twice?
It can, because a timed-out request may have completed on the server. Use an idempotency key: on Smouter, a retry with the same Idempotency-Key on chat-shaped endpoints replays the completed response without charging again. Image endpoints do not honor the key, and zero-retention requests store no replay body, so plan retries for those separately.
Can I fail over a streaming LLM response to another provider?
Not seamlessly. Once tokens have been sent to the user, switching backends means starting a new generation that will not match the partial text. Fail fast before the first token with a short time-to-first-token timeout, then use an idle timeout between chunks and restart or show a retry option if the stream stalls.