Skip to content

LLM model routing: static rules, cascades, classifiers and auto-routing

Published 10 min read

LLM model routing means deciding, per request, which model (and which provider of that model) should answer it, instead of sending everything to one fixed model. The common strategies are static routing by task, cascades that try a cheap model first and escalate, and classifier-based routing. Done well, it cuts cost and improves availability without lowering quality on the requests that matter.

Why route between LLMs at all?

Most production traffic is uneven. A support bot sees a long tail of easy questions and a handful of hard ones; a coding agent mixes file searches and one-line edits with multi-file refactors. Sending all of it to the strongest model you can afford pays frontier prices for work a much smaller model would do identically. Routing is how you stop paying for capability you don’t use, and it doubles as a reliability tool: if you can serve a request from more than one place, one provider’s bad afternoon is not your outage.

There are two different questions hiding under the word “routing”, and mixing them up causes most of the confusion:

  • Which model should answer? Routing across different models: a small open-weight model for classification, a frontier model for hard reasoning. This changes the output, so it needs evaluation.
  • Who should serve that model? Routing across providers for the same model weights, by price, latency or availability. Done correctly, the output distribution is the same; what changes is cost and uptime.

The rest of this guide covers both, starting with the strategies for choosing a model.

What are the main LLM routing strategies?

Most production systems combine at least two of these.
StrategyHow it decidesStrengthsWeaknesses
Static routing by taskYour code maps each feature or endpoint to a modelPredictable, easy to debug and testCoarse: one feature can contain easy and hard requests
Cascade (cheap first, escalate)Try a small model; escalate if a check failsPays frontier prices only on the hard tailAdds latency on escalations; needs a reliable failure check
Classifier routingA small model or trained classifier predicts difficulty or category up frontOne model call per request; adapts within a featureClassifier errors send hard prompts to weak models
LLM-as-judge selectionSeveral models answer; a judge picks the bestHighest quality ceilingPays for every candidate plus the judge
Price and availability routingSame model, choose the provider by live price and healthLower cost and better uptime with no quality changeOnly works when providers truly serve the same weights

Static routing by task

Start here. Give every call site an explicit model chosen for that job: extraction and tagging on a small model, user-facing answers on a mid-tier model, long-horizon planning on a frontier model. Put the mapping in config, not scattered string literals, so you can change it without a deploy. Static routing captures a surprising share of the savings because feature boundaries often line up with difficulty.

Cascades: cheap first, escalate on failure

A cascade sends the request to a cheap model, checks the answer, and escalates to a stronger model only if the check fails. The whole design rests on the check. Good checks are mechanical: the JSON parses and matches the schema, the code compiles and the tests pass, the answer cites a document that exists. Weak checks (“does the model sound confident?”) let bad answers through and the cascade quietly degrades quality. Budget for the latency of escalated requests, which pay for two calls.

Classifier and LLM-judge routing

Classifier routing makes the decision before any expensive call: a small model labels the request (category, difficulty, language, whether tools are needed), and the label maps to a model. It is cheaper than a cascade when most requests are easy, but every misclassification is silent. The heavier variant fans the request out to several models and lets a judge model pick the best answer. That buys quality, not savings, and suits high-value, low-volume calls such as final drafts or code review.

Routing across providers vs routing across models: what’s the difference?

Popular open-weight models, such as DeepSeek V4 or Kimi K3, are served by many providers at different prices, speeds and reliability levels. Routing the same model across those providers is a pricing and availability problem, not a quality problem, provided the provider is actually running the weights it claims at the precision it claims. That proviso is the hard part: a cheap endpoint that quietly serves a different or degraded model turns a cost optimization into a quality regression you never evaluated. Open-weight model APIs goes deeper on the supply side.

Routing across different models is the opposite: availability is easy (just call another model), but the output changes, so it must be evaluated. A fallback from one model family to another during an outage is a model switch, even if it only happens for ten minutes. Make sure the fallback model handles your prompts, tool schemas and output format before you need it. LLM API fallbacks covers the outage side in detail.

How do you evaluate a routing change before switching?

  1. Build a replay set. Sample a few hundred real requests per feature, stratified so the hard tail is represented, with any sensitive data removed.
  2. Define pass criteria per feature. Schema validity, exact-match labels, tests passing, or a rubric scored by a judge model. Write it down before you look at results.
  3. Run old and new side by side. Same prompts, temperature: 0, both routes. Batch jobs are a cheap way to do this; see the LLM batch API guide.
  4. Read the diffs, not just the score. Aggregate scores hide the five requests where the new route fails badly.
  5. Shadow or canary in production. Send a small share of live traffic to the new route and compare cost, latency, error rate and user signals before widening it.

What are the common pitfalls of LLM routing?

  • Silent quality drift. A router that cuts the average cost while failing your highest-value requests is a net loss. Evaluate per segment.
  • Prompt coupling. Prompts tuned for one model family often underperform on another. A routed request may need a per-model prompt variant.
  • Tool and format mismatch. Models differ in tool-call reliability and JSON adherence. Validate structured output after every call, not just on the primary model.
  • Cache fragmentation. Prompt caches are per model and often per provider, so spreading identical traffic across many endpoints can lower cache hit rates. See LLM caching explained.
  • Retries mid-stream. Switching models after a stream has started produces a stitched answer. Decide the route before the first token.
  • Router cost. A classifier or judge is itself a model call. Count it in the cost comparison.

How does model routing work on Smouter?

Smouter is an LLM gateway, so routing is a request parameter rather than code you maintain. There are four tools, from least to most control.

Provider routing for the same model (always on)

Whatever model you name, each request is shopped across a benchmarked pool of providers serving the same weights. Nodes are fingerprinted against reference weights and benchmarked continuously, and nodes that advertise a low price but fail to serve are demoted. The gateway also fails over automatically when a node degrades. This is the cross-provider routing described above; you get it without changing your model id. If you need the model maker’s own endpoint instead, send x-smouter-lane: verified; the value vs verified lanes article explains the trade-off.

`model: "auto"`

Send model: "auto" and Smouter resolves the request to one concrete available model, aiming for the cheapest model that can handle it, before pricing. It is account-gated, its responses are not cached, and it is unavailable to scoped keys that could not authorize every possible resolution. Use it where the cost of an occasional different model is low; pin a model where outputs must be stable.

Fallback lists

Pass models: [...] in the body, or the x-smouter-models header when the body field is absent, with up to 16 ids. Smouter picks the first viable candidate at dispatch. This is not a retry after a model has started generating, which avoids stitched answers. auto and @route ids are not allowed in the list, every candidate is checked against the key’s model allowlist, and the list is never forwarded upstream.

bash
curl https://api.smouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $SMOUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-smouter-tags: production,checkout" \
  -d '{
    "model": "<primary-model-id>",
    "models": ["<primary-model-id>", "<fallback-model-id>"],
    "messages": [{"role": "user", "content": "Summarize this."}]
  }'

Copy real ids from the models page. The x-smouter-tags header labels the request so you can slice usage per feature, as in the tip above.

Programmable routes: `@route-name`

For cascades, guardrails and judge patterns, define a route in the dashboard and call it with model: "@route-name". Stages can optimize (trim the prompt), guardrail (block what should never reach a model), fan out (run several models on the same input), judge (pick the strongest answer) and return. Each stage is a normal per-token request, and billed sub-calls settle under one route execution with one receipt. Responses carry x-smouter-route-name and x-smouter-route-stages so you can see which stages ran. Because your code only knows the route id, you can swap a judge or add a model without shipping a release. Route invocations are non-streaming (stream: true returns 400).

All of these work through any OpenAI-compatible client by changing the base URL; see the OpenAI-compatible API guide and the routing section of the docs for the full parameter reference.

Frequently asked questions

What is LLM model routing?

It is choosing, per request, which model and which provider should answer, instead of sending all traffic to one fixed model. Strategies include static routing by task, cascades that try a cheap model first and escalate, classifier routing, and judge-based selection. Routing the same model across providers is a separate technique aimed at price and availability rather than quality.

What is an LLM cascade?

A cascade sends a request to a cheaper model first, checks the answer, and escalates to a stronger model only if the check fails. It works best when the check is mechanical, such as schema validation or passing tests. Escalated requests pay for two calls and take longer, so cascades save the most when the majority of requests pass on the first try.

Does routing to cheaper models reduce quality?

It can, which is why routing changes need evaluation. Replay a representative sample of real requests through the old and new routes, score them against criteria you set in advance, and read the failures, not just the averages. Routing the same model across providers should not change quality, provided each provider really serves the claimed weights.

What does model auto do on an AI gateway?

On Smouter, model: "auto" resolves each request to one concrete available model, aiming for the cheapest model that can handle it, before the request is priced. Auto responses are not cached and it is not available to scoped keys that cannot authorize every possible model. Pin a specific model when you need stable, reproducible outputs.

What is the difference between fallbacks and routing?

Routing decides which model should answer a healthy request, usually for cost or quality. Fallbacks decide what happens when the preferred option is unavailable. On Smouter, a fallback list picks the first viable candidate at dispatch, before generation starts, so a request never switches models halfway through an answer. The two are complementary and usually configured together.

Should I build my own LLM router or use a gateway?

Static routing by task is a few lines of config and worth owning yourself. Cross-provider price and availability routing needs live pricing, health checks and failover across many providers, which is what gateways exist to do. Cascades and judge pipelines sit in between: build them if they are core to your product, or use a gateway’s route feature to change them without redeploying.