OpenAI-compatible API: what it means and how to switch base_url safely
An OpenAI-compatible API is any endpoint that accepts the same HTTP requests as OpenAI’s chat completions API (POST /v1/chat/completions with model and messages) and returns the same JSON and streaming shapes. Switching to one usually means changing three values in your client: the base URL, the API key and the model id. The code stays the same; the differences live in model ids, tool-calling details, streaming usage and which optional parameters are honored.
What does "OpenAI-compatible" actually mean?
It means wire compatibility, not model compatibility. The request body, headers, response body and server-sent event stream follow the shapes OpenAI published for chat completions, so any client built for that shape (the official SDKs, LangChain, the Vercel AI SDK, most coding agents) can talk to the endpoint without code changes. The model behind it can be anything: an open-weight model, a Claude model, a fine-tune on a private cluster.
In practice a compatible endpoint implements a core set and then varies at the edges. The core is stable and worth relying on:
- Auth:
Authorization: Bearer <key>on every request. - Request:
model,messageswithsystem/user/assistant/toolroles, plus common sampling fields such astemperature,top_pandmax_tokens. - Response: a
choicesarray withmessage.content, afinish_reason, and ausageobject with prompt and completion token counts. - Streaming:
stream: truereturns server-sent events whose chunks carrychoices[0].delta, ending withdata: [DONE]. - Discovery:
GET /v1/modelslists the ids you can send.
Everything beyond that core (tool calling details, structured output, vision inputs, logprobs, the newer Responses API) is where "compatible" turns into "mostly compatible". The rest of this guide is about those edges. If you are still deciding whether a gateway in front of several providers is the right shape for you at all, start with what an LLM gateway is.
How do I point the OpenAI SDK at a different base_url?
Every official OpenAI SDK takes a base URL and an API key in its constructor, and most also read them from environment variables. The examples below use Smouter, whose OpenAI-style base URL is https://api.smouter.ai/v1 and whose keys start with sk-smouter-. Replace <model-id> with an id copied from the models page or from GET /v1/models; ids are matched verbatim.
Python (openai package)
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.smouter.ai/v1",
api_key=os.environ["SMOUTER_API_KEY"],
)
resp = client.chat.completions.create(
model="<model-id>",
messages=[
{"role": "system", "content": "Be concise."},
{"role": "user", "content": "Explain idempotency keys in two sentences."},
],
)
print(resp.choices[0].message.content)
print(resp.usage)Node / TypeScript (openai package)
import OpenAI from 'openai'
const client = new OpenAI({
baseURL: 'https://api.smouter.ai/v1',
apiKey: process.env.SMOUTER_API_KEY,
})
const stream = await client.chat.completions.create({
model: '<model-id>',
messages: [{ role: 'user', content: 'Write a haiku about latency.' }],
stream: true,
})
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? '')
}curl
curl https://api.smouter.ai/v1/chat/completions \
-H "Authorization: Bearer $SMOUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "<model-id>",
"messages": [{"role": "user", "content": "Say hello."}]
}'Many tools do not expose a constructor, only settings fields or environment variables. The usual pair is OPENAI_BASE_URL (older tools read OPENAI_API_BASE) and OPENAI_API_KEY. The any OpenAI client guide lists the variables and SDK settings that cover most tools, and there are per-tool pages for editors and agents such as Cursor and Aider.
What about the Anthropic-compatible API?
Anthropic’s Messages API is the other wire format that tools standardize on. It differs from chat completions in real ways: the system prompt is a top-level system field rather than a message, max_tokens is required, content is a list of typed blocks, tool calls come back as tool_use blocks, auth is usually an x-api-key header, and streaming uses named event types instead of bare deltas. A translation layer can map one onto the other, but tools like Claude Code expect the native shape.
Smouter serves POST /v1/messages natively at base URL https://api.smouter.ai (no /v1 suffix), and accepts the key either as a bearer token or as x-api-key. With the Anthropic Python SDK that looks like this:
import os
import anthropic
client = anthropic.Anthropic(
base_url="https://api.smouter.ai",
api_key=os.environ["SMOUTER_API_KEY"],
)
msg = client.messages.create(
model="<model-id>",
max_tokens=512,
system="Be concise.",
messages=[{"role": "user", "content": "What is a Merkle tree?"}],
)
print(msg.content[0].text)For Claude Code specifically it is two environment variables, ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN, plus a model id; the Claude Code setup guide has the exact config, and using Claude Code with other models covers what changes when the model behind it is not a Claude model.
What usually differs between "compatible" providers?
The request will almost always parse. The failures show up later, in behavior. These are the areas to test before you move production traffic:
| Area | What varies | What to do |
|---|---|---|
| Model ids | Each provider names models its own way: vendor prefixes, version suffixes, date stamps, case. | Never hard-code a list. Read GET /v1/models and copy ids verbatim; keep the id in config, not code. |
| Tool calling | Whether tool_choice values are honored, whether parallel tool calls are emitted, how arguments stream, and how strictly the JSON schema is followed. | Validate tool arguments yourself. Test a multi-tool turn and a turn where the model should call no tool. |
| Structured output | response_format with a JSON schema may be enforced, approximated with JSON mode, or ignored. | Parse and validate every response. Treat schema adherence as a property of the model, not the endpoint. |
| Streaming usage | Some endpoints send token usage in a final chunk only when asked (OpenAI uses stream_options.include_usage), some always, some never. | Do not assume usage arrives on streams. Reconcile cost from the provider’s usage logs or receipts. |
| Unsupported parameters | Fields like n > 1, logprobs, seed, logit_bias or provider-specific extensions may be rejected with a 400 or silently dropped. | Send only what you use. Prefer an endpoint that rejects loudly over one that ignores quietly. |
| Reasoning tokens | On reasoning models, hidden reasoning may count against max_tokens, so a tight budget yields truncated or empty content. | Check finish_reason for length and give reasoning models room. |
| Errors and limits | Error type strings, 429 semantics and rate-limit headers differ. | Branch on HTTP status first, honor retry-after, and log the raw error body. |
To make that concrete with Smouter’s documented behavior: n must be 1; store and provider-specific cache-control extensions are stripped (caching is controlled with Smouter’s own headers instead); unsupported provider-specific fields can still fail when the request is dispatched to a model; reasoning tokens are drawn from the same max_tokens allowance as the answer, and a response that hit the ceiling carries an x-smouter-warning header. The Responses API endpoint is stateless, so a non-empty previous_response_id returns a 400 and you resend the conversation in input. Every detail is in the docs.
One more difference is easy to miss: rate limiting. A gateway or provider may enforce tokens-per-minute as well as requests-per-minute, and a 429 can mean either. Read retry-after rather than guessing a backoff; rate limits and TPM caps explains how Smouter’s per-key and org limits interact, and LLM API fallbacks covers what to do when a model is throttled or down.
Why use an OpenAI-compatible gateway instead of each provider’s own API?
Because one wire format lets you change models with a config edit. When every model sits behind the same base URL and key, you can A/B two models, move a background job to a cheaper model, or add a fallback list without touching your call sites. Billing lands in one place, and per-key controls (spend caps, model allowlists, TPM limits) apply uniformly.
The trade-off is an extra hop and a dependency on the gateway’s translation for anything non-OpenAI underneath. Smouter states sub-50ms gateway overhead and serves 50+ models behind one key, shopping each request across a pool of providers that serve the same model weights; if you want to compare gateways on those terms, see OpenRouter alternatives. If cost is the reason you are switching, how to reduce LLM API costs goes through the levers that matter more than which endpoint you call.
Migration checklist: moving to a new OpenAI-compatible endpoint
- Inventory your call sites. Find every place that constructs a client or reads
OPENAI_BASE_URL/OPENAI_API_KEY, including background workers, eval scripts and third-party tools. - Centralize base URL, key and model id in config. One environment variable each, so rollback is a config change.
- Create a scoped key. A dedicated key per service with a spend cap and, if your traffic uses only a few models, a model allowlist. See create and manage API keys.
- Map model ids. Copy the new ids from the catalog. Keep a mapping table from old id to new id in one place.
- Verify outside your app.
curlthe models list and a one-line chat completion with the new key before touching application code. - Run your parameter audit. List every optional field you send (
n,seed,logprobs,response_format,tools,tool_choice,stream_options) and confirm each is supported for the models you picked. - Test tool calls and structured output on real prompts, not toy ones. Compare against the old provider on a fixed sample.
- Check streaming end to end, including how your client handles a stream that ends early and whether you still get usage numbers.
- Handle errors by status. 400 means fix the request, 401/403 means credentials or scope, 402 means funding, 429 means wait for
retry-after, 5xx means retry or fall back. - Shift traffic gradually and compare cost per request, latency and error rate before you remove the old path.
Frequently asked questions
What is an OpenAI-compatible API?
It is an HTTP API that accepts the same requests and returns the same response and streaming shapes as OpenAI’s chat completions endpoint. Any client written for OpenAI can call it by changing the base URL, the API key and the model id. The model behind the endpoint can come from any vendor, so compatibility describes the wire format, not the model’s behavior.
How do I change the base_url in the OpenAI Python SDK?
Pass it to the constructor: OpenAI(base_url="https://api.smouter.ai/v1", api_key=...). The SDK also reads the OPENAI_BASE_URL environment variable if you prefer not to touch code. Include /v1 in the base URL and leave off /chat/completions, because the SDK appends the endpoint path itself.
Can I use Claude models through an OpenAI-compatible API?
Yes, if the endpoint routes to them. A gateway can accept chat completions requests and serve them with a Claude model, translating between formats. Tools built for Anthropic’s own format, such as Claude Code, should use an Anthropic-compatible Messages endpoint instead; Smouter serves both, at /v1 for OpenAI-style clients and at the bare domain for Anthropic-style clients.
Why do tool calls behave differently on another OpenAI-compatible provider?
The request format is shared, but tool-calling quality depends on the model and on how the serving stack implements tool_choice, parallel calls and argument streaming. Some models follow schemas loosely or emit malformed JSON arguments. Validate arguments in your code and test multi-tool turns on the exact models you plan to use before moving production traffic.
Do OpenAI-compatible APIs return token usage when streaming?
Not consistently. OpenAI includes usage in a final chunk when you set stream_options.include_usage, while other endpoints may always send it, never send it, or ignore the option. Do not build billing on stream usage alone. Reconcile from the provider’s usage logs; Smouter settles a live stream’s final charge after it completes and shows it in Usage.
Is switching to an OpenAI-compatible gateway risky?
The code change is small; the risk is behavioral. Model ids, optional parameters, structured output and rate-limit semantics can differ, so run your real prompts through the new endpoint first, keep base URL and key in config, and shift traffic gradually with the old path ready as a rollback.