LLM batch APIs: how async batch inference works and when to use it
An LLM batch API lets you submit many requests at once and collect the results later, usually within a fixed window such as 24 hours, instead of holding a connection open for each call. You trade latency for throughput and, on many platforms, a lower price. It fits any job where nobody is waiting on the answer: evals, classification, enrichment and bulk summarization.
What is batch inference?
A normal chat completion is synchronous: you send one request, the model generates, you get one response. A batch job inverts that. You upload a list of requests (each one a complete chat-completion body with its own id), the platform accepts the job and returns a job id, and the requests are processed asynchronously whenever capacity allows. You poll the job, and when it is done you download one results file that maps each id back to its output.
The economics come from scheduling freedom. A provider serving interactive traffic has to keep headroom for bursts; batch work can be slotted into the gaps. That is why providers commonly price batch lower than real-time calls, though the size of the discount, and whether there is one at all, varies by platform and model. Check the current terms where you buy rather than assuming a number. For the wider picture of how per-token pricing works, see LLM API pricing explained.
Which workloads fit a batch API, and which don’t?
The test is simple: if a human or a downstream service is blocked waiting for the answer, it is not a batch job. If the output lands in a table, a file or a dashboard that someone reads tomorrow, it probably is.
| Workload | Batch fit | Why |
|---|---|---|
| Model evals and regression suites | Good | Hundreds to thousands of fixed prompts, results compared offline. Determinism and cost matter more than speed. |
| Classification and tagging | Good | Label tickets, reviews or documents in bulk; short outputs, many lines. |
| Data enrichment | Good | Extract fields, normalize addresses, write product descriptions for a catalog. |
| Summarization backlogs | Good | Nightly digests of calls, threads or reports. |
| Embeddings backfills | Depends | Good fit in principle, but many batch endpoints only accept chat lines. Large array calls to the embeddings endpoint are often the simpler path. |
| Chat, copilots, agents | Poor | A user is waiting. Agents also need each answer before choosing the next step. |
| Streaming UIs | Poor | Batch lines run non-streaming; there is nothing to stream to. |
| Anything with a hard SLA under the batch window | Poor | The window is an upper bound, not a promise of speed. |
Independence matters as much as latency. Batch lines run in no guaranteed order and cannot see each other’s outputs, so a pipeline where step two depends on step one has to be split into two jobs, or run synchronously.
How do you structure batch requests? The JSONL and custom_id pattern
Almost every batch API uses the same shape: one JSON object per line (JSONL), each carrying a custom_id, the HTTP method and endpoint, and a body that is exactly what you would have sent to the synchronous endpoint. The custom_id is the only reliable way to join results back to inputs, because results are not guaranteed to come back in submission order.
{"custom_id":"ticket-81723","method":"POST","url":"/v1/chat/completions","body":{"model":"<model-id>","temperature":0,"messages":[{"role":"system","content":"Classify the ticket as billing, bug or other. Reply with one word."},{"role":"user","content":"I was charged twice this month."}]}}
{"custom_id":"ticket-81724","method":"POST","url":"/v1/chat/completions","body":{"model":"<model-id>","temperature":0,"messages":[{"role":"system","content":"Classify the ticket as billing, bug or other. Reply with one word."},{"role":"user","content":"The export button does nothing."}]}}- Make
custom_idmeaningful. Use your own primary key (ticket-81723), not a line number. Then a result can be written straight back to its row, and re-submitting a subset is trivial. - Keep it unique per job. Duplicate ids make the join ambiguous; treat a duplicate as a bug in the job builder.
- Pin sampling. For evals and classification,
temperature: 0keeps reruns comparable. - Cap output. Set a sensible output limit per line. A runaway line costs the same in batch as it would live.
How do you poll a batch job and handle the results?
After submission you get a job id. Poll it on a slow interval (every 30 to 60 seconds is plenty for a job measured in hours) and back off further as it runs. When the job reports completion, download the results once, store the raw file, then parse it. Keeping the raw file means a parsing bug never costs you a rerun.
Results usually arrive as newline-delimited JSON with one record per input line, each tagged with its custom_id. Parse line by line rather than loading the whole file, and expect a mix of successes and per-line errors in the same file. A completed job is not the same as every line succeeding.
How should you handle failures and avoid paying twice?
Batch failures come in three sizes, and each needs its own handling:
- Rejected at submit. Malformed lines, a missing
modelormessages, or a job that exceeds size limits. Validate locally before uploading; it is cheaper than a round trip. - Per-line errors. One line hits a content filter, an unavailable model or a bad parameter. Collect the failed
custom_ids and submit them as a new, smaller job after fixing the cause. - Job-level failure or expiry. The job did not finish inside its window. Lines that completed are typically still billed, so do not blindly resubmit the whole input.
The way to make retries safe is to treat your own datastore as the source of truth. Before building a retry job, filter the input down to ids that have no stored successful result. That makes the whole pipeline idempotent at the custom_id level: running it twice produces the same table and only pays for the missing lines.
Is a batch API always cheaper?
Per token, it often is, because providers commonly offer a batch rate below their real-time rate. But the cheapest token is the one you don’t send. Before reaching for batch, check whether the job could use a smaller model (classification rarely needs a frontier model), whether shared system prompts benefit from prefix caching, and whether repeated inputs could be served from a response cache. How to reduce LLM API costs and LLM caching explained cover those levers; batch stacks on top of them.
Also watch total cost, not unit cost. A batch eval that runs 5,000 prompts against three candidate models is cheap per line and still a real bill. Estimate tokens first: how tokens are counted gives the rough ratios.
How does the Smouter Batch API work?
Smouter exposes POST /v1/batches on the same base URL and key as its chat endpoint. The request is a JSON envelope: completion_window set to 24h, and an input field holding either an array of line objects or a single raw JSONL string. Each line needs a body with model and messages; method and url are optional, but when present they must be POST and /v1/chat/completions. If you omit custom_id, a line defaults to req-<n>, but you should always set your own.
# Submit a job
curl https://api.smouter.ai/v1/batches \
-H "Authorization: Bearer $SMOUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"completion_window":"24h",
"input":[
{"custom_id":"line-1","method":"POST","url":"/v1/chat/completions",
"body":{"model":"'"$MODEL_ID"'","messages":[{"role":"user","content":"Hello"}]}}
]
}'
# List your 20 most recent jobs, or poll one
curl https://api.smouter.ai/v1/batches \
-H "Authorization: Bearer $SMOUTER_API_KEY"
curl https://api.smouter.ai/v1/batches/$BATCH_ID \
-H "Authorization: Bearer $SMOUTER_API_KEY"Copy a live model id from the models page into MODEL_ID. The behaviors worth knowing before you build on it, all from the docs:
- Results are NDJSON. They come back as
application/x-ndjsonwith a trailing newline. Asking before the job is ready returns409 batch_not_ready, so treat 409 as “keep polling”, not as an error. - Lines run non-streaming, and jobs expire after the completion window.
- Cancellation is best effort. The line already in flight may finish; later lines do not start.
- Checks run per line. Each line rechecks the key’s scopes, spend caps, zero-retention state and the org safety policy at execution time. Async batch work is not subject to the per-minute token gate described in rate limits and TPM caps.
- Not for zero-retention keys. Batch inputs and results are stored so you can retrieve them, so a zero-retention key is rejected. If your data needs zero data retention, run it synchronously instead.
- Not for org-billed keys yet. These return
400 org_batch_unsupported; use a personal key for now. Delegated Connect tokens cannot use batch either. - Size limits are configurable. Current line and byte ceilings are returned by
GET /account/batches/config; read them rather than hard-coding them.
Completed jobs earn a discount. The rate shown in the batch dashboard is a ceiling, not a guaranteed figure: the credit is capped per line, so some models refund less than the headline rate, and it is finalized only for completed jobs, never cancelled ones. The batch processing support article covers the dashboard side, including where the current rate is displayed.
One scope note: Smouter batch lines target /v1/chat/completions. For an embeddings backfill, send arrays to POST /v1/embeddings instead (up to 2,048 strings per call) and drive it with a simple worker that tracks which ids are done, using the same idempotency approach as above.
Frequently asked questions
What is the difference between a batch API and a normal LLM API call?
A normal call is synchronous: one request, one response, while you wait. A batch API accepts many requests in one job, processes them asynchronously within a window such as 24 hours, and returns all results in one file. You give up immediate answers in exchange for simpler bulk handling, no per-minute rate limit juggling and, on many platforms, a lower per-token price.
How long does an LLM batch job take?
The completion window, commonly 24 hours, is the upper bound rather than a typical duration. Small jobs often finish much sooner, but you should design as if the full window will be used. If you need a guaranteed answer within minutes, use synchronous requests with concurrency control instead of a batch job.
What is custom_id in a batch request?
It is a string you attach to each line so you can match results back to inputs, since results are not guaranteed to come back in order. Use your own primary key, such as a ticket or document id, and keep it unique within the job. It also makes retries easy: resubmit only the ids that have no stored successful result.
Can I stream responses from a batch API?
No. Batch lines run non-streaming because nothing is connected to receive the stream; the output is written to the results file. If your interface depends on tokens appearing as they are generated, that request belongs on the synchronous chat endpoint with streaming enabled, not in a batch.
Are batch API results billed if the job fails?
Generally, lines that were processed are billed even if the job as a whole fails, expires or is cancelled, because the model did the work. That is why retries should only include lines without a stored success. On Smouter, the batch discount is credited only after a completed job, never a cancelled one.
Can I use a batch API with zero data retention?
Usually not, because batch inputs and results have to be stored until you download them. Smouter rejects batch requests from zero-retention keys for exactly this reason. Sensitive workloads that need zero retention should run as synchronous requests with zero retention enabled per key or per request.