Batch API for Open Models: Half-Price Chat, Embeddings and Responses Jobs with the OpenAI SDK

Most of the work people do with language models is not a chat. It is a spreadsheet of ten thousand support tickets to label, a product catalogue to translate, a document archive to embed, an evaluation set to grade. For that kind of work, the answer does not need to arrive in half a second; it needs to arrive correct, complete and cheap.

That is what the Paralon Batch API is for. You upload a JSONL file of requests, and you get a file of results back, at half the per-token price of the same requests sent live. It speaks OpenAI's Batch API, so the files and batches calls in the official SDKs work unchanged, and it runs every endpoint we serve: chat completions, embeddings and the Responses API, on open models.

We put it through a real workload with the official OpenAI Python SDK before writing this. Here is what came back.

batch	requests	completed	failed	billed vs live price
support-ticket classification, qwen3.8-27b, JSON schema	300	300	0	exactly 50%
ticket embeddings, multilingual-e5-small	100	100	0	exactly 50%
reply drafts through the Responses API	20	20	0	exactly 50%

And the question every team asks first, whether a batch slows down their live traffic: the median latency of a live request was 423 ms just before the 300-ticket batch started and 430 ms while it ran.

Why a batch API at all

If you have ever run a few thousand requests through an API by hand, you know the shape of the script: a thread pool, a rate limiter tuned to your key, a retry loop for 429s and the occasional 502, checkpointing so a crash does not mean starting over, and a merge step at the end. It works, and it is an afternoon of code that has nothing to do with your problem.

A batch API replaces all of it with three calls:

1. Upload a file where each line is one request.
2. Create a batch from it.
3. Download the results when it is done.

The pacing, the retries and the bookkeeping move to our side. Batches run on network capacity that live requests are not using, never take a slot a live request is waiting for, and do not count against your key's per-minute or in-flight limits. That is how the price can be half: you give up an immediate answer, and in exchange you pay for GPU time that would otherwise sit idle.

The whole thing with the OpenAI SDK

The input file is JSONL, one request per line, each with your own custom_id and the body you would have sent live:

{"custom_id": "t0", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen3.8-27b", "temperature": 0, "max_tokens": 30, "response_format": {"type": "json_schema", "json_schema": {"name": "intent", "schema": {"type": "object", "properties": {"intent": {"type": "string", "enum": ["cancel_order", "damaged_item", "order_status", "..."]}}, "required": ["intent"]}}}, "messages": [{"role": "system", "content": "Classify the customer-support message into one intent."}, {"role": "user", "content": "My order #4482 still hasn't arrived..."}]}}

Then, in Python:

from openai import OpenAI
import json, time

client = OpenAI(base_url="https://paraloncloud.com/v1", api_key="prlc_...")

f = client.files.create(file=open("tickets.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=f.id, endpoint="/v1/chat/completions",
                              completion_window="24h")

while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(10)
    batch = client.batches.retrieve(batch.id)
    print(batch.status, batch.request_counts.completed, "/", batch.request_counts.total)

for line in client.files.content(batch.output_file_id).text.splitlines():
    result = json.loads(line)
    intent = json.loads(result["response"]["body"]["choices"][0]["message"]["content"])["intent"]
    print(result["custom_id"], intent)

That is the complete program. There is no Paralon SDK and no Paralon-specific field: the same code runs against OpenAI's API if you change the base URL back.

What we measured

All three batches ran on 15 September 2026 from one premium key, with the script linked at the end.

300 ticket classifications. Every line used JSON-schema output with the twelve intents as an enum, so every response is a valid label by construction. The model worked through all 300 requests in 22 seconds from the first line starting to the last one finishing, with no retries and no failed line. 79% of the labels matched the ones the tickets were generated with; the LLM-as-a-judge guide explains why that dataset's own labels are not a perfect answer key.

100 embeddings. One multilingual-e5-small request per ticket, each returning a 384-dimension vector, executed in 3.5 seconds end to end. For indexing a knowledge base or a ticket archive, that is the part you used to write a pipeline for.

20 Responses API drafts. Each line asked the 27B model, through /v1/responses with instructions, for a two-sentence support reply. All 20 came back with a completed response object in the output file, running side by side in 7 seconds for the whole batch, about 2.3 seconds of generation each.

Billing. For each batch we summed what was charged per request and compared it with the same tokens at list price:

batch	input / output tokens	list price	billed
300 classifications	24,868 / 3,829	$0.009493	$0.004747
100 embeddings	6,271 / 0	$0.000125	$0.000063
20 Responses drafts	1,778 / 1,123	$0.002122	$0.001061

Half, to the rounding of the last digit, charged request by request as each one completes, from the same credit balance as live usage.

Live traffic during the batch. We timed 15 live chat requests just before the classification batch and 15 more while it was running:

	median	slowest
live requests before the batch	423 ms	816 ms
live requests during the batch	430 ms	460 ms

The batch filled capacity around the live requests rather than in front of them, which is the design: a batch line that finds every worker busy steps back and tries again moments later, and a live request never waits behind it.

What it is good for

• Labeling and classification at scale. Tickets, reviews, emails, documents, with JSON-schema output so every answer parses. The document classification guide measures how accurate that is on a public benchmark.
• Embedding a corpus. Build or rebuild a vector index overnight at half price, then serve queries live; the semantic search guide covers the retrieval side.
• Synthetic data and evaluations. Generate training examples or grade model outputs by the thousand; see the synthetic dataset guide.
• Anything agents do offline. Summaries, translations, extraction, drafts through the Responses API, queued instead of streamed.

The details that matter in production

• Validation before anything runs. Every line is checked when the batch is created: valid JSON, a unique custom_id, the right endpoint, a model we serve. If anything is wrong, the batch is created as failed with the problems listed by line number, and nothing is charged. You find out in a second, not after half the file has run.
• Output and error files. Successful responses go to the output file, failures to a separate error file with the status code and message, each line tagged with your custom_id.
• Progress you can poll. request_counts.completed and .failed update while the batch runs.
• Cancel any time. POST /v1/batches/{id}/cancel stops the requests that have not started and delivers the results so far.
• Scoped to the key. Files and batches belong to the API key that created them, so two projects on one account never see each other's jobs. Result files are kept for 30 days.
• Limits. Up to 50,000 requests and 50 MB per file, a 24-hour completion window, and one endpoint per batch. Free keys can use it too, drawing on the 250,000-token trial.

The full reference, including the output line format and every status, is in the Batch API documentation.

Try it

• batch_bench.py runs the three batches above with the OpenAI SDK and prints the timings, accuracy and a before-and-during latency comparison. It uses the 990-ticket dataset from the judge guide.

pip install openai
PRL_KEY=prlc_... python3 batch_bench.py

A free key from the Console is enough to run it. After the trial, batch requests are billed at half the list price shown in the Console, from prepaid credits (billing).

Frequently asked questions

Is it compatible with OpenAI's Batch API? Yes: the same /v1/files upload with purpose="batch", the same /v1/batches create, retrieve, list and cancel, the same input and output line formats. Code written for OpenAI's batches runs by changing the base URL and the key.

Which models and endpoints can I batch? Every chat model and embedding model in GET /v1/models, through /v1/chat/completions, /v1/embeddings or /v1/responses, including JSON-schema output, tool definitions and image input.

How fast does a batch finish? Batches are guaranteed to finish within 24 hours; in practice they run as soon as capacity is free, and the ones above finished in seconds. A very large batch at a busy hour takes longer, because it always gives way to live requests.

Do failed requests cost anything? No. Only completed requests are billed, at 50% of the model's price.

Does a batch use up my rate limit? No. Batch requests do not count against your key's per-minute or in-flight limits, so your live application keeps its full allowance while a batch runs.