Document Classification with an Open LLM API: 7,218 Documents, No Training Data, $0.05 per Thousand

The usual way to classify documents is to label a few thousand of them, train a linear model, and ship it. The usual objection is the first step. This guide measures the shortcut: hand an open 27B model the list of categories and the document, constrain its answer to the list, and count. No labels, no training, one HTTP endpoint.

We ran it on 20 Newsgroups, the 1993 Usenet benchmark every text classification paper has reported on for thirty years, because its test split is public, its twenty labels are known, and some of those labels overlap in ways that show exactly where an LLM classifier fails. Three conditions, all on qwen3.8-27b through the OpenAI-compatible endpoint: zero-shot with one document per request, zero-shot with ten documents per request, and few-shot with two examples per class and ten per request.

condition	documents	accuracy	macro-F1	cost per 1,000
zero-shot, 1 per request	992	69.2%	69.4%	$0.072
zero-shot, 10 per request	7,218	72.2%	71.9%	$0.046
few-shot (2 per class), 10 per request	1,940	72.7%	74.1%	$0.090

The short version: batch ten documents per request, skip the examples, and spend the effort on the taxonomy instead, because that is where the 28% went.

The dataset, and what we stripped from it

20 Newsgroups "bydate" has 11,314 training posts and 7,532 test posts across twenty newsgroups. Five of those twenty are the reason it is a hard benchmark rather than an easy one: alt.atheism, soc.religion.christian and talk.religion.misc argue about the same things from different corners, and talk.politics.guns, talk.politics.mideast and talk.politics.misc do the same for politics. A post about gun legislation belongs in guns by the poster's choice of newsgroup, not by anything in its text.

We stripped everything that would let a model cheat: the header block (which contains the newsgroup name), quoted lines, "X writes:" attributions and signatures after --, then kept the first 400 words. This matches the remove=("headers", "footers", "quotes") option in scikit-learn's loader, and it matters: the same library's documentation reports that a naive Bayes classifier trained on the full training split scores 0.88 macro-F1 with headers in and 0.77 with them out. Posts under twenty words after cleaning were dropped, which left 7,218 of the 7,532.

One more thing to say plainly: a model trained on the open web has seen 20 Newsgroups. It is in the training data of every large model. The errors below are not the errors of a model reciting memorized labels, they cluster exactly where the labels are ambiguous, but treat the absolute numbers as a benchmark reading, not a guarantee for your documents.

The request

The whole classifier is one chat completion with the label set enforced by the schema:

from openai import OpenAI
import json

client = OpenAI(base_url="https://paraloncloud.com/v1", api_key=KEY)

CLASSES = ["alt.atheism", "comp.graphics", "comp.os.ms-windows.misc", ...]  # your taxonomy

SYSTEM = ("You classify Usenet posts from 1993 into exactly one of these 20 newsgroups: "
          + ", ".join(CLASSES) + ". Judge by topic, not by tone. Answer with the category id only, via the schema.")

def classify(posts):                      # a list of up to 10 document strings
    schema = {"type": "object",
              "properties": {"categories": {"type": "array", "minItems": len(posts), "maxItems": len(posts),
                                            "items": {"type": "string", "enum": CLASSES}}},
              "required": ["categories"]}
    user = ("Classify each of the following posts, in order. Return one category per post.\n\n"
            + "\n\n".join(f"### Post {i+1}\n{p}" for i, p in enumerate(posts)))
    r = client.chat.completions.create(
        model="qwen3.8-27b",
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}],
        response_format={"type": "json_schema", "json_schema": {"name": "cls", "schema": schema}},
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
        temperature=0, max_tokens=400)
    assert r.choices[0].finish_reason == "stop", "raise max_tokens"
    return json.loads(r.choices[0].message.content)["categories"]

Three decisions in there carry the result:

• The label set lives in the schema, not only in the prompt. With enum on the item type, the server's guided decoding cannot produce a label that is not in your list, cannot produce two, and cannot add "I think this is probably". Of 7,218 documents in the main run, 7,218 came back with a valid label. There is no parsing step and no "unknown" bucket to clean up later. This is the same structured output mechanism the synthetic data guide used for generation; here it is doing the opposite job.
• Ten documents per request. The system prompt with twenty class names is about 150 tokens. Sent once per document that is 40% of the input; sent once per ten it is 4%. Input tokens per document fell from 399 to 253 and cost per thousand from $0.072 to $0.046. The array has minItems and maxItems equal to the batch size so the model cannot skip one.
• Thinking off. Qwen 3 reasons before answering unless told not to. For a label choice that is a few hundred tokens of deliberation per document, billed at output prices. enable_thinking: false in chat_template_kwargs switches it off at the template level.

What we measured

Full test split, zero-shot, ten per request, 14 requests in flight, 13 September 2026:

documents scored	7,218 of 7,218 (722 requests, 0 failures)
accuracy	72.2%
macro-F1	71.9%
input tokens per document	253
output tokens per document	9.2
latency per request (10 documents), median / p90	6.8 s / 8.6 s
throughput at 14 in flight	70,494 documents per hour
cost at list price	$0.046 per 1,000 documents ($0.33 for the run)

Per-class recall tells the real story. Seventeen classes sit between 59% and 94%. Three do not:

class	recall
rec.sport.baseball	94.3%
rec.sport.hockey	92.2%
misc.forsale	91.7%
comp.windows.x	84.9%
rec.motorcycles	82.1%
rec.autos	81.8%
sci.med	81.2%
comp.sys.ibm.pc.hardware	80.2%
comp.graphics	80.1%
talk.politics.mideast	79.4%
sci.space	78.2%
talk.politics.misc	76.3%
comp.os.ms-windows.misc	75.6%
comp.sys.mac.hardware	74.6%
sci.crypt	66.8%
talk.religion.misc	63.5%
sci.electronics	59.4%
talk.politics.guns	39.5%
soc.religion.christian	34.2%
alt.atheism	14.4%

And where the misses went:

confusion	count
soc.religion.christian → talk.religion.misc	239
alt.atheism → talk.religion.misc	182
talk.politics.guns → talk.politics.misc	166
sci.crypt → talk.politics.misc	84
comp.sys.mac.hardware → comp.sys.ibm.pc.hardware	58
talk.politics.mideast → talk.politics.misc	55
sci.electronics → comp.sys.ibm.pc.hardware	41
sci.space → talk.politics.misc	41

The model has a consistent theory of what talk.religion.misc and talk.politics.misc mean: religion, and politics. It is not wrong. A Christian's post about atonement and an atheist's reply to it are both "religion", and the newsgroup that actually carried them is a fact about 1993, not about the text. The 239 Christian posts filed under religion.misc are the single largest error in the run, and no prompt wording changes the fact that the category names alone do not say which of the three religion groups a given argument belongs to. Counting only the four within-group confusions in the table, which are a third of all the errors in the run, a taxonomy that merged the three religion groups into one and the three politics groups into one would score at least 81% on the remaining sixteen classes, without a single change to the model or the request.

sci.crypt → talk.politics.misc is the same effect on a smaller scale. Half of sci.crypt in 1993 is the Clipper chip debate, which is a political argument about cryptography, and the model files it as politics.

Few-shot examples: two points of F1 for twice the price

The third condition added two training-split examples per class, sixty words each, to the system prompt: about 3,700 tokens, sent with every batch of ten.

	zero-shot, 10 per request	few-shot, 10 per request
accuracy	72.2%	72.7%
macro-F1	71.9%	74.1%
alt.atheism recall	14.4%	22.7%
soc.religion.christian recall	34.2%	50.0%
talk.religion.misc recall	63.5%	53.1%
input tokens per document	253	628
latency per request, median	6.8 s	15.0 s
cost per 1,000 documents	$0.046	$0.090

The examples helped the two classes with the worst recall by showing the model what a soc.religion.christian post looks like, and paid for it in talk.religion.misc, which lost ten points as the boundary moved. Net, macro-F1 rose by 2.2 points and accuracy by 0.5, on a different sample of 1,940 documents, at double the input tokens and double the latency. Two examples per class is not the only few-shot design, and more examples would cost proportionally more, but for a taxonomy whose problem is overlapping definitions, examples are the wrong tool. Definitions are. Writing a one-line definition per class into the prompt ("talk.religion.misc: religious discussion that is not specifically Christian or atheist") is the experiment we did not run here, and it is the first one to run on your own taxonomy; it costs a few hundred tokens per request rather than thousands.

Throughput: what actually sets the ceiling

Three conditions, three different bottlenecks:

• One document per request ran at 992 documents in 213 seconds, 16,764 an hour. Each request took 0.64 s, so the 14 concurrent slots were mostly idle; the limit was the key's requests-per-minute rate, which the script paces itself to.
• Ten per request ran 7,218 documents in 369 seconds, 70,494 an hour. Requests took 6.8 s, so at 14 in flight the ceiling was concurrency, and the rate limit was nowhere near. Same key, 4.2× the throughput, 36% cheaper.
• Few-shot, ten per request: 15 s per request with the same concurrency gives 30,800 an hour. The examples are input tokens the worker has to prefill on every request, and prefill is what the extra seconds are.

For a backlog, then: batch as many documents per request as fit comfortably below the context limit, keep the prompt short, and the per-minute limit stops mattering. A million documents at the zero-shot-ten rate is about 14 hours on one premium key and $46 at list price.

Two failures worth knowing about

Zero requests failed in the two zero-shot runs. In the few-shot run, two of 194 requests came back as invalid JSON, and twenty documents went unscored. The cause was ours: max_tokens was set to 160 for a ten-label answer, the model padded the JSON with newlines on those two responses, and the cap cut it off mid-structure. Guided decoding guarantees the tokens it emits fit the schema; it does not guarantee the schema is complete when your token budget runs out. The published script uses 400 and checks finish_reason.

The other one was self-inflicted too: an early attempt launched the batch runner above the key's rate limit and collected a page of 429 responses. The error body says the limit; the script now paces to it. A free key is 20 requests a minute, which at ten per request is 12,000 documents an hour, and its 250,000-token trial covers about a thousand documents of this size.

What to check before you trust it

1. Look at the confusion table, not the accuracy. On this dataset 28% of documents were misfiled, and at least a third of those went into a sibling class a human would have argued about. If your taxonomy has classes whose names do not settle the boundary, fix the taxonomy or define the classes in the prompt before touching the model.
2. Score against a held-out set you labeled yourself. Two hundred documents is enough to see the shape. The script prints per-class recall and the top confusions; that is the report to read.
3. Batch, but keep the batches under the model's attention. Ten 400-word posts is about 2,500 tokens of documents per request. We did not measure whether fifty per request holds accuracy; measure it if you go there.
4. Temperature 0. Classification is not a place for sampling, and it makes reruns reproducible enough to compare prompts.
5. A trained classifier is still better on a fixed taxonomy. Naive Bayes with 11,314 labeled examples scores 0.77 macro-F1 on this cleaned task; zero-shot here is 0.72. When you have the labels and the taxonomy is stable, train. When you have neither, or the categories change every month, an enum in a schema is a classifier you can change by editing a list.

The script

One Python file, requests only: downloads nothing (point it at the extracted 20 Newsgroups folder or your own label/file.txt tree), cleans, batches, paces to the rate limit, scores, and writes a JSON report with accuracy, macro-F1, per-class recall, confusions, tokens and cost.

• classify_20news.py

curl -O http://qwone.com/~jason/20Newsgroups/20news-bydate.tar.gz && tar xzf 20news-bydate.tar.gz
pip install requests
PRL_KEY=prlc_... MODE=zero10 N=7532 RPM=20 IN_FLIGHT=2 python3 classify_20news.py     # free key
PRL_KEY=prlc_... MODE=zero10 N=7532 RPM=280 IN_FLIGHT=14 python3 classify_20news.py  # premium key

MODE is zero1, zero10 or few10; N caps the number of documents. Point ROOT and TRAIN at any folder-per-class tree and the same report prints for your documents. The generation counterpart is in the synthetic dataset guide, and the same schema trick on images is in the invoice extraction guide. A free key from the Console carries the 250,000-token trial; after that it is $0.12 per million input tokens and $1.70 per million output on a premium key (billing).

Frequently asked questions

Does batching ten documents per request lower accuracy? Not in this run: the ten-per-request condition scored higher than one-per-request (72.2% against 69.2%), on a larger sample. The two samples are different documents, so read that as "no measurable penalty", not as a gain from batching.

Why not ask for a confidence score? You can add a confidence field with an enum of low, medium, high to the schema and route the lows to a person. We did not measure how well the model's self-reported confidence tracks its errors, so measure it before you route on it.

Can it do multi-label? Change the item type from a single enum to an array of enums with minItems: 1 and the schema handles it. Scoring becomes per-label precision and recall rather than accuracy.

Which model? qwen3.8-27b, the same model as the other guides in this series; /v1/models lists what is live. For a taxonomy this size a smaller model is worth testing at a fraction of the price, and the script takes MODEL from the environment.

Where do the labels come from at runtime? From your list. The schema is built from CLASSES on every request, so adding a category is a one-line change with no retraining, which is the whole argument for doing it this way.