LLM-as-a-Judge with an Open Model: 990 Planted Errors, 99.6% Caught, and What the Rubric Changes

Using a model to grade the work of a model is now standard practice, and the usual complaint about it is fair: nobody knows how good the judge is, because grading the judge needs the same labels you were trying to avoid producing. There is a cheap way around that. Take data whose labels you trust, corrupt half of them on purpose, and ask the judge which are wrong. You know the answer for every row, so the judge's recall and false-alarm rate are exact, not estimated.

This guide does that with an open 27B model over an OpenAI-compatible endpoint, on the 990 customer-support tickets the synthetic dataset guide generated: twelve intents, one label each. Half the tickets keep their label; the other half get a deliberately wrong intent from the same taxonomy. The judge sees a ticket and a proposed label and answers correct or incorrect with a one-line reason. We ran it twice, once with a written rubric, a one-line definition per intent plus the rule "correct only if it is the single best fit", and once with the task alone.

	with rubric	without rubric
planted wrong labels caught	491 of 493 (99.6%)	475 of 495 (96.0%)
dataset labels accepted	436 of 494 (88.3%)	478 of 494 (96.8%)
"accuracy" against the dataset	93.9%	96.4%
input / output tokens per verdict	290 / 52	114 / 50
latency per verdict, median	0.98 s	0.84 s
cost per 1,000 verdicts	$0.12	$0.10

Read the first row if you are building quality control, and the second row if you are wondering why the rubric "lowered accuracy". Then read the next section, because the 58 labels the rubric judge rejected are the most useful output of the whole run.

The 58 rejections were mostly right

A judge that rejects labels the dataset calls correct has either made a mistake or found one. We went through the rejections. Forty-three of the 58 are tickets the dataset labels complaint, and they read like this:

"This jacket is garbage. The zipper broke after one wear and the lining is peeling off. Complete waste of $140."

Judge: incorrect. "The message describes specific physical defects (broken zipper, peeling lining). This fits the definition of damaged_item."

"I just got my laptop bag from order #4482 and the zipper is already stuck and the lining is peeling off this is ridiculous I need a fix NOW"

Judge: incorrect. "The message describes a specific product defect upon arrival. While the tone is angry, the primary intent is to report a damaged item."

The rubric defines complaint as "dissatisfaction with service or the company as a whole" and damaged_item as "item arrived broken or defective". By that rubric the judge is right, 43 times. What it found is a systematic fault in the generator: asked in the first guide to write a "complaint", the model reached for a broken product almost every time, because that is what an angry customer complains about. The label was the generator's instruction, not a description of the text it produced. The judge with definitions in hand caught the disagreement; the judge without them accepted 41 of those 43, because without a definition, "complaint" fits an angry message about a zipper perfectly well.

The other fifteen rejections are seven return_request tickets that describe a defect ("arrived with a double-clicking left button on day two, want a replacement", which the rubric also files under damaged_item), seven compound tickets with two requests in them ("I can't log in, and I need to change the shipping address on my order", labelled account_login), where the rubric's single-best-fit rule makes the judge reject a label that covers half the message, and one order mix-up the judge reads as wrong_item. None of them is a clear judge error. The compound ones are a hole in the rubric, which says nothing about tickets with two intents, and that is a finding too.

So the number to report from this run is not 93.9%. It is: 99.6% of planted errors caught, and a review queue of 58 rows, 12% of the trusted half, of which one systematic problem accounts for three quarters. That queue is worth more than the accuracy figure, because fixing one sentence in the generator prompt, or one line in the taxonomy, clears 43 rows at once.

Without the rubric, the judge agrees with the dataset and misses errors

The no-rubric judge accepted 96.8% of the dataset's labels and looks better on the accuracy line. Its cost is in the first row: 20 planted errors got through, and they are the plausible ones.

planted wrong label it accepted	count
a damaged_item ticket labelled return_request	4
a damaged_item ticket labelled complaint	3
a wrong_item ticket labelled complaint	3
a complaint ticket labelled return_request	2
a complaint ticket labelled damaged_item	2

A cracked phone case with "need a replacement or refund" labelled return_request: without a definition that says returns are for items you want to send back and damage is its own category, the judge accepts it. With the rubric, the same judge rejected 491 of 493 planted errors and the two it missed are both complaint tickets labelled damaged_item, which is the same boundary the dataset itself gets wrong.

Its sixteen rejections of correct labels are a different kind of disagreement: compound tickets ("I can't log in, and also change my shipping address", labelled account_login) and exchange-versus-return ("swap for size L", labelled return_request). Reasonable objections, and a different notion of "correct" from the rubric judge's.

That is the finding. The rubric is not a prompt-engineering tweak that moves a score up or down; it is the specification of what the judge is checking. Without one, the judge checks whether the label is defensible. With one, it checks whether the label is the one your taxonomy says. Only the second is quality control.

The request

from openai import OpenAI
import json

client = OpenAI(base_url="https://paraloncloud.com/v1", api_key=KEY)

DEFS = {
  "damaged_item": "item arrived broken or defective",
  "return_request": "wants to send an item back",
  "complaint": "expresses dissatisfaction with service or the company as a whole",
  # ... one line per label in your taxonomy
}
SYSTEM = ("You are a strict quality reviewer for a customer-support dataset. "
          "Decide whether the proposed intent label is the correct one for the message. "
          "Definitions: " + "; ".join(f"{k}: {v}" for k, v in DEFS.items()) + ". "
          "A label is correct only if it is the single best fit; if another label fits clearly better, it is incorrect. "
          "Answer via the schema.")
SCHEMA = {"type": "object",
          "properties": {"verdict": {"type": "string", "enum": ["correct", "incorrect"]},
                         "reason": {"type": "string", "maxLength": 160}},
          "required": ["verdict", "reason"]}

def judge(message, label):
    r = client.chat.completions.create(
        model="qwen3.8-27b",
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": f"Message: {message}\nProposed intent: {label}"}],
        response_format={"type": "json_schema", "json_schema": {"name": "verdict", "schema": SCHEMA}},
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
        temperature=0, max_tokens=120)
    return json.loads(r.choices[0].message.content)

Three details that matter:

• The verdict is an enum in the schema. Guided decoding means the answer is correct or incorrect, never "mostly correct" or a paragraph that starts with "It depends". 1,976 verdicts, 1,976 parsed. Same mechanism as the classification guide, where the enum is the label set.
• The reason is capped at 160 characters. Long enough to be reviewable in a spreadsheet, short enough that it is 50 output tokens and not 300. The reason is 85% of the cost of a verdict ($0.085 of the $0.10 per thousand is output tokens); drop it and a thousand verdicts cost about three cents, but the review queue becomes a list of row numbers.
• Temperature 0, thinking off. A judge should give the same verdict twice; and on a yes/no with a rubric, the model's extended reasoning costs several hundred tokens per verdict for a decision that the definitions already settle.

Calibrate by planting errors

The method, in full, because it is the transferable part:

1. Take a slice of data whose labels you trust: a few hundred rows labelled by a person, or, as here, a generated set where the label was the instruction.
2. For a random half, replace the label with a wrong one drawn from the same taxonomy. Random wrong labels are the easy case; if your taxonomy has neighbours, also plant the neighbour (label a damaged_item as return_request) so recall is measured where it will be lowest.
3. Run the judge over all of it. Recall on the planted half is the judge's ability to catch errors. Rejections on the untouched half are either judge errors or label errors, and you have to look at them to know which; the reason field is what makes that a ten-minute job.
4. Report the recall and the disagreement rate together, and keep the planted set. When you change the rubric or the model, rerun it. In this run the rubric moved recall from 96.0% to 99.6%; the next change to it should be measured the same way.

The script that does all four steps is linked below. It takes a JSONL of {message, label} rows, plants the errors with a fixed seed, and prints recall, acceptance, the missed-error table and a sample of the rejections with reasons.

Throughput and cost

990 verdicts, 14 requests in flight, 13 September 2026:

	with rubric	without
requests failed (connection errors, not retried)	3	1
latency, median / p90	0.98 s / 1.12 s	0.84 s / 0.94 s
verdicts per hour	14,320	14,360
cost per 1,000 verdicts at list price	$0.123	$0.099

A verdict is a short request, so the limit is requests per minute, not tokens: the same 14 in flight that moved 70,000 documents an hour in the classification guide moves 14,000 verdicts here, because there is one verdict per request. Packing ten (message, label) pairs into one request with an array schema, the way the classification guide batches documents, would raise that several times over; we did not measure whether the verdicts stay as good when the judge sees ten at once, so measure it before you do it. A 100,000-row dataset at the single-verdict rate is about seven hours on one premium key and $12.

What to check before you trust it

1. Same model as generator. Here the judge is the same model that wrote the tickets, and it still rejected 58 of its own labels, so for label verification against a rubric the self-agreement worry did not show up. For grading free text on quality, helpfulness or style, a model's preference for its own phrasing is documented and real; use a different model as judge, or calibrate on a human-graded sample first.
2. Planted errors are easier than real ones. A random wrong label is usually far from the text. Real label noise is the neighbour case, and every one of the 20 errors the no-rubric judge missed was a neighbour. Plant neighbours.
3. The rubric is the product. Every line in DEFS is a decision about your taxonomy that the judge will enforce a thousand times an hour. The 43 complaint rejections came from one definition. Write the definitions with whoever owns the taxonomy, not in the judge script.
4. Route rejections to a person, not to an auto-relabel. The judge says which label it would prefer in the reason, and it was right about the zippers, but "the judge disagreed" is a queue, not a correction.

The script and the data

• judge_labels.py: loads the JSONL, plants wrong labels on every other row with a fixed seed, runs the judge with or without the rubric (RUBRIC=1 or 0), paces to the key's rate limit, and writes a results file with the summary and every verdict and reason.
• support_tickets_990.jsonl: the 990 generated tickets from the synthetic data guide, with intent, sentiment and a suggested reply per row. Synthetic throughout; the e-mail addresses in the messages are on example.com.

pip install requests
curl -O https://paraloncloud.com/files/guides/llm-as-a-judge-open-model-api/support_tickets_990.jsonl
PRL_KEY=prlc_... RUBRIC=1 RPM=20 IN_FLIGHT=2 python3 judge_labels.py     # free key
PRL_KEY=prlc_... RUBRIC=1 RPM=240 IN_FLIGHT=14 python3 judge_labels.py   # premium key

Edit INTENTS and DEFS for your own taxonomy and point DATA at your rows. The request format is standard chat completions with response_format; a free key from the Console carries a 250,000-token trial, which is about 700 rubric verdicts, and after that the per-token price on a premium key.

Frequently asked questions

Can the judge grade free-text answers, not just labels? Yes, with a different schema: a score enum (1 to 5) or a pairwise A/B, and a rubric that says what each score means. The calibration method is the same: plant answers you know are bad (truncate them, swap in the answer to a different question) and measure whether the judge catches them. Pairwise comparison with position swapping is more reliable than absolute scores for most open models; measure yours.

Why not just ask the classifier for its confidence? Because the classifier and the judge fail differently. The classifier chose complaint because the generator did; the judge with definitions disagreed. A second opinion with the rubric in front of it is worth more than the first opinion's self-assessment.

Should the judge be a bigger model? For rubric-driven label verification, the 27B model at 99.6% recall left little room. For subtle quality grading, a larger judge helps more, and costs more per verdict; the planted-error method tells you whether the upgrade bought anything.

Do I need the reason field in production? Keep it while the review queue is being read by people, drop it once the disagreement rate is low and stable and you only need counts. It is most of the cost.