OpenJev vs an LLM Judge on 990 Planted Errors: Recall, Calibration, Cost

Two judges, one dataset, exact answers. Eight days ago we measured an open 27B model as an LLM judge on 990 customer-support tickets where we had deliberately corrupted half the labels, so recall and false-alarm rate were facts rather than estimates. Yesterday OpenJev went live on /v1/classify, a Qwen3.5-4B natural language inference cross-encoder that answers with a label and probabilities instead of text. Same tickets, same planted errors, same seed, second judge.

The short version: how you ask matters more than which model you ask. OpenJev given one pair at a time is a mediocre judge. OpenJev asked to pick the best of twelve labels is within 1.4 points of the 27B on catching errors, at 12 pairs per verdict and 3 ms per pair, and its probabilities mean what they say.

The setup

• Data. 990 tickets from the synthetic dataset guide, twelve intents, one label each. Rows at even positions keep their label (495 correct); rows at odd positions get a random wrong intent from the same taxonomy (495 planted errors). Seed 21, the same as the judge guide, so the planted set is identical.
• The 27B judge. Numbers from the guide's rubric run on 13 September: qwen3.8-27b, a one-line definition per intent, "correct only if it is the single best fit", JSON verdict, temperature: 0.
• OpenJev. openjev-4b over /v1/classify. The rubric becomes twelve hypotheses, one per intent, from the same definitions: premise {ticket}, hypothesis The customer asks where an order is or when it arrives, and so on. Two ways to turn that into a verdict:
  • single pair: score only the proposed label; correct if the argmax class is entailment;
  • rerank: score all twelve; correct if the proposed label has the highest entailment probability.

11,880 pairs, through the public API in 93 requests of 128 pairs, 2.2 s each. The worker that day was a single RTX 5060 Ti on a provider's Windows machine, the slowest card in the pool, which is worth knowing when you read the timing row.

Results

	27B judge, with rubric	OpenJev, single pair	OpenJev, rerank 12
planted errors caught	99.6% (491/493)	82.6% (409/495)	98.2% (486/495)
correct labels accepted	88.3%	89.3%	80.2%
accuracy against the dataset	93.9%	86.0%	89.2%
top-1 intent, all twelve	—	—	80.3%
time per verdict	980 ms	17 ms	207 ms (12 pairs)
tokens per verdict	290 in / 52 out	75 in / 0 out	903 in / 0 out
list price per 1,000 verdicts	$0.123	$0.004	$0.045

Two things jump out, and the second explains the first.

Single-pair OpenJev is not a good judge. 82.6% of planted errors caught, 86 accepted that should have been rejected. The misses cluster on vague hypotheses: seven cancel_order tickets labelled product_question, five refund_status as product_question, five order_status as product_question. "Asks about a product before or after buying, not a problem with it" is a definition written for a reader with the other eleven in front of them. Alone, a ticket about an order fits it well enough to entail.

Rerank fixes that. Given all twelve at once, the right intent usually scores higher than the wrong one, and the same tickets are caught. 98.2% recall, nine planted errors through. The nine are neighbours (complaint labelled return_request, complaint labelled damaged_item), the same boundary the 27B missed both of its errors on.

The cost of rerank is on the second row: OpenJev rejects 98 correct labels, the 27B rejected 58. Read on before you call that a loss.

The judges reject the same rows

Of the correct labels rejected by both the 27B (rubric) and OpenJev (rerank), 38 are common to both, and 33 of the 38 are complaint tickets. The judge guide already found that: 43 of its 58 rejections were tickets the generator had labelled complaint that describe a broken zipper or a flickering lamp, which the rubric files under damaged_item. OpenJev makes the same call without being told:

"This jacket is garbage. The zipper broke after one wear and the lining is peeling off." → damaged_item 0.42, complaint 0.15

"…the zipper is already stuck and the lining is peeling off this is ridiculous I need a fix NOW" → damaged_item 0.93, complaint 0.48

Top-1 accuracy for the complaint intent is 6%. Every other intent is between 57% and 100%. That is the taxonomy's problem, not the model's: an abstract intent next to eleven concrete ones loses on every ticket that names a concrete thing. Both judges are telling us to fix the generator prompt, one sentence, and 40 rows come back.

The 42 rejections that are OpenJev's alone are a different story, and a useful one: 12 discount_code tickets it reranked as product_question, 11 refund_status it reranked as order_status. Those are two hypotheses that need a better sentence. The hypothesis is the rubric; you write it, you can fix it. We left them as written so the numbers are the first attempt, not the tuned one.

Calibration: 0.9 means 90%

The question on Hacker News the week Jev launched was whether a classifier's probabilities are probabilities at all. On the proposed pair's entailment probability, over all 990 verdicts:

entailment probability	verdicts	actually correct
0.0–0.1	339	4.7%
0.1–0.2	51	21.6%
0.2–0.3	30	26.7%
0.3–0.4	29	41.4%
0.4–0.5	55	52.7%
0.5–0.6	68	52.9%
0.6–0.7	46	73.9%
0.7–0.8	45	75.6%
0.8–0.9	67	94.0%
0.9–1.0	260	96.9%

Expected calibration error 0.028. Sixty percent of verdicts land in the two outer bins, where the model is right 95% and wrong 95% of the time respectively. The middle is thin and honest: at 0.5 it is a coin flip and it says so. That is what lets you set a threshold and know what it buys:

single pair, accept if entailment ≥	planted errors caught	correct labels accepted
0.3	77.8%	92.9%
0.5	86.5%	84.6%
0.7	95.4%	70.5%
0.9	98.4%	50.9%

And for rerank, accept if the proposed label is within a margin of the top score:

margin	planted errors caught	correct labels accepted
0.00 (strict top-1)	98.2%	80.4%
0.10	96.4%	85.7%
0.20	93.3%	89.5%

Pick the row that matches what a miss costs you. A dataset QA pipeline wants the first; a router that must not bounce customers wants the last. The 27B judge offers no such dial: it returns a word.

Time and money

	27B judge	OpenJev rerank (12 pairs)
verdicts per hour	14,320 (14 in flight)	17,400 (one 5060 Ti, sequential batches)
tokens per verdict	290 in / 52 out	903 in / 0 out
cost, 990 verdicts	$0.122	$0.045

The OpenJev number is the slow-card, one-request-at-a-time number: 93 batches of 128 pairs, 2.2 s each, through the tunnel to a provider's 5060 Ti in Canada. On the 4090 that served the model the day before, the same batch ran in about 0.8 s, and nothing stops a client from keeping several batches in flight. The point is not the exact ratio. It is that a verdict costs twelve short prefills and no generation, so the ceiling is the card, not the token budget.

What to take from it

1. Ask "which", not "is this right". A judge that sees only the proposed label misses what the other labels would have told it. Rerank across the taxonomy, or at least across the neighbours.
2. The hypothesis is the rubric. Two badly worded definitions cost 23 false rejects. Write them the way you would explain the category to a person who has not seen the others.
3. Trust the probabilities, after you check them once. ECE 0.028 here; measure it on 200 rows of your own before you set a threshold in production. The script prints the table.
4. Keep the LLM judge for the reason. The 27B gives a one-line explanation per verdict, and that is what made the 58-row review queue a ten-minute job in the guide. OpenJev gives you a number. For quality control at scale, run OpenJev first and send the disagreements to the 27B.

Reproduce it

The dataset and the 27B judge script are from the judge guide. The OpenJev script plants the same errors with seed 21, sends 11,880 pairs to /v1/classify in batches, and prints both verdict rules, the miss table and the calibration bins. It takes a JSONL of {customer_message, intent} rows; edit DEFS for your taxonomy. A free key's trial covers the run.