Tag

Llm-as-a-judge

2 articles tagged with “Llm-as-a-judge”

The Paralon capybara measuring three holographic bar columns with an emerald caliper, red cubes on the bench as the planted errors

Sep 22, 20268 min read

OpenJev vs an LLM Judge on 990 Planted Errors: Recall, Calibration, Cost

The same 990 support tickets with 495 deliberately wrong labels, judged two ways: OpenJev over /v1/classify and Qwen 3.8 27B as an LLM judge. Asked one pair at a time OpenJev catches 82.6% of the planted errors; asked to pick among all twelve intents it catches 98.2%, rejects the same 33 mislabelled 'complaint' rows the 27B judge rejected, and its probabilities are calibrated to within 3 points. Every number is exact because we planted the errors.

Read

The Paralon capybara seated as a judge with a glowing gavel, weighing two floating sheets on a holographic balance

Guides & Tutorials

Sep 16, 202613 min read

LLM-as-a-Judge with an Open Model: 990 Planted Errors, 99.6% Caught, and What the Rubric Changes

We used Qwen 3.8 27B over an OpenAI-compatible API as a judge of dataset labels, and scored it against errors we planted ourselves so every number is exact. With a written rubric the judge caught 491 of 493 wrong labels and also rejected 58 labels the dataset called correct, 43 of them the same systematic mistake in the generator. Without the rubric it agreed with the dataset more and missed 20 planted errors. The rubric is not a tuning knob; it decides what the judge measures.

Read