A RAG answer is only as good as the sentence you did not check. Retrieval finds passages, the model writes four sentences, and one of them says "within 24 hours" when the passage said "within the monthly window". Nobody reads the passage again. This guide adds the step that does, with the same measured-on-planted-errors method as our judge guide: every sentence of the answer is checked against the passages it came from by an NLI classifier that returns entailed / contradicted / neutral with a probability, in one request per answer, at a few milliseconds per sentence.
The whole pipeline runs on one prlc_ key: /v1/embeddings for retrieval, /v1/chat/completions for the answer, /v1/classify for the check. The corpus is our own documentation, 75 chunks, so you can reproduce it on any folder of markdown.
The pipeline
- Chunk. Split the docs on headings, keep chunks of 60 to 450 words, drop code blocks. 75 chunks.
- Embed.
multilingual-e5-small,passage:prefix for chunks,query:for questions. - Questions. Two per chunk from
qwen3.8-27b("write two short questions a user would ask that this passage answers"), 148 in total, each with a known source chunk. That gives a free retrieval check: top-1 hit the source chunk 107 times out of 148 and top-3 hit it 136 times; the 12 misses were answered from whatever three chunks came back, which is fine for this exercise and would be a finding on a real corpus. - Answer. Top-3 chunks by cosine,
qwen3.8-27b, "answer in 3 to 5 plain sentences using only the passages",temperature: 0. 463 sentences over 148 answers. - Plant. In every second answer, corrupt one sentence, three ways, chosen at random: multiply a number by ten ("250,000 tokens" becomes "2500000 tokens"); insert a negation after the first verb ("is" becomes "is not"); or replace the sentence with a plausible claim that is nowhere in the docs ("Refunds are processed automatically within 24 hours"). 74 planted sentences: 44 replaced, 20 negated, 10 numbers.
- Check. For each sentence, three pairs: premise = each retrieved passage, hypothesis = the sentence. Support is the highest entailment probability across the three; the verdict is the class of the passage that supports it most. One
/v1/classifyrequest per answer, up to 256 pairs.
Step 6, the part this guide is about:
pairs = [{"premise": chunks[ci], "hypothesis": sent}
for sent in sentences(answer) for ci in top3]
r = requests.post("https://paraloncloud.com/v1/classify",
headers={"Authorization": f"Bearer {KEY}"},
json={"model": "openjev-4b", "pairs": pairs}).json()
# probs are [contradiction, entailment, neutral]; group by sentence, keep the best passage
support = {}
for (sent, ci), item in zip(((s, c) for s in sentences(answer) for c in top3), r["data"]):
support[sent] = max(support.get(sent, 0.0), item["probs"][1])
flagged = [s for s, p in support.items() if p < 0.7]
That is the entire check. No rubric, no JSON schema, no parsing of a reason.
Results
463 sentences, 74 planted, 21 September 2026, through the public API:
| accept a sentence if support ≥ | planted caught | clean sentences flagged |
|---|---|---|
| 0.3 | 82.4% (61/74) | 0.8% (3/389) |
| 0.5 | 87.8% (65/74) | 2.1% (8/389) |
| 0.7 | 93.2% (69/74) | 4.6% (18/389) |
| verdict class ≠ entailment | 87.8% | 1.5% |
The /v1/classify part: 1,389 pairs, 251k input tokens, 49 seconds, $0.0125. The 27B answering the 148 questions cost more than checking them.
The same 463 sentences through qwen3.8-27b as a judge, one request per sentence, "supported or unsupported", temperature: 0:
| OpenJev, threshold 0.7 | 27B as judge | |
|---|---|---|
| planted caught | 69 / 74 | 69 / 74 |
| clean flagged | 18 | 0 |
| time | 49 s (one 5060 Ti, sequential) | 141 s (8 in flight) |
| cost | $0.0125 | $0.031 |
Same recall, to the sentence. The judge's zero false alarms are real and worth the money if a false alarm costs you a human review; the classifier's 18 are the price of a threshold, and you can move it.
What each judge misses, by error type
| planted error | count | OpenJev caught (0.7) | 27B caught |
|---|---|---|---|
| replaced with an unsupported claim | 44 | 44 | 44 |
| negation inserted | 20 | 17 | 17 |
| number ×10 | 10 | 8 | 8 |
Identical per type, which says the misses are in the data, not the judge. The two numbers both missed: one is not an error at all (the planted sentence set max_active_rentals to 0 and ten times zero is still zero, a bug in our planting that we left in the count so the table is honest); the other is "2500000 tokens" for a 250,000-token trial, which both models waved through. A multiplied number inside an otherwise faithful sentence is the hardest case for any reader that is not doing arithmetic; if your domain is numbers, extract them and compare, do not classify.
The three negations that got through are of the kind "model availability depends on which provider nodes are not currently online", grammatical, plausible, and wrong. NLI models are known to be weaker on negation than on substitution, and this is what that looks like: 85% instead of 100%.
The replaced claims, the ones a model invents when it runs out of passage, were caught every time by both. That is the common case in production.
What the false alarms look like
The 18 clean sentences OpenJev flagged at 0.7 are paraphrases the passage supports loosely rather than states:
"This process is handled entirely on the Payouts page." (0.42, entailment)
"The key simply becomes unable to start any new rentals until the limit is adjusted." (0.63, entailment)
Note the class: the verdict is still entailment, the probability is just not high, because the passage says it in other words. That is why the "verdict class" row above has fewer false alarms (6) than the 0.7 threshold (18) at the cost of 4 more misses. Two dials, one number, pick the trade you want: a support chatbot flags at 0.5 and lets a human read six sentences a day; a medical assistant flags at 0.7 and lives with rewording.
Where to put it in a real system
- Before the answer is shown. One request per answer adds 50 to 200 ms. Below the threshold, either strip the sentence, regenerate with "use only the passages" and the flagged sentence quoted, or show the sentence with a marker.
- On a sample, as a metric. Run it on 1% of production answers and chart the flagged rate per day. It moves when retrieval breaks, when a model is swapped, or when the docs change under the index.
- On the corpus, once. Ask the model questions from every chunk as this script does and check the answers; the flagged sentences point at chunks that are ambiguous or contradict each other.
Limits, measured or known
- 463 sentences is enough to see the shape, not to quote a decimal. Rerun on your corpus; the script prints the tables.
- English. The classifier's NLI training is English; the launch article has the cross-lingual miss.
- Numbers and negation are the weak spots, above. Extract numbers; treat negation misses as a known 15%.
- Sentences are split on punctuation. A model that writes one 80-word sentence gets one verdict for four claims. Ask for short sentences, as the prompt above does.
- 4,096 tokens per premise. Long passages need chunking anyway.
Reproduce it
rag_check.py does all six steps against a folder of markdown, plants the errors with a fixed seed, and prints the threshold table, the per-type table, and, with JUDGE=1, the 27B comparison. It needs a key with credits for the 27B answers; the classify and embeddings parts fit a free key's trial.
PRL_KEY=prlc_... DOCS=./docs N_Q=148 python3 rag_check.py
For the classifier's own numbers, calibration and how it compares with the 27B on a labelling task, see OpenJev vs an LLM judge on 990 planted errors.
