Benchmarks

Oct 12, 20267 min read

Two RTX 5090s on a 27B Model: 150 tok/s Alone, 1,300 at 32, Next to Four Other Cards

149–156 tok/s

one request, two cards

1,318 tok/s

in total at 32 concurrent requests

Oct 12, 20268 min read

Qwen3.8-27B on an H200: 3,300 tok/s at 128 Requests, After One Flag Stopped It Starting

163–176 tok/s

one request

3,342 tok/s

in total at 128 concurrent requests

Oct 11, 20267 min read

Our Benchmark Read an H200 at 48% of Spec. The Benchmark Was Wrong.

An NVIDIA H200 joined the network and our verification put its memory bandwidth at 2,319 GB/s, under half of the 4,800 on the spec sheet. The rent page warned renters about the card. We rented it ourselves, measured it another way, and found 4,110 GB/s. Here is how a bandwidth test that was right on every consumer card went wrong on a fast one, and the fix.

Sep 28, 20268 min read

RTX PRO 6000 vs Two RTX 4090s on a 27B Model: Equal on One Request, 3x Apart at 32

106 vs 94

tok/s on one request, RTX PRO 6000 vs two RTX 4090s

1,342 vs 435

tok/s in total at 32 concurrent requests

Sep 22, 20268 min read

OpenJev vs an LLM Judge on 990 Planted Errors: Recall, Calibration, Cost

The same 990 support tickets with 495 deliberately wrong labels, judged two ways: OpenJev over /v1/classify and Qwen 3.8 27B as an LLM judge. Asked one pair at a time OpenJev catches 82.6% of the planted errors; asked to pick among all twelve intents it catches 98.2%, rejects the same 33 mislabelled 'complaint' rows the 27B judge rejected, and its probabilities are calibrated to within 3 points. Every number is exact because we planted the errors.

Sep 16, 202615 min read

Route to the GPU That Already Has Your KV Cache

A follow-up question costs 0.54 seconds on the worker that answered the first one and 2.56 seconds on any other — the whole prompt prefilled again. We measured that across a live fleet, built cache-aware routing for two router replicas that must agree, and then found why our hybrid 27B showed a 0% hit rate: vLLM leaves prefix caching off on Mamba-style models unless you pass two arguments. With them, a repeated 13,000-token prompt goes from 7.2 s to 1.6 s — and a third flag brings the same win down to the 1,200-token chats where most traffic lives.

Sep 9, 20268 min read

Two RTX 4090s Over PCIe: Splitting a 27B Model Beat Two Replicas by 2 to 4x

We had a rule that a model which fits on one GPU should run as one replica per GPU. On a two-RTX-4090 machine with a 27B model, that rule was wrong by a factor of two to four, and the reason was not the parallelism at all. Here are the measurements, the rule that replaced it, and how the router now sends a 40k-token prompt to the worker that can hold it.

Sep 6, 202610 min read

One Config Is Always the Weakest Card's Config

A mixed GPU fleet has one setting per model, and it has to be safe on the slowest machine in it. We measured what that costs: an RTX 5090 running at 8% of its memory bandwidth because a 24GB card three racks away needed a flag. Two arguments, applied only to the cards that could take them, made it fifteen times faster. Here is what the engine measures per node, what it changes, and the two things that surprised us.

Aug 31, 20268 min read

The Faster Card Was Half the Speed

Two RTX 4090s on our network, same container, same model, same arguments. One runs Windows with WSL2 and benchmarks better on every hardware axis we measure. It serves inference 48% slower — three times the penalty the published figures report. Here is the measurement, what we ruled out, and why we think large quantized models are the worst case for GPU virtualisation.