Gpu

Oct 22, 20267 min read

B200 vs H200 on a 27B Model: Faster for One User, Slower at 128

201–231 tok/s

one request, the fastest card we have measured

2,867 tok/s

in total at 128 concurrent requests, 14% under the H200

Oct 12, 20267 min read

Two RTX 5090s on a 27B Model: 150 tok/s Alone, 1,300 at 32, Next to Four Other Cards

149–156 tok/s

one request, two cards

1,318 tok/s

in total at 32 concurrent requests

Oct 12, 20268 min read

Qwen3.8-27B on an H200: 3,300 tok/s at 128 Requests, After One Flag Stopped It Starting

163–176 tok/s

one request

3,342 tok/s

in total at 128 concurrent requests

Sep 30, 20269 min read

Our Load Balancer Gave the Fastest GPU the Least Traffic

A new RTX PRO 6000 joined our network and got a quarter of the traffic of a pair of RTX 4090s it outperforms 3x under load. The GPU-name table had no entry for it, and our measured correction could not see the difference, because per-request speed is nearly equal on the two. What we replaced both with, the cold-start mistake we made on the way, and two routing bugs the new weights exposed.

Sep 29, 20267 min read

How Much VRAM Qwen3.8-27B Needs for Its Full 256k Context

Qwen3.8-27B is trained on 262,144 tokens of context, and a full-length request costs only about 9.3 GiB of KV cache in vLLM, because three layers in four keep no per-token cache at all. Here is the arithmetic, what we measured on a 96 GB card and a pair of 24 GB cards, a table from 24 to 96 GB, and why we would not stretch it to 1M with YaRN.

Sep 28, 20268 min read

RTX PRO 6000 vs Two RTX 4090s on a 27B Model: Equal on One Request, 3x Apart at 32

106 vs 94

tok/s on one request, RTX PRO 6000 vs two RTX 4090s

1,342 vs 435

tok/s in total at 32 concurrent requests

Sep 9, 20268 min read

Two RTX 4090s Over PCIe: Splitting a 27B Model Beat Two Replicas by 2 to 4x

We had a rule that a model which fits on one GPU should run as one replica per GPU. On a two-RTX-4090 machine with a 27B model, that rule was wrong by a factor of two to four, and the reason was not the parallelism at all. Here are the measurements, the rule that replaced it, and how the router now sends a 40k-token prompt to the worker that can hold it.