Long-context

Oct 12, 20267 min read

Two RTX 5090s on a 27B Model: 150 tok/s Alone, 1,300 at 32, Next to Four Other Cards

149–156 tok/s

one request, two cards

1,318 tok/s

in total at 32 concurrent requests

Oct 12, 20268 min read

Qwen3.8-27B on an H200: 3,300 tok/s at 128 Requests, After One Flag Stopped It Starting

163–176 tok/s

one request

3,342 tok/s

in total at 128 concurrent requests

Sep 29, 20267 min read

How Much VRAM Qwen3.8-27B Needs for Its Full 256k Context

Qwen3.8-27B is trained on 262,144 tokens of context, and a full-length request costs only about 9.3 GiB of KV cache in vLLM, because three layers in four keep no per-token cache at all. Here is the arithmetic, what we measured on a 96 GB card and a pair of 24 GB cards, a table from 24 to 96 GB, and why we would not stretch it to 1M with YaRN.

Sep 28, 20268 min read

RTX PRO 6000 vs Two RTX 4090s on a 27B Model: Equal on One Request, 3x Apart at 32

106 vs 94

tok/s on one request, RTX PRO 6000 vs two RTX 4090s

1,342 vs 435

tok/s in total at 32 concurrent requests