Vllm

Oct 12, 20267 min read

Two RTX 5090s on a 27B Model: 150 tok/s Alone, 1,300 at 32, Next to Four Other Cards

149–156 tok/s

one request, two cards

1,318 tok/s

in total at 32 concurrent requests

Oct 12, 20268 min read

Qwen3.8-27B on an H200: 3,300 tok/s at 128 Requests, After One Flag Stopped It Starting

163–176 tok/s

one request

3,342 tok/s

in total at 128 concurrent requests

Sep 30, 20269 min read

Our Load Balancer Gave the Fastest GPU the Least Traffic

A new RTX PRO 6000 joined our network and got a quarter of the traffic of a pair of RTX 4090s it outperforms 3x under load. The GPU-name table had no entry for it, and our measured correction could not see the difference, because per-request speed is nearly equal on the two. What we replaced both with, the cold-start mistake we made on the way, and two routing bugs the new weights exposed.

Sep 29, 20267 min read

How Much VRAM Qwen3.8-27B Needs for Its Full 256k Context

Qwen3.8-27B is trained on 262,144 tokens of context, and a full-length request costs only about 9.3 GiB of KV cache in vLLM, because three layers in four keep no per-token cache at all. Here is the arithmetic, what we measured on a 96 GB card and a pair of 24 GB cards, a table from 24 to 96 GB, and why we would not stretch it to 1M with YaRN.

Sep 28, 20268 min read

RTX PRO 6000 vs Two RTX 4090s on a 27B Model: Equal on One Request, 3x Apart at 32

106 vs 94

tok/s on one request, RTX PRO 6000 vs two RTX 4090s

1,342 vs 435

tok/s in total at 32 concurrent requests

Sep 24, 20265 min read

vLLM: "There is no module or parameter named 'visual' in Qwen3_5Model" — loading a Qwen3.5 classification head

Serving OpenJev (Qwen3_5ForSequenceClassification) on vLLM fails at weight loading because the checkpoint carries the vision tower and vLLM's fallback picks the text-only class. The fix is a text-only copy of the checkpoint. Also: 'UVA is not available' on WSL2, and why vllm serve cannot take a Hugging Face subfolder.

Sep 17, 202611 min read

Own GPU vs Pay-per-Token: Where an RTX 4090 Breaks Even Against an LLM API

A measured cost model for running a 27B model on a card you own versus paying per token: what a 4090 actually produces at one and at sixteen concurrent requests, what it draws at the wall loaded and idle, and the utilization at which owning becomes cheaper. Sixteen requests deep, the card breaks even at about 20% utilization in Germany and 15% in the US. One request at a time, it never does. A calculator script takes your own numbers.

Sep 16, 202615 min read

Route to the GPU That Already Has Your KV Cache

A follow-up question costs 0.54 seconds on the worker that answered the first one and 2.56 seconds on any other — the whole prompt prefilled again. We measured that across a live fleet, built cache-aware routing for two router replicas that must agree, and then found why our hybrid 27B showed a 0% hit rate: vLLM leaves prefix caching off on Mamba-style models unless you pass two arguments. With them, a repeated 13,000-token prompt goes from 7.2 s to 1.6 s — and a third flag brings the same win down to the 1,200-token chats where most traffic lives.

Sep 9, 202610 min read

One Model Name, Three Engines: vLLM on Linux, vLLM on WSL and llama.cpp on Apple Silicon Behind One Endpoint

The same model ID is now served by vLLM on Linux GPUs, vLLM on Windows/WSL, and llama.cpp on Apple Silicon Macs, and the caller cannot tell which one answered. Here is what had to be made equal for that to be true, the tokens-per-second we measured on an M1 Ultra and an M4 Max next to the GPUs, the 27B experiment that failed first and the rules it left behind, and a self-update bug that only exists when your inference server is a child process.

Sep 9, 20268 min read

Two RTX 4090s Over PCIe: Splitting a 27B Model Beat Two Replicas by 2 to 4x

We had a rule that a model which fits on one GPU should run as one replica per GPU. On a two-RTX-4090 machine with a 27B model, that rule was wrong by a factor of two to four, and the reason was not the parallelism at all. Here are the measurements, the rule that replaced it, and how the router now sends a 40k-token prompt to the worker that can hold it.

May 19, 20266 min read

One VRAM Number Can't Schedule LLMs Across Mixed Consumer GPUs

What we learned running a 27B coding model across a heterogeneous fleet of consumer GPUs — why the obvious sizing formula silently OOMs, why pipeline parallelism behaves nothing like the benchmarks suggest, and the gotchas nobody writes down.