Inference

Oct 15, 20265 min read

How We Take a GPU Back in Two Minutes Without Dropping a Request

Spot nodes promise their owners the machine back within two minutes. Doing that without cutting off inference requests mid-answer took a draining state, a fix to a heartbeat that kept undoing it, and one switch that the whole platform obeys.

Oct 12, 20267 min read

Two RTX 5090s on a 27B Model: 150 tok/s Alone, 1,300 at 32, Next to Four Other Cards

149–156 tok/s

one request, two cards

1,318 tok/s

in total at 32 concurrent requests

Oct 12, 20268 min read

Qwen3.8-27B on an H200: 3,300 tok/s at 128 Requests, After One Flag Stopped It Starting

163–176 tok/s

one request

3,342 tok/s

in total at 128 concurrent requests

Sep 30, 20269 min read

Our Load Balancer Gave the Fastest GPU the Least Traffic

A new RTX PRO 6000 joined our network and got a quarter of the traffic of a pair of RTX 4090s it outperforms 3x under load. The GPU-name table had no entry for it, and our measured correction could not see the difference, because per-request speed is nearly equal on the two. What we replaced both with, the cold-start mistake we made on the way, and two routing bugs the new weights exposed.

Sep 29, 20267 min read

How Much VRAM Qwen3.8-27B Needs for Its Full 256k Context

Qwen3.8-27B is trained on 262,144 tokens of context, and a full-length request costs only about 9.3 GiB of KV cache in vLLM, because three layers in four keep no per-token cache at all. Here is the arithmetic, what we measured on a 96 GB card and a pair of 24 GB cards, a table from 24 to 96 GB, and why we would not stretch it to 1M with YaRN.

Sep 28, 20268 min read

RTX PRO 6000 vs Two RTX 4090s on a 27B Model: Equal on One Request, 3x Apart at 32

106 vs 94

tok/s on one request, RTX PRO 6000 vs two RTX 4090s

1,342 vs 435

tok/s in total at 32 concurrent requests

Sep 9, 202610 min read

One Model Name, Three Engines: vLLM on Linux, vLLM on WSL and llama.cpp on Apple Silicon Behind One Endpoint

The same model ID is now served by vLLM on Linux GPUs, vLLM on Windows/WSL, and llama.cpp on Apple Silicon Macs, and the caller cannot tell which one answered. Here is what had to be made equal for that to be true, the tokens-per-second we measured on an M1 Ultra and an M4 Max next to the GPUs, the 27B experiment that failed first and the rules it left behind, and a self-update bug that only exists when your inference server is a child process.

Sep 9, 20268 min read

Two RTX 4090s Over PCIe: Splitting a 27B Model Beat Two Replicas by 2 to 4x

We had a rule that a model which fits on one GPU should run as one replica per GPU. On a two-RTX-4090 machine with a 27B model, that rule was wrong by a factor of two to four, and the reason was not the parallelism at all. Here are the measurements, the rule that replaced it, and how the router now sends a 40k-token prompt to the worker that can hold it.

Sep 8, 20266 min read

Pay for the Tokens You Use. Nothing to Cancel.

ParalonCloud's inference API has no subscription, no seats, no monthly minimum and no auto-renewal. You add credits by card or crypto, every request is billed the moment it completes, to the token, and an invoice lands in your inbox. Here is the whole billing model on one page, including the parts that are not perfect yet.

Sep 7, 202610 min read

Every Inference Request Pays the GPU That Ran It

Inference on ParalonCloud is now paid, and it settles one request at a time: the moment a response completes, 80% of what it cost lands in the earnings of whoever owns the GPU that served it. No monthly statement, no batch. Here is how a request turns into money, why the cards we want most are the RTX 5090 and the RTX PRO 6000, and what changes if you call the API.