A provider connected a machine with two RTX 4090s. Our scheduler looked at the 27B model we serve, saw that it fits on a single 24 GB card, and did what its rule said: start one replica per GPU. Two copies of the model, one per card, requests split between them. Textbook data parallelism, the choice every guide recommends when the model fits and you want throughput.
It was the wrong choice, and by a margin we did not expect. This post is the measurement, the reason, and what we changed.
The setup
- Two RTX 4090 (24 GB each) on PCIe. No NVLink, no peer-to-peer.
- vLLM, serving Qwen 3.8 27B in W4A16 (4-bit weights, 16-bit activations), about 17 GB of weights.
- Same prompts for every run, 256-token answers, 1, 8 and 32 concurrent requests, measured through our own proxy so that nothing else touched the node during the run.
Two configurations:
- DP2: two replicas,
--data-parallel-size 2. Each card holds the whole model and serves its own requests. - TP2: one copy split across both cards,
--tensor-parallel-size 2. Every layer's matrices are sharded; each token's computation runs on both GPUs with an all-reduce between them, over PCIe.
The numbers
| concurrent | DP2, total tok/s | TP2, total tok/s | DP2, per request | TP2, per request |
|---|---|---|---|---|
| 1 | 30 | 110 | 30 | 113 |
| 8 | 214 | 422 | 27 | 57 |
| 32 | 294 | 654 | 14 | 25 |
KV cache: 38,229 tokens per replica on DP2, 350,518 tokens on TP2.
Per request, the split was 3.7x faster at one request and still 1.8x faster at thirty-two. In aggregate it was 2.2x at full load. The all-reduce over PCIe, the thing that is supposed to make tensor parallelism a bad idea without NVLink, never became the bottleneck at any concurrency we tested.
We repeated the TP2 run twice more, once under real traffic. The 1 and 8 figures reproduced within 2%; at 32 the loaded run gave 402 tok/s, still above the empty DP2 number.
Why: it was never about the parallelism
The replica did not lose because two GPUs compute faster than one. It lost because a 27B model on a 24 GB card leaves about 5 GB after weights, and that 5 GB has to hold everything else: the KV cache, the activation peak, and the memory for the two features that make vLLM fast on this model.
- CUDA graphs. Capturing them costs memory. On a 24 GB replica we run
--enforce-eagerbecause there is no room; every decode step then goes through the Python scheduler. That alone is a large part of the 30 tok/s. - MTP speculative decoding. Qwen 3.8 ships a multi-token-prediction head that drafts three tokens per step. It is not quantised, and its embedding and output layers alone need close to 4 GB. On 24 GB it does not fit at all; we had measured that earlier and left it off for single cards.
- KV cache. What is left after the above. On the replica, 38k tokens, which the router turned into five concurrent slots.
Split the model and each card holds 8.5 GB of weights instead of 17. The same 24 GB card now has room for graphs, for the MTP head, and for 350k tokens of cache. The speed-up is the sum of three things the replica could not afford, and the PCIe cost of the split is smaller than any one of them.
There is a fourth effect that only shows in production. A data-parallel node reports the KV budget of one replica, so our router gave the whole two-card machine five slots. The tensor-parallel worker reported 350k tokens and got the maximum, twenty-four. The 294 tok/s the replicas reached in the benchmark would never have been reached behind the router.
The rule we replaced
Old rule, in the agent that plans a container on each machine: if the model fits on one GPU, run one replica per GPU; tensor-parallel only with NVLink.
New rule: if the model fits on one GPU but takes at least half of it, and the machine has 2 or 4 GPUs, split it; small models keep the replicas.
The "half of it" threshold is the honest boundary of what we measured. A 3B model on two 4090s already has room for graphs and cache in each replica; splitting it would spend PCIe bandwidth to free memory nobody needs. Eight GPUs and more stay on replicas until we have measured them. And the scheduler that chooses the per-VRAM launch arguments now judges those arguments by what one worker process will see: a card's 24 GB for a replica, the group's 41 GB for a split. It used to judge by the machine's total, which is exactly how the first tensor-parallel start on that node got the 41 GB argument tier on a 24 GB replica and OOMed while loading the embeddings.
What the freed memory bought: context
With one copy across two cards, the same model can be started with a much
longer context. We added VRAM tiers to the model's configuration: above
40 GB effective, --max-model-len 65536; above 48 GB, 131072. On the
two-4090 node, started at 131k:
| real prompt tokens | total time | prefill rate |
|---|---|---|
| 21,759 | 12.4 s | ~1,760 tok/s |
| 57,939 | 47.6 s | ~1,220 tok/s |
| 86,883 | 66.5 s | ~1,310 tok/s |
And the KV cache grew to 492k tokens, enough for almost four full-length 131k sequences at once, or hundreds of ordinary ones.
The model's native context is 262k. We stopped at 131k on purpose: a 256k prompt would be minutes of prefill during which every other request on that worker slows down. On consumer cards over PCIe, 128k is the useful limit.
Routing by context
This created a new problem. The same model now runs with a 32k context on single cards and a 131k context on the split node, under one model name. A 50k-token prompt sent to the wrong worker gets a 400 from vLLM, and a gateway that picks workers by load alone will send it to the wrong one some fraction of the time. A request that works on Tuesday and fails on Wednesday is worse than one that always fails.
So each worker now reports the context it was started with (vLLM exposes
max_model_len on /v1/models), and the gateway estimates what a request
needs, roughly four characters per token plus max_tokens, and keeps it
away from workers it would not fit. Two details matter:
- The estimate only steers, it never refuses. Four characters per token overshoots English prose by about 30% (we measured 5.5) and undershoots code. A request that looks too big for every worker still goes to the largest one, and vLLM's exact token count decides. vLLM checks length before any GPU work, so a wrong guess costs a round trip in milliseconds, not a wasted generation.
- It costs nothing. The context comes from the same Redis hash the gateway already reads per worker; the estimate is one pass over the request body. Worker selection for the test requests took under a millisecond, same as before.
Verified on the node above: a 40k-token prompt skipped the 32k worker and completed on the split node in 23 seconds; a 131k-token prompt got vLLM's 400 in 0.4 seconds, with no GPU time spent.
The published context for the model stays at 32,768 for now. A number that moves with whichever machines are online is a thermometer, not a contract; we raise it when more than one node can hold it, and long context will likely become its own model name with its own price.
One more thing we fixed on the way
While testing, a non-streaming request with a very long answer ran past the
gateway's five-minute limit. The gateway treated that as a failed worker and
retried on two more, each of which generated the same answer for nothing,
and the caller got an error after fifteen minutes. A slow generation is not
a broken worker. It now returns a 504 with the fix in the message, use
stream: true for long answers, and does not retry. Under a 40-request
load on the split node, the 504 came at exactly 300 seconds, retries zero.
What this means if you provide GPUs
If your machine has two RTX 4090s, or two 3090s, or four of either, it now serves the 27B model as one split worker with speculative decoding, CUDA graphs and a 64k or 131k context, and the router sends it the long prompts nobody else can take. Per request it is two to four times faster than the same cards were last week, and it gets many more concurrent slots. Our earlier post on scheduling LLMs across mixed consumer GPUs covers the sizing rules that still apply when the model does not fit on one card at all.
If you call the API: nothing to change. Requests under 32k tokens go where they went before; longer ones now go where they fit.
