RTX PRO 6000 vs Two RTX 4090s on a 27B Model: Equal on One Request, 3x Apart at 32

Two of the machines serving the same 27B model on our network make an unusually clean comparison. One is a single RTX PRO 6000 Blackwell with 96 GB. The other is two RTX 4090s over PCIe, with the model split across both cards. Send each one request and they are nearly equal: 106 tokens per second against 94. Send each 32 at once and one delivers 1,342 tokens per second and the other 435.

If you are choosing a GPU for a model in this class, that gap is the whole decision, and a single-request benchmark will not show it to you. This post is the measurement, what we think explains it, and what the big card does with a 256k-token prompt.

The setup

Both machines run the same container and the same arguments. The only difference is the hardware, and the context window it can afford.

Common to both:

• vLLM 0.27.1.
• Qwen3.8-27B in W4A16 (4-bit weights; 18.9 GiB loaded).
• FP8 KV cache.
• MTP speculative decoding with three draft tokens.
• CUDA graphs.
• Prefix caching (--enable-prefix-caching --mamba-cache-mode align).
• --max-num-seqs 128.
• Reasoning effort low.

The RTX PRO 6000 is a cloud VM we rented on Verda in Helsinki for these tests. It joined our network the way any provider's machine does, with the same docker run command, and our verification measured it before it served anything: 1,468 GB/s of memory bandwidth (92% of the card's 1,597 GB/s spec), 81.1 TFLOPS FP32 and 311.6 TFLOPS FP16 with cuBLAS.

Method, the same as in our tensor-parallel post:

• Every answer is exactly 256 tokens (min_tokens plus ignore_eos).
• Requests go through our own proxy, straight to the worker, at 1, 8 and 32 concurrent.
• Two runs per point, after a warm-up burst.
• Both workers were idle before each run.

Throughput

The two runs at each point agreed within 4%.

At one request the difference is 12%. At eight, 1.7x. At thirty-two, 3.1x. The per-request column shows it from the user's side. With 32 people using the model at once, each of them reads 45 tokens per second on the big card and 17 on the pair.

A note on the pair's last number. On 8 September we published 654 tok/s at 32 concurrent requests for this same machine. Today it measured 425 and 445. The configuration has changed since then; the largest change is the prefix caching flags added on 16 September. We have not isolated which change costs what, so we are reporting today's figure and flagging the difference instead of explaining it.

Why they are equal at one request and not at thirty-two

A single decoding request on a 27B model is limited by memory bandwidth: every token reads the weights once. The pair has two cards reading their half of the weights in parallel. The big card reads the whole thing from faster memory. With MTP drafting three tokens per step, both land near 100 tokens per second. That is the number most single-request benchmarks report, and at that point it tells you almost nothing about capacity.

Under load, three things diverge.

• The split pays per step. In tensor parallelism, every layer of every step needs all-reduces between the two cards over PCIe. The data they move grows with the batch. At one request it is small. At 32 requests it is 32 times as wide, every step.
• Compute starts to matter. Batched decoding turns a bandwidth-bound problem into a compute-bound one. The Blackwell card's measured 311 TFLOPS FP16 is on one die, with no transfer between halves.
• Room. 1.84 million tokens of KV cache against 477 thousand. Our test prompts are short, so this did not limit the 32-request runs. It is what limits real traffic with long prompts, and it is why the big card runs a window twice as long.

We have not profiled the step to split the gap between the first two. What we can say from the numbers is that the pair scales from 94 to 435 as load goes from 1 to 32, a factor of 4.6, and the single card from 106 to 1,342, a factor of 12.7.

What the 256k context costs

Qwen3.8-27B is trained on 262,144 tokens of context. On two 4090s we stopped at 131k because a longer prompt would occupy the worker for minutes. With 96 GB the full window fits with room to spare: vLLM reports enough KV cache for 7.03 simultaneous requests of 262,144 tokens each. The next post explains why this model is so cheap in cache per token.

Prefill, one prompt at a time, time to the first token:

For prompts of the same size the big card prefills about three times as fast as the pair. The rate falls as the prompt grows, because attention cost grows with length. A 256k-token prompt, a large codebase or a long contract in one request, takes a minute and a half before the first token.

The minute and a half is paid once. We sent the same 199,976-token prompt twice:

That is prefix caching on a hybrid model, which vLLM leaves off unless asked; we covered how to turn it on in routing to the GPU that already has your KV cache. For an agent that re-sends a long context on every turn, the second number is the one it lives with. Decoding after a 200k prompt still ran at 112 to 129 tokens per second for a single request.

What this means if you are choosing a GPU

• Measure at the concurrency you will run. At one request these two machines look interchangeable. Two 4090s cost a fraction of an RTX PRO 6000 to buy. For one user, or a batch job running one request at a time, that is a fair trade.
• For serving many users, one large card beats two small ones split over PCIe by a margin that grows with load: 1.7x at 8 requests, 3.1x at 32.
• For long context, memory decides. Both machines run the same model. Only the one with 96 GB can offer its full trained window, and it is the one that prefills three times as fast.

What it changed on our side

Our router used to weigh workers by a table keyed on the GPU name. The table had no entry for this card, so it gave it less traffic than the pair of 4090s. The measurement above is what exposed that; we replaced the table with a load test every worker now runs when it starts, and that is the subject of a separate post.

Long prompts on our API now land on the worker that can hold them: requests beyond 131k go to the 262k window, on the same model name. The published context for qwen3.8-27b stays at 32,768, the length every worker guarantees. The models page of the docs lists what is served.