A pair of RTX 5090s is a configuration we serve our largest model on and had never published a number for. On 12 October we ran the test we run on every card, on a provider's machine with two of them, and on a machine with two RTX A6000s the same morning. With the three cards we had measured the same way before, that makes five, and for the first time they can sit in one table.
The setup
| GPUs | two NVIDIA RTX 5090, 32 GB each, over PCIe, power limit 400 W per card |
| Host | Ryzen 9 7950X3D, Ubuntu 24.04.5, native Linux, driver 595.84 |
| Verified before listing | 1,526 GB/s of memory bandwidth |
| Engine | vLLM 0.31.0 |
| Model | Qwen3.8-27B W4A16 (4-bit GPTQ), split across both cards (tensor parallel, 2) |
| KV cache | FP8, 805,349 tokens by vLLM's figure |
| Context window | 131,072 |
| Also on | MTP speculative decoding with three draft tokens, CUDA graphs, prefix caching, --max-num-seqs 128, reasoning effort low |
Why split at all: our serving rule asks for 28,000 MB for this model, a little more than the 27,715 MB we count as usable on one 32 GB card, so the platform spreads it over the pair. The weights are shared out and the rest of both cards becomes cache, which is where the 131k window comes from. We measured a single RTX 5090 on this model in an earlier post: 94.7 tokens per second on one request.
Method, the same as in our RTX PRO 6000 post and the H200 one:
- Every answer is exactly 256 tokens (
min_tokensplusignore_eos). - Requests go through our own proxy, straight to this worker.
- Two runs per point, after a warm-up burst.
One difference from those two posts: this machine was in the pool and serving other people's requests while we measured. Read its numbers as a floor.
Throughput
| concurrent | total, run 1 | total, run 2 | per request |
|---|---|---|---|
| 1 | 149 tok/s | 156 tok/s | 149 to 156 |
| 8 | 770 tok/s | 785 tok/s | 97 |
| 32 | 1,320 tok/s | 1,315 tok/s | 41 |
| 64 | 1,303 tok/s | 1,284 tok/s | 20 |
The two runs agree within 2% from 8 requests up.
The pair is at its ceiling by 32. Sixty-four requests get the same total as thirty-two, so each one gets half. Between 1 and 8 it scales well: eight times the requests for five times the tokens.
The platform's own load probe, which runs on every worker when it starts, read the same machine at 152 and 145 tok/s for one request on two occasions that day, and 975 and 961 across 16. Our figures and its figures were taken hours apart by different code.
The context
Prefill, one prompt at a time, time to the first token, nothing in the cache:
| prompt tokens | first token | prefill rate |
|---|---|---|
| 23,874 | 6.3 s | 3,803 tok/s |
| 61,804 | 17.6 s | 3,507 tok/s |
| 93,490 | 28.4 s | 3,292 tok/s |
| 124,281 | 40.3 s | 3,085 tok/s |
A prompt that nearly fills the window waits 40 seconds. Sent a second time, the same 124,281-token prompt answered in 0.5 seconds, with 124,224 tokens served from cache.
Five cards, one test
The same model, the same method, our own measurements. Totals in tokens per second.
| 1 request | 8 | 32 | ~94k prompt, first token | measured | |
|---|---|---|---|---|---|
| H200 NVL, one card | 163 to 176 | 1,034 | 2,527 | 12.1 s | 11 Oct |
| two RTX 5090s | 149 to 156 | 778 | 1,318 | 28.4 s | 12 Oct |
| RTX PRO 6000, one card | 106 | 636 | 1,342 | 25.7 s | 28 Sep |
| two RTX 4090s | 94 | 381 | 435 | 66.5 s | 28 Sep |
| two RTX A6000s | 65 to 76 | 430 | 816 | 48.4 s | 12 Oct |
This table is not a controlled experiment, and three things in it are not equal:
- The engine. The two September rows were measured on vLLM 0.27.1, the three October rows on 0.31.0. Some of the distance between a September row and an October row is the card and some is the engine.
- The load. The H200 and the RTX PRO 6000 were serving nothing else. The RTX 5090s and the RTX A6000s were in the pool, taking other requests.
- The cache. The H200 ran an unquantized KV cache, for the reason in its own post. The others ran FP8.
The prompt in the last column is also not the same length on every row: from 86,883 tokens on the RTX 4090s to 93,763 on the H200. The RTX 4090s' prefill figure is from 8 September, three weeks before their throughput figures.
With that said, three things stand out.
For one user, two RTX 5090s are close to an H200. 149 to 156 tokens per second against 163 to 176. The H200 pulls away only when the machine is shared: at 32 requests it delivers nearly twice as much.
At 32 requests two RTX 5090s and one RTX PRO 6000 land in the same place. 1,318 against 1,342. They get there differently: the pair is faster for a single request, the single 96 GB card has the full 262k window where the pair has 131k.
Two RTX A6000s have the most room and the least speed. They hold the model's full 262,144-token window, with 1,766,176 tokens of cache, more than any pair here. A single request on them is less than half as fast as on the RTX 5090s. It is a 2020 card: you rent it for the memory.
What it means if you are choosing
- One person, long sessions, current-generation speed: two RTX 5090s.
- A context longer than 131k: not this pair. An RTX PRO 6000, two RTX A6000s or an H200 take the full window.
- Many people at once: the H200, by a distance.
- The cheapest way to a long context: two RTX A6000s, if you can wait for the first token.
The prices are on each card's page and move with the market: they are what the machines available at that moment ask.
What we did not measure
- A single RTX 5090 on this model with today's engine. Our serving rule does not place it there now, so there was no worker to measure; the 94.7 tok/s above is from September.
- The same engine version on all five. That is what a real comparison needs.
- Two RTX 5090s as two separate replicas instead of one split model. We did that for the RTX 4090 in an earlier post; on 32 GB cards the answer may differ, and we have not run it.
- Output quality, which nothing here speaks to.
