On 22 October a machine with one NVIDIA B200 joined our network. We ran it through the test every card here gets: the same 27B model, the same vLLM and the same arguments as the H200 and the five-card comparison.
We expected it to win everywhere. It did not. For one user it is the fastest card we have measured. With 32 or more requests at once, the older H200 serves more tokens per second. This post is the numbers, the two explanations we tested and had to drop, and the one that is left.
The setup
| GPU | one NVIDIA B200, 183,359 MB, compute capability 10.0, 1,000 W limit, PCIe Gen 5 x16, no NVLink |
| Host | Ubuntu 26.04.1, kernel 7.0, AMD EPYC 9575F, 30 threads, 167 GB RAM, driver 580.178.04, CUDA 13.0 |
| Verified before listing | 5,972 GB/s of memory bandwidth (78% of the 7.7 TB/s spec), 66.6 TFLOPS FP32 |
| Engine | vLLM 0.31.0 |
| Model | Qwen3.8-27B W4A16 (4-bit GPTQ), 19.0 GiB loaded |
| KV cache | FP8, 139.45 GiB, 3,935,088 tokens by vLLM's figure |
| Context window | 262,144 |
| Also on | MTP speculative decoding with three draft tokens, CUDA graphs, prefix caching, --max-num-seqs 128, reasoning effort low |
The B200 is a cloud VM we rented on Verda in Helsinki.
It joined the network with the same docker run command a provider uses, and
the model started from our recipe without a single change: the argument set for
cards of 64 GB and more.
Method, unchanged from the earlier posts:
- Every answer is exactly 256 tokens (
min_tokensplusignore_eos). - Requests go through our own proxy, straight to the worker.
- Two runs per point, after a warm-up burst.
Throughput
| concurrent | B200, run 1 | B200, run 2 | per request |
|---|---|---|---|
| 1 | 231 tok/s | 201 tok/s | 201 to 231 |
| 8 | 1,182 tok/s | 1,221 tok/s | 156 |
| 32 | 2,218 tok/s | 2,209 tok/s | 74 |
| 64 | 2,627 tok/s | 2,642 tok/s | 44 |
| 128 | 2,841 tok/s | 2,893 tok/s | 24 |
Next to the cards measured the same way:
| concurrent | B200 | H200 NVL | RTX PRO 6000 |
|---|---|---|---|
| 1 | 201 to 231 | 163 to 176 | 106 |
| 8 | 1,201 | 1,034 | 636 |
| 32 | 2,213 | 2,527 | 1,342 |
| 64 | 2,635 | 3,067 | — |
| 128 | 2,867 | 3,342 | — |
For one request the B200 is about 30% ahead of the H200. Decoding one stream is limited by how fast the card reads the model's weights, and the B200 reads memory faster: 5,972 GB/s in our verification against 4,062 on the H200. At eight requests it is still 16% ahead. From 32 on the order flips, and at 128 the H200 delivers 14% more.
The two B200 runs agreed within 2% everywhere except the single request, where one stream depends on how many of the three draft tokens are accepted.
Two suspects, both cleared
The worker's log names two things that are different on this card, and both looked like the answer.
The FP8 KV cache. On startup vLLM says FlashAttention 4's Blackwell kernel
for this model's head size does not take an FP8 cache, and the H200 ran with an
unquantized cache. So we started the same container again with
--kv-cache-dtype auto, nothing else changed.
The attention backend. On the B200 vLLM picks FlashInfer, and with it,
it says, the three MTP draft steps cannot be fused. We forced FlashAttention
(--attention-backend FLASH_ATTN), which ran Blackwell's own FlashAttention 4
kernel and did fuse the draft steps.
Both runs were on the same machine, one after the other, sending requests straight to vLLM on the card rather than through our proxy:
| concurrent | as served (FP8 cache, FlashInfer) | unquantized cache | unquantized cache, FlashAttention 4 |
|---|---|---|---|
| 1 | 201 to 231 | 226 to 232 | 212 to 242 |
| 8 | 1,201 | 1,255 | 1,176 |
| 32 | 2,213 | 2,287 | 2,165 |
| 64 | 2,635 | 2,661 | 2,485 |
| 128 | 2,867 | 2,890 | 2,694 |
The unquantized cache gives 1 to 4%, inside what skipping the proxy can explain. FlashAttention 4 is slower than FlashInfer at every point from eight up. Neither closes the gap, so the FP8 cache stays: it holds 3.9 million tokens instead of 2.1 million for the same throughput.
What is left
The model's weights are 4-bit, and vLLM multiplies them with the Marlin kernel on both cards. Marlin was written for Ampere and Hopper. On a B200 it runs, but as code for an older generation, without the new tensor core instructions that make Blackwell fast at large matrix multiplications.
At one request that hardly matters: the time goes into reading weights from memory, and the B200 reads faster. At 128 requests the matrices are large enough for compute to count, and the H200 runs Marlin on the hardware it was written for.
We think this is the cause, but we have not proven it. The test that would is the same model in a format that does not go through Marlin on Blackwell, FP8 or NVFP4, and that is a different model from the one every other card in this series ran. When we run it, it will be its own post.
Long prompts: where the B200 is clearly ahead
Prefill, one prompt at a time, time to the first token:
| prompt tokens | B200 | prefill rate | H200 NVL |
|---|---|---|---|
| 23,599 | 3.3 s | 7,082 tok/s | 2.5 s |
| 61,329 | 8.7 s | 7,033 tok/s | 7.3 s |
| 92,663 | 13.4 s | 6,897 tok/s | 12.1 s |
| 128,127 | 19.1 s | 6,715 tok/s | 18.3 s |
| 197,965 | 31.0 s | 6,392 tok/s | 32.8 s |
| 251,907 | 40.9 s | 6,162 tok/s | 46.2 s (254,470) |
The H200 starts faster on short prompts, but its rate falls from 9,456 to 5,503 tokens per second as the prompt grows. The B200's falls only from 7,082 to 6,162. Past about 200k tokens the B200 is ahead, and on a full 250k prompt by five seconds.
The same 197,652-token prompt sent twice:
| time to first token | tokens served from cache | |
|---|---|---|
| first time | 30.9 s | 0 |
| second time | 0.31 s | 197,600 |
After that prompt, a single request still decoded at 230 to 246 tokens per second.
The cache is where the 183 GB shows. With an FP8 cache vLLM has room for 3,935,088 tokens, which is 15 requests of the full 262,144 at once. The H200, with an unquantized cache, holds 5.9 and the RTX PRO 6000 seven.
What this means if you are choosing a GPU
- One user, or a few, on a mid-size model: the B200 gives the fastest answers we have measured, about 30% ahead of an H200.
- Many short requests on a 4-bit model: the H200 serves more tokens per second from 32 concurrent up, at a lower price.
- Very long contexts, many of them at once: the B200. It prefills a 250k prompt faster than anything else here and keeps 15 of them in cache.
- A 4-bit model is not the B200's best case. Its advantage is meant to come from FP8 and FP4 compute. Our test runs the same 4-bit model on every card so the numbers compare, and that leaves part of this card unused.
