Qwen3.8-27B on an H200: 3,300 tok/s at 128 Requests, After One Flag Stopped It Starting

On 11 October a machine with one H200 NVL joined our network. It passed verification, and for the next half hour it tried to start the same 27B model that runs on every other card we have, and failed every four minutes with the same error. Nothing was wrong with the machine. One argument in our recipe is fine on every GPU generation except this one.

This post is the error, why it happens only on Hopper, the fix, and then what the card does once the model is up.

The error

The worker loaded the weights, started capturing CUDA graphs, and died:

RuntimeError: mha_fwd, /workspace/.deps/vllm-flash-attn-src/hopper/flash_api.cpp:940,
For FP16/BF16 input, output must have the same dtype as inputs.
For FP8 input, output must have dtype BF16

The card had 142.6 GB free out of 143.2, so this was not memory. The path in the message says where to look: hopper/.

Why only on Hopper

Three things have to be true at once.

vLLM picks FlashAttention 3 on Hopper. The worker's log said so: Using FlashAttention version 3. The code that raised the error lives in that kernel's hopper/ directory, built for compute capability 9.0: the H100 and H200. The other cards in our fleet (Ampere, Ada, Blackwell) start the same recipe without reaching it.

We run the KV cache in FP8. --kv-cache-dtype fp8 is in our base recipe because it halves the cache on 24 GB cards, where every gigabyte beside 18.7 GB of weights counts. With it, vLLM quantizes the query to FP8 before calling the attention kernel.

The model loads in FP16. We serve Qwen3.8-27B as a 4-bit GPTQ build (W4A16), and its config says float16.

FlashAttention 3 handles an FP8 query, but it chooses the output type itself: BF16. If the caller hands it an output buffer, that buffer has to match. vLLM's buffer has the model's type, FP16. The check on line 940 of flash_api.cpp compares the two and stops.

So an FP16 model with an FP8 KV cache cannot start on Hopper with this kernel. The same recipe starts on every other card we run. vLLM met the same thing in its own H200 CI, where a test with an FP16 model and an FP8 cache had been skipped; the fix there was to run that case in BF16 (vllm-project/vllm#49231).

The fix

There are two ways out, and either one is a single argument:

• --dtype bfloat16: keep the FP8 cache and run the model in BF16.
• --kv-cache-dtype auto: keep the model as it is and do not quantize the cache.

We took the second. We have not checked how the GPTQ weights behave in BF16, and a card with over 140 GB does not need a compressed cache to hold this model's full context.

The more useful change was where the argument lives. Our recipes already had extra arguments per operating system and per VRAM tier. Neither can say "on this generation of card": an 80 GB H100 and a 96 GB Blackwell card clear the same VRAM tier and need opposite answers. Models now carry a fourth list, keyed by the compute capability the driver reports, applied last:

{"9.0": ["--kv-cache-dtype", "auto"]}

The next start attempt picked it up and the model was ready 2 minutes 32 seconds later.

The setup

Method, the same as in our RTX PRO 6000 post:

• Every answer is exactly 256 tokens (min_tokens plus ignore_eos).
• Requests go through our own proxy, straight to this worker.
• Two runs per point, after a warm-up burst.
• The worker was serving nothing else.

Throughput

The two runs agree within 1% from 8 requests up, and within 8% at one.

The card is close to its ceiling at 64. Doubling the load to 128 adds 9% to the total and halves what each request gets. Between 8 and 32 it still scales well: four times the requests for 2.4 times the tokens.

One thing to know if you measure a freshly started worker: the first burst is cold. In an earlier pass with 400-token answers, the first 8-request burst after start read 468 tok/s and the same burst a minute later read 1,154. Throw the first one away.

Next to the RTX PRO 6000

Two weeks ago we measured an RTX PRO 6000 Blackwell on the same model with the same method:

This is not a controlled comparison and should not be read as one. The H200 ran vLLM 0.31.0 with an unquantized cache; the RTX PRO 6000 ran vLLM 0.27.1 with an FP8 cache. Some of the gap is the card and some is two weeks of engine changes, and we have not separated them. We are showing the two columns because they are the same test, not because the ratio is a property of the hardware.

The full context

Prefill, one prompt at a time, time to the first token, nothing in the cache:

The rate falls as the prompt grows, as it does on every card. A prompt that fills nearly the whole window waits 46 seconds for its first token.

That is paid once. We sent the 199,766-token prompt a second time:

For an agent that sends the same long context on every turn, the second row is the one that matters. How we keep such a conversation on the worker that already holds its cache is in routing to the GPU that has your KV cache.

What the unquantized cache costs

Dropping FP8 is not free. vLLM reports room for 1,542,293 tokens of KV cache on this card. The RTX PRO 6000, with 96 GB and an FP8 cache, reported 1,843,794. The card with half again as much memory holds fewer cached tokens, because each one takes twice the space.

In whole requests at the full 262,144-token window, that is 5.9 at once on the H200 against 7.0 on the smaller card. For short requests it changes nothing you would notice: 128 of them ran side by side above. If you serve many very long prompts at once on Hopper, --dtype bfloat16 with the FP8 cache is the option to test.

What we did not measure

• The BF16 route. We know it is what vLLM's own CI uses; we have not run it.
• The same engine version on another card, which is what a real comparison needs.
• Output quality. Nothing here says whether an unquantized cache changes answers, and we did not look.
• Mixed traffic. Every point above is one kind of request at a time.

If you hit this

If a vLLM worker on an H100 or H200 dies during startup with For FP8 input, output must have dtype BF16, check two things in its startup line: kv_cache_dtype=fp8 and dtype=torch.float16. If both are there, add either --kv-cache-dtype auto or --dtype bfloat16. The same command line will keep working on your other cards, which is exactly why this one is easy to miss until the first Hopper machine arrives.