Qwen3.8-27B is trained on 262,144 tokens of context. The question people actually search for is how much GPU memory that takes. The short answer: much less than you would expect from a 27B model. In vLLM with an FP8 KV cache, a full 262k-token request costs about 9.3 GiB of cache, on top of about 22.6 GiB for everything else.
This guide shows where that number comes from, what we measured on two machines serving this model, and how to size a card for it.
Why the cache is so small: three layers in four keep no per-token cache
A transformer's KV cache grows with the context because every attention layer stores a key and a value vector for every token it has seen. For a dense 27B model that is the dominant cost of long context.
Qwen3.8-27B is not dense in that sense. Of its 64 layers, 48 are linear
attention (Gated DeltaNet). They keep a fixed-size state per request,
however long the request is. Only 16 layers, one in four, are full attention
with a per-token cache. From the model's config.json:
| field | value |
|---|---|
num_hidden_layers | 64 |
full_attention_interval | 4 (so 16 full-attention layers) |
num_key_value_heads | 4 |
head_dim | 256 |
Per token, per full-attention layer, the cache holds a key and a value of 4 heads x 256 dimensions. In FP8 that is 2 x 4 x 256 = 2,048 bytes. Across 16 layers: 32 KiB per token. The MTP head used for speculative decoding adds one more attention layer, 2 KiB.
vLLM's own figure is a little higher, because the cache is allocated in blocks and each request also carries its linear-attention state. On our 96 GB card vLLM reported 65.33 GiB of KV cache and a maximum concurrency of 7.03 requests of 262,144 tokens. That is 9.3 GiB per full-length request, or about 38 KB per token. We size with that figure, not the theoretical 34 KiB.
For comparison, with all 64 layers as full attention and the same heads, the cache would be four times larger: a 262k request would need 32 GiB by the same arithmetic, and more in practice. Without FP8 (a bf16 cache) every figure here doubles.
What else the card has to hold
From the vLLM log on the same 96 GB card, run with
--gpu-memory-utilization 0.92:
| GiB | |
|---|---|
| usable (95.6 GiB x 0.92) | 87.9 |
| weights (W4A16 quantization, MTP head included) | 18.9 |
| CUDA graphs | 0.6 |
| activations, vision encoder, runtime | ~3.1 |
| KV cache | 65.3 |
So on a single card, the fixed cost before any cache is about 22.6 GiB. Everything above that is context.
The sizing table
Two rows are measured on machines that serve this model on our network. The others apply the same two numbers, 22.6 GiB fixed and 9.3 GiB per 262k request, at a utilization of 0.92.
| setup | KV cache | full 262k requests at once | source |
|---|---|---|---|
| 1x 24 GB | ~38k tokens | none; runs at 32k context without MTP or CUDA graphs | measured, earlier post |
| 1x 32 GB | ~6.7 GiB | none; a single 131k request fits | estimate |
| 2x 24 GB, tensor parallel | 476,993 tokens | 1.8 (we run it at 131k) | measured |
| 1x 48 GB | ~21.6 GiB | ~2.3 | estimate |
| 1x 80 GB | ~50.7 GiB | ~5.4 | estimate |
| 1x 96 GB (RTX PRO 6000) | 65.3 GiB, 1,843,794 tokens | 7.03 | measured |
vLLM refuses to start if the cache cannot hold one request of
--max-model-len, so "none" in this table means the server will not come
up at 262k, not that it will be slow.
The count of full-length requests is the pessimistic view. The cache is shared by whatever is in flight, and most requests are short. A 48 GB card at 262k can serve many ordinary chats at once. It is the third long document arriving while two are in progress that waits.
The rule we use
On our network a worker gets --max-model-len 262144 only when the GPU
memory it actually has free is at least 64,000 MB. Below that, down to
40,000 MB, it runs 131,072.
We did not set the line at 48 GB, for two reasons:
- Headroom. At about 2.3 full-length requests, two long prompts nearly fill a 48 GB card. The third waits, and so does every short request queued behind it.
- Free is less than the label. The line is judged on free VRAM, and a 48 GB card with a driver and a desktop on it often reports less than 48,000 MB free. It would move between tiers from one start to the next.
The router already sends a prompt to a worker whose window can hold it, so cards on different tiers serve the same model name without a caller noticing.
How long it takes to fill
Memory is one half of the answer; time is the other. On the RTX PRO 6000, one prompt at a time, from our benchmark:
| prompt tokens | time to first token |
|---|---|
| 25,272 | 5.0 s |
| 100,391 | 25.7 s |
| 199,988 | 68.0 s |
| 256,361 | 99.3 s |
With prefix caching on, the same 200k-token prompt a second time answered in 2.4 seconds. If your workload re-sends a long context every turn, as coding agents do, that is the figure that matters.
Setting it up in vLLM
The flags that matter for long context on this model:
vllm serve Qwen/Qwen3.8-27B \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.92 \
--enable-prefix-caching --mamba-cache-mode align
Use your quantized checkpoint in place of the base model name; we serve a W4A16 build. Then read two lines from the startup log before trusting the setup:
GPU KV cache size: 1,843,794 tokens
Maximum concurrency for 262,144 tokens per request: 7.03x
The second line is this guide's table for your own card. If it reads below
2, lower --max-model-len or expect long requests to queue.
What about 1M tokens?
The model card says the context can be extended to about 1M tokens with YaRN (factor 4.0). Memory is not the obstacle: at about 38 KB per token, one 1M-token request is around 37 GiB of cache, which a 96 GB card holds.
We would still not do it on a general worker, for two reasons:
- YaRN applies to every request. The model card is explicit that vLLM and the other frameworks implement static YaRN, so the scaling applies to every request, and it can hurt performance on short texts. A general-purpose worker serves mostly short texts.
- Prefill would be very slow. Prefill speed fell from 5,070 tokens per second at 25k to 2,581 at 256k on our card. We did not run 1M, but that trend puts a single 1M-token prompt well past ten minutes.
If you need it, give it its own worker and its own model name, so the short traffic never pays for it.
Summary
- A full 262k request on Qwen3.8-27B costs about 9.3 GiB of KV cache in vLLM with FP8, because only 16 of 64 layers keep per-token cache.
- Budget about 22.6 GiB for the rest on a single card.
- 48 GB runs 262k, but with little headroom. From 64 GB free there is room for several long requests. 96 GB holds seven.
- Check vLLM's "Maximum concurrency" line rather than trusting any table, including this one.
To use the model without running it, qwen3.8-27b is on our
inference API, OpenAI-compatible.
Long prompts are routed to the workers that can hold them.
