Two RTX 4090s serve the same model on our network. One of them is 48% slower, and it is the one with the better hardware.
We found it by accident. Our scheduler had been splitting traffic evenly between them for weeks, because it weights nodes by GPU name and both are called "NVIDIA GeForce RTX 4090". Then we added a column recording what operating system sits underneath the agent's container, and the two nodes stopped looking identical.
The measurement
Both run the same container image, the same 27B model at 4-bit, the same vLLM arguments, the same agent version. Same city, same model row in the database. The only difference we know of is that one is native Linux and the other is Windows running Docker Desktop on WSL2.
Requests were sent one at a time, identical, deterministic, short prompt so the number reflects decode speed rather than prefill:
| samples | tokens/sec | std dev | range | time to first token | |
|---|---|---|---|---|---|
| Native Linux | 26 | 28.98 | 0.79 | 27.9 – 29.9 | 212 ms |
| WSL2 | 11 | 15.14 | 0.10 | 15.0 – 15.3 | 341 ms |
A 47.8% gap. What makes it interesting is the WSL2 column's standard deviation: 0.10 across eleven runs, a spread of 0.3 tokens per second. This is not noise or a bad afternoon. It is a hard, reproducible ceiling.
The card that loses is the better card
Every node on our network runs a verification container that measures the GPU rather than trusting what it claims. Those numbers:
| native Linux | WSL2 | |
|---|---|---|
| FP32 | 5.23 TFLOPS | 5.42 TFLOPS |
| Memory bandwidth | 868.7 GB/s | 921.3 GB/s |
| CPU | Intel i5-14600 | Ryzen 9 7950X3D |
| Inference | 28.98 tok/s | 15.14 tok/s |
The WSL2 machine wins on compute, on memory bandwidth, and on CPU. It delivers a little over half the throughput.
That inversion is what makes this worth writing about. If the slower machine had the slower card, there would be nothing to explain.
Autoregressive decoding is memory-bandwidth-bound: every token requires reading all 18.7 GB of weights. Divide throughput by that and you get how much of its own bandwidth each machine actually uses:
- Native Linux: 28.98 × 18.7 = 542 GB/s of an available 869 — 62%
- WSL2: 15.14 × 18.7 = 283 GB/s of an available 921 — 31%
The faster card uses half as much of itself.
What it is not
The obvious explanations are the boring ones, so we checked them while a generation was actually running.
| power draw | temperature | GPU utilisation | |
|---|---|---|---|
| Native Linux | 218 W | 67 °C | 71% |
| WSL2 | 157 W | 50 °C | 42% |
Not thermal. A 4090 begins throttling around 83 °C; this one is sitting at 50.
Not a power limit. 157 W against a board that will draw over 400.
The utilisation figure is the answer, or at least the shape of it. The GPU is idle 58% of the time in the middle of a generation. It is not working hard and falling behind. It is finishing each piece of work and waiting for the next one to arrive.
Why we think this happens
Decoding a token is not one big computation. It is a long sequence of small kernels — hundreds per layer, thousands per token, each one dispatched from the host to the GPU. When each kernel is small, the cost of dispatching it starts to rival the cost of running it.
WSL2 does not pass a GPU through to the guest the way a hypervisor passes through a PCIe device. It paravirtualises: CUDA calls made inside the Linux guest are marshalled across to the Windows host driver. That indirection is small per call. Multiply it by thousands of calls per second and it stops being small.
A GPU sitting at 42% utilisation, cool and drawing a third of its power budget, is exactly what a dispatch bottleneck looks like from the outside.
This is three times worse than the published numbers
The figures we could find put WSL2's inference overhead at 5–15%. The closest comparison to ours — llama.cpp running a 7B model at Q4_K_M on an RTX 4070 Ti — reports 68 tokens per second native against 58 under WSL2, a 15% gap. The common summary is that WSL2 is "within 5% of native" for inference and that the overhead "is in kernel launch latency, which is negligible."
We measured 48%. When your own result is three times the published one, the first assumption should be that you measured wrong, and our first attempt genuinely was — an earlier version of this comparison ran while a bug let requests pile onto workers unevenly, and we threw those numbers away.
The numbers above come from a clean rerun. So the interesting question is not whether the published figures are wrong. It is why our workload behaves so differently from theirs.
Our best explanation is that the published benchmarks measure the case where launch overhead is cheapest, and we are running the case where it is most expensive:
Model size. A 7B model has fewer layers and does more arithmetic per kernel. A 27B model issues far more, smaller dispatches for the same amount of output.
Quantization. 4-bit weights mean every matrix multiply is preceded by dequantisation. More kernels, each doing less.
CUDA graphs. This is the one we suspect matters most. CUDA graphs exist precisely to amortise launch overhead: record a sequence of kernels once, replay it as a single dispatch. Our configuration runs with --enforce-eager, which disables them — not by choice, but because 18.7 GB of weights plus activations on a 24 GB card leaves under 2 GB, and graph capture wants 784 MB of that. We tried removing the flag. vLLM failed to start with an out-of-memory error inside graph profiling.
So we are running the workload with the most dispatches, in the configuration that does the least to amortise them, on a platform that charges the most per dispatch. If that reasoning is right, the three factors multiply rather than add.
What we are not claiming
One machine per condition. This is two specific computers, not a population, and they differ in more than their operating system — motherboard, PCIe generation, memory speed, cooling. We can rule out thermal throttling and power limits because we measured them. We cannot rule out everything.
The clean experiment is the same physical machine booted natively and then under Windows, with nothing else changed. We do not own either of these nodes — they belong to providers on the network — so that experiment needs someone's cooperation, and we would like to run it.
The claim we are comfortable making is narrower than the headline: on this workload, on these two machines, the WSL2 node loses about half its throughput for reasons that are not heat, not power, and not the card.
Why it matters to a network like ours
We do not buy the hardware. People connect machines they already own, and a large share of consumer GPUs live in Windows desktops. Across our fleet the split is 74 native Linux nodes, 43 on WSL2, and 21 macOS.
Which means roughly a third of the machines that could serve inference may be delivering half of what their specifications promise, and until this week nothing in our system could even tell which ones they were. The scheduler still weights both of these 4090s identically, sending the same share of traffic to a node that returns half the work. Fixing that is the next thing we do.
There is a broader point for anyone renting compute rather than owning it. "RTX 4090" is not a specification. It is a name attached to a card, and the number of tokens per second you get out of it depends on a stack that the name says nothing about. Verifying the hardware — which we do, by measuring rather than asking — turns out not to be enough on its own. This node passed verification with better numbers than the one that outruns it.
If you run a Windows machine with a decent GPU and would be willing to boot it natively for an afternoon so we can measure the same hardware both ways, we would like to hear from you. We will publish whatever it shows, including if it shows we are wrong.
