"Should I buy a 4090 and run the model myself, or pay per token?" comes up in every thread about open models, and the answers are usually two people with different workloads talking past each other. Both are right. The number that decides it is neither the price of the card nor the price per token; it is how many hours a day the card is actually generating, and at how many requests at once.
This guide puts measured numbers on both sides for one concrete case: Qwen
3.8 27B, the model behind qwen3.8-27b on our API, on an RTX 4090 you
own, against the same model at $1.70 per million output tokens over the
API. The throughput figures come from a benchmark we published, the power
figures from the fleet's own telemetry during two of the other guides in
this series, and the cost model is a 60-line script you can rerun with
your electricity price, your card's price and your measured watts.
| cost per 1M output tokens | |
|---|---|
| own RTX 4090, 16 concurrent requests, 100% busy, Germany | $0.40 |
| own RTX 4090, 16 concurrent, busy 25% of the time | $1.39 |
| own RTX 4090, 16 concurrent, busy 10% of the time | $3.35 |
| own RTX 4090, one request at a time, 100% busy | $1.97 |
| rented RTX 4090, 16 concurrent, 100% busy | $0.70 |
API, qwen3.8-27b | $1.70 |
Break-even at sixteen concurrent requests: about 20% utilization with German electricity, 15% with US electricity, meaning roughly four to five hours of full load every day of the card's life. At one request at a time, the card does not break even at 100%.
What a 4090 actually produces
Spec sheets do not say how many tokens a card generates; a benchmark does. In the tensor-parallel test we ran Qwen 3.8 27B quantized to W4A16 with vLLM on two 4090s as two independent replicas, which is the same thing as measuring one 4090 twice. Per card:
| concurrent requests on the card | output tok/s, total | per request |
|---|---|---|
| 1 | 30 | 30 |
| 4 | 107 | 27 |
| 16 | 147 | 14 |
Two things in that table drive everything below. First, a 27B model on a single 24 GB card is memory-bandwidth-bound: one request gets 30 tok/s, and the card only earns its keep when it batches, at 16 requests it makes five times the tokens per hour. Second, 16 is close to the ceiling: the W4A16 weights leave about 38,000 tokens of KV cache on a 24 GB card, and sixteen requests at 2,000 tokens each fill it. A card running one chat at a time produces 108,000 tokens an hour; a card kept sixteen deep produces 529,000.
What it draws at the wall
We sampled the fleet's GPU telemetry, nvidia-smi power and utilization
every ten seconds, on an RTX 4090 and an RTX 5090 while the
classification and
judge runs from this series were going through them. Those runs are
bursty, short requests, so the cards alternate between full utilization
and idle within seconds; we then ran a two-minute saturation test, sixteen
parallel 600-word generations, to see the steady state.
| card | condition | GPU power, mean | max |
|---|---|---|---|
| RTX 4090 | bursty serving, utilization ≥ 90% | 177 W | 214 W |
| RTX 4090 | model loaded, utilization 0% | 116 W | 191 W |
| RTX 4090 | saturated, 16 long generations, utilization 90% | 218 W | 237 W |
| RTX 5090 | bursty serving, utilization ≥ 90% | 359 W | 576 W |
| RTX 5090 | model loaded, utilization 0% | 149 W | 497 W |
| RTX 5090 | saturated, 16 long generations, utilization 97% | 534 W | 553 W |
Utilization is sampled at an instant, so an "idle" row can catch the
edge of a burst; read the means. The 4090 is a 450 W card and never got
near it: decode on a quantized 27B
model does not saturate the compute, and the power follows. The number
people forget is the second row: a card with the model loaded and nothing
to do still draws over 100 W, all day, because the weights sit in VRAM
and the card does not clock down the way a desktop GPU showing a
wallpaper does. The 4090 figures come from one card of a two-card node
running the model as a tensor-parallel split, so a single card carrying
the whole model alone will sit higher; the 5090 in the table is a single
card carrying the whole model and runs within 40 W of its 575 W limit
when saturated. The calculator takes whatever your own nvidia-smi
says, and the sensitivity line below shows what a hotter card does to
the answer.
For the model below we add 80 W for the rest of the machine, CPU, board, fans, PSU losses, and use 300 W at the wall while serving (the saturated figure) and 196 W idle.
The cost model
Four inputs, all of them yours to change:
- The card. New RTX 4090s are no longer produced, and in September 2026 remaining stock and good used cards sit between about $1,300 and $2,750. We use $1,800.
- The rest of the machine. A board, CPU, 64 GB of RAM, a PSU that will run a 4090 for three years: $800.
- Amortization. Three years, 24 hours a day, whether or not it is working: 26,280 hours, so the hardware costs $0.099 per hour of existence.
- Electricity. €0.35 per kWh, about $0.38, for a German household tariff; $0.17 for the US average. Everything below is shown for both.
Then the cost of a million output tokens is the machine's cost for an hour divided by the tokens it produced in that hour. The machine costs $0.213 an hour fully loaded and $0.173 an hour idle in Germany; the two are close because the amortization and the idle draw do not care whether it is working. What changes with utilization is the denominator.
cost per 1M tokens (utilization u) =
( amortization/h + electricity × (u × load_W + (1 − u) × idle_W) / 1000 )
÷ ( u × tokens per hour at full load ) × 1,000,000
At sixteen concurrent requests, 529,000 tokens an hour:
| busy fraction of the card's life | Germany, $/1M | US, $/1M | vs API at $1.70 |
|---|---|---|---|
| 100% | 0.40 | 0.28 | 4.3× to 6.1× cheaper |
| 75% | 0.51 | 0.37 | cheaper |
| 50% | 0.73 | 0.53 | cheaper |
| 25% | 1.39 | 1.03 | cheaper, just |
| break-even | 20% | 15% | |
| 10% | 3.35 | 2.53 | 1.5× to 2× dearer |
| 5% | 6.63 | 5.03 | 3× to 4× dearer |
| 2% | 16.46 | 12.53 | 7× to 10× dearer |
At one request at a time, 108,000 tokens an hour:
| busy fraction | Germany, $/1M | US, $/1M |
|---|---|---|
| 100% | 1.97 | 1.39 |
| 50% | 3.58 | 2.61 |
| 25% | 6.79 | 5.06 |
| break-even | never | 80% |
That second table is the one to read if the plan is "a private assistant for me and my team". A single-user chatbot is one request at a time almost by definition, and a 4090 doing that work is more expensive per token than the API in Germany even if it never sleeps, and needs nineteen hours a day of nonstop generation in the US.
Sensitivity, so nobody has to trust the defaults. A $1,300 used card moves the sixteen-deep German break-even from 20% to 18%. A card that draws 300 W instead of 218 under load, 380 at the wall, moves it to 21%. Counting the API's input tokens as well ($0.12 per million; a workload with three input tokens per output token adds $0.36 per million output tokens to the API side) moves it to 17%. None of these change the shape: the break-even lives between 15% and 25% utilization for batched serving, and above 75% for sequential.
The third option: rent the card by the hour
Between owning and paying per token there is renting the same card when you need it. An RTX 4090 on Paralon lists at $0.37 an hour and an RTX 5090 at $0.56, billed per minute, with no idle cost because you stop it. Sixteen requests deep, a rented 4090 makes tokens at $0.70 per million, a fifth of the API price; one request at a time it is $3.43 per million, twice the API. Renting wins for exactly one shape of work: a batch you control, big enough to keep the card saturated for the whole session, on a model you install yourself with vLLM in the rented container. For a backlog of a hundred million tokens that is the cheapest line on this page. For traffic that arrives when it arrives, the API's per-token bill is the cheapest because you pay nothing for the gaps.
And the reverse of renting: if you own a 4090 that sits at 5% utilization, the other 95% can be rented out instead of drawing 116 W for nobody.
What the dollars do not count
On the owning side:
- Your hours. vLLM upgrades, CUDA driver upgrades, the quantized checkpoint changing, the day the fan bearing goes. None of it is in the $0.099.
- Availability. One card is one card. When it is down, the assistant is down; the API side of this comparison runs on many.
- Model cadence. The API moved from one Qwen generation to the next without anyone reinstalling anything. On your card that is an afternoon, each time.
- Concurrency is not free to obtain. The 147 tok/s assumes sixteen requests are there to batch. If your users produce four at a time, read the 107 row and the break-even moves to about 28%.
On the API side:
- Rate and concurrency limits. A premium key on our API takes 300 requests a minute and 16 in flight; your own card has whatever limit you give it.
- Data leaves the building. For some workloads that is the whole decision, and no spreadsheet changes it.
- Latency. A local card answers a single request at 30 tok/s with no network in front of it; the API adds a round trip and shares the card.
The calculator
One Python file, no dependencies. Every input is a flag; the defaults are the numbers in this guide.
# your card, your tariff, your measured watts and tokens per second
python3 breakeven.py --gpu-price 1500 --host-price 600 --years 3 \
--kwh 0.28 --load-w 290 --idle-w 180 --tok-s 120 --api-out 1.70 --rent-hour 0.37
To get --tok-s and --load-w for your own machine, run vLLM, fire
sixteen concurrent requests at it for two minutes, and read nvidia-smi --query-gpu=power.draw --format=csv -l 5 in the other terminal. Take the
mean under load and the mean five minutes after the last request; those
are the two watt figures. Tokens per second is completion_tokens summed
over the run divided by the wall time.
The API side of the comparison needs no setup: a free key from the Console carries a 250,000-token trial, the list prices are on the inference page, and the classification guide shows what a saturated key looks like from the client side.
Frequently asked questions
Does this change for a 5090? The 5090 has 32 GB and about 1.8× the
memory bandwidth, so more concurrency and more tokens per hour, at a
higher purchase price and, from the table above, roughly double the
power. Rerun the script with the 5090's measured --tok-s; the shape of
the answer is the same and the break-even utilization is similar.
What about a smaller model? A 3B or 4B model on a 4090 makes several
times the tokens per hour, and the API price for the small model is
lower too ($0.20 per million output for qwen3-3b). Both sides scale;
measure --tok-s for your model and pass --api-out for its price.
Why three years? Consumer cards in continuous service are usually
replaced for capacity reasons before they fail; three years is a common
depreciation period and a 4090 bought in 2023 is at that point now. Use
--years 5 if you disagree; it moves the German break-even at sixteen
concurrent to about 16%.
Is the API price stable enough to plan on? Per-token prices for open models have only fallen since 2024, and the comparison gets worse for owning each time they do, because the card's cost does not fall with them.
Can I run the model unquantized? Not on a 24 GB card; 27B parameters in 16-bit is 54 GB. W4A16 is what fits, and it is what the throughput above measures. The API serves the same quantization.
