Our Load Balancer Gave the Fastest GPU the Least Traffic

We added an RTX PRO 6000 to the workers serving our 27B model, sent sixteen requests at once through the public API, and watched where they went. We compared it with a pair of RTX 4090s serving the same model: twelve requests went to the pair, four to the RTX PRO 6000.

The RTX PRO 6000 is the faster machine, by 3.1x at 32 concurrent requests (the benchmark is here). Our load balancer was sending most of the work to the slower one. This post is why, what we replaced, and what we got wrong while replacing it.

How the router chooses

For each request, our gateway filters the workers of the requested model by context window, prefers one that already holds the prompt's prefix in its cache, and otherwise picks two workers at random and sends the request to the one with the lower in-flight requests / weight. This is the power-of-two-choices scheme. Load is measured live, so it is accurate. Weight is the router's estimate of capacity, and it is the only part that is a guess.

Between two given workers, the weight decides everything: under load, the router keeps their in-flight counts in the ratio of their weights. At 3.5 against 1.0, the pair of 4090s was meant to carry three and a half times the RTX PRO 6000's load. In the sixteen-request test it carried three times.

Weight, version one: a table keyed on the GPU name

The first weights came from a table: an RTX 4090 is 1.75, an RTX 5090 2.5, an H100 3.5, and so on. A multi-GPU worker gets the sum, so two 4090s are 3.5. A name the table does not know gets 1.0.

The RTX PRO 6000 was not in the table. That is the obvious bug, and adding a line would have hidden it. The real problem is that no line would be right. Capacity is a property of the card, the host, the engine, the launch arguments and the model together. The same two 4090s that run this 27B model at 435 tok/s under load would look very different on a 4B model. The earlier post One Config Is Always the Weakest Card's Config and the WSL2 measurement are both cases of one card name covering very different machines.

Weight, version two: a correction from live traffic

We knew the table was a guess. On 6 September we added a correction that re-derived each worker's weight every three minutes from its own traffic:

• the median decode speed of its requests since it registered;
• multiplied by how much of that time it was busy;
• divided by the median of the other workers of the same model;
• blended with the table's figure until enough requests had been seen.

It was built for a real case, two identical 4090s delivering 29.6 and 15.7 tok/s, and it handled that case.

On the new pair it did this: RTX PRO 6000 111 tok/s over 8 requests, the 4090 pair 37.5 over 15, weights moved to 1.22 and 1.23. Equal. Two things were wrong with it.

Per-request speed does not measure capacity. On one request the two machines are 106 and 94 tok/s. On eight, 86 and 50 per request, but the RTX PRO 6000 is carrying the same eight at 1.7x the total. Per-request speed mostly tells you how busy the node was when it was measured. The utilization factor was meant to correct for that. It is computed over the whole run, so idle gaps between bursts dilute it until it rounds back to 1. With sparse traffic, the correction measures the traffic pattern, not the machine.

Its starting point was wrong for multi-GPU workers. The blend started from the table's figure for one card name, 1.75. The same pair was registered at 3.5, the sum over two cards. The correction and the registration disagreed about the machine before either had measured it.

Weight, version three: measure capacity directly

Capacity is what a worker delivers when it has work queued, so we measure that, the same way for every worker:

• The burst. 16 concurrent chat requests, each exactly 128 output tokens (min_tokens plus ignore_eos), sent by the node manager through its own proxy to the worker. It never touches billing, the request log or the router's load counters.
• The weight. Total tokens over wall time is the worker's throughput. Its weight is that throughput over the median of the workers serving the same model, clamped to between 0.25 and 8.
• When it runs. Once when a worker registers. Again every six hours, but only when the worker has nothing in flight, so a re-measurement never competes with a customer. On demand, from the admin page.
• Reconnects are free. A worker's connection drops and reconnects a few times a day. The result is kept for six hours per node, model and launch arguments, so a reconnect does not cost a burst. A restart with different arguments is a different worker and is measured again.

An administrator can also pin a node's weight, which survives restarts until cleared. That is the escape hatch for a machine the probe misjudges. We have not needed it yet.

The first measurement gave 992 tok/s for the RTX PRO 6000 and 453 for the pair: weights 1.37 and 0.63.

The mistake: measuring an engine that had not warmed up

A few hours later we restarted the RTX PRO 6000's worker with a longer context window. The registration probe ran five seconds after the worker reported ready and measured 275 tok/s. The weights flipped to 0.76 and 1.24, and the slower machine got the traffic again.

The machine was fine. We ran the identical burst by hand a few minutes later: 1,042, 854 and 1,007 tok/s. A freshly started vLLM engine pays for compilation and for capturing CUDA graphs on its first requests. The first probe in production had happened to land on workers that had been running for hours, which is why the design looked right.

Two lessons came out of it:

• Warm up before you measure. Every probe now sends the same burst twice and keeps only the second. It adds about two seconds of GPU time per probe.
• A cache keeps bad values as faithfully as good ones. The six-hour result cache that makes reconnects cheap also kept the 275 across a restart of the node manager. The fix for a bad reading is a fresh measurement, not a restart.

With the warm-up the numbers are 980 tok/s against 479, weights 1.34 and 0.66, stable across reconnects.

Two routing bugs the new weights exposed

Once the weights were right, the choice between two workers still had two places where they did not count.

Ties were a coin toss. With nothing in flight on either worker, both have a load of 0 / weight = 0. The code picked the first of the two random candidates, so a quiet network split requests 50/50 between a fast worker and a slow one. On this model it barely matters, since the two are within 12% on one request. On a model served by both a GPU and a Mac, whose single-request speeds are far apart, it means half the quiet-hour users wait longer for no reason. Ties now go to the higher weight.

The prefix-cache preference counted requests, not capacity. A worker that already holds a prompt's prefix skips re-reading it. For a long conversation that is the difference between seconds and a fraction of one. We prefer that worker as long as it is not much busier than the freest one, with the allowance growing with prompt length. "Busier" was a raw count of requests in flight:

• RTX PRO 6000 holding the conversation with 10 requests in flight;
• the pair with 3;
• raw count, 10 against 3: the RTX PRO 6000 looked far too busy, and the conversation moved to the slower machine and lost its cache;
• per unit of capacity, 10 / 1.34 = 7.5 against 3 / 0.66 = 4.5: close, and worth staying for.

It also ran the other way: a cached conversation stayed on the slow machine after it was already full for its size. The comparison now uses in-flight / weight on both sides, the same unit as the main choice.

What this costs, and what it does not do

• About four seconds of GPU per worker per measurement, at registration and at most every six hours after.
• When the network is quiet, the fastest worker gets the requests. That is the point: the user gets the faster answer. It also means slower machines see less traffic in quiet hours than they did under the coin toss.
• The probe measures short prompts. It is a decode-heavy load. A worker's capacity for 100k-token prompts is a different number: prefill speed and cache size, which the router handles separately by sending long prompts only to windows that can hold them.
• The burst crosses the worker's connection, so a distant node carries its round trip in the wall time. On a two-second burst that is a few percent.
• The per-worker cap on concurrent requests is still flat. Every worker takes at most 24 at a time, however fast. The RTX PRO 6000 ran 32 in our benchmark at 45 tok/s each. Raising the cap with the weight is the next change, when traffic reaches it.

The general lesson

If a load balancer has a weight, the weight is its model of capacity, and capacity is a load-dependent quantity. Per-request speed and hardware labels are both measured at the wrong load. Measure every worker the same way, at a load where the difference shows, after the engine has warmed up. Re-measure when it is idle, and keep a way to override it.

The router serves our inference API, where all of this happens behind one model name.