Engineering9 min read
Our Load Balancer Gave the Fastest GPU the Least Traffic
A new RTX PRO 6000 joined our network and got a quarter of the traffic of a pair of RTX 4090s it outperforms 3x under load. The GPU-name table had no entry for it, and our measured correction could not see the difference, because per-request speed is nearly equal on the two. What we replaced both with, the cold-start mistake we made on the way, and two routing bugs the new weights exposed.
Read