Spot nodes make a promise to the people who own the hardware: flip the switch to Paused and the machine is yours again within two minutes. For a rental that part is easy to state: the renter gets two minutes' notice, then the container stops.
Inference is harder. A spot GPU can also be serving our OpenAI-compatible API, and at the moment its owner pauses it there may be requests mid-generation: a streamed answer half written, a long prompt still in prefill. Killing the worker would cut those off. Waiting for it to go quiet on its own could take forever on a busy model. The answer is the one every load balancer uses: drain.
One switch, read by everything
The first decision was where the "don't use this machine" state lives. Not in the dashboard, and not separately in each service: one flag on the node, workload_enabled, on by default everywhere.
- Pausing a spot node sets it off, together with rentals.
- Setting it Available turns both back on.
- Every path that can put a model on a node goes through a single function that decides whether to start a worker there, and that function returns immediately when the flag is off. A worker restarted after a rental ends, a worker the recovery loop thinks is missing, a node reporting its hardware after a reboot: all of them pass through the same check.
Render jobs are kept off spot nodes entirely: a frame range that can run for hours has no business on a machine that can be reclaimed.
Draining, in three steps
Our router only sends a request to a worker whose status is ready. That made the first step almost free:
- Stop routing. The worker's status changes to
draining. From the next request on, the router skips it and sends traffic to other workers of the same model. Nothing in the router had to change. - Let what's running finish. Every request in flight already holds a slot on its worker (that's how the router counts load). The node manager watches the worker's slots and waits until there are none left.
- Stop, with a ceiling. When the last request finishes, or when two minutes have passed, whichever comes first, it tells the agent to stop the worker and removes it from the routing table.
The ceiling is what keeps the promise to the owner. A request that is still generating after two minutes is cut; the agent reports the stream as interrupted rather than finished, so it isn't mistaken for a complete answer. On a node that was serving short requests, the drain finishes in seconds.
The heartbeat that kept undoing it
The first version drained nothing. The status went to draining and, a moment later, back to ready.
Workers send a heartbeat, and the heartbeat carries the worker's own view of its status, which is always ready: from inside the container the model server is perfectly healthy. The node manager wrote whatever the heartbeat said. So the drain was overwritten on the next beat and the router kept sending traffic.
The fix was one rule: a heartbeat may refresh a worker's expiry, but it may not move it out of draining. Only the drain itself ends that state, by stopping the worker.
When the node manager restarts mid-drain
A drain is a loop in a running process. If that process restarts halfway (a deploy, a crash), the loop is gone and the worker stays draining forever: never routed to, never stopped, still holding the GPU the owner asked for.
Two things close that gap:
- The periodic recovery loop, which already looked for nodes missing a worker, now also looks for the opposite: a node whose workload is off but which still has a worker registered. It starts the drain again.
- A worker that registers on a node whose workload is already off (a start that was under way when the owner paused) is drained the moment it appears, instead of serving even one request.
Not blaming the owner
The last piece isn't about workers at all. Our reputation scores machines on how their rentals went, and a rental ended by its provider normally counts against them. On a spot node, ending it is the whole point. So a rental records whether it started on a spot node, and spot rentals are left out of both the node and the provider score. The flag is fixed when the rental starts, so a provider can't switch a node to spot halfway through a rental to walk away from it unmarked; the dashboard also refuses to change it while the node is rented.
What it adds up to
From the owner's side it's one switch. From the platform's side it is a flag every placement path respects, a state the router already understood, a heartbeat that no longer gets the last word, and a recovery loop that finishes what a restart interrupted. The renter's side, two minutes' notice and a reclaim_at field in the Rental API, is covered in Spot vs On-Demand GPUs.
