Tag

Pytorch

1 article tagged with “Pytorch”

A training curve resuming from a checkpoint after an interruption

Oct 19, 20264 min read

Checkpoint Your Training So a Spot GPU Can't Cost You Work

A spot GPU can be reclaimed with two minutes' notice and its disk is wiped when the rental ends. Here's a PyTorch training loop that checkpoints off the machine and reacts to the notice, so a reclaim costs minutes, not the run.

Read