You get either 30 seconds or 2 minutes of notice before a cloud provider reclaims a spot instance, depending on the provider and instance type. Many training scripts do nothing during this interval, forfeiting spot's cost benefit. Long-running training works on spot only when that interval is used to save the current training state.
This guide covers the practical details: each provider's signal, how your Python process responds, what storage can receive before termination, and what to verify after restart. There is no theory here, only the implementation choices that matter.
Provider Signal Details
GCP: SIGTERM and 30 Seconds
GCP Spot VMs send SIGTERM to the active process 30 seconds before reclamation. This is the documented delivery method, with GCP describing the interval as "approximately 30 seconds." Actual timing is usually 28-32 seconds. If the process remains alive after 30 seconds, the kernel issues SIGKILL and forcibly terminates the instance.
The signal targets a process, so the recipient must handle it. In a single-process job, that is the Python training script. In a multi-node job launched with torchrun or mpirun, the launcher generally receives SIGTERM first. Training code sees it only when the launcher forwards it, and some launchers do not do so by default. Confirm the launcher's propagation behavior before depending on SIGTERM for multi-node work.
AWS: Metadata Polling and 2 Minutes
AWS Spot Instances do not deliver an interruption signal to the process. Instead, the EC2 Instance Metadata Service endpoint at http://169.254.169.254/latest/meta-data/spot/termination-time starts returning a timestamp two minutes before termination. Until then, it returns HTTP 404. Once notice begins, it returns HTTP 200 and an ISO 8601 timestamp.
Poll this endpoint from a background thread. A 5-second interval balances prompt detection with low overhead:
import threading
import urllib.request
import urllib.error
class SpotInterruptionMonitor:
def __init__(self, callback):
self.callback = callback
self._stop = threading.Event()
def start(self):
t = threading.Thread(target=self._poll, daemon=True)
t.start()
def stop(self):
self._stop.set()
def _poll(self):
url = "http://169.254.169.254/latest/meta-data/spot/termination-time"
while not self._stop.wait(5):
try:
urllib.request.urlopen(url, timeout=2)
# HTTP 200 means interruption notice active
self.callback()
return
except urllib.error.HTTPError as e:
if e.code == 404:
continue # No notice yet
except Exception:
continue # Network error, keep polling
AWS's 2-minute interval provides meaningful room. With typical S3 checkpoint write rates of 500-1000 MB/s on EBS-optimized instances, models with roughly 20-30 billion parameters can save fp16 checkpoints within 2 minutes. Smaller models leave more margin.
Azure: Scheduled Events and 30 Seconds
Azure exposes interruption notices through the Instance Metadata Service Scheduled Events API. Use http://169.254.169.254/metadata/scheduledevents?api-version=2020-07-01 with Metadata: true. The response contains a Preempt event and a NotBefore timestamp. For Spot VMs using Delete eviction, the notice-to-NotBefore interval is approximately 30 seconds. With Deallocate eviction, it can be longer because the VM is stopped and preserved instead of deleted.
Poll every 5 seconds, as with AWS, and select events where EventType == "Preempt".
The Shutdown Handler's Responsibilities
Whether triggered by a signal or metadata notice, the handler should perform exactly three actions, in order:
- Set a flag that the training loop reads at the next safe boundary. Do not attempt I/O directly from a signal handler. Signal handlers run in an undefined context and may interrupt tensor operations mid-execution. Setting a Python threading.Event or a simple boolean flag is safe.
- Wait for the training loop to reach a safe boundary and write the checkpoint. A "safe boundary" is between batches, after the optimizer step, before the next forward pass starts. The training loop should check the flag at least once per batch.
- Exit cleanly. After the checkpoint write completes, call sys.exit(0) or let the process exit normally. Do not raise an exception unless you want it logged as a failure. A clean exit tells the orchestration layer that the shutdown was intentional, not a crash.
GCP time budget for a 30-second interval: SIGTERM arrives at T+0, the flag is set at T+1, and the current batch ends at T+3 to T+8, depending on batch size. Checkpointing begins at T+8 and must finish by T+25, followed by exit at T+26. That allows 17 seconds to write. At 500 MB/s to NVMe, the budget holds 8.5 GB of checkpoint data. A 7B fp16 model needs about 14 GB for weights alone, so use faster storage, a smaller model, or pre-computed checkpoint shards.
Checkpoint Size versus Storage Speed
The calculation is straightforward: fp16 or bf16 weights require 2 bytes per parameter. A 7B model therefore uses 14 GB. Adam optimizer state in fp32 uses 4 bytes per parameter multiplied by 2 for momentum and variance, totaling 56 GB. Weights plus optimizer state equal 70 GB.
"Save the model" can mean different things. Weights alone require 14 GB for inference, while resumable training needs weights and optimizer state, totaling 70 GB. With a 30-second interval and NVMe writing at 2 GB/s, 70 GB requires 35 seconds, which exceeds the window.
From simpler to more involved, the options are: (1) use bf16 optimizer state, reducing the size to 35 GB so it fits; (2) use ZeRO or sharded optimizer state, allowing GPUs to write their shards in parallel and keep total time below 30 seconds; (3) maintain a pre-staged checkpoint buffer that the training loop fills incrementally, leaving the shutdown handler to flush it instead of starting a full write.
For models above 13B parameters within GCP's 30-second interval, option (2) or (3) is necessary. Smaller models may work with option (1), or even the default fp32 optimizer state. Measure checkpoint write time on the storage you will actually use before trusting a narrow interval.
Validating State After Resume
After loading a checkpoint, perform a small validation before restarting training. Load it, run one forward pass on a fixed-seed batch, and compare the loss with the reference stored in that checkpoint. Matching within a floating-point tolerance of 1e-4 indicates validity. A major difference suggests corruption.
Save the reference value as a field in the checkpoint during the original write. Do not calculate it anew during resume, since that would require a second forward pass. Store it when saving:
checkpoint_data = {
'step': step,
'model': model.state_dict(),
'optimizer': optimizer.state_dict(),
'validation_loss_at_save': current_loss.item(),
}
Reload and compare on resume. A difference greater than 5-10% signals possible corruption, so revert to the previous checkpoint. A two-checkpoint rotation keeps the N-1 checkpoint until N passes validation, which limits recomputation when a checkpoint is damaged.
Handling spot interruptions is mainly plumbing: poll metadata in a background thread, let the training loop inspect a flag, write a checkpoint, and exit cleanly. The code is not algorithmically notable, but it separates jobs that fail on interruption from jobs that recover properly. Teams that run reliably on spot implement this plumbing once and reuse it.