A 72-hour training run using 32 GPUs across 4 nodes has a real chance of encountering at least one failure. That is arithmetic, not pessimism: every node carries a baseline hardware risk, each inter-node network path can experience a transient fault, and spot preemption adds another hazard. Across 72 hours, these small probabilities compound into a job-level risk that deserves explicit engineering.
Reliable long-running jobs do not depend on unusually stable hardware or cloud conditions. Their pipelines assume failures will occur and make those events recoverable instead of catastrophic. This post examines common multi-node failure modes, ways to tell them apart, and the work required in both the training framework and orchestration layers.
A Taxonomy of Training Failures
Failures differ, and a single response creates two problems: costly handling for unlikely events or no recovery path for common ones. This is the production taxonomy that matters for multi-node training.
Node Loss or Job Loss
Node loss occurs when one physical or virtual machine becomes unavailable because of spot preemption, hardware failure, a network partition, or an OOM kill. In a 4-node DDP job, losing node 3 leaves 75% of the cluster healthy. The job may resume on 3 nodes, restart on a replacement 4-node cluster, perhaps with a new node 3, or fail completely if the loss spreads.
Standard PyTorch DDP does not recover cleanly from node loss. If rank 2 on node 3 vanishes, NCCL collectives wait for it, eventually hit their timeout, and raise an exception that terminates the other ranks. By default, node loss becomes job loss.
PyTorch Elastic, including TorchElastic and torchrun with --rdzv_backend, permits elastic membership. Training can be reconfigured when the node count changes. A job on 4 nodes can keep running on 3 nodes with a revised global batch size while a replacement is provisioned. The training code must accept a variable world size, which increases implementation work but improves fault tolerance substantially.
NCCL Failure Modes
NCCL failures generally fit three categories, with distinct recovery requirements.
Timeouts occur when NCCL all-reduce or all-gather waits for a peer rank that is slow or unresponsive. The default timeout, set by NCCL_TIMEOUT, is 30 minutes on most setups. A timeout can leave the job frozen for up to 30 minutes before the error spreads. A shorter setting, such as 5-10 minutes, limits that pause but can flag legitimate slow work, including large gradient synchronizations over slow links.
Transport errors indicate packet loss, high latency, or total failure along the network path between nodes. NCCL may report NCCL_ERROR_INTERNAL_ERROR or NCCL_ERROR_UNHANDLED_CUDA_ERROR. These usually indicate a transient network problem, not a GPU fault. Save a checkpoint and restart rather than retrying the operation, which will likely fail again.
A slow peer is a node running well behind its neighbors because of thermal throttling, EBS I/O contention, or co-tenant interference. It may produce no immediate error, yet synchronous all-reduce makes every other node wait. Per-rank step-time monitoring can reveal this issue and is worthwhile for regular multi-node workloads.
OOM and Silent Data Corruption
An out-of-memory error on one GPU can be recoverable when it raises a Python exception: save a checkpoint, then restart with a smaller batch or revised gradient accumulation. An OOM that kills the process cannot be recovered without a checkpoint.
Silent corruption is less common but harder to detect. Certain GPU faults can alter computation without an exception, producing NaN or Inf gradients, nonsensical weight updates, and a diverging loss curve. Monitor loss shape and gradient norm, then stop when values leave expected ranges. One simple check saves a reference forward-pass result at step 0 and compares it periodically. Meaningful divergence signals a problem.
Gradient Checkpointing and Its Tradeoff
Gradient checkpointing, also called activation checkpointing, saves memory by recomputing activations during backpropagation instead of storing them. It cuts activation memory by about 50-80% while adding 20-30% compute per step. This differs from model-state checkpointing, although both use the word checkpoint, a frequent source of documentation confusion.
For fault tolerance, gradient checkpointing determines which hardware can fit a model. A model too large for GPU memory without it cannot run there, while enabling it may make the same model viable. After a failure, a replacement with the same GPU count but different memory may therefore require changed gradient checkpointing settings.
In PyTorch, apply torch.utils.checkpoint.checkpoint() to transformer layers or blocks, or use higher-level wrappers such as those in Hugging Face Accelerate. JAX offers a comparable mechanism through jax.checkpoint. Choosing per-layer or per-N-layers granularity balances memory reduction against compute overhead.
Checkpoint Timing for Multi-Node Jobs
A multi-node job needs a different checkpoint cadence than a single-node job. More components can fail per unit time, and writing state costs more because a larger model state must be saved across nodes.
For a 4-node 32-GPU job training a 13B-parameter model, one full checkpoint saves roughly 26GB of fp16 weights and another 26GB of fp32 Adam state, or about 52GB total. At 500MB/s, writing to S3 or GCS takes about 1.7 minutes. Saving every 10 minutes uses roughly 17% of runtime for I/O, while a 30-minute cadence reduces that share to 6%.
Choose the interval from the acceptable recomputation cost and estimated failure probability. A practical rule for multi-node spot jobs is every 20-30 minutes for runs below 48 hours, or every 45-60 minutes on longer runs when recomputation is acceptable. Keep the SIGTERM emergency checkpoint separate from this regular schedule.
Orchestration and Training Responsibilities
A clear division prevents duplicated logic in the wrong layer and exposes missing recovery work.
The training layer, meaning your PyTorch or JAX code plus TorchElastic when used, should save and restore model state, optimizer state, and step counters. It should also detect and report NCCL errors, change world size for elastic jobs, and watch training metrics for anomalies.
The orchestration layer, SkyPilot here, should detect node loss through cloud metadata polling or heartbeat timeouts, tell every rank to checkpoint by forwarding SIGTERM, provision replacements, provide the checkpoint location to resumed processes, record failure events, and choose the replacement node type and region.
A common production gap appears when orchestration knows a node has disappeared but the remaining training processes do not know to checkpoint and wait. The signal must reach training processes on every node, not only the failed one. Use an explicit heartbeat or signal channel to all processes, or use TorchElastic dynamic rendezvous to manage membership changes directly.
Designing the Restart Protocol
When a multi-node job must restart, the order of operations matters.
- Detect failure (via orchestration layer or training process exception)
- Signal all surviving ranks to checkpoint (if any are still running)
- Confirm checkpoint write completion on all ranks before releasing nodes
- Select replacement hardware (same cloud if available, different cloud if not)
- Stage the checkpoint to the new node's storage (or confirm shared storage access)
- Launch training processes on all nodes with the checkpoint path as a parameter
- Validate that all ranks have loaded the checkpoint and step counters match before resuming training
Step 7 is often skipped. If a run resumes with two ranks at step 1000 and another at step 980 because the failed node wrote only part of its checkpoint, gradient synchronization errors can follow and be difficult to diagnose. Check checkpoint consistency before resuming.
The constraint is straightforward: distributed fault tolerance adds unavoidable work to training code. For short or inexpensive jobs, restarting from scratch may be simpler, so on-demand instances can avoid that extra machinery. The investment makes sense when job duration and GPU cost make recovery from a checkpoint substantially better than starting over.