GPU spend is often the line item that catches ML teams off guard. Compute gets budgeted, but not the difference between the price paid and the lower price available when someone checks all three major clouds in real time. That gap can produce 30-40% savings. The core idea is routing, not a complex savings scheme.
This post examines where GPU budgets leak, how multi-cloud spot routing works, and which tradeoffs matter before adopting it. The 30-40% figure is not universal. Results vary with your cloud mix, required instance families, regions, and tolerance for training interruptions. The mechanism is real, and the math is simple.
Where GPU Budgets Leak
Most teams begin with 1 cloud provider. They learn one instance family, such as p4d.24xlarge on AWS or a2-megagpu-16g on GCP, then configure IAM roles and networking. The infrastructure runs. The bill keeps rising.
The first leak is the on-demand premium. Spot capacity for A100-class hardware is often 60-70% cheaper than comparable on-demand capacity. Teams that use on-demand for every training job pay full price while providers discount more than half the cost. The tradeoff is reliability because spot instances can be interrupted. We will return to that, but the price gap is large enough to change how reliability costs should be judged.
The second leak comes from relying on spot capacity in 1 cloud. Spot prices and availability change. When a region runs short of capacity, availability falls and prices rise. A pipeline tied to that cloud must either wait, leaving engineers idle and experiments delayed, or use costly on-demand instances. A single-cloud setup offers no alternative when the market turns unfavorable.
The third leak is harder to notice: paying one rate throughout the day. Spot prices change by hour and region. A job submitted at 2pm EST pays the price at 2pm EST. Waiting three hours to use GCP's us-central1 capacity might have reduced the cost by 20%. Manual monitoring takes time, so teams usually pay the higher rate.
What Job Routing Does
Multi-cloud spot routing checks current prices across providers and zones before each placement, then selects the cheapest option that satisfies the job's requirements. The concept is simple. The difficult parts are credential management, mapping equivalent instance types, and scheduler logic for migration when a cheaper option becomes available during a run.
An internal 30-day benchmark compared single-cloud on-demand with automatic multi-cloud spot routing for LLM fine-tuning. The routed jobs averaged about 35% lower cost per GPU-hour. Results varied: some jobs approached 50% savings, while jobs requiring scarce capacity at premium spot prices came closer to 10%. The average held because the router could choose among alternatives.
For a concrete mid-scale example, a team using 200 GPU-hours per day at an average on-demand price of $3.50/GPU-hour spends about $255,000 per year. A 35% cut on routable work, assuming 70% of jobs tolerate interruption, equals about $60,000-80,000 in yearly savings. These figures are illustrative, not guaranteed. Teams with fewer interruption-tolerant jobs or less cloud access will save less.
Spot or Reserved Capacity
Reserved instances are the usual alternative to spot. You commit for one-year or three-year terms and receive a discounted on-demand rate, commonly 30-40% lower. The savings may look similar to spot, but the mechanisms and risk profiles differ.
Reserved instances provide availability in the committed region and instance family, so interruption is not a concern. Utilization is the drawback. You pay for reserved capacity even when no job uses it. Aggressive reservations can leave GPUs idle during off-peak periods, weekends, and gaps between experiments. Idle time can consume the discount.
Spot routing charges only for capacity used, but interruption risk remains. Good checkpoint and resume design makes that risk manageable, as covered in another post. Spot savings and reserved discounts can work together. Keep steady, predictable jobs on reserved capacity, and send experimental or batch work to spot across clouds. Either approach alone is less suitable.
Cross-Provider Price Differences
Teams are often surprised by the price gap between apparently equivalent hardware across clouds. An A100 80GB instance on AWS, GCP, and Azure does not have one consistent price. In many regions and time windows, the prices are not even close.
By mid-2025, spot prices for A100-class capacity, using 8-GPU instances, showed sustained 20-40% variation across the three major providers, depending on region and time. GCP's us-central1 and AWS's us-east-1 often competed closely. Azure's H100 spot pricing was notably higher in several periods, partly because of quota allocation. Prices move, so a router that does not check live rates misses this variation.
Price differences also appear between regions within one provider. At a given moment, AWS us-east-1 and us-west-2 spot prices for the same instance type can differ by 15-25%. Routing between regions in one cloud is a simpler form of the same opportunity.
Requirements for Automatic Migration
Checkpoint infrastructure is the part teams most often underestimate. Choosing the cheapest available instance is easy. Moving a job automatically when conditions change, or recovering after a spot interruption, requires training code that can reliably save and restore its state.
PyTorch's built-in checkpointing, including optimizer state, model weights, and epoch or step counters, is the starting point. Distributed runs also need coordination across all nodes before migration. For most long training runs, a practical interval is every 10-20 minutes. Shorter intervals increase I/O overhead, while longer ones increase recomputation after interruption.
SkyPilot manages the migration trigger and replaces nodes, but the training code must understand checkpoints. Teams with clean checkpoint and resume logic can capture the full routing benefit. Monolithic scripts that cannot pause without losing the run receive only initial placement savings, not migration savings.
What Routing Does Not Do
Multi-cloud routing does not replace sound workload architecture. A poorly structured job running on 1 node when it should be distributed will not become cheaper merely by moving to a cheaper node. Routing efficiency compounds with architectural efficiency: a well-structured distributed job sent to the cheapest spot capacity produces greater absolute savings because it has more GPU-hours to reduce.
Routing also leaves data egress costs intact. If training data sits in S3 and the job runs on GCP, each batch can incur cross-cloud egress. For small datasets, that cost may be minor. For large pretraining datasets, it can reduce the compute savings. Include data locality in the routing policy so the scheduler can account for it.
Start Without Full Migration
A full pipeline migration is not required on day 1. Begin by finding jobs that already support checkpoints and tolerate interruption, add access to 1 cloud provider beyond the primary one, and route only those jobs. Track the cost difference for two to four weeks before expanding.
Credentials are usually the first practical obstacle. AWS and GCP use different authentication systems, and each needs service accounts with suitable IAM permissions before the router can place jobs. Setup is a one-time cost that teams often underestimate as a time commitment, then overestimate as a continuing burden after completion.
The 30-40% figure describes teams that broadly use spot routing for experimental and batch work. Adoption is gradual. The first month often shows the largest percentage improvement because the starting point is frequently all on-demand spending. Later gains come from more precise policies, improved checkpoint timing, and routing additional workload types.