GPU spend grows in two ways. More experiments and larger models increase compute needs. Poor purchasing decisions add an efficiency tax: idle reserved capacity, on-demand jobs that could use spot, and interrupted experiments that restart because they lack checkpointing.
These five strategies target separate causes of GPU waste. Their effects compound because the categories interact. Spot routing lowers on-demand spend, which improves per-experiment cost tracking and can influence how engineers design experiments. They are not isolated settings, but parts of one system.
No strategy fits every team. Idle termination thresholds should reflect job submission habits, while spot usage should match workload tolerance for interruptions. Treat these as adjustable frameworks, not fixed recipes.
Strategy 1: Route Experimental Jobs to Spot
For many ML teams, moving experimental and iterative jobs from on-demand to spot offers the largest single saving. On A100 and H100 instance families, spot pricing averages 60-70% below on-demand rates. This requires reliable checkpoint and resume logic in the training code.
Start by finding checkpoint-capable jobs, such as single-node fine-tuning. Set up access to at least two cloud providers or two regions within one provider, then send those jobs to spot. Track actual savings over two to four weeks and expand based on the results.
For a team whose workload is mostly single-node fine-tuning, a reasonable three-month goal is 60-75% of GPU-hours on spot. The remaining 25-40% may include interruption-sensitive deadline work, jobs without adequate checkpoint and resume support, and multi-node jobs lacking cross-node migration.
Suppose a team spends $30,000/month on on-demand A100 compute. Moving 70% to spot at an average 65% discount produces roughly $13,650/month in savings. Actual results depend on the on-demand baseline, spot mix, and interruptions that require on-demand fallback.
Strategy 2: End Idle Instances
Running reserved or on-demand instances consume money even when no training is underway. A job may finish while its engineer reviews results, leaving the instance idle for hours. Across a team of eight engineers using mixed capacity, this becomes a recurring source of waste.
Automate the response. Check GPU utilization on running instances every 5-10 minutes, then alert or terminate after a grace period when utilization stays below the chosen threshold. On training instances, idle often means below 5% GPU utilization for more than 15-20 minutes. Short dips during data loading or checkpoint writes should not count, which is why the duration test matters.
Automatic termination needs a clear consent model. One option is alert-and-wait: notify the engineer and terminate after 30 minutes without a response. Another is terminate-and-notify: shut down the idle instance and send the notice afterward. The first is safer; the second saves more when detection is trusted.
For teams running 24/7 on reserved capacity, idle termination has limited value because reserved capacity costs the same while idle. For per-job on-demand or spot capacity, it cuts billed hours directly.
Strategy 3: Schedule Batch Jobs Off-Peak
Spot prices vary by hour and day. Weeknights from 11pm-6am US time and weekends generally offer lower prices and more availability than business hours. Differences depend on provider and region, but 10-25% off-peak savings is a realistic range from our observations.
Target batch jobs that can wait, including weekly evaluations, dataset processing, and scheduled fine-tuning with flexible deadlines. A queue that holds these jobs for off-peak windows can capture the price difference automatically.
The jobs must have real timing flexibility. A weekly evaluation due by Monday morning has about 60 hours to run. A production-serving fine-tune needed Tuesday morning has 24 hours. Jobs with under 8 hours of flexibility are poor candidates for off-peak scheduling.
A sophisticated scheduler is unnecessary at first. A basic queue can release jobs only in configured windows, check spot prices at release, and defer jobs above a chosen threshold. More advanced systems can use historical price patterns to select release times.
Strategy 4: Use Mixed Instance Families
Many ML teams train on one instance family, often A100 or H100. That choice makes spot availability and pricing depend on a single market.
Many fine-tuning jobs can use several GPU families with modest batch-size and precision changes. A configuration using 8xA100 80GB may also run on 8xH100 80GB, with possible batch-size reduction, 16xA100 40GB, with adjusted gradient accumulation, or 4xH100 80GB if the model fits on half the GPUs.
Letting the scheduler choose among equivalent configurations expands the available spot options. If A100 80GB spot is unavailable or costly in the target region while H100 80GB has capacity, the job can use H100. This applies multi-cloud spot routing across instance families, not only cloud providers.
The limitation is framework support for changing world sizes. DDP with fixed batch-size assumptions can fail when GPU counts differ between runs. Gradient accumulation is the usual way to keep effective batch size consistent. Jobs unable to adapt to different GPU counts do not fit this strategy.
Strategy 5: Track Cost per Experiment
The first four strategies change infrastructure. This one changes team behavior with infrastructure support. Per-experiment attribution gives every training run a visible dollar cost for its submitting engineer, without manual lookup.
Record the instance type, start time, end time, and GPU count for each job. Multiply billed hours by the spot or on-demand rate at submission, attach the result to the MLflow or W&B experiment record, and show it on the experiment page.
Seeing experiment costs can change engineering choices. A 500-run sweep on 8xA100 looks different when its record shows $4,200. Running a smaller model pilot on a 1-GPU instance, then scaling the top-3 candidates to a full 8-GPU run, costs $180 and can provide similar information. Visibility prompts this realization more effectively than policy alone.
We have found that cost visibility changes behavior more effectively than caps alone. Caps can make engineers focus on avoiding the limit rather than using compute efficiently. Visibility supports informed choices. Together, caps, visibility, and easy override provide the benefits of both.
These five approaches do not cover every case. Large hyperparameter searches, dataset ablations, and iterative RLHF fine-tuning cycles may need additional tactics. For most ML platform workloads we have seen, start with spot routing because it usually delivers the largest immediate savings. Add the remaining strategies according to the waste patterns your team finds.