ML infrastructure leads selecting GPU capacity face three pricing models with distinct tradeoffs. On-demand, reserved, and spot each suit particular workload types. Using one model everywhere can mean paying for reliability you do not need or exposing interruption-sensitive jobs to unnecessary risk.
This guide offers a decision framework, not a live pricing sheet. Rates change, while the reasoning behind each model changes more slowly. We cover how each option works, which workloads suit it, and how to assign jobs to pricing tiers. Dollar figures come from internal benchmark data from early 2025 and show relative scale. Confirm current provider pricing before committing.
On-Demand: Flexible, Full-Price Compute
On-demand instances require no commitment, avoid interruption risk, and remain available within your account quota. You pay the provider's listed hourly rate for the hours used.
On-demand fits jobs where interruption is costly, including model releases, deadline work, and jobs without checkpointing. It also suits uncertain runs, where an experiment may last 2 hours or 20, plus one-off work whose spot or reserved setup costs exceed its savings.
For most training, on-demand costs 60-75% more than spot on identical hardware. On 8xA100 instances, the average premium is $2.50-$3.00/GPU-hr. At 100 GPU-hours per day, that adds $91,000-$109,500 per year versus spot, assuming similar interruption overhead. A working checkpoint system can often handle interruptions at far lower cost.
Use on-demand as a fallback tier, not the primary one. When spot capacity is unavailable but a job must run, on-demand fits. As the default for every workload, it is expensive.
Reserved Instances: Commit for a Discount
Reserved instances on AWS and committed use discounts on GCP require 1-year or 3-year commitments to specific instance types, in return for 30-50% off on-demand rates. Azure provides similar programs under other names.
Reservation economics depend on utilization. A reserved instance running at 90%+ utilization for 365 days captures nearly the full discount. If it sits idle on weekends and between experiments, it may run at 50% utilization, effectively halving the discount when savings are divided by purchased compute hours.
Reservations suit predictable demand: production inference running 24/7, weekly fine-tuning pipelines, and workloads whose GPU use can be forecast over a 12-month window. They fit poorly during research and experimentation, when utilization varies and changing model designs may require another instance family in 6 months.
A key trap is reserving capacity in one region or availability zone. Spot supply changes, but reservations stay tied to their placement. If another region or provider offers better spot capacity, the reservation cannot use it. Reservations also cannot prevent an instance-family mismatch. Moving from A100 to H100 mid-commitment still leaves the A100 reservation billing.
Spot Instances: Lowest Cost, With Conditions
Spot instances use provider capacity left over after on-demand and reserved demand. Rates vary with demand and can reach 60-75% below on-demand for A100 and H100 families. Providers may reclaim an instance with 30 seconds to 2 minutes of notice as capacity needs rise.
Spot is a strong default for experiments, iterative fine-tuning, batch work, overnight pipelines, and jobs with working checkpoint and resume support. Avoid it for strict-latency production inference, interruption-sensitive business work, or training without checkpointing that would lose all progress.
The risk attributed to spot usually belongs elsewhere. Training code that cannot recover from interruption is the problem, not the pricing model. Regular checkpoints and orchestration that restarts jobs reduce that risk. An interruption then becomes a 10-20 minute delay instead of a catastrophic failure.
A GPU Pricing Decision
Assign each workload category to a pricing tier by answering these four questions:
| Question | If Yes | If No |
|---|---|---|
| Can this job checkpoint and resume without data loss? | Spot is viable | On-demand only |
| Is utilization predictable and consistently above 80%? | Reserved may be cost-effective | Reserved is a liability |
| Does failure block a time-sensitive deliverable (release, demo)? | On-demand fallback needed | Spot with auto-resume is sufficient |
| Is the instance type and region fixed for 12+ months? | Reserved commitment is lower risk | Reserved commitment risks wasted spend |
Most ML teams use three tiers: spot for experiments and work without deadlines, on-demand when spot capacity is unavailable, and reserved for a limited set of steady production workloads with stable utilization and instance needs.
Savings Plans or Instance Reservations
AWS Savings Plans offer a flexible alternative to instance reservations. Instead of selecting one instance type, you commit to a dollar amount of hourly compute spend, with the discount applying across families and regions. For ML teams moving from p4d to newer H100-capable instances, Compute Savings Plans provide reservation-level discounts without locking in one family.
GCP committed use discounts follow a similar model: resource commitments for vCPU and memory can span instance types. For GPU workloads, GCP also offers commitments tied to GPU type rather than instance type, giving teams more flexibility as new GPU generations arrive.
Savings Plans require forecasting spending instead of instance counts. Committing too little misses discount potential, while committing too much leaves unused compute credits. For teams with highly variable monthly spend, calibration may be harder than with reservations, which specify exactly what was purchased.
Effective $/GPU-hr Is What Counts
Track effective $/GPU-hr across the complete workload, rather than the listed rate for one pricing tier. This measure includes idle reserved capacity, spot interruption overhead, on-demand fallback use, and actual utilization.
A team paying $4.00/GPU-hr on-demand at 50% utilization effectively pays $8.00/GPU-hr for delivered compute. A team paying $1.20/GPU-hr on spot at 90% utilization, with a 5% interruption rate and 20-minute checkpoints, pays $1.27/GPU-hr after recomputation overhead. That is a sixfold difference, not threefold, because utilization magnifies pricing differences.
This framework remains useful across different cloud mixes and changing rates. Measure the cost per GPU-hour delivered, rather than the instance price, to clarify the right tier allocation.