SkyPilot began with a simple premise: compare cloud prices, launch on the cheapest option, and move a job when a lower price appears. The prototype took a week. Production work took much longer because the difficult cases were outside that happy path: managing credentials across different IAM systems, comparing only approximately equivalent instances, handling networking assumptions in training frameworks, and keeping scheduler state consistent during migration.
This article covers the architectural choices that proved important, including our early mistakes. It is for engineers building or assessing multi-cloud orchestration layers, rather than teams seeking a quick-start guide.
Credential Management: The Hardest Quiet Problem
Cloud providers authenticate differently. AWS relies on IAM roles and instance profiles. GCP uses service accounts with JSON key files or workload identity federation. Azure provides managed identities and service principals. The differences affect token lifetimes, permission scope, credential attachment to running instances, and runtime token refresh APIs.
The key design question is where credentials reside: in the orchestrator or on compute instances. Keeping them in the SkyPilot control plane and injecting them at launch simplifies reasoning and rotation, but concentrates exposure. A compromised orchestrator could reveal every cloud credential. Provider-native credentials, such as IAM instance profiles on AWS or attached service accounts on GCP, limit access to an instance. However, the orchestrator then needs broad permission to create instances and apply particular credential bindings.
We use a hybrid approach. The orchestrator receives only the permissions needed to launch, terminate, and describe instances. At launch, it injects short-lived data credentials for S3, GCS, and Azure Blob through environment variables or mounted files, then rotates them before expiry. A compromised instance can read training data, but cannot create more instances or reach other cloud accounts.
Credential rotation across clouds becomes visible when a token expires during training. AWS temporary STS credentials last 1-12 hours, so a 24-hour run requires refresh logic. Building that path initially is cleaner than adding it after the first expiration failure at hour 13.
Instance Equivalence Is Always Approximate
The scheduler must recognize that AWS p4d.24xlarge, a GCP a2-highgpu-8g, and a Nebius 8xA100 can satisfy a request for "8x A100 80GB." They are similar, not identical. Memory bandwidth varies by hardware revision, as do NVLink layout and GPU interconnect bandwidth. CPU count, system memory, storage type, and storage throughput can also differ substantially for the same GPU count.
For most fine-tuning jobs, these differences fall within normal run-to-run variation. Workloads sensitive to GPU interconnect bandwidth, such as large all-reduce operations in multi-node distributed training, may see throughput change by 5-15% when moved between NVLink topologies. This matters less in practice because such workloads usually span multiple nodes, and migrating multiple nodes is more involved than moving one.
Maintain an equivalence table that maps requirements such as GPU family, GPU count, minimum GPU memory, and minimum system RAM to provider-specific instance types. Job authors should also be able to mark a job "interconnect-sensitive," telling the scheduler to avoid moving it to different hardware topology during a run.
Networking: Where Assumptions Break
Training frameworks often assume a particular network layout. Those assumptions can remain hidden in one cloud and fail as soon as a job spans providers.
NCCL initialization is a frequent problem. NCCL finds peer GPU nodes through IP addresses, which works automatically inside one VPC. Across VPCs on different providers, it needs explicit rendezvous settings: workers must receive the head node's reachable IP, and that address must work across the cloud boundary.
Single-node jobs, with every GPU on one instance, avoid this issue. Multi-node jobs can use a VPN or tunnel between clouds, adding latency and cost; a shared rendezvous service, adding complexity; or one-cloud placement for multi-node jobs while routing single-node jobs across clouds. The last option is the pragmatic default until a workload clearly benefits from cross-cloud multi-node execution. For most jobs, the networking complexity costs more than the gain.
Data locality creates another networking problem. If a dataset sits in S3 us-east-1 while the job runs on GCP us-central1, every batch crosses clouds. S3 egress costs $0.09/GB for the first 10TB. A pretraining job reading 10TB per epoch can lose its spot-routing savings to transfer charges. Placement should consider dataset location and favor the same cloud when egress exceeds the spot discount or cross-cloud I/O risks becoming the throughput bottleneck.
Keeping Scheduler State Consistent
During a move from provider A to provider B, the scheduler must know the stopped step, checkpoint location, new instance address, and credentials required by that instance. Stale or conflicting values can prevent restart or make the job resume from the wrong point.
Job state must not depend on one cloud's control plane. SkyPilot keeps a record in its control plane rather than AWS CloudWatch or GCP Logging. The record includes the current step, last checkpoint path, instance ID and cloud, submission parameters, and migration history. Checkpoint writes and state updates must be atomic, avoiding a period when a checkpoint exists but the scheduler has not recorded it.
Our early failure looked like this: an AWS job wrote a checkpoint to S3, then the instance was reclaimed before the scheduler finished updating its record. On restart, the scheduler saw the old checkpoint and repeated N completed steps. That was wasteful but harmless. We fixed it by storing the checkpoint path inside the checkpoint file, making it self-describing, then checking that file against the state record on resume. If they differ, the file's step counter wins.
The Spot Decision Loop
The orchestrator must repeatedly assess whether the current placement remains worthwhile or a cheaper option has appeared. It polls spot prices across configured providers and regions, then compares the alternatives with current running cost and migration cost.
Migration cost includes more than checkpoint time. A spot instance under heavy demand may take 2-10 minutes to acquire. The total also includes transferring local state, setting up the environment, pulling the Docker image, writing the checkpoint, and the gap between stopping the old instance and resuming training. With a 20-minute migration cost, a 5% price improvement is not enough for a job lasting 4-24 hours, since overhead consumes much of the saving.
One useful heuristic is to migrate only when the price gap represents at least 30 minutes of equivalent compute over the expected remaining runtime. That reduces thrashing when prices are close while preserving worthwhile savings when a substantially cheaper option appears.
What We Learned the Hard Way
Several early choices failed. We first kept spot-price history in orchestrator memory, leaving the scheduler without decision data after a restart. Price history must be persistent. We also selected the single cheapest instance globally, which caused simultaneous submissions to converge on one option, producing failed spot requests and fallbacks. A ranked set of acceptable options works better, with choices spread according to demand. Finally, we treated unavailable instances as a rare launch error. Availability checks belong in placement itself, because an instance can be absent rather than merely expensive.
Multi-cloud orchestration does not require unusual algorithms. Its difficulty comes from combining independent systems, including IAM, networking, storage, spot markets, and instance availability, each with distinct failure modes. A useful orchestrator must handle their combinations gracefully. The important choices concern state ownership, failure propagation, and which assumptions become fixed behavior instead of configuration.