Using one cloud for ML training means accepting that provider's spot pricing, capacity, and instance quotas. With multiple clouds, a costly or unavailable AWS p4d job can move to GCP A2, while limited GCP capacity in us-central1 can shift the job to us-east1 or AWS. Comparing providers opens a broader spot market.
The initial setup takes several steps: configure credentials for both providers, define a portable training job, and create an image that runs on each. This guide covers the process from start to finish. It assumes Python 3.9+ and pip are installed, SkyPilot is not configured, and AWS and GCP accounts have GPU quota.
This guide uses AWS and GCP. Azure follows the same general pattern, but service principals replace IAM roles and create a third credential path. Start with two providers, then add the third.
Step 1: Configure AWS
SkyPilot follows the AWS CLI credential chain. For a quick start, create an IAM user with programmatic access and use ~/.aws/credentials. IAM roles with short-lived credentials are safer for production, but the credentials file works for initial setup.
In the AWS Console, open IAM > Users > Add user and enable programmatic access. Attach AmazonEC2FullAccess and IAMReadOnlyAccess, then download the access key ID and secret.
# Install AWS CLI if not already present
pip install awscli
# Configure credentials
aws configure
# AWS Access Key ID: <your key id>
# AWS Secret Access Key: <your secret>
# Default region name: us-east-1
# Default output format: json
Confirm the credentials work:
aws ec2 describe-instances --region us-east-1
# Should return an InstanceReservations list (possibly empty if no instances running)
Review GPU quota before launching. In the AWS Console, open EC2 > Limits and search for "p4d" and "p5". A new account may have a p4d.24xlarge limit of 0 or 8 vCPUs, which may not support a full instance. Request more quota for the instance families you need. AWS GPU quota requests typically take 1-3 business days.
Step 2: Configure GCP
GCP accepts service account credentials or application default credentials (ADC). For SkyPilot, a JSON key for a service account is the most portable choice.
In the GCP Console, open IAM & Admin > Service Accounts > Create Service Account. Use a descriptive name, such as skypilot-training-sa, and grant Compute Admin, Service Account User, and Storage Admin for checkpoint writes to GCS.
# Install Google Cloud CLI if not already present
# Instructions at cloud.google.com/sdk/docs/install
# Authenticate with your Google account (needed to create the key)
gcloud auth login
# Create a service account key (replace PROJECT_ID and SA_EMAIL)
gcloud iam service-accounts keys create ~/skypilot-gcp-key.json \
--iam-account=skypilot-training-sa@PROJECT_ID.iam.gserviceaccount.com
# Set the environment variable SkyPilot will use
export GOOGLE_APPLICATION_CREDENTIALS=~/skypilot-gcp-key.json
Place the export line in ~/.bashrc or ~/.zshrc to retain it between sessions. Then verify GCP access:
gcloud compute instances list --project=PROJECT_ID
# Should return an empty list if no instances exist
Review GCP GPU quota under IAM & Admin > Quotas. Filter for "NVIDIA_A100_GPUS" in your target regions, with us-central1 often offering the most capacity. Request additional GPU quota; these requests can take several days.
Step 3: Install SkyPilot
pip install skypilot[aws,gcp]
# Verify installation and credential detection
sky check
Run sky check to confirm authentication for each configured provider and view the available instance families. A successful result looks like this:
Checking credentials to enable clouds for SkyPilot.
AWS: enabled
GCP: enabled
SkyPilot will use the above clouds.
If a provider is marked "disabled," the output identifies the missing credential or permission. Resolve those errors before continuing.
Step 4: Create a Cross-Cloud Image
Each provider may use a different VM image, but a container keeps the training environment consistent across them. Containerizing the code is the cleanest way to keep the job portable.
A minimal PyTorch Dockerfile:
FROM nvcr.io/nvidia/pytorch:24.01-py3
WORKDIR /workspace
# Install your requirements
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy training code
COPY train.py .
COPY dataset_utils.py .
# Default entrypoint for SkyPilot jobs
ENTRYPOINT ["python", "train.py"]
Build the image and push it to a registry reachable from both providers. Docker Hub is the simplest choice. AWS ECR and GCP Artifact Registry support private images but need extra authentication in the job spec.
docker build -t your-dockerhub-user/ml-training:latest .
docker push your-dockerhub-user/ml-training:latest
Containers are optional. SkyPilot can install dependencies on each VM with a setup script. Containers improve reproducibility and launch speed because setup happens during image creation, not at every job start, though either method works.
Step 5: Define a Multi-Cloud Job
SkyPilot YAML job specs declare resource needs and the training command. For portability, request accelerator capacity rather than a fixed instance type, allowing SkyPilot to find the cheapest available option across providers.
# train-job.yaml
name: finetune-llm
resources:
accelerators: A100:8
use_spot: true
any_of:
- cloud: aws
region: us-east-1
- cloud: aws
region: us-west-2
- cloud: gcp
region: us-central1
file_mounts:
/data:
source: s3://your-training-data-bucket/dataset/
mode: COPY
setup: |
# Runs once at VM startup if not using a container
pip install -r /workspace/requirements.txt
run: |
python /workspace/train.py \
--data-dir /data \
--output-dir /checkpoints \
--batch-size 32 \
--num-epochs 5
The line accelerators: A100:8 requests "8 A100 GPUs" without naming a specific instance. SkyPilot maps that request to p4d.24xlarge on AWS or a2-highgpu-8g on GCP. The any_of block orders acceptable provider and region combinations and enables spot. SkyPilot compares current spot availability and prices before choosing an option.
Step 6: Make Data Available
Training data must be reachable from both AWS and GCP instances. The main choices are:
Copy at launch: When the dataset fits on local disk, file_mounts with mode: COPY copies data from S3 or GCS to the VM at startup. It is simple and avoids ongoing egress during training, but larger datasets increase startup time.
Mount with s5cmd or gcsfuse: Mounting object storage as a filesystem avoids copying the full dataset. Use gcsfuse for GCS, or s3fs and goofys for S3. Access is slower than local NVMe but can work when each sample is read once per epoch.
Store copies in both providers: Teams that run jobs regularly on AWS and GCP can keep training data in both S3 and GCS. Duplicate storage often costs less than repeated cross-cloud egress for large datasets used across many jobs.
Cross-cloud egress has a cost and can matter for large datasets. The right choice depends on dataset size, access frequency, and whether simple startup or lower transfer cost matters more. Profile the real access pattern before optimizing.
Step 7: Submit and Observe
# Submit the job
sky launch train-job.yaml
# Check status
sky status
# Stream logs
sky logs my-cluster-name
# SSH into the running instance if needed
sky ssh my-cluster-name
At first launch, SkyPilot reports the selected provider and region, along with the spot price and availability checks. For an 8xA100 spot request, it typically checks two to four provider and region combinations before choosing one.
After completion or spot preemption, sky status reports the job state. With checkpoint and resume logic, you can resubmit after preemption and continue from the latest checkpoint. Without it, resubmission starts over.
Limits of This Setup
Multi-node jobs, meaning more than one instance running together, require NCCL rendezvous and additional networking work beyond this single-node guide. Every rank must reach a shared rendezvous address at startup, requiring cross-cloud VPC connectivity or a single provider's VPC. One distributed run cannot place nodes across clouds without that connectivity.
Azure adds a third credential route through service principals with az login and az configure, so handle it separately from AWS and GCP. After credentials are ready, the job spec syntax is the same.
The first multi-cloud submission is usually the most difficult. Credential failures, quota limits, and VPC issues commonly appear then rather than during routine use. Set aside 30-60 minutes for initial debugging; later cross-provider submissions should be routine.