A team paying $40,000/month for GPU compute at an average 45% utilization is effectively paying $22,000/month for capacity doing no training. That pattern is common. Many workloads start around 40-60% utilization, but the causes are usually diagnosable and fixable.
GPU utilization tells you that a problem exists, not which problem. A 45% reading may indicate that the GPU waits half the time for data, I/O, or CPU preprocessing. It may mean the GPU is active but underfilled because the batch is too small or tensor operations miss tensor-core tile sizes. Or 45% may be the genuine compute limit, with no meaningful overhead to remove. Each case needs a different remedy, so instrumentation must separate them.
What nvidia-smi Actually Measures
The utilization reported by nvidia-smi, and by its NVML foundation, is not the share of tensor cores running. It is the portion of a sampling window in which at least one CUDA kernel executes. A GPU running one small kernel for 1 microsecond every 10 microseconds reports 10%, whether that kernel uses all available resources or only a few SM cores.
This distinction matters. A 90% reading can still hide major tensor-core underuse when kernels are memory-bound or poorly matched to tensor-core requirements. Conversely, 70% may deliver nearly all useful compute when the remaining 30% consists of unavoidable pipeline stalls.
Metrics that explain more than raw utilization:
- SM (Streaming Multiprocessor) occupancy: what fraction of the SM's available warps are active. Low SM occupancy often indicates kernels with too-small thread blocks or insufficient parallelism.
- Memory bandwidth utilization: what fraction of the GPU's peak memory bandwidth is being used. High memory bandwidth with low compute utilization indicates a memory-bound operation.
- Tensor core utilization: only available via DCGM (Data Center GPU Manager), not nvidia-smi. This is the most direct measure of how effectively the GPU is performing matrix operations.
Add dcgm-exporter Instrumentation
DCGM (Data Center GPU Manager) exposes deeper GPU metrics unavailable through nvidia-smi. When the DCGM toolkit is installed, as is standard on AWS p4d/p5 and GCP A2/A3 instances, dcgm-exporter exports those metrics in Prometheus format.
# Install dcgm-exporter (example for Ubuntu)
# DCGM is typically pre-installed on GPU cloud instances
# Start dcgm-exporter
dcgm-exporter &
# Verify it is running and exporting metrics
curl localhost:9400/metrics | grep DCGM_FI
Track these DCGM metrics for each training run:
DCGM_FI_DEV_GPU_UTIL # GPU utilization (same as nvidia-smi)
DCGM_FI_DEV_MEM_COPY_UTIL # Memory transfer utilization
DCGM_FI_DEV_TENSOR_ACTIVE # Tensor core active fraction (key metric)
DCGM_FI_DEV_SM_ACTIVE # SM active fraction
DCGM_FI_DEV_FB_USED # GPU memory used (MiB)
DCGM_FI_DEV_POWER_USAGE # GPU power draw (Watts)
In a well-tuned transformer run, DCGM_FI_DEV_TENSOR_ACTIVE should exceed 50-60%, while DCGM_FI_DEV_SM_ACTIVE should exceed 70%. Much lower readings indicate optimization opportunities. Readings near those levels generally indicate a reasonably tuned workload.
Three Main Causes of Low Utilization
Most underused GPUs fit three patterns. Finding the pattern points to the appropriate fix.
I/O-Bound: Slow Data Loading
Symptoms: utilization repeatedly rises to 90-100% and then falls near zero, in a rhythm matching batch loads. The GPU is waiting while the CPU prepares the next batch.
Diagnosis: time the data-loading and forward-pass sections:
import time
import torch
data_load_times = []
compute_times = []
for batch_idx, batch in enumerate(dataloader):
t0 = time.perf_counter()
batch = batch.to(device)
t1 = time.perf_counter()
outputs = model(batch['input_ids'])
loss = criterion(outputs, batch['labels'])
loss.backward()
optimizer.step()
optimizer.zero_grad()
t2 = time.perf_counter()
data_load_times.append(t1 - t0)
compute_times.append(t2 - t1)
print(f"Avg data load: {sum(data_load_times)/len(data_load_times)*1000:.1f}ms")
print(f"Avg compute: {sum(compute_times)/len(compute_times)*1000:.1f}ms")
When data loading exceeds 20% of compute time, the workload is I/O-bound. Increase DataLoader(num_workers=) to typically 4-8 workers per GPU, set pin_memory=True for faster CPU-to-GPU transfers, prefetch with prefetch_factor, or preprocess and cache data on local NVMe instead of reading network storage each epoch.
Memory-Bound: An Undersized Batch
Symptoms: utilization remains at 40-60%, memory bandwidth is high, and tensor-core activity is low. More time goes to moving data than computing.
NVIDIA tensor cores handle matrix multiplication in transformer attention and MLP layers and have efficient minimum tile sizes. On V100, A100, and H100, fp16/bf16 dimensions work best as multiples of 128, while fp32 favors multiples of 64. If effective batch size, batch size x sequence length x hidden dimension, produces misaligned matrices, tensor-core utilization falls.
Practical fix: raise the batch size until memory bandwidth and compute activity balance. When model size limits fine-tuning batches, use gradient accumulation to raise effective batch size within memory limits. A batch of 4 with 8 accumulation steps is not equivalent to batch 32 for tensor-core use because each step still has size 4, but it uses less memory than native batch 32 when that causes OOM.
Suboptimal Compute Throughput: Fusion and Precision
Symptoms: GPU and tensor-core utilization are high, yet tokens-per-second or samples-per-second trails the hardware's theoretical maximum. The GPU is busy, but inefficient operations consume that time.
Two frequent causes exist. First, fp32 may be used where bf16 or fp16 is suitable: A100 tensor cores deliver 312 TFLOPS in bf16 versus 77 TFLOPS in fp32. Training with fp32 weights or attention leaves 4x compute throughput unused. Wrap the forward and backward pass in torch.autocast(device_type='cuda', dtype=torch.bfloat16).
Second, standard PyTorch attention may be used instead of FlashAttention. Standard torch.nn.functional.scaled_dot_product_attention sends the attention matrix through GPU memory, while FlashAttention processes tiles in SRAM without materializing the full matrix. For sequences longer than 512 tokens, FlashAttention is significantly faster. With suitable hardware, A100/H100, and dtype, fp16/bf16, it is available as torch.nn.functional.scaled_dot_product_attention in PyTorch 2.0+.
Attribute Each Run in Your Tracker
GPU metrics help when tied to experiments rather than only to instances. Review experiment history to find runs with low utilization, then compare them with training-code changes, configuration choices, and NCCL communication.
With Weights & Biases or MLflow, record GPU metrics beside training metrics:
import wandb
import pynvml
pynvml.nvmlInit()
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
def log_gpu_metrics(step):
utilization = pynvml.nvmlDeviceGetUtilizationRates(handle)
memory_info = pynvml.nvmlDeviceGetMemoryInfo(handle)
wandb.log({
"gpu/utilization_percent": utilization.gpu,
"gpu/memory_used_mib": memory_info.used // (1024 * 1024),
"gpu/memory_total_mib": memory_info.total // (1024 * 1024),
}, step=step)
Call log_gpu_metrics every N steps. Every 50-100 steps is enough, while every step adds overhead. Over time, the per-run history reveals regressions, such as utilization falling from 80% to 50%, and improvements.
An Honest Baseline
Not every workload reaches 90%+ utilization. Sparse models, mixture-of-experts with uneven routing, and some RL fine-tuning loops have less predictable utilization than dense transformer training. Pipelines combining CPU-heavy preprocessing with GPU work, including some multimodal training and online augmentation, also have lower averages by design.
For dense transformer fine-tuning on well-tuned hardware, a reasonable target is 70-80% GPU utilization from nvidia-smi, with tensor-core utilization above 50% from DCGM. Runs consistently below 50% merit investigation. Runs at 70-80% are probably near the practical optimum without architectural changes. Reaching 95%+ may require hardware-specific kernel tuning, often not worth the engineering cost for most teams.
Setup takes modest engineering time and can pay back quickly. Raising one job category from 40% to 75% utilization increases compute value by 87% without extra spending.