Five ML engineers sharing GPUs will eventually clash over scheduling. One person's 48-hour run can delay another's 2-hour test, while 12 weekend submissions consume capacity needed by three colleagues on Monday. These disputes are not personal. They are the expected result of a queue without stated rules.
Queue rules shape GPU utilization and perceived fairness: are GPUs idle while work waits, and does everyone accept the allocation? A policy for three engineers differs from one for ten, and neither fits thirty. This guide compares workable patterns at each size and the tradeoffs to consider before implementation.
FIFO: Simple, Fair, and Wrong at Scale
First in, first out is what systems use when no other rule exists. The model is easy to explain: process jobs in submission order. For ML teams larger than a handful of engineers, that simplicity often creates poor results.
The issue is uneven job length. ML queues may contain 15-minute tuning tests alongside 72-hour fine-tuning runs. With FIFO, a 15-minute job submitted behind a 48-hour run waits 48 hours. One person's long task can therefore slow everyone's iteration. Where fast research feedback matters, that delay carries a real cost.
FIFO is usually adequate for 2-3 engineers with similar workloads and enough GPUs to make waiting uncommon. When any of those assumptions fail, friction follows.
Priority Queuing: Better, With Consistent Definitions
Priority queues give every job a level and run higher levels first. Building one is easy. Agreement on what qualifies as high priority, followed by consistent use, is the difficult part.
Priority models worth considering:
Release-blocking versus exploratory: Use two levels. Mark work blocking a deliverable, such as a model release, demo, or paper deadline, as high priority. Keep exploration normal. The rule is easy to review and helps prevent every job from becoming urgent.
Runtime tiers: Group jobs by expected duration: under 1 hour, 1-24 hours, or over 24 hours. Run fast work ahead of background work, regardless of submission order. This reduces FIFO blocking without asking engineers to make subjective priority calls.
Avoid self-assigned priority with no review. Within two weeks, every job can become high priority. Showing each submitter, job, and selected level gives the team a useful check against that drift.
Preemption: Use This Tool Carefully
Preemption pauses a lower-priority job so a higher-priority one can use its GPU. Save the displaced job at a checkpoint, then place it back in line for later resumption.
Preemption can help, but misuse causes disruption. A long fine-tuning run may be interrupted by a short test, then interrupted again during recovery. Checkpoint and resume work can consume a large share of elapsed time. Excessive preemption reduces delivered GPU compute and erodes confidence in the queue.
Use a cautious rule: preempt only when the waiting job has waited beyond X minutes, the active job has run beyond Y minutes and has a recent checkpoint, and the priority gap is greater than one tier. Never interrupt a job running less than 15 minutes, even for a newly arrived higher-priority task. Let it finish, since the savings seldom repay the overhead and disruption.
Every training job in a preemptible queue needs reliable checkpoint and resume support. Jobs that cannot recover from interruption may be corrupted. Requiring checkpoint readiness before joining the shared queue is reasonable and encourages the needed behavior.
Per-User Budget Caps
A budget cap limits each user's GPU-hour spending for a day, week, or month. Submissions beyond that limit wait for capacity or until the cap resets.
Caps mainly create awareness. With a finite allowance, engineers tend to remove duplicate runs, choose hyperparameters more carefully, and test small pilots before launching full jobs. The important message is that compute is shared, limited, and costly for each experiment.
Soft caps are generally preferable to hard caps. A soft cap warns the user and asks for an explicit override, while a hard cap blocks submission. Hard blocks encourage workarounds, such as alternate identifiers, reset waiting, or combining many small jobs into fewer large ones. Soft limits plus visible attribution produce much of the behavior change without inviting those tactics.
Cost Visibility: The Highest-Impact Intervention
Before adding complicated queue rules, show each engineer the live cost of every run and identify who submitted it.
A report saying "your 17 jobs last week cost $840 in GPU compute" can change habits more consistently than queue rules. Engineers may find that familiar experiment designs cost 10x more than alternatives. A search on 8-GPU instances costs 8x as much as the same search on 1-GPU instances when wall time matches. The difference is easy to miss without clear reporting.
Cost tracking does not need an advanced scheduler. Recording instance type, billed hours, and submitter ID can provide useful visibility. The harder tasks are accurate allocation, such as dividing a shared instance among jobs, and useful follow-through, such as connecting costs to job settings. Ship the basic report before adding advanced features.
Queue Policy at Different Team Scales
Three to five engineers with similar workloads: use FIFO plus duration priority for quick, blocking tests. Avoid preemption unless GPUs are critically scarce. Add soft budget caps and weekly cost reports.
Six to fifteen engineers with varied work, including research, production pipelines, and evaluation: use two priority levels, release-blocking and exploratory, with runtime tiers inside each. Permit cautious preemption for release-blocking jobs. Add per-user soft caps with visible attribution, and reserve separate capacity for scheduled production pipelines.
Beyond fifteen engineers, fair-share scheduling can justify its added complexity. Each user receives an equal long-term share, while unused capacity can move to others. The scheduler must record historical use and adjust allocations over time, which calls for real infrastructure work. At this size, an ML platform team often owns the scheduler instead of each team building its own.
Queue rules cannot fix a lasting GPU shortage. When demand stays above supply, waiting remains a problem under any policy. Rules can make contention fairer and use capacity more efficiently, but they cannot create missing GPU hours. The remaining options are additional capacity, spot routing for lower-cost capacity, or designs that need less compute per experiment.