AI21 moved its GPU fleet management from manual negotiation to Kueue on GKE. The cluster pools around 10,000 GPUs across multiple teams. Kueue now admits, prioritizes, and preempts workloads automatically. This replaced a manual channel where teams requested and freed capacity. The shift cut time-to-start for high-priority workloads by 83%. AI21 collaborated with Google to add Admission Fair Sharing and Topology Aware Scheduling to Kueue. These features solved fairness gaps and GPU fragmentation issues. The system now supports guaranteed, shared, and on-demand capacity pools. Workloads are routed based on labels for spend, preemptibility, and priority. The design ensures fairness, order, efficiency, and technical integrity.
What changed
GPU allocation was negotiated manually in an internal messaging channel.
What it unlocks
Automated admission, prioritization, and preemption of GPU workloads across teams.
- ~10,000 GPUs in one shared GKE cluster
- 83% reduction in time-to-start for high-priority workloads
- 8 GPUs per machine
What you need to act on it
- Kubernetes cluster with Kueue installed
- GKE environment
- GPU workloads labeled for spend, preemptibility, and priority
Sources