Kubernetes was not designed for GPUs. It was designed for stateless HTTP services running on commodity compute — the canonical cloud-native workload of the mid-2010s. A GPU was, to the original scheduler, an opaque device count: a pod could request one nvidia.com/gpu resource, and the scheduler would find a node with at least one available and place the pod there. That was the full extent of the system’s knowledge about the hardware. No topology, no partitioning, no understanding of the difference between a training job that needs eight tightly coupled GPUs and an inference process that can run on a fraction of one.

The gap between this model and what AI infrastructure actually requires has driven, over the past several years, a significant body of engineering work — most of it rooted in scheduling problems that distributed-systems research has studied for decades. The result is an evolving stack of Kubernetes extensions that bring the orchestrator substantially closer to what the underlying hardware demands.

Why GPU Scheduling Is Structurally Different

Scheduling general-purpose compute is a bin-packing problem with relatively forgiving constraints. A CPU pod requests some number of cores and some amount of memory; the scheduler finds a node with sufficient headroom; the pod runs. If a node fails, the pod is rescheduled elsewhere. The workload is usually stateless, and the scheduler does not need to know whether two pods communicate or whether their relative placement on the network affects throughput.

GPU workloads break these assumptions in several ways.

Distributed training is the clearest case. A large language model training run might require 64 or 256 GPUs operating as a single synchronized parallel computation. All of the participating processes — one per GPU, typically — must start simultaneously or the job cannot proceed. This is the gang scheduling problem, which batch and grid computing systems addressed in academic research going back to the 1990s: a job consisting of multiple coordinated tasks that must be co-scheduled, not individually placed. If even one of the 256 requested GPUs is unavailable, the entire job must wait. A scheduler that places tasks individually, without awareness of the gang constraint, will partially schedule the job, hold the placed GPUs idle, and prevent other jobs from using them — the worst possible outcome.

Topology matters in a way it rarely does for CPU workloads. Two GPUs on the same physical host, connected via NVLink, communicate at hundreds of gigabytes per second. Two GPUs on different hosts, connected via InfiniBand, communicate at a fraction of that. Two GPUs that must cross a network switch — slower still. For data-parallel training, where gradient synchronization between GPUs accounts for a significant fraction of wall-clock time, the difference between topology-aware and topology-oblivious placement can be measured in training time and, at scale, in substantial cost.

Finally, GPU utilization under naive scheduling is poor. Production deployments without careful scheduling have observed GPU utilization rates around 13%: a GPU nominally assigned to a workload may be idle for most of the time while the workload does CPU-side preprocessing, data loading, or waits for the next mini-batch. A resource worth tens of thousands of dollars sits effectively unused because the scheduler has no mechanism for sharing it across workloads.

The DRA Architecture: From Count to Attribute

The Kubernetes API that exposed GPU resources for most of the project’s history — the extended resource model introduced around Kubernetes 1.11 — was a count. A node had N GPUs; a pod requested some integer number; the scheduler subtracted. No awareness of GPU model, memory capacity, interconnect topology, or partitioning state.

Dynamic Resource Allocation (DRA), developed through the Kubernetes SIG-Node and SIG-Scheduling working groups and reaching general availability in Kubernetes 1.34, replaces the count model with an attribute-based claim system. A workload declares a ResourceClaim: it needs a GPU with at least 40 gigabytes of memory, or a set of GPUs connected via NVLink, or a fractional instance of a specific GPU model. Device drivers — implemented by GPU vendors and contributed to the Kubernetes ecosystem — expose their hardware’s attributes into the ResourceClaim API. The scheduler gains a global view of actual device characteristics and cluster topology, enabling it to evaluate claims against available hardware rather than subtracting from an opaque count.

IBM Research, which has published applied work on DRA-based just-in-time GPU slicing, has framed the model’s ambition as extending well beyond GPUs — the ResourceClaim architecture is designed to generalize to FPGAs, network interface cards, and other specialized accelerators, not only to graphics processors. At KubeCon Europe 2026, NVIDIA donated its DRA driver implementation to the Cloud Native Computing Foundation, signaling the vendor’s commitment to a community-governed resource allocation framework rather than proprietary scheduling extensions.

Red Hat shipped DRA as generally available in Red Hat OpenShift 4.21, and Google Cloud published integration guidance for DRA-based GPU scheduling in GKE. The feature’s progression from alpha to GA across Kubernetes 1.26 through 1.34 reflects both the technical complexity of the problem and the care required to extend a stable API surface that large-scale production deployments depend on.

Gang Scheduling and the KAI Scheduler

For the gang scheduling problem, NVIDIA open-sourced the KAI Scheduler in 2025 — a rebranded and donated version of the Run:ai scheduler the company acquired. KAI runs alongside the default kube-scheduler as a secondary scheduler: the default scheduler handles CPU-bound and general workloads; KAI handles AI batch jobs that require coordinated GPU allocation.

KAI’s core scheduling guarantee is atomicity for gang workloads. A distributed training job specifying eight pods will not partially schedule: either all eight are placed simultaneously, or none are placed. This eliminates the GPU idle-hold problem that naive scheduling produces. The scheduler also implements fair-share queuing across teams and projects — multiple teams submitting training jobs share the cluster’s GPU capacity according to configured quotas, with preemption and priority mechanisms that prevent a single large job from monopolizing resources indefinitely.

Topology-aware placement is a first-class feature: KAI can be configured to prefer placing gang members on nodes connected via NVLink, or within the same InfiniBand fabric group. Production case studies from CNCF member organizations have documented GPU utilization improvements from approximately 13% to 37% when moving from naive scheduling to gang-aware, topology-conscious placement — nearly tripling efficiency. Some implementations with additional tuning have reported utilization above 80%.

KAI integrates with KubeRay, the Kubernetes operator for Ray clusters, which is the dominant framework for distributed training and large-scale inference. Through this integration, a Ray cluster’s resource requests flow through KAI’s gang scheduling logic, with DRA providing the underlying device attribute awareness.

Partitioning: MIG, Time-Slicing, and Fractional Allocation

Not every workload needs a full GPU. An inference server handling real-time API requests may require stable, isolated compute but nothing close to the full memory and compute capacity of a modern accelerator. Running one inference process per GPU is expensive and underutilizes the hardware.

NVIDIA’s Multi-Instance GPU (MIG) feature, available on Ampere and newer architectures, partitions a physical GPU into isolated instances at the hardware level. Each MIG instance has dedicated compute cores, memory, and cache; workloads running on separate MIG instances cannot interfere with each other’s performance or access each other’s memory. This is hardware-enforced isolation, not software scheduling — appropriate for production inference with strict SLA requirements.

Time-slicing is a softer sharing mechanism: the GPU’s processing time is divided among multiple processes through rapid context switching. Latency jitter is higher than with MIG, and there is no memory isolation, but time-slicing works on older hardware that does not support MIG and is adequate for development workloads and batch inference where occasional latency spikes are acceptable.

DRA ties these partitioning mechanisms into the Kubernetes scheduling layer: a pod can claim a specific MIG partition profile, and the scheduler will place it on a node where that profile is available, without requiring the operator to manually configure static partitions across the cluster.

Continuity with Prior Research

What is now being built into Kubernetes production AI infrastructure — gang scheduling, topology-aware placement, multi-tenant fair-share queuing, fractional resource allocation — maps directly onto problems that batch and cluster-computing research addressed in the grid-computing era. The Maui Cluster Scheduler, PBS Pro, LSF, and the scheduling literature of the 2000s all engaged with variants of the same questions: how do you schedule coordinated multi-node jobs across a shared resource pool, maintain fairness among competing users, and avoid the pathological idle-resource states that naive placement produces?

The vocabulary has shifted — pods instead of jobs, nodes instead of grid resources, Gang PodGroups instead of parallel job arrays — but the structural problems are the same. What Kubernetes brings that earlier grid systems did not is a general, vendor-neutral, declarative API surface over which scheduling logic can be expressed, extended, and composed. The Resource Claim and DRA model is, in this light, a generalization of the resource-description frameworks that grid middleware attempted: a way for workloads to describe what hardware they need, and for the system to match that description against available resources across a shared pool. We trace this specific lineage — from the Globus Toolkit and the DRMAA standard through the metascheduler model — in more depth in our commentary on grid computing’s research thread into modern resource orchestration.

IDC’s analysis placed global AI infrastructure spending at $82 billion in a single quarter in mid-2025, growing at 166% year-over-year. That spending increasingly flows through specialized GPU providers as well as the hyperscalers — the neocloud model this site has covered separately — which makes scheduling efficiency a shared concern across a more fragmented set of infrastructure operators, not just an internal hyperscaler optimization. At that scale of investment, the scheduling and utilization efficiency of the infrastructure matters enormously — a 2x improvement in GPU utilization across a large cluster is a 2x reduction in hardware spend for the same computational throughput. The research problems are not academic; they are directly load-bearing in the economics of AI development.

The State of the Stack

The current production stack for Kubernetes-based AI infrastructure runs roughly as follows: DRA provides the device-attribute API and topology visibility; KAI (or similar gang-aware secondary schedulers like Volcano) provides coordinated multi-pod placement and fair-share queuing; KubeRay or similar distributed training operators coordinate the framework-level cluster lifecycle; MIG partitioning handles the inference isolation problem; and DRA-aware fractional allocation handles intermediate cases.

This is a functional but still-maturing stack. Debugging distributed training failures in Kubernetes — diagnosing why a 256-GPU gang job is failing to schedule, or why cross-rack communication is slower than expected — requires tooling that does not exist in mature form. The operational complexity is real, and the managed Kubernetes AI services from major cloud providers are, in part, an answer to it: GKE, AKS, and EKS all offer managed GPU node pools with pre-configured scheduling extensions that reduce the operational surface for teams that do not need to own the full stack.

The trajectory is clear. Kubernetes became the default substrate for containerized services over a period of roughly five years. It is now becoming the default substrate for GPU-based AI infrastructure over a similar period, driven by the same forces: vendor-neutral governance through CNCF, a composable extension model, and a community large enough to solve the hard operational problems in the open.

Frequently Asked Questions

Why can’t Kubernetes just treat GPUs like CPUs for scheduling?

CPU scheduling is a forgiving bin-packing problem — pods request cores and memory, and the scheduler finds headroom. GPU workloads need gang scheduling (all-or-nothing placement for coordinated multi-GPU jobs), topology awareness (NVLink versus InfiniBand versus cross-switch bandwidth), and attribute-based matching (memory size, interconnect, partition profile) that a simple integer device count cannot express.

What is Dynamic Resource Allocation (DRA) in Kubernetes?

DRA is a Kubernetes API, reaching general availability in Kubernetes 1.34, that replaces the old integer GPU-count model with an attribute-based ResourceClaim system. A workload declares what it needs — GPU memory, interconnect, a specific partition profile — and vendor-supplied device drivers expose real hardware attributes for the scheduler to match against.

What is gang scheduling and why does AI training need it?

Gang scheduling requires that all processes in a coordinated job — for example, all 256 GPUs in a distributed training run — start simultaneously, or none do. Without it, a scheduler can partially place a job, holding GPUs idle while waiting for the rest, which blocks other workloads from using that capacity. NVIDIA’s KAI Scheduler and Volcano both implement this as an atomic placement guarantee.

How much can better GPU scheduling actually improve utilization?

Production case studies from CNCF member organizations have documented GPU utilization rising from roughly 13% under naive scheduling to 37% with gang-aware, topology-conscious placement — nearly tripling efficiency — with some further-tuned deployments reporting above 80%. At AI-infrastructure spending levels in the tens of billions per quarter, that efficiency gap is a direct, large-scale hardware-cost question.

Does Kubernetes GPU scheduling have any connection to older grid-computing research?

Yes — gang scheduling, topology-aware placement, and fair-share queuing all map onto problems that batch and grid-computing schedulers (Maui, PBS Pro, LSF) studied starting in the 1990s and 2000s. Kubernetes’ contribution is less a new scheduling theory than a general, vendor-neutral, declarative API surface for expressing it at much larger scale.

Further Reading

Back to Blog