Kubernetes was designed to place self-contained, restartable pods one at a time. A large language model training job is the opposite: thousands of accelerators that must start together, communicate over a topology-sensitive fabric, and stop when any one of them fails. Most of the friction between Kubernetes and LLM workloads comes from that mismatch, and the research literature on GPU cluster scheduling has been describing it for years before the Kubernetes project began closing it.

This survey sorts the literature and the current Kubernetes ecosystem by the problem each piece addresses, then marks where gaps remain. It draws only on published papers and project documentation, and it names the limits of the evidence where they exist.

What the research schedulers established

Three papers set the vocabulary. Gandiva: Introspective Cluster Scheduling for Deep Learning (OSDI 2018) observed that deep learning jobs are predictable from one iteration to the next, then used that to time-slice GPUs across jobs and migrate jobs to better-fitting hardware. Its authors reported a 26 percent improvement in aggregate utilization on a real 180-GPU cluster workload. Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning (OSDI 2021) argued that allocating GPUs and tuning training parameters such as batch size and learning rate separately leaves performance on the table, so a scheduler should adapt both together.

The common thread is that the scheduler needs knowledge of the workload’s structure. A generic bin-packer sees a job as a request for N units of a resource. These systems see it as an iterative computation whose throughput depends on placement, parallelism and hyperparameters.

LLM training sharpened the problem in a different direction: failure. The NSDI 2024 paper Characterization of Large Language Model Development in the Datacenter analyzed six months of workload traces from the Acme GPU datacenter and reported that LLM development differs from earlier deep learning workloads in resource-utilization patterns and in the impact of job failures. It proposed fault-tolerant pretraining with automatic recovery and a decoupled scheduling approach for evaluation jobs. Meta’s Llama 3 report gives a sense of scale: during a 54-day run on 16,384 H100 GPUs, the team recorded 419 unexpected interruptions, roughly one every three hours, with more than half tied to GPUs or their HBM3 memory, while keeping effective training time above 90 percent. A scheduler for this class of job is part resource allocator and part recovery system.

Where Kubernetes stands

Gang scheduling

The default kube-scheduler places pods individually, so a 64-pod job can end up with 63 pods running and one pending, holding 63 GPUs while doing nothing. The ecosystem answered with add-on systems. Volcano, a CNCF project, provides PodGroups with a minimum member count and queue-based fair share with preemption. Kueue, a Kubernetes SIG project, handles quota and admission, and provides gang admission, but by the description in comparative write-ups it does not itself perform pod-level placement or fabric-aware placement.

Core Kubernetes is now absorbing the idea. The Workload API arrived in v1.35, v1.36 split it into Workload and PodGroup APIs under scheduling.k8s.io/v1alpha2, and the project’s own v1.37 announcement says the core Workload and PodGroup APIs, Workload-Aware Preemption and shared DRA ResourceClaims for PodGroups all graduate to beta. The longer history is in our piece on Kubernetes v1.36’s workload-aware scheduling. The shortfall that remains is practical: beta APIs in a new release do not describe the clusters most operators run today, and they do not by themselves encode the topology and failure behavior discussed next.

Topology

Gang scheduling answers whether all pods start. It does not answer where. Collective communication in training is sensitive to which nodes share a rack or network block. Kueue’s Topology Aware Scheduling models a data center as levels such as block, rack and hostname, lets a job request that its pods fit in a single rack through an annotation, and enforces its computed placement by gating pods and injecting node selectors. That is a workaround layered on top of the scheduler rather than a scheduler-native placement decision, which means two components must agree about the same nodes.

Device allocation

Dynamic Resource Allocation graduated to general availability in Kubernetes v1.34. It lets workloads describe the properties of the devices they need and leaves the allocation decision to the scheduler, replacing the older model in which a GPU is an opaque integer count. For LLM serving, where a replica may need a specific memory size or a partition of a card, this matters more than it does for training. What DRA does not do is decide job-level policy; it makes device requests expressive, not workloads aware of one another.

The gaps that remain

Reading the research against the ecosystem, four gaps stand out. They are editorial judgments drawn from the sources cited, not measured results.

Failure is still an application concern. The Acme and Llama 3 evidence shows that interruptions are a routine event in large runs. Kubernetes can restart pods, but the logic that diagnoses a bad GPU, excludes it, and resumes from a checkpoint lives in training frameworks and operator tooling. A scheduler that treated recovery as a first-class operation would look closer to what the characterization paper proposes than to a pod controller.

Scheduling by goodput is absent. Pollux’s central claim was that the scheduler should reason about training efficiency, not only resource counts. The Kubernetes scheduling documentation and the queueing layers described above do not, as far as we found, include a goodput objective. Quota systems and fair-share queues divide GPUs; they do not ask whether a job is using the GPUs it holds well.

Training and serving compete without a shared model. Training is gang-scheduled and long-lived. Serving is latency-sensitive and elastic. Reclaiming a serving replica’s GPU for training, or the reverse, requires preemption rules that understand both. Workload-Aware Preemption is an early step; the evidence base for how well it behaves under mixed LLM load is thin because it is new.

Isolation is soft. Comparative write-ups of Volcano, Kueue and Run:ai-derived schedulers note that these systems typically rely on namespace-based separation, which does not prevent noisy-neighbor effects on shared GPUs. Sharing mechanisms for inference are an active research subject, including this year’s SOSP paper on energy-conscious GPU sharing for inference serving, though that paper’s claims are best checked against its final text.

What this means for a platform team

The research suggests treating Kubernetes scheduling for LLMs as a layered decision. Use the core scheduler and DRA for device-aware placement. Add a queueing layer, such as Kueue or Volcano, where multi-team fairness and gang admission are the problem. Decide explicitly who owns failure handling, because no layer does by default. And measure goodput, not allocation: a cluster that reports high GPU allocation while jobs stall on failures and restarts is meeting the scheduler’s metric and missing the workload’s.

The system-level descriptions above rest on project documentation plus comparisons written by vendors and practitioners, which vary in rigor, so verify version-specific claims against the release notes of the Kubernetes and Kueue versions you actually run.