KubeCon + CloudNativeCon Japan 2026 opened today in Yokohama. The headline for platform engineers is not a new framework or a flashy product launch — it is a structural fix to a nine-year-old problem. Dynamic Resource Allocation (DRA) is now generally available, the three major GPU vendors have donated their drivers to CNCF, and the 5% GPU utilization crisis that has been quietly hemorrhaging cloud budgets finally has a real answer. If you run AI workloads on Kubernetes and have not looked at DRA yet, this week is a reasonable time to start.
The Integer Problem Nobody Wanted to Admit
When Kubernetes shipped the Device Plugin API in 2017, it solved an immediate problem: how to expose specialized hardware to pods. The solution was workable but crude. Workloads requested GPUs as integer counts — nvidia.com/gpu: 1 — and the scheduler obliged by finding a node with a free GPU and placing the pod there. Which GPU, with how much memory, connected to what topology? The scheduler had no idea, and no way to find out.
Nine years later, Cast AI’s 2026 State of Kubernetes Optimization Report put a number on the consequences: average GPU utilization across tens of thousands of production clusters sits at 5%. At AWS H100 pricing, a 20-GPU cluster running at that utilization rate wastes roughly $94,000 per month in idle silicon. The device plugin model did not cause all of that waste, but it is a structural contributor. You cannot share what you cannot describe.
What DRA Actually Gives You
DRA replaces the integer device count with a proper declarative API. The core object is the ResourceClaim, which describes what a workload needs from hardware — attributes, capacity thresholds, connectivity requirements — using CEL (Common Expression Language) selectors. The scheduler reads these claims against ResourceSlice objects published by DRA drivers on each node, then places pods only where the hardware can actually satisfy the request.
Here is what that looks like in practice. The old approach:
resources:
limits:
nvidia.com/gpu: 1The new approach with a ResourceClaim:
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
name: gpu-claim
spec:
devices:
requests:
- name: gpu
exactly:
deviceClassName: gpu.nvidia.com
selectors:
- cel:
expression: >
device.capacity["gpu.resource.nvidia.com"].memory
.compareTo(quantity("24Gi")) >= 0
- cel:
expression: >
device.attributes["gpu.resource.nvidia.com"].nvlink == trueThis requests a GPU with at least 24 GB of VRAM and NVLink connectivity. If no node can satisfy it, the scheduler tells you immediately rather than silently queuing the pod behind an unsuitable device. The ResourceClaim object can also be shared across multiple pods — useful for inference workloads where several replicas can time-share a single GPU — or templated so each pod in a distributed training job gets its own dedicated device.
Three Vendors Locked In Within Six Months
At KubeCon Europe in March, NVIDIA donated its GPU DRA driver to CNCF and simultaneously became a CNCF Platinum Member. The driver handles per-node GPU discovery, publishes hardware attributes to the Kubernetes API, binds ResourceClaims to specific GPU instances, and manages MIG partitions and NVLink fabric configuration at allocation time. Google open-sourced its TPU DRA driver at the same event, bringing TPU workloads under the same API. AMD followed with an ROCm GPU DRA driver.
When the three dominant AI hardware vendors standardize on the same Kubernetes API in a six-month window, that is not a trend to watch — it is a settled interface. DRA is how GPUs and TPUs will be scheduled in Kubernetes going forward.
Distributed Training: Add KAI Scheduler
DRA handles resource description and allocation. For distributed training jobs, there is still the question of scheduling coordination. NVIDIA’s open-source KAI Scheduler fills that gap as a secondary Kubernetes scheduler designed for AI/ML workloads at scale.
The most important feature is gang scheduling: all pods in a distributed job start together or none start. This eliminates the silent GPU waste pattern where seven of eight training pods schedule immediately while the eighth waits hours for a free node — during which time the first seven pods are running but producing nothing. KAI also supports fair-share queuing across teams, GPU sharing for concurrent workloads, and topology-aware placement that respects NVLink fabric topology. It integrates with KubeRay for Ray clusters.
Where to Start
DRA has been enabled by default since Kubernetes 1.35. If you are already on 1.35 or later (or Red Hat OpenShift 4.21+), the infrastructure is there. The practical steps:
- Install the NVIDIA GPU DRA driver from the CNCF-managed repository (or the AMD/Google equivalent for your hardware).
- Create
DeviceClassresources that describe the GPU types available in your cluster. - Migrate workloads from
nvidia.com/gpu: Nresource limits toResourceClaimreferences — incrementally, not all at once. - For distributed training workloads, evaluate KAI Scheduler alongside your existing kube-scheduler setup.
The old device plugin model still works in Kubernetes 1.35. Migration is not forced. But teams staying on legacy device plugins are opting out of shared GPU access, attribute-based scheduling, and fallback allocation logic — which at H100 prices is an increasingly expensive choice to leave on the table.
The Bottom Line
DRA did not ship last week. It has been in development since 2022 and reached GA in August 2025. What makes this week relevant is the convergence: three GPU vendors with production-ready drivers, KubeCon Japan spotlighting the cloud-native AI infrastructure story, and an industry utilization number (5%) that makes the cost of inaction concrete. The API is stable, the drivers are there, and the migration path is incremental. There is no longer a reasonable argument to delay.
