By ByteBot
Kubernetes 1.37 “Garhwal” landed on August 26 with 67 enhancements, and one of them directly hits your infrastructure budget: gang scheduling for AI/ML workloads is now Beta and on by default. The full-or-nothing guarantee that your 64 GPU pods all schedule together — or none of them do — is no longer an external scheduler concern. It’s built in. That alone makes this release worth evaluating for teams running training jobs on Kubernetes.
The rest of the release is quieter but just as consequential. DRA Extended Resources graduate to GA, HPA can finally scale to zero natively, and Pod Certificates make mTLS workload identity possible without cert-manager or SPIFFE. The nftables clock is also running: 1.37 starts warning you that 1.40 will flip the default. Three releases from now is not far.
Gang Scheduling: AI/ML Workloads Get a First-Class Primitive
The CompositePodGroup API (scheduling.k8s.io/v1alpha3) arrives in Alpha, built for hierarchical workloads like JobSet and LeaderWorkerSet deployments where different groups carry separate gang, topology, and disruption requirements. Gang scheduling itself graduates to Beta and is enabled by default.
The cost story is straightforward: if a 64-GPU training job schedules only 48 of its pods, those 48 GPUs sit mostly idle while the job waits for the remaining 16. At $3–12 per GPU-hour, partial pod groups are expensive. The gang scheduling guarantee eliminates this class of waste.
There is one breaking change before you upgrade. The v1alpha2 PodGroup API is gone in 1.37. If you tested gang scheduling on 1.36, delete all v1alpha2 PodGroup manifests before upgrading — the API server will reject them and prevent startup validation from completing.
DRA Extended Resources Graduate to GA
Dynamic Resource Allocation has been building toward this for a few releases. In 1.37, three features reach stable: DRA Extended Resources, Device Taints and Tolerations, and Resource Claim Status.
The practical impact: GPU allocation no longer requires both a DRA driver and a legacy device plugin running in parallel. A DRA driver alone is sufficient. For teams already invested in DRA-based GPU management, this removes the last justification for maintaining the device plugin sidecar.
Device Taints work exactly like node taints, which is the right design choice. Mark a degraded GPU with a taint, set the appropriate toleration on workloads that can run on degraded hardware, and let the scheduler handle the rest. For GPU maintenance — firmware updates, thermal issues, ECC errors — this replaces manual pod eviction workflows.
HPA Scale-to-Zero: Native, No KEDA Required
The HorizontalPodAutoscaler now accepts minReplicas: 0 when paired with at least one object or external metric. This is Beta and enabled by default. Inference endpoints, queue consumers, and batch processors that spend most of their time idle can scale down to zero and recover when demand returns.
GPU inference endpoints are the obvious target. Production Kubernetes clusters average around 5% GPU utilization — scale-to-zero turns that problem from a billing line item into a scheduler concern.
Does this make KEDA redundant? For straightforward queue or external metric patterns, it might. KEDA still has a much wider event source catalog. But the maintenance overhead of running KEDA purely for scale-to-zero is now hard to justify.
Pod Certificates and ClusterTrustBundles: mTLS Without the Plumbing
Both KEP-4317 and KEP-3257 graduate to GA, and together they change the answer to “how do we do zero-trust mTLS between pods?” The answer is now: use Kubernetes.
Kubelet generates each pod’s private key, projects short-lived X.509 certificates into the container filesystem before the workload starts, and handles rotation automatically. The private key never leaves the node. ClusterTrustBundles distribute the trusted root certificates workloads need to verify each other — also via projected volumes, also validated by the Kubernetes API.
Cert-manager and SPIFFE/SPIRE have served this purpose well. With Pod Certificates GA, the infrastructure argument for deploying them as standalone systems weakens significantly. They still offer richer policy controls; for most workloads, the built-in path is now simpler and sufficient.
nftables: 1.40 Is Three Releases Away, Start Now
Kubernetes 1.37 adds deprecation warnings when kube-proxy defaults to iptables because you haven’t explicitly configured a mode. It also starts warning about ipvs, which is scheduled to be disabled by default in 1.40 and removed in 1.43.
The action item is immediate: explicitly set mode: iptables or mode: nftables in your kube-proxy configuration. This silences the warnings and, more importantly, prevents an unintended backend change during a future upgrade. If you’re running kernel 5.13 or newer, start testing nftables in a non-production cluster now.
What to Test Before Upgrading
Check for v1alpha2 PodGroup resources if you have ever experimented with gang scheduling — those will break on upgrade. Review kube-proxy mode configuration across your clusters. For teams adding GPU workloads, the DRA and gang scheduling combination in 1.37 makes this more than a routine bump.
The official release post and full enhancement list are on kubernetes.io. The DRA deep-dive and HPA scale-to-zero post are worth reading alongside the main announcement. Fairwinds has a solid practitioner upgrade overview covering what to watch during rollout.
