Kubernetes 1.37: Scale GPU Workloads to Zero and Save

Kubernetes 1.37 “Garhwal” shipped on August 26 with 67 enhancements, but one change matters more than the rest for teams running GPU workloads: HorizontalPodAutoscaler can now scale deployments to zero replicas natively. No KEDA, no custom controllers, no workarounds. Set minReplicas: 0, point the HPA at an external queue metric, and your cluster stops draining your budget the moment the job queue empties. Combined with Dynamic Resource Allocation finally reaching GA, this is the most GPU-relevant Kubernetes release since the project started taking AI infrastructure seriously.

HPA Scale to Zero Is Now Default-On

The HPAScaleToZero feature gate graduates to Beta in v1.37 — and it’s enabled by default, so you don’t need to flip anything. The mechanics are straightforward: traditional HPA relies on CPU and memory metrics that only exist when pods are running, which makes zero replicas impossible. The v1.37 implementation breaks that dependency by using external or object metrics — queue depth, pending jobs, any signal that lives outside your pods.

The typical pattern for GPU batch workloads looks like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: gpu-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: gpu-worker
  minReplicas: 0
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: queue_consumer_lag
      target:
        type: AverageValue
        averageValue: "30"

You’ll need a metrics adapter exposing your queue metric through the External Metrics API — the Prometheus Adapter is the standard choice. The main limitation: this does not work for HTTP services. Kubernetes has no mechanism to buffer incoming requests during a cold start, so if your workload needs to be available on demand, you’re still on KEDA or a buffering layer. But for queue consumers and batch processors — exactly the workloads that eat GPU budgets when idle — native HPA scale-to-zero is the simpler path forward.

DRA Extended Resources Hit GA After Three Releases

Dynamic Resource Allocation has been building toward this for three consecutive releases: Alpha in v1.35, Beta in v1.36, and now GA in v1.37. The specific milestone is DRA Extended Resource Support, which means a DRA driver can now satisfy traditional nvidia.com/gpu extended resource requests without requiring a separate device plugin. Your existing pod specs don’t change. The allocation logic backend switches to DRA, which is the part that matters.

The other GA graduation in this release is Device Taints and Tolerations. If a GPU in your cluster degrades — bad memory, thermal throttling, a failed firmware update — a DRA driver or cluster admin can taint that specific device. It won’t be assigned to new pods, and if you configure it, existing pods using that device get evicted. This is the same model Kubernetes has used for nodes for years, finally applied at the hardware device level. For clusters running expensive GPU hardware 24/7, this is genuine operational value.

Pod Certificates Graduate to GA: Drop cert-manager for the Simple Cases

Service Account JWTs have a fundamental problem: they’re bearer tokens. Possessing the token means you can impersonate the workload that owns it. Pod Certificates, now GA in v1.37, solve this with asymmetric cryptography. The private key never leaves the kubelet. The certificate gets mounted into the container. If a peer service receives your certificate, they still can’t impersonate you — proof-of-possession is the primitive, not token possession.

The implementation is kubelet-native: certificates are written to the pod filesystem before the container starts and are automatically rotated. You don’t need cert-manager, Istio, or any external controller for the basic case. The ClusterTrustBundle API handles distributing CA certificates cluster-wide so workloads can verify each other’s certificates. Not every team needs this immediately, but if you’ve been deploying cert-manager specifically to avoid bearer token risks, the built-in solution is here.

Three Things to Check Before You Upgrade

v1.37 has three changes that will silently break production clusters if you skip the pre-flight checklist.

IPVS is now on a formal deprecation clock. Running with mode: ipvs in kube-proxy will log warnings starting in v1.37. The full timeline: disabled by default in v1.40, removed entirely in v1.43. Check your current mode now:

kubectl get cm -n kube-system kube-proxy -o yaml | grep mode

If you see ipvs, start planning a migration to nftables. The Tigera migration guide is a practical starting point. nftables has been GA since v1.33 and offers better performance through incremental rule updates rather than full rewrites.

cgroup v1 is now a hard failure. The kubelet hard-fails on cgroup v1 nodes in v1.37. If any of your nodes are still on cgroup v1, they will not join or operate correctly after the upgrade. This is not a warning — it’s a blocker. Migrate to cgroup v2 before upgrading.

Static pods lost API object references. Static pods can no longer reference Secrets or ConfigMaps directly. If any of your static pod specs include these references, they will break on upgrade.

Also Worth Knowing

etcd RangeStream graduates to Beta (requires etcd v3.7+). It converts large list reads from a batch-and-dump approach to a streaming model, which prevents the memory spikes that cause OOM kills in clusters with thousands of pods. CBOR serialization also moves to Beta, delivering 8x faster encoding and 2x faster decoding compared to JSON — significant if your cluster processes high volumes of API requests.

Upgrade Path

The full release notes are on the Kubernetes blog. v1.37 follows standard support: three minor versions receive patches simultaneously, so v1.34 users should be looking at upgrading now. Run the IPVS check, verify your nodes are on cgroup v2, audit static pod specs for Secret/ConfigMap references, and then upgrade. The GPU features — HPA scale-to-zero and DRA GA — are ready to use immediately on a stock v1.37 cluster.