By ByteBot
KubeCon North America just added an AI Inference and Agentic track to its November lineup, the first dedicated AI inference programming in the conference history. Kubernetes is no longer just a container platform. It is becoming the AI inference operating system. The project at the center of that shift is llm-d, and if you are running LLMs in production on Kubernetes, you need to understand what it does and why your existing setup is probably costing you GPU budget.
The Problem Vanilla Kubernetes Cannot Solve
Standard Kubernetes knows nothing about what is happening inside a vLLM pod. Round-robin load balancing routes requests to whichever replica is next in rotation, regardless of what that pod actually has cached. The result: the receiving pod throws away any previous computation and re-runs the entire input from scratch. In a production RAG application where every user sends the same system prompt, your GPU fleet is recalculating the same expensive prefill thousands of times per day.
Compounding this is the prefill-decode asymmetry. The prefill phase is compute-bound and relatively short. The decode phase, generating tokens one at a time, is memory-bandwidth-bound and can run for seconds. When both phases run on the same GPU pool with no separation, long decode jobs block the compute that new prefill requests need. Time-to-first-token spikes. Throughput collapses under load. This is not a tuning problem. It is an architectural one.
What llm-d Actually Does
llm-d, where the d stands for distributed, is an open-source framework that wraps vLLM with Kubernetes-native intelligence. IBM Research, Red Hat, and Google Cloud donated it to the CNCF as a sandbox project in March 2026, with NVIDIA, CoreWeave, AMD, Hugging Face, Intel, and Mistral AI as contributors.
Prefill and decode disaggregation: llm-d splits the inference fleet into dedicated prefill pods and dedicated decode pods. After a prefill pod finishes, it transfers the request and its KV cache to a decode pod via NVLink or InfiniBand. Each pool scales independently.
KV cache-aware routing: The Endpoint Picker component scrapes each vLLM replica Prometheus metrics in real-time, tracking KV cache utilization, queue depth, and which replicas have which prompt prefixes cached. When a request arrives, it routes to the pod most likely to serve it from existing cache, skipping prefill recomputation entirely.
Hierarchical KV offloading and scale-to-zero: Introduced in llm-d 0.5, hot KV cache stays on GPU memory while less-accessed cache moves to CPU RAM or NVMe. Inference pools can also scale to zero replicas during idle periods, making inference financially viable for dev clusters and internal tools.
The Numbers
llm-d 0.5 benchmarks on Qwen3-32B across 8 vLLM pods and 16 NVIDIA H100 GPUs show 109% higher throughput and 99% lower P50 TTFT versus a baseline Kubernetes service. The baseline degrades rapidly under load; llm-d sustains near-zero TTFT while scaling to approximately 120,000 output tokens per second. These numbers come from a realistic multi-replica setup under production-representative load, not a micro-benchmark.
The Bigger Picture
llm-d is one layer of a larger stack taking shape. The emerging Kubernetes AI inference consensus in 2026: vLLM for the inference engine, KServe for model serving and autoscaling, Kueue for GPU job scheduling, Ray for distributed orchestration, and llm-d as the intelligent routing and disaggregation layer. The KubeCon NA 2026 announcement puts this entire stack on stage for the first time. KubeCon NA runs November 9 through 12 in Salt Lake City, with a co-located Cloud Native AI and Inference Day on November 9.
The CNCF survey data makes the stakes plain: 82% of container users run Kubernetes in production, and 66% of organizations using generative AI workloads already rely on it. The AI inference coordination layer was the missing piece. llm-d is what fills it.
What to Do Now
If you are running vLLM on Kubernetes today, llm-d is an enhancement rather than a replacement. It wraps your existing deployment and adds routing intelligence on top. The llm-d GitHub repository and IBM Research donation post are the best starting points. If you are planning inference infrastructure from scratch, the CNCF sandbox path with llm-d, KServe, vLLM, and Kueue is now the vendor-neutral, governance-backed standard.
