Most teams running LLMs on Kubernetes are quietly burning money. They deploy five or ten vLLM replicas behind a load balancer and call it production. The load balancer round-robins requests. Each replica recomputes the same prompt prefixes it already processed an hour ago. KV cache — the whole point of prefix caching — gets wasted because the router has no idea what’s cached where.
llm-d 1.0, which shipped October 20 and is getting its KubeCon North America spotlight this week, fixes that. It is an open-source control plane that sits in front of a vLLM fleet, subscribes to each replica’s KV-cache events in real time, and routes requests to the pod that already holds the prompt’s prefix. On identical hardware, that routing decision alone cuts median time-to-first-token by more than 85% and raises throughput 2.4x.
Control Plane, Not Engine
The confusion that derails most llm-d discussions: people assume it replaces vLLM. It doesn’t. vLLM stays the inference engine — it runs the model, manages PagedAttention, executes tokens. llm-d is the orchestration layer above it. Think of vLLM as your Linux kernel and llm-d as the scheduler deciding which process runs on which CPU.
What llm-d adds is a single endpoint in front of the entire fleet, a live distributed KV cache index, and a flow-control policy. Install it and your ten vLLM pods behave like one intelligent system instead of ten oblivious strangers sharing a load balancer.
How the Routing Actually Works
vLLM exposes KV-cache allocation and eviction events over ZMQ. llm-d’s Endpoint Picker subscribes to those event streams from every pod in the fleet. When a request arrives, the router checks the index — which replica already holds tokens for this prefix? — and sends it there instead of guessing randomly.
When GPU memory fills up, llm-d spills the KV cache down a tiered hierarchy: host CPU RAM, local SSD, then remote storage (Managed Lustre, Mooncake, or SGLang’s HiCache). The cache stays alive longer. More requests hit it.
For heavy multi-step workloads, llm-d can disaggregate prefill from decode — running prompt processing and token generation on separate worker pods. This lowers time-to-first-token on long prompts and steadies per-token latency under load.
What 1.0 Actually Added
The 1.0 release was not a rebranding of an existing stable version. Four things are genuinely new:
- Stable v1 API: EPPConfig and InferenceObjective CRDs moved to
llm-d.ai/v1. You can now standardize CI/CD pipelines on this without breaking changes. - Session-aware routing: Multi-turn conversations get pinned to the replica holding their KV cache. Agent workloads with 10-turn context chains stop paying the full recompute penalty on every turn.
- Zero-downtime upgrades: DisaggregatedSet drains one shard at a time while the others serve traffic. Rolling inference upgrades without request drops.
- Multi-tenant QoS: Priority and fairness flow control so latency-sensitive API traffic and bursty background agent jobs share the same cluster without strangling each other.
IBM’s Cost Proof Point
Performance numbers from a blog post are one thing. IBM Research published a concrete result: they ran an AI agent factory on llm-d to explore quantum error correction chip designs, hit a 98% prefix-cache hit rate, and paid $0.11 per million tokens. The equivalent proprietary frontier model configuration cost $1.27 per million tokens — 11.5x more. That gap is not a rounding error.
The 98% cache hit rate is what makes the economics work. That kind of number is not achievable with round-robin routing. It requires the router to know the cache state of every replica.
Who Should Actually Care
llm-d is not for teams running a single vLLM pod. It is infrastructure for:
- Platform teams with 10+ GPU nodes serving shared inference endpoints
- Teams with repetitive prompts — agentic workloads, RAG pipelines, code generation with long system prompts
- Multi-tenant deployments where different teams need different SLOs on the same cluster
- Anyone paying for recompute they should not be paying for
If Kubernetes is already your operational home for model serving, the installation path is two Helm commands:
helm repo add llm-d https://llm-d.github.io/charts
helm install quickstart llm-d/llm-d --set model=meta-llama-3-8b-instruct --set shards=4Prerequisites: Kubernetes 1.29+, cert-manager, Gateway API CRDs v1.3.0+, and the NVIDIA GPU Operator. The GitHub repo and Google Cloud’s technical breakdown cover the deeper architecture. The vLLM integration docs walk through the specific configuration options on the vLLM side.
llm-d is a CNCF Sandbox project with 500+ contributors backed by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. The v1 API is stable. KubeCon NA has three sessions dedicated to it this week. If you are running a vLLM fleet and not thinking about inference routing, this is the week that changes.
