On November 12, eight companies that compete fiercely on AI hardware — NVIDIA, AMD, Intel, Google, Meta, Cloudflare, Modal, and RadixArk — announced they would jointly develop SGLang, the open-source inference engine already running on 400,000+ GPUs in production. The SGLang Consortium, launched at the first SGLang Summit in San Francisco, is essentially a Linux Foundation moment for AI inference: competing hardware vendors agreeing to cooperate on the software stack they all depend on.
Inference Is Now the Expensive Part
In 2025, inference consumed roughly half of all AI compute. In 2026, that figure is projected to hit two-thirds. Training has largely been claimed by the hyperscalers — if you’re not OpenAI or Google DeepMind, you’re buying access to a finished model, not training one. Inference is where the ongoing compute bill lives, and it’s where hardware fragmentation creates real friction.
NVIDIA’s CUDA stack has a 20-year head start on training. At the inference layer, that gap is narrower. Hardware-agnostic software is both technically feasible and economically attractive to every vendor that isn’t NVIDIA. The SGLang Consortium is that software.
What SGLang Actually Is
SGLang isn’t another wrapper around vLLM. Its core technical differentiator is RadixAttention — a prefix caching mechanism that eliminates redundant computation when requests share context. That’s exactly what agent workloads look like: every call to your coding agent reuses a large system prompt. Every RAG query shares the same document chunks. SGLang turns that reuse into free throughput. It also leads on constrained JSON generation, which is table stakes for any structured output pipeline.
The numbers are real. On Llama 3.1 8B, SGLang delivers 29% higher throughput and 23% faster time-to-first-token compared to vLLM. On newer models like Qwen2.5-7B, the gap has narrowed — vLLM now leads on raw chat throughput. Performance depends on workload, which is the honest answer anyone giving you a “just use X” recommendation is glossing over.
The Governance Model That Might Actually Work
What separates this from a press release is the governance structure. SGLang’s maintainers retain final authority over technical decisions and code acceptance — the consortium cannot force features in. Members commit to designating a technical representative and attending two annual roadmap meetings. This is the CNCF model: the same structure that made Kubernetes a neutral cross-vendor infrastructure layer instead of a Google product.
The real test arrives when AMD or Intel needs a hardware-specific optimization and asks NVIDIA-employed contributors to help ship it. Ying Sheng, SGLang’s co-author and RadixArk CEO, put the goal plainly: “The teams running SGLang at the largest scale see what inference needs next before anyone else does.” That’s true. Whether they can coordinate on it across competing interests is the open question.
What Developers Should Take From This
Nothing ships with this announcement. The cautious read from the community is correct: real validation means Day-0 kernel support landing on AMD and Intel hardware, not a press release. That said, the direction is clear.
If you’re deploying inference today, here’s the practical split:
- Use vLLM for standard chat serving, broadest model support, fastest onboarding. It has 96,500 GitHub stars and the deeper community.
- Use SGLang for agent workloads with shared prefixes, structured JSON output, DeepSeek V3 or other MoE models, and large-batch inference where RadixAttention pays off. That’s a 10–30% end-to-end improvement for the right workloads.
If you’re serving at scale — tens of thousands of requests per day — it’s worth benchmarking both on your actual traffic pattern. The gap between them is workload-dependent and not trivially predictable from published numbers.
Open Source Wins the Infrastructure Layer, Again
The pattern here is familiar. Kubernetes was Google’s internal tool until the Linux Foundation made it everyone’s infrastructure. Prometheus, Envoy, OpenTelemetry — the same story. Competing companies find it cheaper to collaborate on shared infrastructure than to build redundant proprietary versions. AI inference is following the same arc.
The question isn’t whether open-source inference software wins. That’s already happening. The question is whether eight companies with conflicting hardware roadmaps can actually ship coherent software together. The CNCF has done it before. The SGLang Consortium has the structure. Now it needs to ship. For more context on choosing between today’s inference engines, Red Hat’s comparison guide is a solid starting point.
