By ByteBot
OpenAI revealed benchmark results for its first custom inference chip — Jalapeño — at Hot Chips 2026 on August 25. The chip, co-developed with Broadcom on TSMC’s 3nm node, delivers 1.5x–1.9x more AI work per watt and up to 3.6x lower latency than Nvidia’s Blackwell GB300 across three models: GPT-OSS, DeepSeek R1, and Kimi K2.5 1T. More striking than the hardware specs: OpenAI built its own kernel programming language, Gluon, that bypasses CUDA entirely. This is the first full-stack challenge to Nvidia’s inference dominance from a lab that generates enough AI traffic to actually test the theory at scale.
What the Benchmarks Actually Show
The numbers are notable. At concurrency-1 on Kimi K2.5, Jalapeño delivers 700 tokens per second per user — against the next-best chip’s 100 tok/s. That’s a 7x gap where it matters most: low-latency, single-user inference. On GPT-OSS, Jalapeño’s iso-interactivity throughput per megawatt is described as nearly double GB200’s highest throughput point. Richard Ho, OpenAI’s hardware head, put it plainly: “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly.”
The tests were run on SemiAnalysis’ InferenceX benchmark across the three models, covering latency and throughput scenarios that matter for production API traffic at scale. The chip itself is a reticle-sized ASIC: 13.4 PFLOPs of MXFP4 compute per die, 15.4 TB/s of HBM4 bandwidth, and a 700W TDP — lower than Nvidia Rubin’s 900–1,150W. A full two-rack system runs 128 Jalapeño ASICs at 160 kW total, scalable to 2,048 chips across 16 racks.
Gluon and the Real CUDA Question
The hardware story is interesting. The software story is more important. OpenAI developed Gluon — a kernel programming language built on top of Triton — specifically to program Jalapeño. It preserves Triton’s SPMD programming model while exposing lower-level hardware abstractions that CUDA typically hides. What makes this significant is how OpenAI bootstrapped the kernel library: Codex generated functional and efficient kernels without human kernel engineering, including MLA kernels for benchmarking DeepSeek. The chip’s software stack went from zero to production-ready in under nine months after silicon returned from the fab.
This is what actually threatens the CUDA moat. Not the chip itself — custom silicon has existed for years from Google, Amazon, and Apple. What’s different is the mechanism: AI-generated kernels on a Triton-based stack make the usual “but you’d have to rebuild the entire software ecosystem” argument weaker. SemiAnalysis noted that if Jalapeño succeeds, it signals that “the industry’s obsession over universal compilers is invalidated by frontier AI models.” That’s a significant claim. It’s also plausible.
Related: NVIDIA CUDA Targets RISC-V: What the Server Play Means for Devs
The Caveats Worth Reading
Several things the headlines are underplaying. First: the benchmarks compare Jalapeño against Blackwell (GB300), not Nvidia’s Vera Rubin — which is already shipping to customers and uses the same HBM4 technology. SemiAnalysis itself called the Blackwell comparison “somewhat incomplete and unfair.” Against Rubin, the comparison on per-token cost and TCO is roughly equivalent, not a decisive Jalapeño win. Second: all InferenceX results use single-turn 8k context. No AgentX results exist — no multi-turn, long-context, or prefix-cache-heavy workloads that dominate actual production agentic deployments. Third: the benchmarks were run inside OpenAI’s labs without independent verification.
On Hacker News, chip professionals pushed back on the development timeline: a nine-month tape-out cycle “is fairly typical (even somewhat unimpressive)” depending on how milestones are defined. The more honest read is that OpenAI executed quickly but not impossibly so. Deployment timeline: tiny volumes by end of 2026, broader production ramp in 2027, gigawatt-scale by H1 2028.
What This Means for Developers
If you’re calling OpenAI’s API today, nothing changes. No configuration flags, no SDK updates, no code changes required. Jalapeño is internal infrastructure — not for external sale. Self-hosted developers running their own Nvidia or AMD clusters are entirely unaffected. The payoff is indirect and delayed: if Jalapeño delivers its claimed 50% reduction in inference cost, OpenAI has room to cut API pricing or raise rate limits without sacrificing margins. Given how high-frequency agentic workflows are currently cost-prohibitive at scale, that would unlock a real change in what’s economically viable to build. According to Pinggy’s analysis, watch for any pricing movements from late 2026 onward.
Key Takeaways
- Jalapeño posted 1.5x–1.9x efficiency gains over Nvidia Blackwell GB300 at Hot Chips 2026, but the fairer comparison against Vera Rubin shows rough parity on per-token cost
- Gluon, OpenAI’s Triton-based kernel language, plus AI-generated kernels via Codex, is the more consequential story: it demonstrates a path to bypassing CUDA without rebuilding a decade of ecosystem from scratch
- All benchmarks are single-turn, vendor-reported, and not independently verified — production multi-turn agentic results do not exist yet
- API developers see no immediate changes; self-hosted developers are unaffected; the real impact is indirect API cost reductions, likely visible in 2027 pricing rounds
