By ByteBot
Nvidia today released two tools aimed at the part of the AI agent stack nobody wants to pay for: the execution layer. Nemotron 3.5 Lightning is a 30-billion-parameter mixture-of-experts model that activates only 3 billion parameters per inference step, delivering up to four times the output speed of comparable dense models. Alongside it, NeMo Switchyard is an open-source routing library that automatically directs agent requests to the most capable and cost-effective model available — without requiring any changes to existing agent code. Both are live today on Hugging Face and GitHub.
What Makes Nvidia Nemotron Lightning Useful for Agents
The pitch is specific: Nemotron 3.5 Lightning is not a general-purpose reasoning model. Nvidia positions it as the execution layer in multi-agent systems — the component that handles high-volume, repetitive subtasks while frontier models handle planning. The mixture-of-experts architecture makes this practical. With only 3B active parameters per inference step despite having 30B total, the model runs efficiently on a single H100 while maintaining accuracy on specialized tasks. Nvidia claims it ranks 24th out of 133 models on the Artificial Analysis Intelligence Index — well above the median of 9 for models of similar size.
The model ships with speculative decoding baked in. Two draft models accompany it: DSpark for low-concurrency deployments and DFlash for comparison testing. Weights are fully open under the OpenMDW-1.1 license, with training data and post-training recipes included — not just weights. Nvidia VP of Generative AI Kari Briski noted that one partner trained a custom domain router in approximately two hours for $85 in compute costs. That kind of economics matters when you’re building production pipelines.
NeMo Switchyard Solves a Problem Developers Patch Manually
Most production agent systems today include some form of model routing logic — a decision about whether this request needs GPT-5, Claude Opus, or a smaller local model. Teams typically implement this with ad-hoc prompt engineering, custom middleware, and manual cost thresholds. NeMo Switchyard turns that into infrastructure. It’s a Rust-based proxy that sits between your agent and your model providers, translating between OpenAI Chat, Anthropic Messages, and OpenAI Responses API formats. Agents keep their existing code unchanged. Switchyard handles the rest.
Three tuning-free routing algorithms ship immediately: an LLM classifier that uses a judge model to assign requests, a stage router that matches model capability to task phase based on recent tool activity, and an escalation router that starts with cheaper models and upgrades only when difficulty persists. A more sophisticated tunable prefill router is also available for teams willing to invest in training data. Integration points include LangChain, LiteLLM, Kong AI gateways, vLLM, and any OpenAI-compatible endpoint.
Related: llm-d Joins CNCF: Kubernetes Gets Serious About AI Inference
The Enterprise Numbers Are Real — With Caveats
Nvidia’s announcement includes some striking figures from early adopters. Ramp cut AI inference costs by 58% while matching frontier model performance. LangChain achieved 74% lower cost across 145 multi-turn Deep Agents tasks. Boomi reported 100% domain-routing accuracy. Cognition AI saw a 28% cost reduction. These are Nvidia-reported numbers from partner deployments, not independent benchmarks, so treat them as directional rather than universal. However, the pattern is consistent enough to be credible: routing intelligently between models based on task complexity delivers meaningful cost reductions at scale.
The honest caveat is that Nemotron 3.5 Lightning is not the strongest model at its parameter count. Developer discussion on Hacker News noted it trails Qwen 3.6 35B-A3B by roughly 20% on several standard benchmarks — despite both models having roughly 3B active parameters. The FP4 quantization version also drops noticeably in coding benchmark accuracy compared to BF16, so teams using Lightning for code review should stick to the BF16 checkpoint. Early testers noted “over-thinker” behavior — the model generates multiple solution attempts before settling on a final output, adding latency in single-turn scenarios.
Where to Start
Nemotron 3.5 Lightning is available through Hugging Face, ModelScope, OpenRouter, and NVIDIA NIM. NeMo Switchyard is live on GitHub today. The path most teams will take is to deploy the escalation router first — no training required, and it provides immediate routing based on difficulty without complex configuration. Nvidia’s infrastructure play here is clear: more inference demand benefits their GPU business directly, which gives open-sourcing a financial incentive that’s likely to last.
Key Takeaways
- Nemotron 3.5 Lightning is a purpose-built execution model, not a reasoning model — use it for high-volume subtasks, not complex planning
- NeMo Switchyard provides infrastructure-level model routing without application rewrites, starting with three tuning-free algorithms
- Enterprise adopters report 28-74% cost reductions; treat these as directional, not guaranteed
- The FP4 quantization checkpoint loses accuracy on coding benchmarks — stick to BF16 for code review workloads
- Both tools are available now: weights under OpenMDW-1.1, routing library fully open-source on GitHub
