Nvidia's Switchyard router reshuffles AI models mid-task, cutting task costs to a third in its own tests
Our take
Nvidia’s entrance into the open-weight AI model and routing space with Nemotron 3.5 Lightning and NeMo Switchyard represents a significant shift in how enterprises approach the escalating costs of running always-on AI agents. The inherent challenge – balancing performance with affordability – has long plagued organizations deploying these systems. Sending every task to a powerful "frontier" model rapidly inflates operational expenses, while attempting to build bespoke routing logic to direct simpler tasks to cheaper alternatives quickly becomes a complex and ongoing engineering burden. This situation is increasingly relevant as organizations explore generative AI capabilities, as highlighted by recent discussions around [Why AI-driven purchase intent so rarely becomes a completed sale], where efficient processing is vital for delivering timely and relevant customer experiences. Moreover, Nvidia’s move echoes the broader trend of accelerating AI development in Europe, as seen in [Mistral AI wants to build 1 gigawatt of European compute by 2030 — and lock in customers now], demonstrating a strategic push toward greater compute infrastructure and localized AI innovation.
The brilliance of Nvidia’s solution lies in its holistic approach, tackling the problem from both the model and routing layers simultaneously. Rather than simply offering a cheaper model or a smarter router in isolation, they’ve created a system where the two components are intrinsically linked and open-sourced. This is a departure from the current landscape, where developers often piece together solutions from various providers, integrating routing frameworks like OpenRouter or LiteLLM with models from different sources. The reported results – cost reductions of up to 74% in LangChain’s testing, while maintaining accuracy – are compelling evidence of the potential for this integrated approach. While Nvidia isn’t claiming Lightning is a general intelligence leader, its optimized architecture and ability to match Qwen3.6-35B's accuracy at a faster completion rate, particularly when paired with Switchyard, positions it as a valuable tool for specialized agent tasks, especially when cost optimization is a priority. The ease of integration, with existing frameworks like Kong already supporting Switchyard, further lowers the barrier to adoption.
What makes this particularly noteworthy is Nvidia’s recognition that the optimal model for a given task isn’t static. Agent workflows evolve as they interact with tools, encounter errors, or handle routine versus complex requests. Switchyard’s ability to dynamically route tasks based on agent state, rather than pre-defined categories, is a crucial advancement. The direct incorporation of model verbosity into the routing decision—predicting how many tokens a model will generate—further optimizes cost efficiency. This aligns with a broader industry trend, as illustrated by [LTX-2.5 can generate a 10-second AI video from an image in just 6.8 seconds on Nvidia superchips — and it's open weights], where speed and efficiency are becoming increasingly paramount alongside raw model capabilities. Nvidia's ecosystem-focused approach, partnering with existing routing providers, suggests they are not aiming to replace existing infrastructure but rather to augment and improve it.
Ultimately, Nvidia’s move underscores a fundamental shift in the AI landscape. The focus is moving beyond simply chasing the "best" model to building efficient, adaptable systems that leverage a diverse range of models for different tasks. The commoditization of open weights means that the differentiator will increasingly be the intelligence of the system – its ability to optimally route and manage these models in real-time. The question now is whether other major players in the AI space will follow Nvidia’s lead and embrace this integrated, open-source approach to agentic AI, or if the race will continue to be dominated by the pursuit of ever-larger, more powerful, but ultimately more expensive, frontier models.
Enterprises running always-on AI agents keep hitting the same tradeoff. Send every task to a frontier model and the bill climbs fast. Build custom routing logic to send easy tasks to cheaper models and that becomes its own engineering project, one that has to be maintained every time a workflow changes.
Nvidia is proposing a fix that touches both ends of that problem at once. The company is out on Tuesday with Nemotron 3.5 Lightning, a 30-billion-parameter open mixture-of-experts model built for high-volume, specialized agent tasks, alongside NeMo Switchyard, an open-source library that routes each step of an agent workflow to whichever model fits it best.
The headline numbers: According to Nvidia, Lightning delivers up to 4x faster output than comparable models in its class, completing agentic tasks roughly 30% faster than Qwen3.6-35B at matching accuracy. Paired through Switchyard, Nvidia says the combination holds frontier-level task completion while cutting benchmark costs to roughly a third of running Opus 4.8 alone.
The timing puts Nvidia in the middle of the busiest open-weight stretch the industry has seen in months. Alibaba, Moonshot, Zhipu and DeepSeek have all shipped competitive open models out of China since the spring, several landing at or near frontier performance while undercutting US labs on size or price. Meta added to that pressure by releasing its own 30-billion-parameter open agentic model, Muse Glimmer. Open weights have gone from a differentiator to table stakes in a matter of months, and Nvidia's release lands squarely inside that shift rather than ahead of it.
The pairing is the point. A model alone doesn't solve the cost problem, and a router alone has nothing efficient to route to. Nvidia is betting that open source, applied at both the model layer and the routing layer, is what actually moves the cost needle on agentic AI, not a single cheaper model and not a smarter router bolted onto someone else's stack.
Switchyard's real rivals aren't other open models — they're Not Diamond, which already powers OpenRouter's Auto mode, and RouteLLM, the open-source framework from UC Berkeley and LMSYS. Neither ships its own model. Nvidia's bet is that owning both sides of the decision, under one open license, is what a router-only or model-only competitor can't match.
"That is the power of a system of models, matching the right model to each step of the workflow," Kari Briski, vice president of generative AI at Nvidia, said in a briefing.
How the router actually changes the workflow
Model routing isn't a new category. OpenRouter, LiteLLM and a handful of standalone routing startups already let developers point traffic across multiple providers. Switchyard plugs into several of them rather than replacing them outright.
The core problem Switchyard solves is that the right model changes as an agent moves through a task. An agent's state shifts as tools return results, errors show up, or a step turns out to be routine rather than complex, and a fixed model choice can't adapt to any of that.
Briski described routing strategies that respond to that shifting state rather than a static task category.
"It has many types of routing strategies," Briski said. "You can have a random router, which is not that great, or you can have an agent state route or a classifier route. Depending on your routing strategy, it wants to choose the best model. In some cases you want to go with a model like Lightning for really efficient tasks, and the router will actually choose Lightning if it's set up in your pool of models."
Cost enters the routing decision directly, not as an afterthought. In response to a question from VentureBeat, Briski said Switchyard can evaluate model verbosity, meaning how many tokens a given model tends to produce for a task, and use that prediction to steer work toward the cheaper option before the call is made.
The part that keeps this from becoming its own integration project is where Switchyard sits. Nvidia split its partners into two groups: agent frameworks that call Switchyard directly, including Cognition, LangChain and Nous Research, and LLM gateways that have built Switchyard support into their own products, including Kong, LiteLLM and OpenRouter. Kong ships Switchyard natively inside Kong AI Gateway. Briski pointed to that same list of gateway partners when describing how the library fits into the existing routing ecosystem.
"We are an ecosystem lover, and we want to make sure that we are integrated," Briski said. "We've partnered with OpenRouter, LiteLLM and Kong, and they've already integrated our routing algorithm, so you can pick it up right where you're already using the best tools."
Nvidia shared results from nine companies testing Switchyard, several with specific figures attached. LangChain reported a 74% cost reduction across 145 multi-turn Deep Agents tasks by routing just 7% of calls to a frontier model, at a 6% accuracy tradeoff. Ramp said it matched a frontier model's performance on Ramp SWE-Bench while cutting costs 58% and runtime 33%. Cognition integrated Switchyard's staged router into Devin Desktop for internal use and reported near-frontier performance on FrontierCode Main while cutting mean cost 28% relative to routing everything to a single frontier model.
Lightning's architecture and performance gains
Nemotron 3.5 Lightning is a standalone open model in its own right, built for high-volume, specialized agent tasks rather than general-purpose use.
It extends the hybrid Mamba-Transformer, latent mixture-of-experts architecture Nvidia introduced with the Nemotron 3 family in December 2025, the same line behind Nemotron 3 Super, which Nvidia uses as Lightning's own baseline in its post-training comparisons. Positioned within a routing setup like Switchyard, it's built to sit at the fast, cheap end of the decision rather than the frontier end, but it runs and ships independent of any router.
According to the Artificial Analysis Intelligence Index, a general capability benchmark spanning nine evaluations, Lightning scores 24, tied with gpt-oss-120b and behind Nemotron 3 Super, Gemma 4 31B, Claude 4.5 Haiku and Mistral Medium 3.5, all at 30. Lightning isn't a general-intelligence leader in its size class, and Nvidia isn't claiming it is.
The actual claim is narrower: according to PinchBench data supplied by Nvidia, Lightning matches Qwen3.6-35B's accuracy roughly 30% faster and beats Gemma 4 26B's accuracy at a similar completion time on PinchBench, a real-world agent task benchmark spanning coding, research and file management. That's a speed-to-accuracy tradeoff, not a capability win.
Post-training is where Nvidia says the bigger gains show up. The company shared before-and-after figures from four early-access partners: CrowdStrike's malicious-content recall against a Nemotron 3 Super baseline, CodeRabbit's coding router against a GPT 5.4 Nano baseline, Harvey and Trajectory's legal task completion against an Opus 4.6 baseline, and Lila Sciences' energy simulation work against an Opus 4.8 baseline. CodeRabbit's case is the most specific: Nvidia says the standard NeMo Auto model recipe, trained for one epoch, built into a working router agent for $85 in about two hours.
What this means for enterprises
There is no shortage of competitive offerings in the growing market for open models. The new Nemotron Lightning release will be yet another option for organizations to consider.
On the model side, Lightning's own benchmark chart picks Qwen3.6-35B as its direct comparison point. Asked by VentureBeat directly how Lightning compares to Chinese models more broadly, Briski didn't offer a head-to-head benchmark, pointing instead to openness and customizability as the differentiator.
"Our value proposition is not just open and it's very customizable," Briski said.
For enterprises building agentic infrastructure, three trends stand out:
The routing decision is becoming dynamic instead of static. Enterprises that built agent pipelines around a single default model are being pushed toward per-step routing based on live signals like agent state and token cost, not a fixed assignment set at design time.
Open source is now a cost lever at two layers, not one. Pairing an open model with an open router a vendor controls end to end is a newer argument than cheaper weights alone, and worth watching for whether other labs follow the same pattern.
The competitive question shifts from best model to best system. As routing libraries mature, the differentiator moves from which model an enterprise defaults to, toward how well its routing layer matches models to tasks in production, a harder thing to benchmark and a harder thing to market.
Read on the original site
Open the publisher's page for the full experience