A proxy that lets one agent use several models

NVIDIA has released NeMo Switchyard as a standalone open-source proxy for assigning difficult planning to stronger models and repetitive execution to smaller, cheaper ones. The agent keeps using its native API format while Switchyard routes each request across providers and models. The design targets a basic cost problem: not every turn in a long agent run needs the most capable model.

“Not every step in an agent workflow needs the same model.” — NVIDIA AI

The source tweet proposes frontier models for complex reasoning and planning, with Nemotron Lightning handling high-volume specialized execution. FxTwitter recorded 160 likes, 25 reposts, and about 13,500 views. More important than the engagement is the implementation: Switchyard sits directly in the request path rather than merely recommending a model.

Three API formats translated through one route

According to the public repository, Switchyard translates OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Claude Code, Codex CLI, and OpenClaw can keep their existing request format while traffic is served by vLLM, NVIDIA NIM, Ollama, or another OpenAI-compatible endpoint. Routing options include random splits, an LLM classifier, signal-driven stage routing, and escalation that judges a weaker model’s first answer before calling a stronger tier.

Operational metrics cover requests, errors, latency, tokens, and routing overhead through Prometheus. That makes the same proxy useful for A/B tests and cost-performance comparisons. The project uses the Apache 2.0 license and has roughly 1,600 GitHub stars. Its own documentation labels the software pre-alpha and experimental, warning that APIs and algorithms may change substantially before version 1.0.

What to watch next

Switchyard’s value depends on whether routing reduces cost and latency without erasing quality. Useful benchmarks should report accuracy, token spend, and routing overhead against a single strong-model baseline. Production adoption will also depend on backend failover, consistent model choice across long conversations, and complete translation of tool calls and streaming responses.

#nemo

NVIDIA Switchyard Routes Agent Tasks Across Weak and Strong Models