I tried NVIDIA's new LLM routing infrastructure NeMo Switchyard

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

I had just published two articles about running NVIDIA LLM Router on DGX Spark, when immediately afterward I was told that "a new routing platform has been released that also incorporates LLM Router's algorithms." NeMo Switchyard, v0.1.0, was released on July 1, 2026 (Japan time).

It's a bit of a bittersweet feeling since I was preparing follow-up content for those articles, but as I actually started using it, I kept discovering that "features I had struggled to build myself for LLM Router are already included out of the box." In this article, I'll run Switchyard on my Mac and DGX Spark to verify whether the pitfalls I encountered during LLM Router testing have truly been resolved.

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3/

https://dev.classmethod.jp/articles/dgx-spark-nvidia-llm-router-v3-training/

What Is NeMo Switchyard?

It is a routing proxy that distributes LLM traffic, published under the NVIDIA-NeMo organization on GitHub. It's a Python package installable via pip install nemo-switchyard, with internals structured in two layers: a Rust core built with maturin, wrapped by Python. The license is Apache 2.0, the version is 0.1.0, and the development status is explicitly listed as Alpha.

https://github.com/NVIDIA-NeMo/Switchyard

A documentation site is also available.

https://nvidia-nemo.github.io/Switchyard/

I had the opportunity to ask NVIDIA about the relationship with LLM Router, and the response I received was that it's not simply a successor, but rather "a more formal product encompassing various routing technologies and the infrastructure to host and improve them." The algorithms from the LLM Router Blueprint are currently being ported to Switchyard as well.

For those who have used LLM Router, here's a table comparing the two:

I think the two major differences are "no trained router" and "protocol conversion is built-in." LLM Router was designed to train a custom classifier that passes Qwen embeddings through PCA and an MLP. Switchyard replaces that with signals from LLM queries and tool execution histories in agent workflows. Since GPU is no longer needed, it runs as-is on a Mac.

Routing: Choose from 4 Methods

The documentation covers 4 routing methods.

llm-routing summarizes the last 4 turns of conversation, passes them to a classifier model, categorizes them into 4 categories (simple / medium / complex / reasoning), then maps them to the weak / strong tier. It uses tool calling for the decision, and falls back to the default tier if confidence drops below a threshold or classification fails — a fail-open design. Three classification policies are built in — general, coding_agent, and openclaw — and it's interesting that policies for coding and for resident assistants are provided out of the box.

cascade is even more elaborate, evaluating tool execution result signals — such as error severity, test pass/fail status, and number of file edits — across 3 layers. First, clear-cut situations are decided immediately (critical errors go to strong, finishing tasks with all tests passing go to weak), next a weighted score makes the judgment, and only when confidence is lacking does it consult the LLM classifier. The only dial the user touches is confidence_threshold, and the documentation states that the recommended value of 0.5 was calibrated on SWE-Bench Pro.

From Installation to Serve

Python 3.12 or later is required. This time I set up the environment with uv.

uv init switchyard-handson && cd switchyard-handson
uv add "nemo-switchyard[server,cli]"

Wheels are available for Linux x86_64 / aarch64 as well as macOS arm64, so it installs directly on an Apple Silicon Mac.

The configuration has a 3-layer structure: endpoints (provider connections), targets (upstream models), and profiles (routing policies exposed to clients). I assigned GLM-5.2 to the strong tier and DeepSeek V4 Flash to the weak tier. The classifier handling routing decisions references the same target as weak. I actually made a painful mistake once in selecting the classifier model, and the configuration you see now is the second iteration incorporating that lesson — but I'll cover the full story together in the latter half.

endpoints:
  openrouter:
    base_url: https://openrouter.ai/api/v1
    api_key: ${OPENROUTER_API_KEY}

targets:
  strong:
    endpoint: openrouter
    model: z-ai/glm-5.2
    format: openai
  weak:
    endpoint: openrouter
    model: deepseek/deepseek-v4-flash
    format: openai

profiles:
  fast:
    type: passthrough
    target: weak
  smart:
    type: llm-routing
    profile_name: coding_agent
    strong: strong
    weak: weak
    classifier: weak

I only ran into 2 issues. The profile type name uses hyphens: random-routing, while random_routing (with underscore) that appears in the quickstart example is a separate system for the old route bundle format. Also, classifier in llm-routing takes a target ID as a string. Both had error messages that specifically told you the expected format — "expected one of strong, weak, ..." and "expected a string" — so they were quick fixes. Passing the same ID as weak to classifier isn't laziness; if you define two targets for the same model, you get a duplicate registration error. The approach is to have one target referenced by both the classifier role and the response role.

Since the quickstart had no note about this naming difference between the two formats, I sent a PR to upstream to add it to the documentation.

https://github.com/NVIDIA-NeMo/Switchyard/pull/19

Starting is a single command.

uv run switchyard serve --config profiles.yaml --port 4000

This exposes all three APIs — OpenAI Chat Completions, Anthropic Messages, and OpenAI Responses — on the same port. Even if the client-side format differs from the upstream format, they are mutually converted through an internal intermediate representation.

/v1/models Returns 3 Types of IDs

Looking at GET /v1/models right after starting serve is the best way to understand Switchyard's design philosophy. The returned model list contains IDs of different kinds coexisting together.

{"id": "smart", "display_name": "llm-routing", ...}
{"id": "strong", "display_name": "z-ai/glm-5.2", ...}
{"id": "z-ai/glm-5.2", "display_name": "target strong", ...}

Specifying a profile ID (smart) in the model field activates routing, while specifying a target ID (strong) or upstream model name bypasses routing and locks to that model. In other words, clients can choose between "route me" and "use this specific model" simply by which model name they use.

Seeing this, I couldn't help but stare into the distance. Because this was exactly the biggest obstacle in the verification I was preparing as a practical follow-up to the LLM Router series.

Has the LLM Router "Model Name Ignored" Issue Been Resolved?

LLM Router had a spec where it didn't look at the model in the request body. Even if a client explicitly specified model: claude-opus-4-8, it would be overridden by auto routing — making it incompatible with clients like Claude Code that send different model names per task type. In my testing, I worked around this by applying a model name bypass patch to the fork.

I ran the same test against Switchyard. I sent 5 variations of the same prompt "Say OK only." to an llm-routing profile, changing only the model name specified.

When routing is wanted, it routes; when locking is wanted, it locks. Behavior that previously required modifying forked code in LLM Router is officially supported from the start. For this single point alone, I think migration is worth it.

Connect Claude Code in 1 Command with switchyard launch claude

Since Claude Code is an Anthropic API-exclusive agent, connecting it to an OpenAI-compatible routing proxy traditionally required inserting a conversion proxy like CCR (Claude Code Router). Since Switchyard has Anthropic Messages conversion built in, this becomes a single command.

switchyard launch claude

A proxy starts on an available port, and Claude Code launches with environment variables like ANTHROPIC_BASE_URL already swapped out. The default configuration uses a verified trio of Claude Opus 4.7 (strong), Kimi K2.6 (weak), and Gemini 3.5 Flash (classifier), and a status footer at the bottom of the screen shows per-tier request counts and token counts in real time. Sending a mix of simple instructions and heavy queries while watching the numbers in the footer show the distribution is quite satisfying.

When I asked within the session "What model are you currently running on?", it returned the route ID switchyard-deterministic-.... Claude Code itself has no knowledge that it's running in a proxied routing configuration, while behind the scenes Kimi K2.6 is responding. This transparency is the core of the launcher.

A smoke test is available for verifying connectivity. It automatically checks 8 stages, from credential resolution to proxy startup to actual Claude Code responses.

[1/8] Resolving credentials...        OK
[2/8] Reaching backend...             OK (GET /models 200, 289ms)
[3/8] Probing /v1/messages support... OK (native passthrough)
[4/8] Starting proxy...               OK (127.0.0.1:51068)
[5/8] Locating claude binary...       OK
[6/8] Round-tripping chat completion... OK (reply='ok')
[7/8] Spawning claude with proxy env...  OK (10463ms)
[8/8] Tearing down proxy...           OK
verify claude: PASS (model=moonshotai/kimi-k2.6, 14322ms)

To switch between multiple routing configurations, pass a route bundle YAML. The registered routes appear in Claude Code's /model picker, enabling mid-conversation switches like "normally leave it to the classifier, but force Sonnet for the hard parts." One caveat here: Claude Code's picker only shows models whose IDs start with claude or anthropic. Switchyard automatically generates aliases with a claude- prefix, but naming your routes like claude-smart from the start avoids confusion with what appears in the picker.

Opening the picker in practice showed the backend catalog models lined up with claude- prefix below the registered routes. I had over 340 on my end. This means you can directly switch to models not defined as routes, on a whim.

Note that route bundles are treated as the legacy format and show a deprecated warning at startup. Yet there's still no option to pass the new-format profile config to the launcher, so this is currently the only way to switch between multiple routes in the picker. Startup via bundle can take close to a minute to start listening, and I initially mistook it for a hang. These are gaps befitting a transitional v0.1.0. Note that this old/new relationship gets completely swapped around three weeks after article publication. I'll touch on that in the addendum at the end.

Does /effort Finally Work?

The other pitfall I hit during LLM Router testing was actually on the CCR side. Claude Code can adjust thinking depth in 5 levels with the /effort command, but that value goes into output_config.effort in the body, and the thinking field always has {type: "adaptive"} attached regardless of effort level. CCR's think judgment only looks at the presence of thinking, so even lightweight /effort low requests were being sent directly to expensive models. Workaround required writing a custom router to replace the judgment logic.

I reproduced the same situation in Switchyard. I sent Anthropic Messages requests to an llm-routing profile with thinking: {type: "adaptive"} attached, varying output_config.effort across 5 levels.

Since the prompt is "Say OK only." across all cases, weak is the correct answer if routing by content. Routing didn't misbehave even with the thinking field attached. Switchyard's conversion layer converts requests into an intermediate representation first, where output_config.effort is treated as a first-class field. It seems safe to say there's structurally no place for a simplistic judgment of "has thinking = heavy processing."

Building a Fully Local Routing Setup on a Single DGX Spark

Since aarch64 wheels are available, I tried it on DGX Spark too. Just creating a venv and installing nemo-switchyard[server] gets it running — the import works as-is even on GB10's aarch64 environment.

Since I was at it, I built a fully local configuration using no external APIs. I set up two models in ollama to act as tiers.

targets:
  strong:
    endpoint: ollama # http://localhost:11434/v1
    model: qwen3.6:35b
  weak:
    endpoint: ollama
    model: qwen3:1.7b

When I asked the llm-routing profile "What is 2+2?", the 1.7B answered immediately, and when I asked "Prove the undecidability of the halting problem using diagonalization," it switched to the 35B. Everything, including the routing judgment itself, runs entirely within a single DGX Spark. For those who've felt "it's wasteful to wake up the 35B for a simple question" in local LLM operation, this is quite a compelling setup.

I learned one thing from this. Initially I assigned the 1.7B to the classifier too, but small models ignored the forced tool_choice specification and responded with plain text, causing all classifications to fail. Since it's fail-open design, routing itself doesn't stop and just keeps flowing to the default tier — but the reason I noticed was that the classifier error count showed up directly in the stats API. Assigning a model of sufficient scale to reliably handle tool calling to the classifier seems to be a key practical point.

Switching My Everyday Hermes Agent Usage to Switchyard

I had been running Hermes Agent through LLM Router. Since I was at it, I switched this everyday traffic to Switchyard as well. The change was just replacing base_url in the connection config file.

model:
  default: hermes # profile name on the Switchyard side
  provider: custom
  base_url: http://localhost:4000/v1
  api_key: dummy
  api_mode: chat_completions

Switchyard's serve doesn't require authentication on the client side, so a dummy API key works fine. I assigned the openclaw policy for resident assistants to the profile. In the operational profiles.yaml, I embedded model names into tier names for clarity when reviewing later.

targets:
  weak-ds:
    endpoint: openrouter
    model: deepseek/deepseek-v4-flash
    format: openai
  strong-glm:
    endpoint: openrouter
    model: z-ai/glm-5.2
    format: openai

profiles:
  hermes:
    type: llm-routing
    profile_name: openclaw
    strong: strong-glm
    weak: weak-ds
    classifier: weak-ds
    fallback_target_on_evict: strong-glm

Here I hit one pitfall. llm-routing's fallback_target_on_evict, when omitted, looks for a target named strong. The moment I changed the tier name to strong-glm, a startup error occurred — so when using tier names other than the default strong / weak, explicit specification is required.

I ran everyday traffic through this configuration for about 15 hours overnight. Stats showed 56 routing decisions, 39 dispatches to the weak model, and 0 errors. The total DeepSeek V4 Flash usage over the period was 95 requests, approximately 2.53 million tokens, and $0.25 on OpenRouter actuals — and it was satisfying that the Switchyard count (56 classification requests + 39 weak body requests = 95) matched the billed request count exactly. GLM-5.2 on the strong side also ran with 0 errors, tool calling and streaming included. The routing decision overhead has a median of about 7.2 seconds, but since Hermes traffic is primarily cron and async message responses, this hasn't caused any real issues.

Additionally, I moved scheduled delivery jobs where I don't want to sacrifice quality to specify the target ID like strong-glm directly as the model name, bypassing routing. The usage pattern from the first half of the article — "use profile ID when you want routing, use target ID when you want to pin" — translates directly into an operational tool. Note that the scheduled news delivery job remains on LLM Router, so I have a configuration that lets me compare "LLM Router operation" and "Switchyard operation" running in parallel over the same period.

There's one more problem I found because I put it into actual operation. OpenRouter recorded 2.53 million tokens, yet the weak token count in Switchyard's stats was only a small fraction of that. After investigating, I found the implementation doesn't include streaming response usage in stats (only buffered responses are aggregated), and I was able to confirm that even when upstream sends usage frames, they get dropped. Since agent traffic is almost entirely streaming, cost aggregation becomes completely invisible in real operation. I reported this issue with repro steps and the root cause.

https://github.com/NVIDIA-NeMo/Switchyard/issues/18

What happened with this issue is covered in the addendum at the end.

I Actually Got the Classifier Selection Wrong Once

To be honest, the tier configuration I've shown so far is the second iteration. Initially, for easy comparison with the model pool from the LLM Router articles, I set strong to Claude Sonnet 4.6, weak to Nemotron 3 Nano, and used Gemini 3.5 Flash as the classifier. The reasoning was "Flash is in the name, so it must be a cheap model suitable for judgment."

Running Hermes for half a day with this initial configuration gave flawless behavior: 125 requests with 0 errors, 98.4% routed to weak, and routing judgment overhead at a median of about 2.1 seconds. However, looking at the OpenRouter aggregations, while weak body usage was about 5.67 million tokens at $0.32, classifier Gemini 3.5 Flash was about 590,000 tokens at $0.70. The classifier was using more than twice the cost of the actual responses. Since llm-routing sends the last 4 turns of conversation to the classifier every turn, the agent's long context rides along directly, averaging over 4,000 tokens per call. I understood the reason for sticky (which fixes the tier after the first decision) and cascade (which avoids calling the classifier at all) the hard way — through my wallet.

However, it wasn't just a traffic characteristics problem. Realigning the unit price table revealed it was a model selection error to begin with.

According to OpenRouter's model information, Gemini 3.5 Flash has mandatory reasoning — meaning there's no way to stop it from thinking. Checking the stats from that time, of 32,360 completion tokens from the classifier, 65% — 21,184 tokens — were reasoning. For a job that just needs to classify into 4 categories and return via tool calling, I was continuously paying $9.00/M in thinking fees every single time. The lesson: selecting by name impression while skipping the unit price check was my downfall.

What's frustrating is that this information was already in my local knowledge base. In the LLM Router testing two weeks prior, I had recorded "using a reasoning model as a judge makes thinking unstoppable and judgment heavy." Just a few days ago in OCR model selection, I had also noted "Gemini 3.5 Flash has mandatory reasoning and is overkill for simple tasks." Recording something but failing to retrieve it at the moment of writing configuration — that's meaningless. A lesson that stings.

So I unified weak and classifier to a single model in two roles — DeepSeek V4 Flash — and reselected strong as GLM-5.2, which became the current configuration used throughout this article. GLM-5.2 scores on par with Sonnet 4.6 on Artificial Analysis metrics, while being positioned at roughly 1/3 the input price and 1/5 the output price. Here's a comparison of the classifier-related metrics before and after the switch:

The cost per decision is roughly 1/12. Since I unified to the same model as weak, billing from OpenRouter no longer separates classifier charges, so the post-switch value is an estimate multiplying stats token counts by the official unit price. The estimate for 56 decisions over one night comes to $0.02, bringing the cost ratio from "classifier uses 2x the body" down to "classifier uses 1/10 of the body."

However, it didn't come for free. The median decision latency tripled, from 2.1 seconds to 7.2 seconds. It was surprising to get slower after dropping reasoning — but the work of processing a prompt averaging 4,500 tokens every time doesn't change, so the raw response speed of the model and provider shows up directly. If placed in front of an interactive agent, decision cost and decision latency need to be weighed on separate axes.

The tier swap itself required only a few lines of YAML changes and a serve restart. Start with a working configuration, put it into operation, and swap models while watching stats and billing. Being able to iterate this way naturally is a benefit of the routing proxy becoming a pip library.

Things I'm Keeping an Eye On

I've written a lot of positives, so let me honestly summarize the current caveats too.

First, the development status is Alpha, and known issues are published. At v0.1.0, there are two: cases where token counting becomes 0 in Codex integration, and cases where requests with tools fail when routed to an upstream with a fixed tool schema. The latter is something you could hit in agent operation with heavy tool use, so it's safest to ensure all tier models in your routing setup support tool calling.

Missing format: specification also needs attention. If omitted, it's treated as OpenAI format, and the documentation explicitly states that cache_control for prompt caching is stripped when sending to Claude-family models. Make sure to specify format: anthropic for targets that use Claude as upstream.

My personal version of LLM Router would pick one of 9 models from a pool using a trained classifier — a many-to-one choice. Switchyard's built-in routing is designed around a strong/weak binary choice plus a classifier for all methods, so if you want more granular routing, you'd define multiple routes and have the caller explicitly select by model name or the /model picker. Automatic judgment is narrowed to 2 choices; many-way selection is left to explicit caller specification. This is similar to the design I introduced with Sakana Fugu, where the client is responsible for calling fugu vs fugu-ultra, with orchestration happening internally on the called side.

https://dev.classmethod.jp/articles/sakana-fugu-ga-first-touch/

So will many-way automatic routing never return? Reading the code, I found something interesting. While the documentation covers 4 routing methods, the source already has a type implemented for integrating LMSYS's RouteLLM (a learning-based router using matrix factorization) as a profile. It looks like the trained router has disappeared, but the receptacle for learning-based approaches is properly prepared. This aligns with NVIDIA's response that "LLM Router algorithms are being ported to Switchyard." As someone with LLM Router training assets, this is a point I'd want to dig into in a follow-up. ...Or so I wrote, but this plan didn't pan out. See the addendum at the end.

Post-Publication Updates (Added 2026-08-05, Supplemented 2026-08-08 / 2026-08-12)

What Changed in This One Month

The problem mentioned in the latter half of the article — streaming response tokens not appearing in stats — was fixed on July 14. Since agent traffic is almost entirely streaming, this makes cost aggregation usable in real operation.

There's also a follow-up on the documentation PR I sent about the naming difference between random-routing and random_routing. The maintainer indicated they'd prefer to unify on the code side rather than explain it in documentation, so I resubmitted it as a code fix (PR #22) that accepts both hyphen and underscore forms, and it was merged on July 7. The stumbling block introduced in the main body became an upstream fix directly from the report.

Configuration saw major movement. The route bundle that I described as "deprecated as legacy format" is what remained, while the profile config I was using as the new format was removed. From late July onward, it was rewritten with a standalone Rust server and TOML configuration. The RouteLLM integration I said I wanted to dig into in a follow-up was also deleted on July 16.

However, pip install nemo-switchyard still installs 0.1.0 today, and the steps in this article still work as written. If you go look at main on GitHub, it's a completely different thing, so those starting now should first clarify whether they're working with the PyPI version or main.

After that, this Rust version was officially released as v0.2.0 on August 10, 2026, and became installable from crates.io and PyPI. Since everything from the configuration syntax to the classifier algorithms is different from v0.1.0 in this article, those starting a new installation should refer to the new article I wrote as a first-touch for v0.2.0 (published 2026-08-12).

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-v020-rust-first-touch/

Tried opencode and Fireworks on Internal Workloads

At the time I wrote the article, my local Hermes Agent was the only production deployment, but since late July I've been running another workload internally. The configuration places Switchyard behind the OSS coding agent opencode and routes to Fireworks AI models. It's a two-role setup with DeepSeek V4 Pro assigned to strong, and DeepSeek V4 Flash to both weak and classifier, with session affinity enabled.

I ran an A/B test on coding tasks. The same task set was run across three arms — auto (with routing), strong-fixed, and weak-fixed — for a total of 39 runs. All three arms scored perfectly with zero failures, and auto was approximately 27% cheaper than strong-fixed, including the cost of the classifier. Looking at just the base cost excluding the classifier, it's 40%.

That said, I'll be upfront about the conditions. Since the task set produced perfect scores even with weak-fixed, this is not proof that "routing preserved quality" — it's proof that it "reduced cost without breaking anything." Measuring quality differences would require harder tasks involving design decisions spanning multiple files. Latency was also slower with auto, with a median real-time difference of 30 seconds versus 21 seconds.

Session affinity worked straightforwardly. Classifier calls dropped by 56%, and tier switches within a session were zero. On the other hand, the same task can be pinned to either weak or strong depending on the session, so there's some variance in cost estimates.

When using DeepSeek models on Fireworks, you need to write extra_body: {} empty in the configuration. This is because the vLLM-specific parameters that Switchyard automatically appends are rejected by Fireworks, resulting in HTTP 400 errors. This has been reported upstream, but for now the safest approach is to work around it on the configuration side.

The full configuration has been published as a Docker bundle. Those who want to try the same combination can run it from here.

https://github.com/himorishige/switchyard-opencode-bundle

I've continued updating the configuration since then, switching weak to the official DeepSeek V4 Flash-0731 and strong to Kimi K3. This is also the current default configuration in the bundle. The full picture as a team AI environment — from the reasoning behind limiting to two models, to settings that keep data from leaking externally, to observing routing logs — is summarized in the next article (published 2026-08-08).

https://dev.classmethod.jp/articles/open-weight-team-ai-environment/

Summary

I ran NeMo Switchyard on Mac and DGX Spark and re-investigated the two pitfalls encountered during LLM Router verification. The issue where model names were ignored has been officially resolved as a distinction between profile/target ID usage, and the /effort misfiring no longer occurs at the design level of the conversion layer. Both the patch to the fork and the custom router I had built as a CCR are now unnecessary.

It installs via pip with no GPU required, connects to Claude Code with a single command, and can handle fully local routing on DGX Spark. While it still has some rough edges typical of an Alpha release, compared to the overhead of "forking a Blueprint and nurturing it" for an LLM Router, the barrier to adoption feels dramatically lower.

The classifier cost was reduced to roughly one-twelfth by reselecting the model. Next time, I'd like to look at comparing session affinity and stage_router — which reduce the number of classifications themselves — using real data from the internal workload mentioned in the addendum.