AI Model Routing in 2026: Measure the Router, Not Just the Model

Oscar Gallo

Published on July 16, 2026 · Updated on September 15, 2026

Model routing is becoming a product control. It solves a different problem from multi-agent orchestration, and it needs its own evaluation.

Choosing one smarter default model does not solve every workload.

The useful question is whether the system sends each job to the right model.

That distinction matters because model routing has moved into mainstream developer products. On September 14, GitHub said that efficiency, balance, and intelligence tiers were rolling out in Visual Studio Code, Copilot CLI, and the GitHub Copilot app. All three use the same model pool available to the user, and Copilot evaluates each prompt before choosing a model.

Four days earlier, OpenAI launched its Agents API in public beta. It puts context management, tool search, programmatic tool calls, and subagent coordination inside a managed harness.

The model still matters. It is no longer the only useful unit of evaluation.

You need to measure the model, the router, the harness, and the work they receive.

A model leaderboard is not an architecture

On Episode 14 of Human in the Loop, Matt and I discussed what gets lost when every model launch becomes a horse race.

Models now vary across price, latency, modality, context handling, tool use, and reasoning behavior. A benchmark winner can still be the wrong default for a routine classification job. A cheap model can become expensive after retries, failed tool calls, or human cleanup.

Think about a software company. You would not hire six principal engineers to rename folders, update button copy, and sort support tickets. You would also not ask the newest intern to make every architectural decision.

AI systems need the same discipline. Choose the lowest-total-cost route only among options that meet explicit quality, latency, and policy requirements, then prove that it did.

What is multi-agent orchestration?

Multi-agent orchestration coordinates several AI agents around one outcome.

For this article, I use "agent" as shorthand for a role executed with its own instructions, tools, working context, and handoff or stop rules. Implementations can keep durable state in the agent, session, or runtime. Several agents may use the same underlying model, and one agent may use several models.

A central agent might plan the work, delegate independent tasks, track progress, and combine the results. Other systems hand work between peer agents. The important part is ownership: which agent is responsible for each part of the workflow?

The closest human equivalent is a team:

• A product role defines the outcome.
• An architecture role proposes the smallest safe change.
• Implementation roles build independent parts.
• A test role checks the result against acceptance criteria.
• A human retains release authority.

OpenAI's agent-building guide recommends an incremental path. A single agent with clear tools is often enough. Split the workflow when prompts, tool selection, or responsibilities become too difficult to manage reliably. Microsoft's architecture guidance gives the same warning from another angle: multi-agent systems add coordination overhead, latency, cost, and failure modes.

More agents do not automatically produce better work. Five confused agents give you five places to fail.

What is AI model routing?

Model routing chooses which model should process a request.

A router can use explicit rules, a task classifier, an optimization policy, or a vendor-managed selector. It can choose a model before an agent starts, make a different choice inside an agent's workflow, or escalate after a deterministic check fails.

In an AI system:

• The orchestrator decides which role owns the work.
• The router decides which model should attempt it.

Those are separate decisions. A research agent and a coding agent can call the same model. A single coding agent can route a file search to a fast model, implementation to a stronger model, and image inspection to a multimodal model.

Routing can reduce cost or response time while preserving quality. It can also make all three worse. The result depends on route accuracy, fallback behavior, retries, and the test used to define "good enough."

The hard part is not connecting three APIs. The hard part is defining success well enough to know when the router chose correctly.

Multi-agent orchestration vs. model routing

The two approaches can work together, but neither requires the other.

You can orchestrate several agents that all use one model. You can also route between several models inside a single-agent workflow.

Model routing moved into the product

GitHub's Copilot auto-selection rollout makes the routing decision visible to the user. Efficiency favors lower cost, balance considers cost, quality, and latency together, and intelligence prioritizes quality for complex tasks. The eligible model pool stays the same across the three tiers; the routing objective changes.

That is a useful design lesson. "Pick the best model" is incomplete until you define what best means for this request.

OpenAI's public-beta Agents API shows the same shift at the harness level. Developers choose the task, model, tools, and execution environment while OpenAI operates the harness. The API also supports independent subagents running in parallel. A team can outsource part of the orchestration machinery, but it still owns the task definition, evaluation, permissions, and release decision.

Routing and orchestration are becoming platform features. Your responsibility does not disappear when a vendor supplies them.

When one agent is enough

Multi-agent demos look impressive. Several terminal windows start moving at once. Agents talk to each other. Progress bars fill the screen.

None of this proves the system works.

Start with one agent when:

• One prompt and a small set of tools can complete the job.
• Every step depends on the previous step.
• The workflow still changes every week.
• Your team lacks a clear quality test.

Single-agent systems are easier to inspect, test, and maintain. They also expose weak requirements sooner. If one agent does not understand the job, adding four more agents spreads the confusion.

Microsoft puts the principle plainly in its Agent Framework overview: if a normal function can handle the task, use the function instead of an AI agent.

AI should earn its place in the architecture. So should every extra agent.

When parallel agents earn their cost

Parallel multi-agent orchestration becomes useful when the work can be split into meaningful, independent directions.

Sequential or handoff agents can still earn their place when stages need different prompts or tools, separate security boundaries, focused context, or an explicit review gate. Their benefit is specialization or isolation, not parallel speed, so compare them with a single-agent baseline and include the added latency and cost.

Research is a clear example. One agent searches technical sources while another studies competitors and a third gathers customer language. A lead agent can then compare the findings and resolve gaps.

Anthropic uses this orchestrator-worker pattern in its multi-agent research system. Its lead agent creates three to five subagents in parallel, and those subagents also call tools in parallel. Anthropic reports that this cut research time by up to 90% for complex queries in its system. The same post says its agents used about four times as many tokens as chat interactions and its multi-agent systems used about 15 times as many tokens. That is a result from Anthropic's implementation, not a promise for every multi-agent workflow, and Anthropic says the task must be valuable enough to justify the added cost.

The pattern can also fit:

• Code review across independent security, performance, and accessibility checks
• Due diligence across legal, financial, and product documents
• Research that needs several source families inspected at the same time
• Product work with front-end and back-end changes that have a stable contract

Parallel agents can reduce elapsed time when the branches are genuinely independent and the coordination cost stays below the gain. Dependent stages may still justify separate agents for specialization, isolation, or quality control, but the system should not claim a parallel speed benefit it cannot produce.

A practical architecture for software work

Suppose you want an AI system to build a product feature.

A measured first version could look like this:

1. A single lead agent reads the request and the repository rules.
2. It turns the request into acceptance criteria and identifies independent work.
3. Separate workers run only when those tasks can proceed without conflicting edits.
4. Automated tests, type checks, and linting evaluate the combined result.
5. A human reviews the diff and retains release authority.

Routing can happen at any of those steps. A fast model may classify files or summarize logs. A stronger model may inspect an ambiguous architecture decision. A multimodal model may review a screenshot. Escalation should follow an observable failure, a policy boundary, or a calibrated signal, not a model saying that it feels confident.

Your premium model becomes a specialist. It does not need to be the entire payroll.

The hidden cost of adding agents

Every new agent adds more than another model call. It also adds context, permissions, output to verify, retries, timeouts, logs, and another place where an error can affect the final result.

Agent count is not a success metric. Measure the outcome:

• Did the workflow finish?
• Did the result pass the same acceptance test as the baseline?
• How long did the full run take at the median and at the slow end?
• What did the accepted result cost after retries and tool calls?
• How often did a human need to intervene?
• Which route or agent caused each failure?

Without those answers, you have a demo with an invoice.

Measure the router, not just the models

A router needs its own evaluation. Microsoft's model-router guidance starts with a baseline, representative prompts, and workload-specific criteria for quality, cost, latency, and policy. A practical comparison uses three configurations on the same tasks:

1. Always use the strongest model you can afford.
2. Always use the cheaper default model.
3. Let the router choose between them using the proposed fallback policy.

Keep the prompts, tools, environment, and acceptance criteria fixed. Record accepted-outcome rate, end-to-end cost, median and tail latency, fallback frequency, human corrections, and wrong-route failures.

If the router cuts token cost but creates more retries, it may be more expensive. If it improves average latency but sends rare high-impact work to the wrong model, it may be unsafe. If it matches the premium baseline on accepted outcomes and costs less, then it has earned its place.

Specific Labs makes a related point in its September 2026 Real-SWE benchmark. It evaluates model-and-harness combinations on private production codebases rather than treating a model as an isolated contestant. In its published results, higher estimated rollout cost did not guarantee a higher resolution rate. That benchmark does not prove your router will work. It does show why model price or benchmark rank alone is not enough.

These are publisher-reported results from a private-code benchmark. The public page cannot make the private tasks independently reproducible, and the rollout costs are estimates.

At low volume, routing purely for inference savings may not repay its implementation and evaluation cost. Capability or policy routing can still be justified.

Build routing rules you can explain

Start with rules that a reviewer can inspect:

• Route low-risk classification and extraction to the fast model.
• Route ambiguous architecture or policy-sensitive work to a stronger model.
• Route image-heavy work to a model tested on that modality.
• Escalate after a failed schema check, test, policy gate, or other observable condition.
• Log the selected route, fallback reason, result, elapsed time, and total cost.

Do not treat an uncalibrated confidence score as proof. A router that says "92% confident" is useful only if that number predicts success on your own task distribution.

Version the routing policy alongside the model and harness. When a provider changes a model pool, price, or selector, rerun the evaluation before treating the new behavior as equivalent.

The harness is part of the product

An agent harness can manage instructions, context, tools, memory, permissions, retries, routing, and feedback. Those choices can change the result even when the model name stays fixed.

OpenAI's Agents API announcement is unusually direct about this. The managed harness compacts context, searches for tools, supports programmatic tool calls, and coordinates subagents. A model comparison that ignores those behaviors can tell you which engine won a lab test while missing which system completes your work.

Evaluate the unit you deploy:

• Model version
• Routing policy
• Harness version and configuration
• Tool set and permissions
• Task distribution
• Acceptance test

Change one of those and you have changed the system.

Earn the second agent and every route

Before adding another agent, answer five questions:

1. What exact job does this agent own?
2. Why does the current workflow fail to complete it reliably?
3. Can the new work run independently, or does it create another handoff?
4. Which model meets the quality bar at the lowest total cost?
5. How will you test the final result?

Before adding a router, add one more: what baseline must it beat?

Build one measured workflow. Track accepted outcomes, cost, and elapsed time. Add the second agent or the second model only after the first system exposes a clear, testable gap.

The architecture wins when it improves the target outcome without violating the other acceptance criteria. The model name is only one part of that result.