---
title: "Glossary: Model routing"
canonical_url: "https://tokenobserve.com/glossary/model-routing"
published: 2026-09-02
last_updated: 2026-10-21
---

# Glossary: Model routing

Model routing is the resolution, at the moment of the call, of a requested model name to a concrete provider and model, together with the ordered list of alternatives that may serve it if the first one fails. It lets a client that only knows one model name be pointed at a different vendor, region or price without the client changing, and it is where a multi-provider estate’s data-handling and cost constraints are applied.

## Also called

- LLM routing
- provider routing
- model gateway routing

## In practice

The mechanism is small. Rules are matched by pattern against the requested model name and evaluated in priority order; the first match wins; each rule names a primary target — a provider and the model id that provider knows — plus an ordered list of fallbacks. When no rule matches, the request passes through to a default provider. The one detail worth copying is how that default is chosen: prefer a provider that has an exact, non-wildcard price row for the requested model, because an exact row is the evidence that this provider serves that model natively, and fall back to a pattern row only when no exact one exists. Choosing by configuration order instead sends a model to whichever upstream happens to be listed first, which works until someone reorders the list.

Routing looks like load balancing and is not, because the candidates are not interchangeable. They have different prices, different context windows, different request dialects and — decisively — different data-handling terms. That makes the fallback chain a place where a control can be quietly defeated, so every candidate, primary and fallback alike, has to be filtered through the caller’s data policy before it can be used. Enterprise procurement does not ask one data question but three independent ones: is there a zero-retention agreement, are our payloads excluded from training, and where is this processed. Collapsing them into a single flag forces the operator to silently decide which one it means and then be wrong about the other two, and it makes an EU-pinned agent and a zero-retention agent indistinguishable to the router.

Dialect compatibility is the second thing routing has to preserve. Callers send vendor-specific fields, and those fields do not survive a change of provider family: an Anthropic-shaped extra means nothing to a Gemini endpoint, and forwarding it either errors or, worse, is quietly ignored so the request executes without the constraint the caller attached. A configured fallback that would drop or reinterpret such a field belongs out of the chain, with a compatible candidate promoted in its place; when nothing compatible remains, refusing before the request leaves is more transparent than serving a silently different request.

Two accounting consequences follow, and both are commonly got wrong. Because route rules, substitution and failover all change what may leave, a spend ceiling has to be applied after routing and against every candidate the chain could execute — not against the model name the caller typed, which by then may not correspond to anything that will be billed. And once a call has succeeded, the route must be narrowed to the provider that answered before the usage is priced. A failed-over request priced against the original primary attributes spend to a vendor that never ran it, and is wrong in dollars too whenever the fallback prices differently.

What routing cannot do is worth saying beside what it can. It cannot verify a vendor’s retention or training claim; those flags are operator-asserted configuration, and the router faithfully enforces whatever it was told. It cannot make two models behaviourally equivalent — equivalence of governance, meaning that the same rules bind on every upstream, is a stronger and far more checkable claim than equivalence of answers, and it is the one to insist on. And a pass-through decision offers no fallbacks at all, which is the correct behaviour: nothing has been configured about where that model should go if its provider fails, and inventing a destination would be routing traffic by guess.

## Example: One rule, three candidates, and the one the policy removes

A rule matches `gpt-4*` and names a primary on a US-hosted provider with two fallbacks: an EU-hosted deployment of the same model family and an aggregator. The calling agent’s data policy requires EU processing and no training on payloads. Resolution drops the US primary before anything is sent, promotes the EU deployment to primary and keeps the aggregator behind it only if its recorded flags satisfy both constraints. Both surviving candidates are then priced, because the ceiling has to hold for whichever one runs. If the aggregator has no active price row and the agent has any USD ceiling configured, the call is refused before egress rather than admitted against a fallback nobody can price.

## Routinely confused with

Load balancing: A load balancer spreads traffic across interchangeable replicas of one service, and any replica answering is as good as any other. Model routing chooses between vendors that differ in price, context window, dialect and contractual data handling, so the choice changes both the answer and the invoice and has to be recorded rather than merely made.

Tier routing: Model routing decides where a named model goes. Tier routing decides which model is named, by classifying the request first. Tier routing runs ahead of routing and its output is fed through the same rules, the same provider selection and the same data policy. A substitution that skipped them would be a cost optimisation with a governance hole in it.