Model routing
also called LLM routing · provider routing · model gateway routing
The mechanism is small. Rules are matched by pattern against the requested model name and evaluated in priority order; the first match wins; each rule names a primary target — a provider and the model id that provider knows — plus an ordered list of fallbacks. When no rule matches, the request passes through to a default provider. The one detail worth copying is how that default is chosen: prefer a provider that has an exact, non-wildcard price row for the requested model, because an exact row is the evidence that this provider serves that model natively, and fall back to a pattern row only when no exact one exists. Choosing by configuration order instead sends a model to whichever upstream happens to be listed first, which works until someone reorders the list.
Routing looks like load balancing and is not, because the candidates are not interchangeable. They have different prices, different context windows, different request dialects and — decisively — different data-handling terms. That makes the fallback chain a place where a control can be quietly defeated, so every candidate, primary and fallback alike, has to be filtered through the caller’s data policy before it can be used. Enterprise procurement does not ask one data question but three independent ones: is there a zero-retention agreement, are our payloads excluded from training, and where is this processed. Collapsing them into a single flag forces the operator to silently decide which one it means and then be wrong about the other two, and it makes an EU-pinned agent and a zero-retention agent indistinguishable to the router.
Dialect compatibility is the second thing routing has to preserve. Callers send vendor-specific fields, and those fields do not survive a change of provider family: an Anthropic-shaped extra means nothing to a Gemini endpoint, and forwarding it either errors or, worse, is quietly ignored so the request executes without the constraint the caller attached. A configured fallback that would drop or reinterpret such a field belongs out of the chain, with a compatible candidate promoted in its place; when nothing compatible remains, refusing before the request leaves is more honest than serving a silently different request.
Two accounting consequences follow, and both are commonly got wrong. Because route rules, substitution and failover all change what may actually leave, a spend ceiling has to be applied after routing and against every candidate the chain could execute — not against the model name the caller typed, which by then may not correspond to anything that will be billed. And once a call has succeeded, the route must be narrowed to the provider that actually answered before the usage is priced. A failed-over request priced against the original primary attributes spend to a vendor that never ran it, and is wrong in dollars too whenever the fallback prices differently.
What routing cannot do is worth saying beside what it can. It cannot verify a vendor’s retention or training claim; those flags are operator-asserted configuration, and the router faithfully enforces whatever it was told. It cannot make two models behaviourally equivalent — equivalence of governance, meaning that the same rules bind on every upstream, is a stronger and far more checkable claim than equivalence of answers, and it is the one to insist on. And a pass-through decision offers no fallbacks at all, which is the correct behaviour: nothing has been configured about where that model should go if its provider fails, and inventing a destination would be routing traffic by guess.
One rule, three candidates, and the one the policy removes
A rule matches `gpt-4*` and names a primary on a US-hosted provider with two fallbacks: an EU-hosted deployment of the same model family and an aggregator. The calling agent’s data policy requires EU processing and no training on payloads. Resolution drops the US primary before anything is sent, promotes the EU deployment to primary and keeps the aggregator behind it only if its recorded flags satisfy both constraints. Both surviving candidates are then priced, because the ceiling has to hold for whichever one runs. If the aggregator has no active price row and the agent has any USD ceiling configured, the call is refused before egress rather than admitted against a fallback nobody can price.
What model routing is routinely confused with
- Load balancing
- A load balancer spreads traffic across interchangeable replicas of one service, and any replica answering is as good as any other. Model routing chooses between vendors that differ in price, context window, dialect and contractual data handling, so the choice changes both the answer and the invoice and has to be recorded rather than merely made.
- Tier routing
- Model routing decides where a named model goes. Tier routing decides which model is named, by classifying the request first. Tier routing runs ahead of routing and its output is fed through the same rules, the same provider selection and the same data policy — a substitution that skipped them would be a cost optimisation with a governance hole in it.
Related terms
Typed failover
Typed failover is a fallback policy that decides whether to try the next provider from the class of the failure rather than from a retry count: transient classes — a timeout, a rate limit, a server error — move to the next candidate, while failures that are properties of the request or the credential stop where they are. It exists because some failures are identical at every vendor, and because one of them is a refusal that a second attempt would convert into an apparent success.
Tier routing
Tier routing is the substitution of a cheaper model for the one a caller named, decided before the call by classifying what the request is actually asking the model to do. It is a cost control rather than a quality feature, so the two properties that make it defensible are that it never routes above the tier that was requested, and that it records why it moved.
Token cost accounting
Token cost accounting is the practice of turning a provider’s reported token usage into a defensible monetary figure for each model call: resolving the price row for the model and provider that actually served the request, splitting the reported usage into token classes that do not overlap, and applying the right rate to each. It is harder than multiplying tokens by a rate because providers disagree about whether cached prompt tokens are reported inside or outside the prompt total, and because the model that gets billed is not always the model that was asked for.
Circuit breaker
A circuit breaker is a state machine in front of a remote dependency that stops sending it traffic once a threshold of consecutive failures is reached, waits a fixed cooldown, then allows one probe request to decide whether to resume. Its purpose is to fail immediately against a dependency already known to be down, instead of paying a full timeout on every request until it recovers.
Where Token Observe does this
The definition above is the field's, not the product's. This is the part of the product that implements it, for a reader who wants to see one.
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Nothing verifies a provider’s ZDR or training claim
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
One billable egress: no retry, no failover
The terms next to this one
Where the money goes, why providers disagree about how to count it, and what a spend control has to do to be a control rather than an alert.
Definitions are the easy part.
The glossary is written to be useful whether or not you ever buy anything. If you have got to the point of deciding how to implement one of these in your own estate, say what your agents do and you will get a straight answer about what it would actually take.
no form · no qualification step · no sales desk · the other three ways in