--- title: "Capability: Model routing" canonical_url: "https://tokenobserve.com/platform/model-routing" published: 2026-09-02 last_updated: 2026-10-21 --- # Capability: Model routing Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal. ## What it is Token Observe routes one agent across six model providers and applies the same permissions, policies, redaction, budgets and tracing to all of them. OpenAI, Anthropic, Gemini, OpenRouter, Bedrock and Azure are first class, plus any OpenAI-compatible endpoint you register. Every candidate in a fallback chain is filtered through the agent's data policy first, so failing over cannot bypass a residency constraint. Failover is typed rather than counted: a timeout, a 429 or a 5xx moves to the next provider, while a content-policy refusal, an authentication failure, an over-long context and a malformed request stop where they are. After the call the route narrows to whichever provider served it, so the ledger prices against the vendor that will invoice you. ## Facts - First-class upstreams: OpenAI, Anthropic, Gemini, OpenRouter, Bedrock, Azure - Failover classes: Seven; three fail over, four deliberately do not - Data policy: Retention, training and region, checked on every fallback - Circuit breaker: Five consecutive transient failures to open, 30s to a probe ## The limit Operator-asserted: Nothing verifies a provider's ZDR or training claim ## Why it exists: A fallback chain is a governance surface. Configure one and you have already built the mechanism that can quietly defeat your own controls. One vendor declines a request, the chain sends the same payload to the next, the second answers, and the trace records a success. The retry count could not tell a provider being down from one saying no. There is a second problem, and it decides whether your policies mean anything. An estate that adds Gemini beside OpenAI depends on a rule written once binding both, and one that fires on OpenAI but not Gemini is worse than none: the operator believes they are covered. ## How it works - Match a rule, then filter every candidate: Rules are sorted by priority and the first matching model pattern wins, naming one target and an ordered list of fallbacks. One predicate filters the primary and every fallback against the agent's data policy. - Price the chain, then retry by class: An agent with any USD ceiling is refused before egress when the resolved target, or any fallback behind it, has no price row, and it then gets one network attempt for the whole chain because a timeout may already have been billed. Everything else is retried twice, and five transient failures open that provider's breaker. - Narrow the route to the provider that served: Once a call succeeds the route is rebuilt around the provider that answered, and that is what metering prices against. This was a real defect: metering used to get the original route, so a failed-over request was priced wrong. ## Where it runs in the request path Step 8 of the request path ## What it does not do - The data-handling flags on a provider are operator-asserted. Nothing checks that a row marked zero-data-retention has an agreement behind it. - Failure classification is lossy. A 5xx that is really a refusal is failed over (the laundering this design exists to prevent), so the control depends on upstream error hygiene. - Region matching is string equality with no hierarchy: an agent requiring eu will not match a provider tagged eu-west-1, and EU or UK cannot be expressed. - There is no mid-stream failover. It applies to opening the stream; once bytes reach the client a retry would emit the opening of the answer twice, so the stream ends. - Embeddings are proxied to OpenAI and OpenRouter rows only. Anthropic, Google, Bedrock and Azure targets fail closed before the credential resolves. - Routing is not one-directional. A rule an operator writes can point a request at a costlier target, and no check compares the prices. Only the optional tier optimiser goes strictly downward. ## Read next - https://tokenobserve.com/platform/policy-engine - https://tokenobserve.com/platform/spend-controls - https://tokenobserve.com/platform/flight-recorder ## Questions and answers Q: What happens when a provider refuses on content-policy grounds? A: It stops there. Sending the same payload to the next vendor is a second attempt at the same action, and the trace would record a success with the objection hidden. An authentication failure, a malformed request and an over-long context stop for a different reason: each fails the same way everywhere. Q: Can an agent be pinned to providers in one region? A: Yes. An agent's data policy carries three independent requirements (retention, training and region), and every one it states must hold on the primary target and on every fallback. When nothing satisfies it the request is refused with a 403, never downgraded. Two limits travel with it: the flags are operator-asserted, and region matching has no hierarchy. Q: How do you know a policy behaves the same on Gemini as on OpenAI? A: It is tested per provider kind. A suite runs six guarantees for each governed kind, every assertion is made at the provider, and each refusal test carries a positive control. The long-form detail on this capability (the anchored sections, the worked policies and the error envelopes) is on the page itself at https://tokenobserve.com/platform/model-routing.