Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
A fallback chain is a governance surface.
Configure one and you have already built the mechanism that can quietly defeat your own controls. One vendor declines a request, the chain sends the same payload to the next, the second answers, and the trace records a success. The retry count could not tell a provider being down from one saying no.
There is a second problem, and it decides whether your policies mean anything. An estate that adds Gemini beside OpenAI depends on a rule written once binding both, and one that fires on OpenAI but not Gemini is worse than none: the operator believes they are covered.
How it works
Step 8 of the request path. The order is encoded in the evaluator, and where each step sits is the reason the guarantee holds.
- 01
Match a rule, then filter every candidate
Rules are sorted by priority and the first matching model pattern wins, naming one target and an ordered list of fallbacks. One predicate filters the primary and every fallback against the agent's data policy. - 02
Price the chain, then retry by class
An agent with any USD ceiling is refused before egress when the resolved target, or any fallback behind it, has no price row, and it then gets one network attempt for the whole chain because a timeout may already have been billed. Everything else is retried twice, and five transient failures open that provider's breaker. - 03
Narrow the route to the provider that served
Once a call succeeds the route is rebuilt around the provider that answered, and that is what metering prices against. This was a real defect: metering used to get the original route, so a failed-over request was priced wrong.
- beat 01Name the caller
- 1Authenticate
- 2Resolve the agent
Nothing is decided for an anonymous caller. The key resolves to one registered agent with a named owner, and its roles, its ceilings and any stop switch in force are read live from the store on every request, not from a cached copy, so a suspension takes effect on the next call instead of after a redeploy.
- beat 02Start the record, then read what is being sent
- 3Open the trace
- 4Sanitise
- 5Scan
The trace is opened before anything can refuse the request, which is why a blocked call is recorded as thoroughly as one that went through. Only then is the text normalised and scanned, in that order, so the detectors see what the model will read rather than what a human reviewer can see.
- beat 03Decide once
- 6Govern
- 6bIntersect with the named human
- 7Enact
One function returns one verdict: allow, block, or hold it for a named person to approve. Every check lives in that one place rather than scattered along the path, which is what makes it possible to read your rules and know what they do.
- beat 04Send it, and only then spend the money
- 8Route
- 9Call upstream
Routing fixes which provider and model will serve the call, so where an agent has a USD ceiling it is tested against the price you will pay, not the one the caller typed. Routing is also the last step that can still refuse locally: after it, the request has left your network.
- beat 05Govern the answer, then count the cost
- 10Govern the response
- 11Meter and record
A tool call the model proposes is judged before it reaches your agent. Metering runs last and on every ending, including the refusals and the streams a client abandoned halfway, because the tokens were spent either way and a partial trace is still evidence.
Step names and their grouping are the product’s own, and the order is encoded in the evaluator rather than drawn for the page.
The class of the failure decides, not a counter.
Upstream failures land in a closed union of seven classes, and exactly three of them move the request on. Three of the four that stop fail identically at every vendor, so trying the next only pays a second vendor to return the same error. The fourth is deliberate: a content-policy refusal might well have succeeded elsewhere, and that is why it is not tried. The switch is exhaustive, so an eighth class does not compile until its behaviour is decided.
timeout rate_limited server_error
→ fail over to the next provider in the chain
context_too_long → stop (a property of the payload)
content_policy → stop (a signal, not an outage)
auth → stop (fails the same everywhere)
invalid_request → stop (malformed at every vendor)Three data constraints, checked on every fallback.
Each agent carries three data-handling requirements rather than one flag, because procurement asks three independent questions: a vendor can be barred from training while still retaining payloads for abuse monitoring, and residency is neither. Every constraint must hold before a provider may serve that agent, and when nothing does, routing refuses with a 403 rather than downgrading. A Bedrock row declaring a region must also agree with its endpoint and its signing region.
The weakness that matters most to a CISO is that those provider-side fields are operator-asserted, not verified: nothing checks that a row marked zero-data-retention has a signed agreement behind it, so the control is as good as the person who filled in the form.
Equivalence is a test, not a sentence in a brochure.
A table-driven suite runs six guarantees against each of the six governed kinds: an ungranted model is refused and the provider never sees it; a card number reaches the upstream masked; an agent over its cap is refused before egress; a kill switch silences the upstream; the trace carries a cost that reconciles against that provider's own price row; and an approval gate blocks, then opens. Every assertion is made at the provider, so an empty call list is evidence the payload never left.
The table is keyed on the provider-kind type, so adding a kind stops the file compiling. That guard failed silently once. For two releases the typecheck configuration read only the source directory, so it sat at four of six kinds while the documentation claimed six.
- OpenAIChat Completions, Responses and embeddings
- AnthropicMessages, including the x-api-key dialect
- Google GeminiNative Gemini ingress
- OpenRouterWith caller-controlled route selectors refused before egress
- Amazon BedrockFor estates that will not egress to a model vendor directly
- Azure OpenAIDeployment-name routing under the same policies
- Your own endpointAny OpenAI-compatible endpoint you host
Cloud setup offers OpenAI, Anthropic, xAI and Google. Use separate billing and inference credentials.
Each line closes off an assumption a reader would otherwise carry into a proof of concept.
What this does not doWhat this leans on
Policy engine
One deterministic verdict on every governed request: allow, block, redact, or park it for a human.
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
Where this comes up
Jobs that draw on it
Guides that lean on it
The term, defined
Model routing is the resolution, at the moment of the call, of a requested model name to a concrete provider and model, together with the ordered list of alternatives that may serve it if the first one fails. It lets a client that only knows one model name be pointed at a different vendor, region or price without the client changing, and it is where a multi-provider estate’s data-handling and cost constraints are applied.
What happens when a provider refuses on content-policy grounds?
It stops there. Sending the same payload to the next vendor is a second attempt at the same action, and the trace would record a success with the objection hidden. An authentication failure, a malformed request and an over-long context stop for a different reason: each fails the same way everywhere.
Can an agent be pinned to providers in one region?
Yes. An agent's data policy carries three independent requirements (retention, training and region), and every one it states must hold on the primary target and on every fallback. When nothing satisfies it the request is refused with a 403, never downgraded. Two limits travel with it: the flags are operator-asserted, and region matching has no hierarchy.
How do you know a policy behaves the same on Gemini as on OpenAI?
It is tested per provider kind. A suite runs six guarantees for each governed kind, every assertion is made at the provider, and each refusal test carries a positive control.
Prefer to ask a person? Write to James
Bring us the agent you are least comfortable with.
Create a cloud account and workspace. No card or server installation. Model calls and Google BigQuery scans can incur separate provider charges.