Model routing

If you have configured a fallback chain, you have already built the mechanism that can quietly defeat your own controls. One vendor’s safety system declines a request; the chain sends the same payload to the next vendor; the second one answers; the trace records a success and the objection is nowhere in the evidence. Nothing was bypassed on purpose. The retry count simply did not know the difference between a provider being down and a provider saying no.

The same shape of error costs money in three other ways. A bad credential fails identically everywhere that credential is used, so failing over hides a broken key behind a more expensive provider until the invoice arrives. An over-long context is a property of the payload, not the provider, and fallbacks usually have similar or smaller windows, so failing over pays a full input-token charge to receive the same error twice. A malformed request is malformed at every vendor.

The second problem is equivalence, and it is the one that decides whether any of your policies mean anything. An estate that adds Gemini or Bedrock beside OpenAI now depends on the claim that a rule written once binds all of them. That claim is easy to make and easy to be wrong about: a policy that fires on OpenAI but not on Gemini is worse than no policy, because the operator believes they are covered. Token Observe answers it by asserting every guarantee once per provider kind rather than once for the default upstream, and by asserting it at the provider — what the upstream was actually handed — rather than at the response the client got back.

The third is that enterprise procurement does not ask one data-handling question. It asks three, and they are independent in practice: is there a zero-data-retention agreement, are our payloads excluded from training, and where is this processed. A single retention flag makes the operator silently decide which of the three it means and then be wrong about the other two, and it conflates a residency constraint — a legal and geographic property — with a retention one, so an EU-pinned agent and a zero-retention agent become indistinguishable to the router.

Failover is decided by what went wrong, not by how many times it has gone wrong. Upstream failures are classified into a closed union of seven classes, and exactly three of them — timeout, rate-limited and server error — move the request to the next provider in the chain. Content-policy refusal, authentication failure, over-long context and invalid request all stop where they are. Three of those fail identically at every vendor, so trying the next one only pays a second vendor to return the same error. The fourth is the deliberate one: a content-policy refusal might well have succeeded elsewhere, and that is precisely why it is not tried — a second attempt at a declined action, recorded on the trace as a success, is the opposite of evidence.

The alternative was considered and rejected on the record. Retrying every failure down the chain maximises apparent availability and is what a pure availability product should do; three failure modes make it unacceptable for a governance product. A content-policy refusal is a governance signal, and a second attempt at the same action recorded as a success is the opposite of evidence. An authentication failure means the vendor key is wrong, revoked or rate-scoped, and failing over masks a broken key behind a more expensive provider. An over-long context is a property of the payload, and fallbacks usually have similar or smaller windows.

The switch that makes the decision is exhaustive with no default branch, so adding an eighth class is a compile error until someone has decided its failover behaviour. That is the whole enforcement mechanism, and it is deliberately a language-level one rather than a review convention.

The limits are stated in the same register. Classification is a lossy mapping from heterogeneous provider error shapes onto seven classes and it will get cases wrong: a content refusal returned as a bare HTTP 400 is indistinguishable from a malformed request, and while both correctly decline to fail over, the reason recorded on the trace will be the wrong one. A provider that returns a 5xx for what is really a refusal will be failed over, producing exactly the laundering this design exists to prevent — the control depends on upstream error hygiene Token Observe does not control. And availability is genuinely traded away here: requests that a dumber gateway would have completed on a second provider will fail.

Every governance guarantee is asserted once per provider kind, not once for the default upstream and taken on trust for the rest. A table-driven suite runs six guarantees against each of the six governed kinds: a model the agent has no grant for is refused and the provider never sees it; a card number in the prompt reaches the upstream masked; an agent over its daily cap is refused before egress; a kill switch silences the upstream rather than telling the client a story; the trace carries that provider’s name, its own price row and a cost that reconciles arithmetically; and a request gated on human approval blocks, then executes exactly once the approval is granted.

Two properties are what make those assertions mean what they say. Blocked is asserted at the provider rather than at the response — each fake upstream records what it was asked to do, so an empty call list is evidence that the payload never left, where a 403 alone would not distinguish a refusal from a call that happened and was then hidden. And every refusal test carries a positive control, because without one, the provider was never called would also pass if routing never reached that provider at all, which would make the entire suite green and worthless.

A second suite proves the stronger statement an operator actually relies on: one policy, written once, binds every upstream in the same deployment. Six providers are registered in one process, a single redaction policy is created, and the raw card number appears in nothing any of them received; then one kill switch silences all six. A third suite checks the real provider registry can build a live client for every governed kind from a configured, enabled, credentialed row — because the governance suites inject their fakes straight into the registry, and that shortcut would otherwise happily hide a provider Token Observe cannot actually reach.

The guard against the table going stale is that it is a record keyed on the provider-kind type, so adding a kind stops the file compiling until its author has said how the same six guarantees are met. That guard failed silently once, and the confession is the first thing in the test file: for two releases the typecheck configuration included only the source directory, so nothing ever read the test file, and the table sat at four of the six governed kinds. Bedrock and Azure OpenAI were reachable through the gateway and asserted nowhere, while the documentation claimed equivalence across every kind. A compile-time guard that nothing compiles is a comment.

Request fields that would move a routing or processing decision outside Token Observe are rejected before egress rather than forwarded. On the OpenRouter dialect that means the caller-controlled models, provider, route, plugins, transforms and web-search options fields: they select vendor-side fallback models, override provider selection and data-policy enforcement, or attach vendor-side processing, search egress and charges that Token Observe cannot authorise or price. The refusal names the field and says what to configure instead — fallback targets and provider constraints belong on a route rule, where the data policy and the price ceiling still reach them.

The same reasoning covers model-name modifiers. A retained target ending in one of the vendor’s routing suffixes enables web-search egress or changes sub-provider selection without making either governable, so those are refused too — but checked after route resolution rather than on the inbound body, so an operator alias that happens to end in the same text and maps to a safe base model still works. Ordinary OpenAI-compatible extras are deliberately not on the list: a seed, a response format or a reasoning-effort setting does not choose a different provider or attach an unpriced processor.

Cheaper-tier substitution is the other place a routing decision could have escaped, and it does not. It is off unless an operator turns it on, and when it is on the only thing it changes is which model name the rest of resolution sees: route rules, provider selection and the data policy all still apply to the substituted model, because a cost optimisation that skipped them would be a governance hole. If no route exists for the substituted model — a disabled provider, or one this agent’s data policy forbids — the caller is served the model they asked for instead, because a cost optimisation must never take a request down.

The substitution itself is deliberately conservative. It is a heuristic over signals in the request, not a classifier model, because a router that had to call a model to decide which model to call would spend a meaningful share of the saving it exists to produce and would add a second upstream dependency to the request path. It is pure and deterministic, so a decision recorded on a trace can be reproduced from that trace months later. It never routes above the tier the caller asked for, the agent’s tier ceiling applies regardless of everything else, and a downgrade needs both an explicit opt-in and a confident classification — two agreeing signals, not one weak one tipping a borderline score. Spending more of someone’s money than they asked for is not a cost-optimisation feature.

Whatever it decides is written to the trace, including when it decides nothing. An operator who has turned tier routing on needs to see the swap that was not made as much as the one that was, because otherwise the router doing nothing and the router being switched off look identical from the outside.