Typed failover

A retry counter cannot tell the difference between a provider being down and a provider saying no. Both arrive as a non-2xx response. That means the fallback chain built to absorb an outage is also the mechanism that will replay a declined request at a vendor with a different safety configuration, take the second answer, and record a success — with the objection nowhere in the record. Nothing was bypassed on purpose; the counter simply never knew what it was counting. Classifying the failure first, and letting the class decide, is the smallest change that closes it.

Four classes should not fail over, each for its own reason. A content-policy refusal is a governance signal: one vendor’s safety system declined, and sending the same payload onward is a second attempt at the same action. An authentication failure means the credential is wrong, revoked or scoped, so it fails identically everywhere that credential is used and failing over hides a broken key behind a more expensive provider until the invoice arrives. An over-long context is a property of the payload rather than the provider, and fallbacks usually have similar or smaller windows, so trying one pays a full input-token charge to receive the same error. A malformed request is malformed at every vendor. Note that the first of those is the only one that might genuinely have succeeded elsewhere — which is exactly why it is the one not tried.

Three classes should fail over: timeout, rate limit and server error. These are transient and provider-specific, and they are the outages a chain exists for. One ambiguity is worth naming rather than burying: a timeout cannot prove the upstream did not execute and bill the request, so failing one over risks paying two vendors for one call. That is an acceptable trade for ordinary traffic and not for a call under a hard money ceiling, which is why a strictly budgeted request is often capped at a single potentially billable attempt across the whole chain — no retry, no failover — and the ceiling is paid for in availability.

Make the classes a closed set and the decision exhaustive. A switch over a closed union with no default branch means that adding an eighth class fails to compile until somebody has decided its failover behaviour, which is a language-level guard rather than a review convention that erodes. Classification itself should read the status first and the body only for what the status leaves undecided: 408 is a timeout, 429 is a rate limit, any 5xx is a server error, 401 and 403 are authentication, and only then is the message text consulted to tell a context-length complaint from a content refusal from an ordinary bad request. One case needs explicit handling: a provider that returns a blocked prompt as a successful HTTP response carrying a block reason and no content must be mapped to the refusal class deliberately, or it is indistinguishable from a success.

The limits are the reason to trust the design rather than a reason to doubt it. Classification is a lossy mapping from heterogeneous vendor error shapes onto a handful of classes and it will get cases wrong — a content refusal returned as a bare 400 looks exactly like a malformed request, and although both correctly decline to fail over, the reason recorded in the record will be the wrong one. A provider that returns a 5xx for what is really a refusal will be failed over, reproducing the very laundering the design prevents, and no downstream component can fix upstream error hygiene. An unclassifiable exception treated as a server error is failover-eligible, which is the availability-safe reading and the evidence-unsafe one. And availability is genuinely traded away: some requests that a counting gateway would have completed on a second provider now fail, and that will be reported as a bug by the people whose requests they are.