Tier routing
also called model tiering · dynamic model selection · model downgrade · cost-based routing
The opportunity is structural rather than clever. A fleet pins one frontier model in configuration and then sends it everything: classification, field extraction, reformatting, translation, one-line lookups and the occasional genuinely hard question. The largest available saving is not a discount but serving the calls that never needed the expensive model on a cheap one. The subset is usually larger than anyone expects, and identifying it is a scoring problem rather than a machine-learning one.
It has to be decided without calling a model. A router that consults an LLM to choose an LLM spends a meaningful share of the saving it exists to produce and adds a second upstream dependency to the request path. What works instead is a small set of signals that genuinely separate cheap work from expensive work: what the latest user turn asks for, using verbs that name the shape of the work, not domain nouns (extract, classify, translate against analyse, debug, derive); whether tools were offered, the strongest structural signal there is, because a model handed tools is being asked to decide and act rather than answer; total prompt size; conversation depth; the presence of source code; and the caller’s own output cap, since an answer capped at a few dozen tokens buys a label, not an argument. Keep the keyword lists short: every extra term is another chance to fire on a word that happened to appear, and the two error directions are not symmetric.
Five rules make the result safe to run in production. Never route above the requested tier: upgrading spends money nobody authorised. Make a downgrade opt-in per agent, and require the classification to clear its threshold with room to spare rather than by a single point, so one weak signal cannot move a tier. Give the residual verdict — the one returned when nothing decisive fired — a confidence too low to move anything: a tier changed because the signals cancelled out is a silent behaviour change an operator is right to object to. Let a per-agent ceiling apply always, whatever the classifier said and whatever the caller asked for; that is what stops a cheap classifier costing frontier prices. And treat a model the price table does not name as the most expensive thing available, or an unpriced model walks straight through the ceiling that exists to bound it. Mixed requests take the expensive reading: extract the stack frames and then diagnose the crash is a diagnosis.
Choosing the replacement and recording the choice are the rest of the work. Rank candidates within the tier by a blended rate weighted towards input — agent traffic is input-heavy, so a straight average of the two legs ranks models by a workload nobody runs — and never bill from that blend. Break ties deterministically, by provider priority then model name, so the same request always routes the same way and a decision recorded months ago can be reproduced. Skip wildcard price rows: a row covering a family is not a model anybody can name, and routing to one sends a literal asterisk upstream. Then publish the record — requested model, classified tier, confidence, selected model, estimated saving, and the reasons — because an unexplained model substitution is indistinguishable from a bug, and a cost saving nobody can audit is a claim rather than a measurement.
The limits belong beside the saving. The saving figure is an estimate from pre-flight token counts and should be clamped at zero, so a swap can never report a negative saving as a positive one or invent one from a missing price row. The classifier is a heuristic with no calibration against labelled data, so its confidence is agreement between signals rather than a probability. It will be wrong in both directions, and the errors cost different things: a false hard call wastes money, a false trivial call degrades an answer somebody paid for, which is why the bias runs towards caution. And where the substituted model would route somewhere the original would not have gone — a disabled provider, or one the agent’s data policy forbids — serve the model the caller asked for: a cost optimisation must never take a request down.
Two requests, one pinned frontier model
Both arrive naming the same expensive model, from the same agent, under the same configuration. The first is 140 characters — classify this ticket as billing, technical or other — with no tools offered, a single turn and max_tokens of 8. A simple verb, a short prompt with no tools, a tiny output cap and no prior context all point the same way, and several agreeing signals is what clears the confidence bar a downgrade requires, so it is served on the cheapest available economy model and the record says which signals moved it. The second asks the model to diagnose why a deploy failed, carries 40 KB of logs and offers six tools. A reasoning verb, tools offered and a large prompt classify it as reasoning, which is not cheaper than what was requested, so the frontier model stands and the record says that too.
What tier routing is routinely confused with
- Model routing
- Model routing decides where a named model goes; tier routing decides which model is named. Tier routing runs first and its choice is then subject to the ordinary route rules, provider selection and data policy — otherwise the cheap path is the ungoverned path.
- Model cascading
- A cascade calls the cheap model first and escalates to an expensive one when the answer looks poor, so it pays for both calls on every escalation and needs something — a judge model, a schema check, a confidence score — to decide what poor means. Tier routing decides once, before any call, and never pays twice.
Related terms
Model routing
Model routing is the resolution, at the moment of the call, of a requested model name to a concrete provider and model, together with the ordered list of alternatives that may serve it if the first one fails. It lets a client that only knows one model name be pointed at a different vendor, region or price without the client changing, and it is where a multi-provider estate’s data-handling and cost constraints are actually applied.
Token cost accounting
Token cost accounting is the practice of turning a provider’s reported token usage into a defensible monetary figure for each model call: resolving the price row for the model and provider that actually served the request, splitting the reported usage into token classes that do not overlap, and applying the right rate to each. It is harder than multiplying tokens by a rate because providers disagree about whether cached prompt tokens are reported inside or outside the prompt total, and because the model that gets billed is not always the model that was asked for.
Prompt caching
Prompt caching is a provider feature that stores the processed form of a repeated prompt prefix so that later requests beginning with the same bytes are charged at a reduced input rate instead of the full one. It changes the billing shape of a call rather than its content: the prefix still counts as input tokens and still occupies the context window, but the tokens move into a cheap cache-read bucket and, on the call that populates the cache, into a cache-write bucket that on some vendors costs more than uncached input.
Where Token Observe does this
The definition above is the field's, not the product's. This is the part of the product that implements it, for a reader who wants to see one.
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Nothing verifies a provider’s ZDR or training claim
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
One billable egress: no retry, no failover
The terms next to this one
Where the money goes, why providers disagree about how to count it, and what a spend control has to do to be a control rather than an alert.
Definitions are the easy part.
The glossary is written to be useful whether or not you ever buy anything. If you have got to the point of deciding how to implement one of these in your own estate, say what your agents do and you will get a straight answer about what it would actually take.
no form · no qualification step · no sales desk · the other three ways in