LLM cost control
One potentially billable egress: no retry, no failover on that call
every questionOn this page
A budget that only reports is not a budget
Most agent cost tooling is a dashboard. It tells you what you spent after you spent it, which is genuinely useful in a monthly review and useless at two in the morning when a retry loop is a third of the way through the month’s budget. The control a finance owner actually asks for is the one that refuses the call.
Refusing a call is a harder engineering problem than reporting on it, and that is why so much tooling stops at reporting. To refuse, you need a number before the request leaves, which means you need to know what the request will cost before you know what it did — an estimate rather than a measurement. That estimate has to be an upper bound rather than a guess, because a ceiling may be conservative and may not be optimistic: an estimate that comes in low lets a request through that should have been refused, and the money is gone by the time you find out.
It also means the price has to exist. This sounds trivial and it is the single most consequential failure in this area, because the failure is silent. A model with no price row prices at zero; a request estimated at zero passes every ceiling; and an estate spending nothing and an estate spending unmetered emit identical bytes. The customer finds out from the vendor invoice.
The shape of the answer, then, is: normalise the provider’s token accounting so the numbers mean the same thing across vendors; resolve the price at admission against everything the route could reach; reserve the estimate atomically against the windows; and fail closed when the price cannot be resolved. Alerting sits on top of that rather than instead of it.
Cache-token accounting, which is where cost figures usually go wrong
Providers disagree about whether cached prompt tokens are inside or outside the prompt total, and there is no convention to fall back on. Anthropic reports cache_read_input_tokens and cache_creation_input_tokens exclusive of input_tokens — they sit alongside it, not inside it. OpenAI’s prompt_tokens_details.cached_tokens is already inside prompt_tokens, and Gemini’s cached-content count is already inside its prompt count. A ledger that adds Anthropic’s buckets to an OpenAI-shaped total double-counts a cached prefix on every single turn of every long-running agent, and one that subtracts an inclusive count from an exclusive total undercounts it the same way.
The fix is structural rather than a set of per-provider corrections scattered through the code. Normalise every provider’s usage into buckets that are mutually exclusive by construction — uncached input, cache read, cache write, output — at the adapter boundary, before any cost arithmetic happens, and then re-assert that invariant at every boundary the numbers cross afterwards. The property you want is that no bucket contains another, so the sum is meaningful and a later reader cannot reintroduce the ambiguity.
Treat provider usage as untrusted wire data even where a type definition calls it a number. A count that is not a non-negative safe integer is refused outright rather than clamped: a negative value would subtract from a budget, and a non-finite one would turn the whole ledger into NaN from that row onward. An OpenAI-shaped response claiming more cached tokens than prompt tokens is rejected rather than silently corrected, because a silent correction hides a bug in something you do not control.
Streaming needs one more rule. Provider streaming counters are cumulative, but some compatible endpoints emit more than one usage frame and a later frame can omit or regress a bucket. Keeping the greatest validated value seen for each bucket is a conservative merge: it can never hand back a budget credit, and it never mistakes a repeated total for an incremental delta. The naive alternatives — take the last frame, or add the frames together — are wrong in opposite directions and both are wrong quietly.
One last accounting note that shows up on the invoice. Where a price row names no cache columns, the honest default is to bill cache reads and cache writes at the full input rate; that over-bills slightly and never under-bills. Long cache writes need care too: some vendors bill a five-minute cache write at a premium over input and a one-hour write at a larger premium, so a request carrying an explicit cache lifetime other than the short default should be priced at the higher tier rather than at the friendly one, and an unrecognised lifetime should get the same treatment rather than an optimistic assumption.
- Anthropic and Bedrock
- Cache buckets are reported alongside the input count, not inside it. Bedrock’s own invocation metrics follow the same exclusive convention; reading them as an inclusive total bills a cached prefix twice.
- OpenAI
- Cached tokens are subtracted out of the prompt total to produce the uncached-input bucket. A cached count larger than the prompt total is an error rather than a silent correction.
- Gemini
- Cached content is inside the prompt count and normalises the same way as OpenAI. Thinking tokens sit outside the candidate count but bill at the output rate, so they belong in the output bucket rather than being dropped.
- OpenRouter
- The one upstream that reports an authoritative charge for a call. Where it is present, prefer it over local arithmetic so the ledger matches the invoice — and keep the field absent rather than zero when it is unknown, so free stays distinguishable from not reported.
Resolve the price at admission, against everything the route could reach
A budget is only enforceable against what may actually leave, which means the price has to be resolved after routing rather than from the model name in the request body. Route rules can send a requested model somewhere else. Tier routing can substitute a cheaper one. Compatibility filtering can drop a target. Failover can promote a fallback mid-request. Pricing only the name the caller typed leaves a rerouted target or an unpriced fallback as a zero-dollar escape hatch.
So the check prices the resolved model and provider for the primary target and for each fallback in turn, and takes the most expensive input rate and the most expensive output rate across that whole candidate set, cache columns included. That is deliberately pessimistic. If failover promotes a more expensive provider mid-request, the reservation already covered it; the alternative is a reservation that failover can invalidate, which is a ceiling that stops binding at exactly the moment things are going wrong.
This is also the reason the money verdict is taken last rather than first. Permissions, rate limits and policy can be decided in a pure pass with no input and output; a hard money decision cannot be made until routing has fixed the complete provider chain and its prices, and routing is the expensive step. Splitting the two lets everything that does not depend on the route be decided before the route is resolved, and defers only money.
The estimate itself should be an upper bound rather than a friendly approximation. Counting the UTF-8 bytes of the complete serialised outbound request — tool definitions, JSON-schema keys, tool arguments, passthrough configuration, every message boundary — plus a fixed allowance for provider-added framing gives a bound that a tokenizer cannot exceed, because it cannot emit more ordinary tokens than there are bytes. A characters-per-token heuristic is the wrong shape here: it is an average, and averages are optimistic half the time.
Price the output leg too. Pricing input alone lets a request asking for 100,000 output tokens pass a small per-request ceiling and then blow through it on the way back. Price the caller’s stated output cap, and where the caller names none, price an assumed cap and stamp that same value onto the outbound request, so an upstream cannot answer past the amount that was reserved.
serialised outbound request 41,208 bytes
+ framing overhead 256 tokens
= input upper bound 41,464 tokens
caller output cap 2,000 tokens
candidate set primary + 2 fallbacks
highest input rate in the set $3.00 / MTok
highest output rate in the set $15.00 / MTok
reserved before egress $0.154392
hourly ceiling $5.00
spend in the rolling hour $4.91 (2 running traces included)
projected $5.064392 -> refused before egressThe unpriced model, and why it has to fail closed
This deserves its own section because it is the defect that motivated the whole design, and it is the one worth checking for in whatever you are running today.
In an earlier build of Token Observe the shipped price rows loaded only under the demo seeder — a command the setup documentation explicitly tells production operators not to run. The documented production install therefore started with an empty price table. An unknown model produced a zero estimate, every trace recorded no cost, and every per-request ceiling admitted every request. The control was off while appearing to be on. The code comment of the day said so in as many words, and the product’s own notes call it the single worst defect it has had.
The fix has two halves and both are worth copying. The catalogue now loads at boot on the documented production path, additively: a row whose key of model, provider and effective date is already present is skipped, so a restart can never overwrite a price an operator corrected by hand. The cost of that choice is stated rather than hidden — a shipped price is never refreshed in place on upgrade, which means stale-but-present. That is accepted deliberately, because absent means zero and zero means a disarmed ceiling.
The second half is the refusal. Where a candidate on the resolved route has no active price row and the agent has any spend ceiling configured, the request is refused before egress with a typed error naming the model, the provider that would have served it and how many further fallbacks are also unpriced. An agent with no ceiling configured is explicitly unbudgeted and keeps recording what it can. The distinction matters: failing closed for everyone would turn a missing price row into an outage for agents that never asked for a budget, and failing open for everyone is the defect this exists to prevent.
Price rows also need to be versioned rather than overwritten, with an effective window, so that a superseded row still exists and a completed call can be repriced later against the rate that was actually in force. And the price used to meter a completed call should be the row pinned at admission for the provider that actually served it, not a fresh read: re-reading at completion lets a catalogue sync landing mid-call reprice an admitted request, including down to zero.
Hard ceilings, alerting, and the trade each one makes
The ceilings worth having are four in money and three in rate. Per request, compared against the pre-flight estimate for that one call. Per rolling hour, per UTC day and per UTC month, each blocking both when the window is already at the limit and when the projection — spend so far plus this call’s estimate — would cross it, so a single large request cannot step over a limit it was already close to. Then requests, tool calls and tokens per minute, which bound the loop rather than the invoice and are the ones that catch a runaway before the money does.
Use UTC day and month boundaries rather than local ones. The monthly figure then lines up with the budget report and with a vendor’s billing period, rather than drifting against both by a timezone offset that nobody remembers is there.
The admission has to be atomic per subject, and this is the part most home-grown implementations get wrong. One transaction reads the windows excluding the current request, tests whether adding the estimate would cross a ceiling, and writes the reservation only if it would not. Without that, several concurrent callers each read the same pre-reservation window, each see room, and all proceed. This is not a hypothetical: it is a correction Token Observe made after the spend window aggregated completed traces only, so requests already in flight were invisible and a cap that one request would have breached was passed by several.
In-flight spend has to count, including requests parked awaiting a human decision. A running or awaiting-approval trace contributes the greater of its billed cost and its reservation. The subtle case is approvals: a denied or expired approval closes its trace and clears the estimate, while an approved-but-unredeemed approval keeps its reservation, because the action has not happened yet and the money is still about to be spent. Its expiry is what releases it.
Alerting is the complement. A post-call check that publishes a warning once a window reaches 80 per cent of its ceiling and an exceeded event at 100 per cent exists to tell a human, not to enforce — the pre-flight check has already refused anything that would breach. Route both into whatever your team actually reads, and put a warn-only policy below each hard ceiling so an anomaly is visible before work stops.
The trade a hard ceiling makes should be stated to whoever owns availability. A request under a spend ceiling is allowed at most one potentially billable network attempt, because a timeout cannot prove the vendor did not complete and bill the call — so a retry or a failover would let one reservation cover several independently billable attempts. That means no retry and no failover on that call, and the reservation is retained in full on an ambiguous failure rather than released as a free call. An agent with no spend ceiling does not pay that cost. It is a real availability trade in exchange for a real spend boundary, and it is better argued in advance than discovered during an incident.
- Per request
- Catches the single pathological call — an unbounded retrieval, a runaway context. It is inert on a seat-based subscription, where the marginal cost of one call is not knowable at the moment of deciding.
- Rolling hour
- The window that catches a loop. A daily ceiling notices a runaway several hours after it started; an hourly one notices it in minutes.
- UTC day and month
- The windows that line up with reporting and with a vendor’s billing period. Both block on the window already being at the limit or on the projection crossing it, whichever comes first.
- Requests, tool calls and tokens per minute
- Bound the behaviour rather than the invoice, and refuse with a typed error naming the limit type, the configured value and the observed figure so the agent’s own logs explain the refusal.
What none of this bounds, and where the money actually leaks
Metering is not reconciliation. Every figure here is computed from the provider’s own reported usage against price rows you hold, and a shipped price row is a list price captured on a date that will go stale. It will not match a vendor invoice to the cent, and the places it diverges are enterprise discounts, committed-use pricing, minimum commitments and mid-month rate changes. Treat the ledger as the operational control and the invoice as the accounting truth, and reconcile them deliberately rather than expecting them to agree.
Ceilings bind per subject rather than per pool. There is no team-level or fleet-level budget in Token Observe; the team and fleet figures in a budget report are the sum of the per-agent ceilings that exist, published beside a count of the agents that have none — which is the number worth looking at, because an agent with no ceiling is not covered by any of this.
Cheaper routing is a separate lever with a separate honesty problem. Serving a cheaper model than the caller asked for saves money and changes the answer, so it has to be opted into per agent rather than assumed, and a ceiling on the tier an agent may reach is worth more than a router’s cleverness. Retrospective savings analysis is worth running before you enable anything, with the caveat carried in the number: it can tell you what the cheaper model would have cost, and it cannot tell you the cheaper model would have answered acceptably. Nothing short of re-running the work and judging both outputs can.
And the largest leak is usually not in the gateway at all. A vendor credential used directly — an engineer with a personal API key, a service account nobody registered, a coding assistant on a laptop pointed at the vendor default — is spend no ceiling in your governance layer can see, because the request never arrives. That is a discovery problem before it is a budget problem, and the honest sequence is to find the ungoverned traffic first and then decide what to do about it.
How to put LLM cost control into practice
- 01
Normalise token accounting per provider before pricing anything
Convert every provider’s usage into mutually exclusive buckets — uncached input, cache read, cache write, output — at the adapter boundary, and validate every value as a non-negative safe integer. Getting this wrong misprices cache-heavy traffic by 50 to 90 per cent in either direction. - 02
Load a price table on the documented production path
Not in a seeder, not in a demo command. Check the table is populated in production specifically, because an empty price table produces zero estimates and a zero estimate silently disarms every ceiling above it. - 03
Resolve the price after routing, across every reachable candidate
Price the resolved model and provider for the primary and each fallback, and take the highest input and output rates in the set. A fallback nobody priced is a zero-dollar escape hatch that opens exactly when the primary is failing. - 04
Estimate as an upper bound, on both legs
Bound the input by the byte length of the complete serialised request plus a framing allowance, and price the output at the caller’s cap — stamping an assumed cap onto the request where the caller names none, so the upstream cannot answer past what was reserved. - 05
Reserve atomically, per subject, counting work in flight
One transaction reads the windows, tests the projection and writes the reservation. Count running and awaiting-approval traces at the greater of their billed cost and their reservation, or concurrent callers will each see room that only one of them has. - 06
Fail closed on an unpriced route for any budgeted agent
Refuse before egress with an error naming the unpriced model and provider, and leave explicitly unbudgeted agents unaffected. The alternative was tried and it is how a ceiling ends up switched off while the console still shows it. - 07
Put alerting below the ceiling, not instead of it
Publish a warning at 80 per cent of each window and an exceeded event at 100 per cent into whatever your team actually reads, and add a warn-only rule under each hard ceiling so an anomaly is visible before work stops.
Where this argument meets an implementation
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
Shadow AI radar
Five evidence sources for AI activity that never touched the gateway, and a coverage model that refuses to call a dead feed a clean estate.
Those pages are one product's implementation of what this guide argues for; describe what your agents actually do and you will get a straight answer about whether you need any of it, including when the answer is no.
Talk it throughWhy do cached tokens make agent cost figures wrong so often?
Because providers do not agree on where they sit, and agent traffic is the traffic most affected. Anthropic reports cache reads and cache writes alongside its input count; OpenAI’s cached-token detail and Gemini’s cached-content count are already inside theirs. Adding one convention’s buckets to the other’s total double-counts the cached prefix on every turn, and agents are overwhelmingly cache-heavy because the system prompt and retrieved context repeat on each turn. The error runs to 50 to 90 per cent on cache-heavy traffic, which is enough to make a per-request ceiling either useless or permanently tripped depending on the direction.
Can a hard budget ceiling be bypassed by concurrency?
Only if the admission is not atomic, which is the common defect. The check has to read the spend windows, test the projection and write the reservation inside one transaction scoped to that subject, so two callers cannot both decide against the same pre-reservation window. Requests still in flight must count as well, at the greater of their billed cost and their reservation, including requests parked awaiting a human decision. Token Observe’s own spend window originally aggregated completed traces only, so concurrent requests each saw zero in-flight spend and all passed a cap that one of them would have breached; the fix is the transaction plus the in-flight accounting together.
What happens when a model has no price row?
For any agent with a spend ceiling configured, the request should be refused before egress rather than estimated at zero — in Token Observe the refusal names the unpriced model, the provider that would have served it and how many further fallbacks are also unpriced. Agents with no ceiling are explicitly unbudgeted and are unaffected. The reason this is worth failing closed over is that the alternative was tried: an unpriced model produced a zero estimate, which admitted every request under every per-request ceiling on the documented production install, and a control that is off while appearing to be on is the failure mode a spend control cannot have.
Do hard budgets hurt availability?
They cost one specific thing, and it is worth agreeing to it in advance. A request under a spend ceiling is allowed at most one potentially billable network attempt across the whole provider chain, because a timeout cannot prove the vendor did not complete and bill the call — so retrying or failing over would let one reservation cover several independently billable attempts. That means no retry and no failover on a budgeted call, and the reservation is kept in full on an ambiguous failure rather than released as free. An agent with no spend ceiling keeps ordinary retry and failover behaviour, which is the lever if a particular workload needs availability more than it needs a boundary.
Is it safe to let a router serve a cheaper model automatically?
It is safe if it is opt-in, never upgrades, and is capped. Serving a different model than the caller asked for changes the answer they get, so it should be a per-agent decision rather than a default, and a ceiling on the tier an agent may reach is the control that actually binds — because a ceiling stops a cheap classifier quietly reaching a frontier model, which an opt-in downgrade flag does not. Treat any retrospective savings figure as a ceiling rather than a forecast: it can model what the cheaper model would have cost at the same token counts, and it cannot tell you the cheaper model would have answered acceptably.
Prefer to ask a person? Write to us →
Bring us the question this guide did not answer.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in