Prompt caching
also called prompt cache · cached input tokens · context caching · prefix caching
Caching exists because agent traffic is shaped nothing like chat traffic. A system prompt, a block of tool definitions and a retrieved context repeat verbatim on every turn, while the new user turn is a few hundred tokens; the repeated part is often ninety per cent or more of the input leg. Charging it at full rate on every turn prices the same computation over and over. On the price rows in circulation, a cache read is commonly around a tenth of the full input rate, so the saving on the repeated portion is large enough to change which model an estate can afford to run.
The write side is where the arithmetic stops being obvious. Some vendors charge nothing to populate the cache; others charge more than the full input rate for it — a five-minute entry at roughly 1.25 times input and a one-hour entry at roughly twice — so the first call in a cached series can cost more than the equivalent uncached call. With a write at 1.25 times input and reads at a tenth, the extra 0.25 of a prefix is repaid by the first re-read, which saves 0.9; with a write at twice input, it takes two. That means caching pays only when the prefix is genuinely read again before the entry expires, and entries expire in minutes. A low-traffic agent with a large system prompt and a five-minute window can pay the write charge on every single call and never take a read.
That is also the commonest way caching goes wrong, and it is silent. The cache is keyed on an exact byte prefix, so anything that perturbs those bytes turns every request into a miss plus a write. A timestamp or a session id at the top of the system prompt does it. So does reordering tool definitions, serialising a JSON object with non-deterministic key order, or applying a transformation at the gateway that produces a slightly different result each time. The responses are identical, the latency is a little worse, and only the usage fields show what happened — which is why anything that rewrites a prompt in flight has to do so deterministically, and why the cache marker has to sit at the end of a prefix that is byte-stable across turns rather than in the middle of something that changes.
For anyone metering the traffic, caching is also the reason a cost ledger can be catastrophically wrong while looking perfectly healthy. Providers disagree about whether the cached count is reported inside or outside the prompt total, and the two conventions are not distinguishable from the numbers alone. Getting it backwards misprices cache-heavy traffic by roughly half to nine tenths. Time-to-live tiers add a second trap: where a vendor prices a long-lived write higher than a short-lived one and the price table has a single cache-write column, a request carrying a one-hour marker is under-priced unless the marker is read out of the request and the higher rate applied — and an unrecognised time-to-live should take the more expensive tier rather than the optimistic one.
The limits are worth stating plainly. Caching does not reduce the number of tokens the model reads, so it relieves cost and latency and does nothing at all for context-window pressure. Vendors impose a minimum cacheable prefix length and allow only a small number of cache breakpoints per request, so a gateway sitting in the middle may have to collapse several caller-supplied breakpoints into one — which is defensible when the surviving marker lands at the end of the shared prefix, and lossy when it does not. And a cache hit is never guaranteed: hit rate is a property of your traffic pattern and the vendor’s eviction behaviour, not something a caller can assert, so any forecast built on an assumed hit rate should be quoted as a range.
When caching costs more than not caching
An agent sends a 30,000-token prefix — system prompt plus tool definitions — forty times an hour. Uncached, that prefix costs forty times the full input rate. Cached, it costs one write at 1.25 times the rate plus thirty-nine reads at a tenth: about 5.15 units against 40, an 87 per cent reduction on the repeated portion. Now put a current timestamp in the first line of the system prompt. Every request is a miss followed by a write: forty writes at 1.25, or 50 units — 25 per cent more than caching nothing at all. The answers are unchanged, no error is logged, and the only visible symptom is that the cache-read bucket in the usage payload is permanently zero.
What prompt caching is routinely confused with
- Response caching
- A response cache returns a stored answer for an identical request without calling the provider at all: no tokens, no latency, no fresh generation. Prompt caching always calls the provider and always generates a new completion — it only reduces the price of re-reading the prefix. The two also have different safety rules: a response cache must be keyed on the sanitised request and must never serve a reply that was shaped by a policy, a redaction or an approval, because that reply must not outlive the decision that shaped it.
- Context window
- The context window is the hard limit on how much the model can read. A cached prefix still occupies it in full — caching makes the same tokens cheaper, never fewer — so a request that is too long stays too long no matter how well it caches.
Related terms
Token cost accounting
Token cost accounting is the practice of turning a provider’s reported token usage into a defensible monetary figure for each model call: resolving the price row for the model and provider that actually served the request, splitting the reported usage into token classes that do not overlap, and applying the right rate to each. It is harder than multiplying tokens by a rate because providers disagree about whether cached prompt tokens are reported inside or outside the prompt total, and because the model that gets billed is not always the model that was asked for.
Tier routing
Tier routing is the substitution of a cheaper model for the one a caller named, decided before the call by classifying what the request is actually asking the model to do. It is a cost control rather than a quality feature, so the two properties that make it defensible are that it never routes above the tier that was requested, and that it records why it moved.
Model routing
Model routing is the resolution, at the moment of the call, of a requested model name to a concrete provider and model, together with the ordered list of alternatives that may serve it if the first one fails. It lets a client that only knows one model name be pointed at a different vendor, region or price without the client changing, and it is where a multi-provider estate’s data-handling and cost constraints are actually applied.
Where Token Observe does this
The definition above is the field's, not the product's. This is the part of the product that implements it, for a reader who wants to see one.
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
One billable egress: no retry, no failover
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Nothing verifies a provider’s ZDR or training claim
The terms next to this one
Where the money goes, why providers disagree about how to count it, and what a spend control has to do to be a control rather than an alert.
Definitions are the easy part.
The glossary is written to be useful whether or not you ever buy anything. If you have got to the point of deciding how to implement one of these in your own estate, say what your agents do and you will get a straight answer about what it would actually take.
no form · no qualification step · no sales desk · the other three ways in