Token Observe vs LLM gateways
Decide whether you need to carry model requests, explain the AI bill, or govern one agent’s next call.
On operating the request path, the gateways are ahead, and not narrowly.
A production gateway is an operations product, and these vendors have spent years on the part Token Observe has not finished: multi-replica deployment, rolling upgrades, regional distribution and an availability commitment somebody signs. Token Observe is a single-writer SQLite process on one host at its current target scale, with no replica, no clustering and no vendor-operated uptime SLA. If the control you are shopping for has to be highly available on day one, that is a real reason to choose differently.
Breadth of connectivity is the second, on each vendor’s own published description rather than anything measured here. Kong publishes production gateway operation for LLM, MCP and A2A traffic on a data plane many organisations already run, with access control on MCP tools set per tool against consumers and consumer groups. Cloudflare describes its AI Gateway as running on its edge network, a topology Token Observe does not have at all. And if you already run LiteLLM, it sits in this slot (its budgets page describes reserving an estimated cost before a request reaches the provider), and for an agent that needs a model allow-list and a spend ceiling it may already be everything you need.
Token Observe and LLM gateways, row by row
Where these two differ, one row at a time.
Ask which provider accounts, currencies and traffic paths the existing tool covers, and retain its source evidence.
Connect supported organization billing sources. Read collection freshness and reported costs separately from invoices and gateway estimates.
Carrying the call: routing, key custody, quotas, retries, telemetry, and in some, access rules on which caller may use which tool.
Deciding the call: authority before it, evidence after it, refusal inline.
Worth testing per product: is a content-policy refusal treated as a retryable error?
Seven typed failure classes. A 429, a timeout or a 5xx moves on; a content refusal, an auth failure, an over-long context and a malformed request stop where they are.
Where a product offers one, ask what the approval is bound to.
A 403 carrying an approval id, bound to the SHA-256 of the canonical action plus its execution context, single-use, expiring at 60 minutes by default.
Logs and analytics you can query.
A hash-chained audit log, an Ed25519 anchor published off-box, and an export sealed with a SHA-256 digest carrying the chain verdict. Tamper-evident, not tamper-proof.
Managed edge or your cluster, with the vendor’s availability posture.
A hosted cloud pilot with optional customer hosting. One writer per workspace, shared cloud capacity and no availability SLA.
- beat 01Name the caller
- 1Authenticate
- 2Resolve the agent
Nothing is decided for an anonymous caller. The key resolves to one registered agent with a named owner, and its roles, its ceilings and any stop switch in force are read live from the store on every request, not from a cached copy, so a suspension takes effect on the next call instead of after a redeploy.
- beat 02Start the record, then read what is being sent
- 3Open the trace
- 4Sanitise
- 5Scan
The trace is opened before anything can refuse the request, which is why a blocked call is recorded as thoroughly as one that went through. Only then is the text normalised and scanned, in that order, so the detectors see what the model will read rather than what a human reviewer can see.
- beat 03Decide once
- 6Govern
- 6bIntersect with the named human
- 7Enact
One function returns one verdict: allow, block, or hold it for a named person to approve. Every check lives in that one place rather than scattered along the path, which is what makes it possible to read your rules and know what they do.
- beat 04Send it, and only then spend the money
- 8Route
- 9Call upstream
Routing fixes which provider and model will serve the call, so where an agent has a USD ceiling it is tested against the price you will pay, not the one the caller typed. Routing is also the last step that can still refuse locally: after it, the request has left your network.
- beat 05Govern the answer, then count the cost
- 10Govern the response
- 11Meter and record
A tool call the model proposes is judged before it reaches your agent. Metering runs last and on every ending, including the refusals and the streams a client abandoned halfway, because the tokens were spent either way and a partial trace is still evidence.
Step names and their grouping are the product’s own, and the order is encoded in the evaluator rather than drawn for the page.
One place to apply your rules.
A control you can bypass by turning it off is not a control, which is why Token Observe sits in the request path and fails closed. To refuse a payload before it reaches a provider, something has to hold it first, and the base-URL swap is the cheapest way to get there. The consequence is stated in the same breath: being in the path makes Token Observe’s own availability a governance property of your environment, so decide in advance what happens when the gateway is unavailable. What the path buys is a single decision point — eleven ordered steps per governed request, so a blocked request is still traced, smuggled Unicode is stripped before anything reads the payload, and one verdict is taken rather than several middlewares each deciding something different.
- In front of your gateway
- Token Observe holds the agent credential and the policy decision and routes to your existing proxy as an OpenAI-compatible upstream. You keep one place for provider keys.
- Beside your gateway
- Consequential agents point at Token Observe, everything else stays where it is. The cheapest starting shape, and it matches the product’s own pilot boundary of five to fifty agents.
When your existing gateway is enough.
The base URL is already swapped, the keys are already central, the spend is already counted, and ripping that out has no governance outcome attached to it. What decides whether you need more is not the proxy, it is the actions: if every governed call ends in text a human reads, a routing proxy plus a spend counter is proportionate.
Put three questions in writing to whichever proxy you run. They are the failure patterns this product’s engineering notes record solving, not claims about any named product. Does the fallback chain treat a content-policy refusal as retryable, so the next provider’s answer returns as a success and nothing says a refusal happened? Does cache-token accounting add Anthropic’s buckets to a total that already includes them? Does the approval authorise an action type rather than an exact payload?
Which of the two to put in.
Read the left one first. If two of those describe you, buy the other thing. If the right column describes you, the next step is thirty days with the source on your own hardware.
When to choose LLM gateways
- Your requirement is connectivity and cost visibility, and nobody has yet asked you to prove what an agent did.
- You need high availability today. Token Observe is a single-writer process on one host, with no replica, no clustering and no uptime SLA.
- The traffic is chat or drafting, where a human reads every output, so an inline refusal buys less than the outage risk of a fail-closed dependency.
- You need the proxy at the network edge in every region you serve, which is a topology Token Observe does not support and does not claim to.
When to choose Token Observe
- The agents take actions somebody has to answer for — a refund, a deployment, an email, a database write — and a 200 is not proof it happened.
- You need a human decision bound to one exact payload rather than an action type, single-use and expiring, with the approver recorded against the trace.
- You run more than one provider and the same rule has to fire identically on all of them.
- Someone will eventually ask who says the head you are showing me is the head, and a log file is not an answer.
If the left-hand column describes you, that is still worth an email: a short reply costs both of us less than an evaluation that ends in the same place.
See the thirty-day evaluationThe 6 named products in this category
This page argues about the category. If you have already shortlisted a product, these go head to head with it.
LiteLLM
Keep LiteLLM where it meets your gateway needs. Compare Token Observe when your IT team also needs provider billing evidence and scoped agent decisions.
Portkey (Prisma AIRS AI Gateway)
Compare the AI bill you can explain and the routed requests you need to govern, alongside your existing Portkey gateway.
Kong AI Gateway
Kong decides which caller may use which tool. Token Observe decides whether this one call, with these arguments, goes ahead, and checks afterwards that it happened. If you already run Kong, the first is nearly free and the second is the reason to read further.
Cloudflare AI Gateway
Cloudflare’s gateway decides what the payload contains. Token Observe decides whether the agent that sent it was allowed to.
Agent Router
Both hold the request. One charges the token budget once the response completes; the other reserves the money before the request leaves your network.
MuleSoft Omni Gateway
Both refuse the call inline, and both now say they govern agents. MuleSoft governs an agent as an instance in API Manager; Token Observe governs it as a record with a named owner, a budget and a human gate.
The other comparisons
The other 6 categories, each opening with the case for the alternative.
Is Token Observe a gateway or not?
In delivery, yes: you point an agent at it by changing one base URL and one key. That is the mechanism, not the claim. Routing, retries, caching, quotas and cost dashboards are listed in the product’s own roadmap as table stakes that must not define the product. What distinguishes it is the decision point: deny-by-default action-level permissions, one policy verdict per request, approvals bound to an exact payload, and a hash-chained record. It also runs beside a gateway you already have, in either order.
Does the vendor see our prompts or our traffic?
Cloud hosting holds workspace evidence and supplied credentials. For customer hosting, no. Token Observe is self-hosted and bring-your-own-key: the vendor receives no telemetry, no phone-home data, no prompts, no keys and no trace database. Governed payloads leave your network only for the model and tool providers you configure, after policy and redaction. The limit: no claim is made that the vendor is legally never a processor, because contracts and support handling still need your counsel’s analysis.
Have you benchmarked these gateways head to head?
No. Every claim here about another product is that vendor’s own public material, untested; there is no reference deployment and no witnessed bake-off. The claims about Token Observe are different: the licence sets out a thirty-day evaluation so your security team, or a third party you commission, can read, run and attack it on your own deployment. Publication is drafted more narrowly than testing — benchmark results with no pre-approval provided the version and configuration tested are named, and security findings after coordinated disclosure. That licence is a template pending counsel in England and Wales, not an executed grant.
Tell us which way you are leaning, and why.
Create a cloud account and workspace. No card or server installation. Model calls and Google BigQuery scans can incur separate provider charges.