Token Observe vs LLM gateways
Token Observe is a gateway in delivery. The gateway is how it arrives, not what it is for.
Kong AI Gateway, Envoy AI Gateway, Portkey, LiteLLM, Cloudflare AI Gateway, MuleSoft AI Gateway. Every claim about them on this page is that vendor's own published material, untested by us.
the other comparisonsOn this page
On operating the request path, the gateways are ahead, and not narrowly
A production gateway is an operations product, and the vendors named here have spent years on the part Token Observe has not finished: multi-replica deployment, rolling upgrades, regional distribution, backpressure and an availability commitment somebody signs. Token Observe is a single-writer SQLite process on one host at its current target scale. There is no replica, no clustering and no vendor-operated uptime SLA, and the reason none is offered is stated plainly in the product’s own support document — the vendor does not operate your deployment, has no telemetry from it, and an uptime number from a party with no access to any of that would be unmeasurable by either side. If the control you are shopping for has to be highly available on day one, that is a real reason to choose differently, and it is the reason this page leads with rather than buries.
Breadth of connectivity is the second place they win, and the specifics here are each vendor’s own published description rather than anything Token Observe has measured. Kong publishes production gateway operation for LLM, MCP and A2A traffic with authentication, ACLs, limits and telemetry, on a data plane many organisations already run. MuleSoft publishes an existing API estate exposed as governed tools, multi-provider routing and spend controls, identity propagation and bidirectional A2A enforcement on an enterprise gateway — an advantage that is largest for a team already running MuleSoft. Envoy AI Gateway and Portkey each publish their own routing, credential and telemetry feature sets. Cloudflare’s AI Gateway is described by Cloudflare as running on its edge network, which is a topology Token Observe does not have at all; how much of your traffic that actually covers is a question for Cloudflare. And if you already run LiteLLM, the plain point is that it sits in exactly this slot, its proxy is distributed under an open-source licence, and for an agent that needs a model allow-list and a spend counter it may already be everything you need. Verify each of those with the vendor.
The claims summarised above are vendor-authored and Token Observe has not tested any of them. There has been no independently witnessed bake-off against any product on this page, and the product’s own market benchmark says so in the same words: public product claims are vendor-authored and have not been independently tested, and the named list is a comparison with the most relevant category leaders rather than a claim that the market contains only these vendors. Where a claim below reads as an absence in another product, read it instead as a question to put to that vendor in writing. Take the rows below as a description of what Token Observe does, and test the other column against the vendor rather than against this page.
Token Observe and LLM gateways, row by row
One card per dimension rather than a three-column table, because the two sides are rarely the same length and a table of them is a horizontal scroller on a phone.
Carrying the call: routing, key custody, quotas, retries, telemetry.
Deciding the call: authority before it, evidence after it, refusal inline.
Worth testing per product: is a content-policy refusal treated as a retryable error?
Seven typed failure classes. A 429, a timeout or a 5xx moves on; a content refusal, an auth failure, an over-long context and a malformed request stop where they are.
Where a product offers one, ask what the approval is bound to.
A 403 carrying an approval id, bound to the SHA-256 of the canonical action plus its execution context, single-use, expiring at 60 minutes by default.
Provider-reported usage. Worth asking how each product normalises cache tokens before it totals them.
Normalised into mutually exclusive buckets before any arithmetic, because Anthropic reports cache reads and writes outside the input total and OpenAI and Gemini report them inside it.
Logs and analytics you can query.
A hash-chained audit log, an Ed25519 anchor published on a schedule to a sink you site outside the database administrator’s control, and an export sealed with a SHA-256 digest that carries the chain verdict. Tamper-evident, not tamper-proof.
Managed edge or your cluster, with the vendor’s availability posture.
Self-hosted only, in your network, on your keys. One writer, one host, no replica, no SLA.
Why the gateway shape is the delivery mechanism and not the argument
A control you can bypass by turning it off is not a control, which is why Token Observe sits in the request path and fails closed. That decision is the reason the product looks like a gateway: to refuse a payload before it reaches a provider, something has to hold the payload before it reaches a provider. The base-URL swap is the cheapest way to get there, because it governs an existing agent without asking anyone to refactor it, and the product’s onboarding claim goes no further than that — for supported OpenAI-compatible, Anthropic and Gemini ingress, normally a base-URL change rather than an application refactor.
The consequence is stated in the same breath as the claim. Being in the path makes Token Observe’s own availability a governance property of your environment: if it stops, governed agents cannot call models. The product’s support documentation asks you to plan for that in advance — run it close to the agents, watch the readiness endpoint, and decide before you need to what happens when the fail-closed gateway is unavailable. A design-partner gate will not pass until that emergency decision has a named owner.
What the path buys is a single decision point rather than a scattering of them. Eleven ordered steps run per governed request, and the order is load-bearing: authenticate, resolve the agent record, open a trace so even a blocked request is recorded, sanitise Unicode so smuggled invisible characters are stripped before anything reads the payload, scan for sensitive data and injection, take one governance verdict, enact it, route honouring the agent’s data policy, call upstream, govern any tool call the model proposes on the way back, then meter and record. A gateway can add most of those as plugins. The question a buyer should ask is not whether the features exist but whether they compose into one verdict that can be explained afterwards.
Integrate above or beside the gateway you already run
The product’s own roadmap gives the rule of engagement directly: integrate above or beside Kong, do not compete on connectivity alone, and make effect assurance portable across gateways. Nothing about Token Observe assumes it is the only proxy in the estate. OpenRouter, which is a gateway in its own right, is registered as one of six first-class upstreams rather than treated as a competitor, and any OpenAI-compatible endpoint you register — including one of your own — is a routable target.
The practical arrangement in an estate that already runs a gateway is to leave it in place and route only the agents that take consequential actions through Token Observe, either in front of the existing proxy or behind it. Two proxies in series is a second failure domain and a second hop of latency, so this is a decision to make deliberately rather than by default; the honest version of the advice is that a chat assistant does not need both, and a refund agent might.
- In front of your gateway
- Token Observe holds the agent credential and the policy decision, and routes to your existing proxy as an OpenAI-compatible upstream. You keep one place for provider keys and regional routing; the governance verdict happens before that hop.
- Beside your gateway
- Consequential agents point at Token Observe, everything else keeps pointing where it points now. This is the cheapest starting shape, and it matches the product’s own pilot boundary of roughly five to fifty agents owned by one platform team.
- Policy exported outwards
- Scope and trigger matching can be compiled to digest-locked OPA Rego with reproducible positive and negative witnesses, so another enforcement point can be shown to match. The artifact deliberately excludes permissions, budgets, approval consumption, kill switches, action precedence and side effects — Token Observe stays authoritative for those.
If you already run LiteLLM, the honest advice is to keep it
LiteLLM sits in exactly the slot Token Observe asks for, so the objection is real rather than rhetorical: the base URL is already swapped, the keys are already central, the spend is already counted. Ripping it out to install something else is work with no governance outcome attached to it, and there is no reason to recommend that.
The question that decides whether you need anything more is not about the proxy, it is about the actions. If every governed call ends in text that a human reads before anything happens, a routing proxy plus a spend counter is a proportionate control and Token Observe would be an expensive way to add nothing. If a call can end in a refund being issued, a pull request being merged, an email being sent or a row being written, the interesting question becomes whether the thing that was authorised is the thing that happened — and that is a different product, not a bigger version of the same one.
Three questions are worth putting to whichever proxy you run, and they are questions rather than findings — these are the failure patterns Token Observe’s own engineering notes record having had to solve, not claims about LiteLLM or about any other named product, none of which has been tested here. Does the fallback chain treat a provider’s content-policy refusal as a retryable error, so the next provider’s answer comes back as a success and nothing in the record says a refusal happened? Does cache-token accounting add Anthropic’s cache buckets to a total that already includes them, or fail to add OpenAI’s, and report a cost the invoice disagrees with? Does the approval flow authorise an action type rather than an exact payload, so a retry with one argument changed is still allowed? Ask in writing; the answers are specific and checkable, and a good product will have them ready.
Which of the two you should actually put in.
Both columns are real answers and both are the same length on the page. Read the left one first: if it describes your estate, it is the cheaper decision and this page has done its job.
When to choose LLM gateways
- Your requirement is connectivity and cost visibility — one key store, one retry policy, one dashboard across several models — and nobody has yet asked you to prove what an agent did.
- You need high availability today. Token Observe is a single-writer process on one host at this scale, with no replica, no clustering and no vendor-operated uptime SLA.
- The traffic is chat or drafting, where a human reads every output before anything happens, so an inline refusal buys you less than the outage risk of a fail-closed dependency.
- You need the proxy at the network edge in every region you serve, which is a topology Token Observe does not support and does not claim to.
When to choose Token Observe
- The agents take actions somebody has to answer for — a refund, a deployment, an email, a ticket transition, a database write — and an API returning 200 is not acceptable proof that it happened.
- You need a human decision bound to one exact payload rather than to an action type, single-use and expiring, with the approver recorded against the trace.
- You run more than one provider and the same rule has to fire identically on all of them, because a policy that fires on OpenAI but not on Gemini is worse than no policy.
- Someone will eventually ask who says the head you are showing me is the head, and a log file is not an answer to that question.
If the left-hand column is the one that describes you, that is still worth an email: a straight answer about which of these to buy costs both of us less than an evaluation that ends in the same place.
Ask which one fitsThe 6 named products in this category
This page argues about the category. If you have already shortlisted a product, these go head to head with it — every claim about the other product paraphrased from that vendor's own published material on a stated date, and none of it independently tested.
LiteLLM
If you already run LiteLLM, keep it. The question that decides whether you need anything more is about the actions your agents take, not about the proxy.
Portkey
Both hold the payload before it reaches a provider. One is built to carry it to more than 250 models; the other is built to refuse it and prove afterwards who said it could go.
Kong AI Gateway
Kong governs the traffic. Token Observe governs the action. If you already run Kong, the first one is nearly free and the second one is the only reason to read further.
Cloudflare AI Gateway
Cloudflare’s gateway decides what the payload contains. Token Observe decides whether the agent that sent it was allowed to.
Envoy AI Gateway
Both hold the request. One charges the token budget once the response completes; the other reserves the money before the request leaves your network.
MuleSoft AI Gateway
Both refuse the call inline. One refuses on behalf of an endpoint, the other on behalf of an agent that has an owner.
The other comparisons
Same template, same order, same concession first. Claims about every named product on all of them are that vendor's own and have not been independently tested.
Token Observe vs LLM observability
One refuses the call inline. The other scores it afterwards. Most estates need both, and they are not substitutes.
Token Observe vs AI security platforms
Token Observe is not a complete AI security suite, and the product’s own strategy document forbids selling it as one.
Token Observe vs cloud-native controls
If every agent, model and tool lives in one cloud, use that cloud’s controls. The argument here is for the estate that does not.
Token Observe vs building it yourself
For a small estate, a few hundred lines of proxy is usually the right answer. The cost arrives later, and it arrives in specific places.
Token Observe vs doing nothing
With three agents, no regulated data and no incident, doing nothing is often the correct decision. This page is about what changes it.
Is Token Observe a gateway or not?
In delivery, yes: you point an agent at it by changing one base URL and one key, and it holds the request before it reaches a provider. That is the mechanism, not the claim. Provider routing, retries, caching, quotas and cost dashboards are listed in the product’s own roadmap as table stakes that must not define the product, and Token Observe ships them because an agent estate needs them, not because they distinguish it. What is meant to distinguish it is what happens at the decision point: deny-by-default action-level permissions, one policy verdict per request, approvals bound to an exact payload, and a hash-chained record of the whole thing.
Can Token Observe run alongside Kong, Envoy or LiteLLM?
Yes, in either order, and the product’s roadmap explicitly prefers that to competing on connectivity. Token Observe can route to any OpenAI-compatible endpoint you register, so an existing gateway becomes an upstream; equally, an existing gateway can forward to Token Observe for the agents that need governing. Both arrangements add a hop and a second failure domain, so the usual starting shape in a pilot is narrower: route only the agents that take consequential actions through Token Observe and leave the rest where they are. Scope and trigger matching can also be compiled to digest-locked OPA Rego with witnesses, if you need another enforcement point to be shown to agree.
How much latency does the governance step add?
The only measured figure the product publishes is a laboratory baseline, and it travels with its conditions: 206.2 successful requests per second and 71.2 ms p50, 163.8 ms p95, 223.4 ms p99, measured over 30.143 seconds at concurrency 16, from an immutable commit, on an Apple M1 Max with 64 GB against a mock upstream on Node 20. That measures Token Observe’s inline work rather than end-user response time — a mock upstream excludes provider latency, streaming, retries and failover — and thirty seconds is not a soak. The product’s own capacity document refuses to turn it into a throughput commitment, and so does this page. Partner sizing needs a sustained soak on your request mix.
Have you benchmarked these gateways head to head?
No. Every claim on this page about another product is taken from that vendor’s own public material, has not been independently tested, and the product’s market benchmark carries the same caveat in its own words. There is no reference deployment and no independently witnessed competitor bake-off; both are named in the product’s own launch gates as evidence that does not yet exist. The claims about Token Observe are a different matter — the licence sets out a 30-day evaluation specifically so a prospective customer’s security team can read, run and attack the software before a purchase order is raised, with no gag clause and no pre-approval of results. One caveat travels with every sentence about that licence: the published licence is a template pending review by counsel in England and Wales, not an executed grant of rights, so read it as the intended terms rather than as the signed ones.
Does the vendor see our prompts or our traffic?
No. Token Observe is self-hosted and bring-your-own-key: the vendor receives no product telemetry, no phone-home data, no prompts, no keys and no trace database. Governed payloads leave your network only for the model and tool providers you configure, after policy and redaction, and the runtime data flow is documented so you can verify that rather than take it on assurance. The honest boundary alongside that: no claim is made that the vendor is legally never a processor, because contracts and support handling still need your counsel’s analysis even when runtime phone-home is zero.
Tell us which way you are leaning, and why.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in