Token Observe vs Envoy AI Gateway
Both hold the request. One charges the token budget once the response completes; the other reserves the money before the request leaves your network.
2 September 2026
- Envoy AI Gateway — home
- Concepts — architecture overview
- Capabilities
- Traffic handling
- Usage-based rate limiting
- Quota policy
- Security
- Upstream authentication
- MCP gateway
- Observability
- Release notes — v1.0 general availability
- A reference architecture for adopters of Envoy AI Gateway
- Scaling the AI Gateway controller
- Gateway configuration — GatewayConfig
- Getting started
- envoyproxy/ai-gateway on GitHub — Apache-2.0
Their claims, not our testing. Verify anything that decides it for you.
On this page
For connectivity at scale on Kubernetes, Envoy AI Gateway is the better purchase, and it is free
Envoy AI Gateway is built on a data plane that a great many organisations already run, and everything downstream of that is an advantage Token Observe does not have. Their architecture documentation describes “a modern cloud-native architecture with distinct control and data planes”, with Envoy Gateway and the AI Gateway controller in the control plane and Envoy Proxy plus the AI Gateway external processor in the data plane; their scaling page documents running multiple AI Gateway controller replicas, paired with a horizontal pod autoscaler for dynamic workloads, and notes that the extension server serves traffic on every replica including non-leader pods; the v1.0 release notes declare the v1beta1 control-plane API stable for the 1.x series — it “will remain backward compatible for the entire 1.x series”, with breaking changes reserved for a future 2.0 — and name sixteen LLM providers reachable “all behind one OpenAI-compatible API”. The repository is Apache-2.0. Token Observe is, at its current target scale, a single-writer SQLite process on one host: no replica, no clustering, and no vendor-operated availability SLA, for the reason its own support document gives — the vendor does not operate your deployment, has no telemetry from it, and an uptime number from a party with no access to either would be unmeasurable by both sides. If high availability is a day-one requirement, that settles it, and it should.
The second advantage is credential handling, and it is specific. Their upstream authentication page describes long-lived credentials, “like API keys”, as “stored in Kubernetes secrets and managed by the administrator”, and short-lived credentials generated per request for three providers: AWS Bedrock “uses OIDC integration with AWS STS to generate temporary credentials for each request”, Azure OpenAI uses Entra ID to provide short-lived access tokens, and GCP Vertex AI “uses GCP workload federation with Google STS”. Alongside that, their security page inherits Envoy Gateway’s SecurityPolicy — JWT validation and claim assertion, mutual TLS, external authorisation, OIDC login, basic auth and API keys, and IP allowlists and denylists — attached “to the Gateway and/or generated HTTPRoutes”. For a platform team whose rule is that no long-lived provider key exists anywhere, that federation is theirs, it is documented, and it is a good reason to run them. It is also worth noting that if you already operate Envoy Gateway, adding the AI Gateway extends something your team already knows how to run rather than introducing a new failure domain, which is a class of advantage no feature grid captures.
Every claim in the other column of this page is taken from the project’s own site, documentation and repository as read on 2 September 2026, and none of it has been tested against a running deployment. There has been no bake-off. Where a row below says a capability is not described in their published documentation, read that as a statement about the documentation on that date and as a question to put to the project in writing, not as a finding about their code: an open-source project moves faster than a comparison page, and this one changes weekly by its maintainers’ own account of their Monday meetings.
Token Observe and Envoy AI Gateway, capability by capability
The Envoy AI Gateway column paraphrases Envoy AI Gateway’s own published material as it stood on 2 September 2026. None of it has been independently tested here, products in this category ship quickly, and a capability that is absent from a vendor’s documentation is not the same thing as a capability the product lacks. Check anything that decides it for you against their own current documentation.
Where it sits
Their architecture page describes “a modern cloud-native architecture with distinct control and data planes”: Envoy Gateway and the AI Gateway controller in the control plane, and Envoy Proxy plus the AI Gateway external processor in the data plane — “the component that sits in the request path and processes the requests”.
One process between the agent and the provider, running eleven ordered steps per governed request. The order is load-bearing: sanitise before scan, scan before verdict, verdict before route.
Kubernetes custom resources alongside Envoy Gateway — AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, GatewayConfig and MCPRoute — on the v1beta1 API declared stable at v1.0, with the providers reachable “all behind one OpenAI-compatible API”.
Change OPENAI_BASE_URL or ANTHROPIC_BASE_URL and one key. For supported OpenAI-compatible, Anthropic and Gemini ingress that is normally a base-URL change rather than an application refactor.
Two different assumptions about the reader. Theirs is a platform team with a cluster and a Gateway API; Token Observe’s is a team that wants an existing agent governed without touching its code.
Policies attach to the Gateway and/or the generated HTTPRoutes, per their security page; their two-tier pattern puts authentication, top-level routing and global rate limiting at the tier-one gateway and fine-grained self-hosted model access at tier two.
An agent record: an id, a named human owner, a team, a declared purpose, a risk tier, a lifecycle status and a budget. The registry entry is the same record the gateway enforces against.
Sixteen LLM providers named in the v1.0 release notes — OpenAI, Azure OpenAI, Gemini, Vertex AI, Bedrock, Anthropic, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok and Tetrate Agent Router Service.
Six first-class upstreams — OpenAI, Anthropic, Gemini, OpenRouter, Bedrock and Azure OpenAI — plus any OpenAI-compatible endpoint you register, including one of your own.
Breadth is theirs and it is not close. The claim Token Observe makes instead is equivalence: a table-driven test over every provider kind fails to compile until someone has stated how the same guarantees are met on a new one, because a policy that fires on OpenAI but not on Gemini is worse than no policy.
An MCP gateway to “aggregate multiple MCP servers into a single unified endpoint”, which prefixes tool names with the backend name, filters with a toolSelector taking exact names or regular expressions, and authorises per tool against JWT scopes, claims and CEL rules.
One Streamable HTTP endpoint in front of every registered upstream server, tools namespaced server.tool and filtered to the agent’s grants, and every tools/call authorised again at execution because filtering a list is a usability feature rather than access control.
What it enforces
Envoy Gateway SecurityPolicy attached to the Gateway or HTTPRoutes: JWT validation and claim assertion, mutual TLS, external authorisation for custom logic, OIDC login, basic auth and API keys, and IP allowlists and denylists.
A bearer agent token stored as SHA-256 and compared in constant time, then action-level permissions that are deny-by-default, where an explicit deny beats every allow in any role and a delegation chain intersects at every hop rather than unioning.
QuotaPolicy sets per-model token budgets over sliding windows of 1 second, 1 minute, 1 hour or 1 day; “for each completed request, the token cost is computed using the configured cost expression” and charged to the bucket, and an exhausted quota rejects with 429.
USD ceilings per request, per rolling hour, per UTC day and per UTC month, decided after routing has fixed the chain, priced across every provider and fallback that chain could execute, and reserved before egress inside one per-agent transaction.
The row this page turns on. Their budget is charged from what has completed; Token Observe’s is reserved from what could happen. Theirs costs nothing to run and admits the request that crosses the line; Token Observe’s needs a maintained price table and refuses that request.
Tokens, with six token types tracked on their quota page — input, output, total, cached input, cache creation input and reasoning — plus a CEL cost expression for custom weighting. Their worked example is “input_tokens + cached_input_tokens / 10u + output_tokens * 6u”, charging cached input at a tenth and output at six times.
US dollars, from a catalogue of fifty shipped price rows loaded additively at boot on the documented production path. A budgeted agent whose resolved route has an unpriced reachable target is refused with a 409 before egress rather than priced at zero.
Their rate-limiting page states that AI Gateway “does not interrupt an already admitted stream if that stream pushes the token count over the configured limit”; subsequent requests receive 429 “until enough usage expires from the rate-limit window”.
A request under a hard USD ceiling is allowed at most one potentially billable egress — no retry, no failover — because a timeout cannot prove the vendor did not complete and bill the call, and the reservation is retained in full on an ambiguous failure.
Their reference-architecture post describes safety checks and output validation rules as something you “implement” centrally at the gateway, and custom authorisation logic as something you “add … via Envoy’s extension filters”. A shipped detector for personal data or secrets is not described in their published documentation as of 2026-09-02.
Eleven data classes detected inline after Unicode sanitisation, three of them checksum-validated — Luhn, IBAN mod-97, NHS mod-11 — then tokenised, redacted or blocked before the payload leaves your network. Detection is heuristic, and injection scoring is nine weighted patterns rather than a model.
QuotaPolicy supports a shadow mode, which their documentation says lets you test quotas “without rejecting traffic” before enforcement is enabled.
The same idea, one layer up: every policy can run in shadow mode first, and where the deployment turns the gate on, no rule may begin enforcing until a backtest of that exact rule has been replayed against recorded traffic and acknowledged by a named person.
Genuine agreement, arrived at independently. Both projects concluded that a control switched on blind causes the outage it was bought to prevent.
What it records
“Prometheus metrics following OpenTelemetry Gen AI semantic conventions”, covering token usage, latency and model performance.
A ledger row per governed request, priced against the provider that actually served it after the route is narrowed post-call, so the figure matches the vendor that will invoice you rather than the one tried first.
“OpenTelemetry integration with OpenInference semantic conventions” for LLM request tracing, and the same stack applied to MCP requests.
OTLP over HTTP ingested as an input — bounded JSON and protobuf across logs, traces and metrics — rather than as a competing tracing backend. Building another tracing and evaluation product is a stated non-goal.
Not a contest. If you want spans in your existing collector, take theirs; Token Observe consumes telemetry as the observed view in a reconciliation, and does not try to be where your engineers debug.
Their observability page states that “AI metadata produced by the AI gateway (model name, token usage, etc.) can be included in the Envoy Access Logs”.
A trace per governed request whose id is minted before the verdict, so a blocked request is recorded too, searchable in plain English through a validated filter object over allow-listed fields — never SQL.
The v1.0 notes describe “Prometheus metrics for token usage, time-to-first-token, and inter-token latency, with separate accounting for reasoning tokens so cost attribution stays accurate for thinking models”, and the capabilities page describes provider-agnostic prompt caching through a unified cache_control API.
Provider usage normalised into mutually exclusive cache buckets before any arithmetic, because Anthropic reports cache reads and writes outside the input total while OpenAI and Gemini report them inside it, and treating one convention as the other misprices cache-heavy traffic by 50 to 90 per cent.
Their published observability material describes Prometheus metrics, OpenTelemetry tracing and access logs. A tamper-evidence mechanism over those outputs, or a sealed evidence export, is not described in their published documentation as of 2026-09-02.
A hash-chained audit log over every governance-plane change, HMAC-SHA256 under a key held off the box when one is configured, with optional Ed25519 anchoring published to a sink outside the database administrator’s reach. Tamper-evident, not tamper-proof; exports are SHA-256 digest-sealed and are not themselves signed.
Two different readers. An access log is written for the team operating the gateway. The audit chain is written for someone who will ask, a year later, whether the record they are being shown was altered.
How it deploys
Kubernetes. Their getting-started section lists setting up a Kubernetes cluster, installing the required tools and setting up Envoy Gateway before installing Envoy AI Gateway.
Self-hosted, in your network, on your keys: one Node process and one SQLite file in WAL mode. PostgreSQL is implemented behind the store ports as an evaluation alternative, and is explicitly not a supported HA topology or multi-replica claim.
Kubernetes-native. Their scaling page documents running multiple AI Gateway controller replicas and pairing them with a horizontal pod autoscaler for dynamic workloads, with leader election applying only to the Kubernetes controller portion while the extension server serves traffic on every replica; GatewayConfig separately sets the external processor’s resources and environment per gateway.
One writer on one host at the current target scale, sized for a pilot of roughly five to fifty agents owned by one platform team. No replica, no clustering, no vendor-operated uptime SLA.
API keys “stored in Kubernetes secrets and managed by the administrator” for long-lived credentials, plus per-request short-lived credentials for Bedrock through AWS STS, Azure OpenAI through Entra ID and Vertex AI through GCP workload federation.
Bring-your-own-key, held in your deployment. The vendor receives no product telemetry, no phone-home data, no prompts, no keys and no trace database; governed payloads leave your network only for the providers you configure, after policy and redaction.
You do, in your cluster. Their home page describes the project as “the result of the community coming together to address GenAI traffic handling needs using Envoy”, and points people to Slack, GitHub and Monday community meetings.
You do, in your network, under a published support model with severity definitions and response targets — including a plain statement of why there is no availability SLA yet.
The v1beta1 control-plane API is declared stable at v1.0 for the 1.x series: the release notes say it “will remain backward compatible for the entire 1.x series”, with breaking changes reserved for a future 2.0, and record that upgrading from v0.7 was a drop-in change needing no edits to existing v1beta1 resources.
No commitment of that shape is published. Broad, production-critical general availability is currently a no-go on the product’s own decision record, and version-to-version behaviour should be read from the change log and the known-issues list rather than from a stability promise.
What it costs
Apache-2.0, on the envoyproxy/ai-gateway repository.
Commercial source-available: use, modify and self-host under a licence, with redistribution and offering it as a competing hosted service not permitted. The published licence is a template pending review by counsel rather than an executed grant of rights.
No price is published on their site, and the project is presented as open source. A commercial licensing model is not described in their published documentation as of 2026-09-02.
No published price either. What is published is the shape of the evaluation rather than the number at the end of it.
The cluster, the Envoy Gateway installation and the operations team that keeps them running. On their published material the software carries no fee.
A fail-closed dependency in your request path, one host to run and back up, and the operational conversation that comes with both. If it is down, governed agents cannot call models.
The source and the manifests are public under Apache-2.0, so an evaluation needs no commercial conversation with anybody.
The licence sets out a 30-day evaluation specifically so a security team can read, run and attack the software before a purchase order, with no gag clause and no pre-approval of results. There has been no independent penetration test, and the known-issues list naming the attacks that still work is published on purpose.
The token budget is charged after the response; the USD budget is reserved before the request
Their usage-based rate limiting page is unusually direct about its own mechanism, which is why this page can be direct about the comparison. Envoy “checks the token budget using usage that has already been charged”, and “token usage is charged after the response completes”; the quota page repeats it in the same register — for each completed request, the token cost is computed using the configured cost expression and charged against the bucket. Streaming is called out explicitly: AI Gateway “does not interrupt an already admitted stream if that stream pushes the token count over the configured limit”, and later requests get 429 until enough usage expires from the window. Nothing about that is hidden and nothing about it is careless. It is the design a data plane wants, because it needs no price catalogue, no estimate and no lock, and it costs one metadata read on the way in.
What it buys you is fair-share throughput: a tenant that has been consuming heavily gets throttled, promptly, at the next request. What it does not attempt is a hard ceiling in currency on the request that crosses the line, and the vocabulary makes that clear — the limit is written in tokens, weighted by a CEL expression, not in dollars against a per-agent budget. If your requirement is that no single team may exceed a token allowance on a shared cluster, that is exactly the control, and it is free.
Token Observe makes the opposite trade because it was built for a different failure. The USD verdict is deliberately the last one taken: permissions, rate limits and policy are decided first, then the route is resolved, then every provider and fallback that route could execute is priced, and the highest rate in the reachable candidate set is reserved against the agent’s hour, day and month windows inside a single per-agent transaction — BEGIN IMMEDIATE on SQLite, an advisory lock on PostgreSQL — so two concurrent callers cannot both decide against the same pre-reservation window. The estimate is conservative on both legs: the input bound is the UTF-8 byte length of the serialised outbound request plus 256 tokens of framing, and the output leg is priced at the caller’s max_tokens or 4,096 when none is named. A ceiling may be conservative; it may not be optimistic.
The reason that design exists is a defect in Token Observe’s own history rather than an argument against anybody else. The shipped price rows once loaded only under the demo seeder, which the setup documentation tells production operators not to run, so the documented production install had an empty price table: every trace recorded $0 and every per-request ceiling admitted every request. The control was off while appearing to be on, and an estate spending nothing and an estate spending unmetered emit identical bytes. That is why a budgeted agent whose resolved route has an unpriced reachable target is now refused with a 409 before egress, and why the price catalogue is treated as part of the control rather than as reporting furniture. The cost of that is stated in the same breath: a hard-budgeted call gets one potentially billable egress and no failover, because a timeout cannot prove the vendor did not complete and bill the call.
- Their sequence
- Admit against usage already charged → serve the response → compute the cost expression → charge the bucket → reject later requests with 429 until the window drains.
- Token Observe’s sequence
- Kill switch → lifecycle → permissions → policy and rate limits → resolve the route → price every reachable candidate → reserve atomically → egress once → meter against the price pinned at admission.
- What each one is good at
- Theirs is cheap, needs no price data and throttles a heavy tenant promptly. Token Observe’s refuses the specific call that would cross a currency ceiling, and pays for it with a maintained price table and a fail-closed refusal when a route cannot be priced.
Where a human decision has to happen before the action
This is the capability the two products are furthest apart on, and it is written here as prose rather than as a grid row for a reason that matters more than the row would. Envoy AI Gateway’s published documentation as of 2 September 2026 describes access control through Envoy Gateway SecurityPolicy, quota and usage-based rate limiting, upstream authentication, MCP tool authorisation through JWT scopes, claims and CEL rules, and observability through Prometheus, OpenTelemetry and access logs. A mechanism that parks one exact proposed action on a named person and holds the request until they decide is not described in that material. That is a statement about their documentation on that date and nothing more — put it to the project in writing, because an undocumented capability and an absent one look identical from outside, and their reference architecture explicitly invites you to add your own authorisation logic through Envoy’s extension filters, which is a place such a thing could plausibly be built.
What Token Observe does here is specific enough to be checkable. A policy whose action is require_approval refuses the request with a 403, mints an approval record bound to the SHA-256 of the canonicalised action plus the execution context it was proposed in — this subject, with these grants, through this delegation chain, making this call with these arguments — and holds the trace open until somebody decides. Change one argument and the hash no longer matches, and the retry is refused as a mismatch rather than allowed as near enough. Consumption is a compare-and-set, so two concurrent retries cannot both execute, and the approval expires: 60 minutes by default, and anywhere from one minute to seven days by policy.
The limit belongs beside the claim, because it is the thing operators get wrong. Approving does not push anything to the agent. There is no callback; an approval is a permission the agent redeems by repeating the identical request with its id, and the console says so in those words because it once did not, and an operator who believes the work is unblocked goes back to their day while it is not. The same honesty applies to reach: response-side evaluation of proposed tool calls extends the gate to agents that execute tools in their own process, and the architecture calls that defence in depth rather than a guarantee. Token Observe can only refuse a proposal it is shown.
A data plane you extend, against a decision point that arrives opinionated
Their reference-architecture post describes the shape of the product accurately and without overclaiming: you “implement safety checks and output validation rules that enable your team to control quality and compliance centrally, rather than embedding these checks individually within applications”, you “set usage-based guardrails directly in Envoy AI Gateway to prevent cost overruns”, and you “add your own authorization logic and/or custom functionality via Envoy’s extension filters”. The tier-one gateway centralises coarse-grained policies — authentication, top-level routing, global rate limiting — and provides what they call a single control point for platform-wide governance. That is a foundation you build the specific control on, which is the right architecture for a general data plane serving sixteen providers and an unknown set of requirements, and it is why the project can be as widely adopted as it is.
Token Observe arrives with the specific control already decided, and with the corresponding loss of flexibility. One pure function evaluates every governed request — the model gateway, the tool path, a backtest replay and a bundle compiled for a developer laptop all call the same evaluator, so a rule means the same thing wherever it is evaluated. A policy is a trigger, an action and a scope. Triggers match on the tool and its argument values, the model and its estimated input size, accumulated spend, request and token rate, the data classes detected in the payload, the prompt-injection score and the source it came from, or the hour of day in UTC. Actions are block, require approval, redact, warn and suspend the agent, resolved to a single verdict in which a block beats an approval and an approval beats a redaction. You cannot write an arbitrary filter; you can write those.
Which of the two you want follows from who is going to maintain it. A platform team with Envoy expertise and a clear internal specification will build a better-fitting control on extension filters than any vendor’s policy language would give them, and will keep it running on infrastructure they already operate. A team whose requirement arrived from a risk committee rather than from an architecture review usually wants the opinionated version, because the argument they need to win is about evidence rather than about topology, and “here is the verdict, the rule that produced it, the human who approved it and the chain entry that proves neither was edited afterwards” is a different artefact from a metrics dashboard, however good the dashboard.
Two MCP gateways, asking two different questions of a tool call
Both products put a gateway in front of Model Context Protocol servers, and the overlap is real enough to be worth separating carefully. Theirs is documented to “aggregate multiple MCP servers into a single unified endpoint”, prefixes tool names with the backend name so routing is unambiguous, filters the exposed set with a toolSelector taking either exact names or regular expressions, implements the MCP Authorization specification with issuer configuration and audience validation, and evaluates access against JWT scopes and claims with CEL expressions and a default fallback action — with OpenTelemetry tracing and Prometheus metrics on every MCP request, using the same stack as their LLM traffic.
Token Observe’s gateway asks a narrower question and a later one. Tools reach an agent namespaced server.tool and filtered to that agent’s grants, but the filtering is treated as usability rather than as the control: every tools/call is authorised again at execution, because a client can guess a tool name, and a tool the caller may not use is answered as unknown rather than forbidden, because confirming that it exists is an inventory disclosure. Each tool’s name, description and input schema is hashed when an operator approves it and re-checked on every catalogue refresh, so an upstream that quietly rewrites a descriptor — which is a rewrite of the model’s instruction surface, not of its data — is quarantined and refused until a human approves it again. An unpinned tool stays usable, because a gateway that refused every unreviewed tool would not be adopted.
The third difference is on the way back. A tool result is scanned as its own source and weighted 1.25 times the same words typed by a person, on the reasoning that the author of a tool result is data rather than a principal, and it is re-evaluated against data-class and injection rules before the model sees it. That is the channel through which agents are actually hijacked, and it is the part of the tool path that neither authorisation nor telemetry addresses. Their material describes authorisation and observability on MCP requests; result inspection of that kind is not described in their published documentation as of 2 September 2026, and is worth asking about rather than assuming either way.
- Their strengths on this path
- Multiplexing many servers behind one endpoint, backend-prefixed tool names, regex or exact tool selection, the MCP Authorization specification with issuer and audience validation, and CEL rules over JWT scopes and claims.
- Token Observe’s strengths on this path
- Re-authorisation at execution rather than at listing, descriptor pinning with quarantine on drift, and scanning the tool result as an untrusted source before the model reads it.
- The limit on both
- Neither can refuse a tool call that never arrives at it. In Token Observe an agent that routes nothing through the gateway is a shadow-AI radar problem, and saying which of the two is doing the work is the difference between a refund that did not happen and a refund you found out about.
Where each one is the right answer
Choose Envoy AI Gateway when
- You already run Envoy Gateway, and adding AI traffic to a data plane your team operates is a smaller change than introducing a second proxy with its own failure domain.
- You need high availability today. They run on Kubernetes on Envoy Proxy, and their scaling page documents multiple controller replicas with horizontal pod autoscaling for production; Token Observe is a single-writer process on one host with no replica and no uptime SLA.
- Provider breadth is the requirement. Their v1.0 notes name sixteen providers behind one OpenAI-compatible API; Token Observe has six first-class upstreams plus whatever OpenAI-compatible endpoints you register.
- Your credential rule is that no long-lived provider key exists anywhere, and per-request short-lived credentials through AWS STS, Entra ID or GCP workload federation are what you are shopping for.
- The control you need is a token allowance per tenant on shared infrastructure, which their QuotaPolicy expresses directly, in tokens, with CEL weighting, and at no licence cost.
Choose Token Observe when
- The ceiling has to be in currency and has to hold on the request that would cross it, rather than throttling the next one after the cost is known.
- An action has to wait for a named human, bound to that exact payload — single-use, expiring, and refused as a mismatch if one argument changes on the retry.
- The payload itself is the risk: a customer’s card number or an API key must not leave your network, so detection and redaction have to happen inline rather than be observed afterwards.
- The reader of the record is an auditor, and a metrics series is not an answer to the question of whether the record was altered after the fact.
- You run several providers and need the same rule to fire identically on all of them, with the fallback chain refusing to launder a content-policy refusal into an apparent success.
When you would run both
Running both is the normal answer, and the cleanest arrangement puts Token Observe in front of Envoy AI Gateway rather than instead of it. Because their sixteen providers sit behind one OpenAI-compatible API, that API is registrable as a Token Observe upstream like any other: the agent points at Token Observe, which resolves permissions, evaluates policy, redacts, reserves the budget and parks anything that needs a human, then forwards the allowed request to Envoy AI Gateway, which keeps doing what it is better at — provider routing across the full sixteen, short-lived upstream credentials, failover, prompt caching, Prometheus metrics and OpenTelemetry spans in the collector your team already reads. The reverse order also works, with the tier-one Envoy gateway handling client authentication and top-level routing and forwarding the agents that take consequential actions to Token Observe. Both arrangements add a hop and a second failure domain, so decide it deliberately: the honest version of the advice is that a chat assistant does not need both, and a refund agent might. If you want another enforcement point to be shown to agree with the first, scope and trigger matching can be compiled to digest-locked OPA Rego with reproducible positive and negative witnesses — though that artefact deliberately excludes permissions, budgets, approval consumption, kill switches and action precedence, which stay authoritative in Token Observe.
The category argument sits above this one: Token Observe and llm gateways covers what the whole category does and does not do, which is the better page to read if you have not yet shortlisted a product.
The others in the same slot
LiteLLM
If you already run LiteLLM, keep it. The question that decides whether you need anything more is about the actions your agents take, not about the proxy.
Portkey
Both hold the payload before it reaches a provider. One is built to carry it to more than 250 models; the other is built to refuse it and prove afterwards who said it could go.
Kong AI Gateway
Kong governs the traffic. Token Observe governs the action. If you already run Kong, the first one is nearly free and the second one is the only reason to read further.
Cloudflare AI Gateway
Cloudflare’s gateway decides what the payload contains. Token Observe decides whether the agent that sent it was allowed to.
MuleSoft AI Gateway
Both refuse the call inline. One refuses on behalf of an endpoint, the other on behalf of an agent that has an owner.
Envoy AI Gateway has quotas and rate limits. Why add budgets on top?
Because the two limits are written in different units and take effect at different moments. Their QuotaPolicy is expressed in tokens over sliding windows of a second, a minute, an hour or a day, weighted by CEL, and their documentation states that the token cost is computed for each completed request and charged to the bucket, so the request that crosses the line is served and the 429 lands on a later one — with streams explicitly not interrupted. Token Observe’s ceiling is in dollars, per request and per rolling hour, UTC day and UTC month, and it is reserved before egress against every provider and fallback the resolved route could execute. If your question is which team is using more than its share, theirs answers it at no cost. If your question is can this agent spend more than £500 this month under any circumstances, the answer has to be decided before the call, and it has to know the price.
Is this a like-for-like replacement?
No, and treating it as one would be a mistake in either direction. Envoy AI Gateway is a Kubernetes-native data plane built on Envoy Gateway with sixteen providers behind one API, horizontal scaling, per-request short-lived upstream credentials for three cloud providers, and a control-plane API declared stable at v1.0. Token Observe is a single self-hosted process that decides a governed request and records why: deny-by-default action-level permissions, one policy verdict per request, approvals bound to an exact payload, a hash-chained audit log. There is no version of Token Observe that is a better Envoy data plane, and the product’s own roadmap says to integrate above or beside gateways rather than to compete on connectivity.
Does Envoy AI Gateway do PII redaction or prompt-injection scanning?
Their published documentation as of 2 September 2026 does not describe one, and that is a statement about the documentation rather than about the code. What it does describe is the place such a thing would go: their reference architecture invites you to implement safety checks and output validation rules centrally at the gateway and to add your own authorisation logic through Envoy’s extension filters, and their security page documents external authorisation for business-specific logic. Ask the project directly, and ask specifically whether anything scans a tool result rather than only the prompt. Token Observe detects eleven data classes inline, three of them checksum-validated, and scores prompt injection with nine weighted patterns — heuristics, not a classifier, and a novel phrasing that matches none of them scores zero.
Have you tested Envoy AI Gateway against Token Observe?
No. Every claim on this page about Envoy AI Gateway is a paraphrase of their own site, documentation or repository as read on 2 September 2026, and none of it has been verified against a running deployment. There has been no independently witnessed bake-off against any product, and that gap is named in the product’s own launch gates as evidence that does not yet exist. The claims about Token Observe are checkable in a different way: the licence sets out a 30-day evaluation so a prospective customer’s security team can read, run and attack the software before a purchase order, with no gag clause and no pre-approval of results — with the caveat that the published licence is a template pending review by counsel in England and Wales rather than an executed grant of rights.
Their project is Apache-2.0 and free. What is the argument for paying?
There is no argument for paying if what you need is connectivity, and this page opens by saying so. The argument starts where an agent can take an action somebody has to answer for — a refund, a deployment, an email, a database write — and an HTTP 200 stops being acceptable proof that what was authorised is what happened. At that point you need a human decision bound to one exact payload, a currency ceiling that holds on the call that would cross it, a redaction pass before egress, and a record whose integrity you can demonstrate rather than assert. Whether that is worth a licence is a question about your risk register rather than about either product, and the free option remains free while you decide.
Prefer to ask a person? Write to us →
Tell us which one you are already running.
If Envoy AI Gateway is already in your stack, the useful question is not which to buy but what each is for, and where the seam between them sits. Say what you have and you will get a straight answer — including when the answer is that you do not need a second thing.
no form · no qualification step · no sales desk · the other three ways in