LLM gateway
also called AI proxy · LLM proxy · model gateway · LLM router · AI model gateway
The work a gateway does is unglamorous and genuinely valuable. It normalises provider dialects, so an application written against one vendor’s request shape can reach several. It holds the provider keys, so no application ever sees one and rotating a key is one operation rather than a deployment per service. It routes — by model name, by cost, by availability — and it fails over when a provider is unavailable. It applies rate limits and quotas per caller. It caches, where the traffic tolerates it. And it attributes usage, which is the part most teams actually buy it for, because a single provider bill with one line on it cannot answer which team, which application or which agent spent the money.
The adoption property is what makes the category so large. Because integration is a base URL and a credential, a gateway can be introduced into an estate without asking any application team to change code, which means it can be introduced quickly and removed quickly. That is a real architectural advantage and it is also the reason the slot is contested: the same one-line change is how a governance layer, a security inspection layer or a cost-control layer arrives, so several quite different products compete for the same environment variable. Kong Agent Gateway, Envoy AI Gateway, Portkey, LiteLLM, Cloudflare AI Gateway and MuleSoft AI Gateway are the names that come up most; how each behaves is a question for its own documentation, and nothing here has been tested against any of them.
Three places leak, and they are the places to press a vendor on. The first is money. A quota expressed in requests bounds volume, not spend, because what a call costs depends on which model actually served it after failover and on how that provider counts cached, reasoning and batch tokens — providers genuinely differ, and a gateway that estimates cost from the caller’s request rather than from what the route could reach will under-count exactly when it matters. The second is streaming. A streamed response has no moment at which the whole response exists, no way to recall a byte already written, and no status line left to answer with once the first byte is out; response-side controls that work on a buffered reply have to be rebuilt for it, and a gateway that quietly disables them for streams has disabled them for most production traffic. The third is failover semantics, which is a governance property masquerading as an availability one: a 429, a timeout and an upstream 5xx are worth retrying elsewhere, while a content-policy refusal, an authentication failure, an invalid request and an over-long context fail identically at every provider, so failing over on them either pays a second vendor for the same error or launders a refusal into a success — and the trace then records a clean, completed call that the first provider declined to perform.
What a gateway does not settle is authority. Carrying a request is not deciding it, and the questions that decide it are about the caller rather than the traffic: is this agent permitted this action, was that permission delegated by somebody with the standing to delegate it, and is anyone other than the agent’s own operator able to check afterwards what it did. A gateway can be extended to answer those, and several are being extended in exactly that direction; the distinction is worth keeping because it is what determines whether an outage in the component is an availability incident or a governance one. Token Observe occupies this slot deliberately — the same base-URL change, the same routing and quota table stakes — and treats the connectivity features as the price of entry rather than the reason to install it.
A gateway also sits on the critical path of every model call, so its availability becomes the estate’s availability for anything model-backed. That argues for running it close to the agents and deciding in advance whether it should refuse or step aside when it cannot do its job. If it is carrying traffic, stepping aside may be defensible. If it is deciding, stepping aside means the control was absent and nothing recorded that it was.
What the first week of a gateway usually shows
A platform team points twelve services at a gateway by changing one base URL and one key in each. Nothing about the applications changes and, at first, neither does the bill. What changes is the arithmetic they can do. Within a week they can see that two services account for four fifths of the spend, that one of them is calling the largest available model for a classification task a small one would do, that a retry loop in a third service is issuing three identical requests for every user action, and that a fourth is calling a provider nobody remembers approving. None of those is a governance decision and all four are the input to one. This is the honest case for a gateway: it does not decide anything, and until it exists nobody in the organisation can see enough to decide.
What llm gateway is routinely confused with
- AI gateway
- In common use, an LLM gateway is defined by what it carries — model calls — and an AI gateway by what it decides, usually across model calls, tool traffic and agent-to-agent traffic together. Vendors use the two names interchangeably, so the label is not evidence; ask which traffic types are terminated and whether the component can refuse one.
- API gateway
- The same architectural position, different failure modes. LLM traffic brings token-based cost that is not knowable from the request alone, streaming responses that cannot be inspected after the fact, non-deterministic outputs that break response caching assumptions, and dialect translation between providers. A general API gateway handles none of those without extension.
- AI control plane
- The gateway carries and the control plane decides. A gateway that also holds the agent registry, the permission model and the policy set is doing both jobs, which is common and fine — but the two are worth separating when evaluating, because a strong gateway with a weak record of authority fails silently rather than loudly.
Related terms
AI gateway
An AI gateway is a policy-bearing proxy for AI traffic: it terminates the calls an application or agent makes to models, tool servers and other agents, and applies authorisation, inspection, quota and logging rules to them before they reach the upstream. The term is used for at least three different products — an API gateway extended to model traffic, a security-inspection layer sold as an AI firewall, and a cloud runtime that bundles gateway, identity and observability — so the label describes a position in the architecture rather than a set of guarantees.
AI control plane
An AI control plane is the layer that holds the authoritative configuration for an organisation’s AI agents — which agents exist, who owns each one, what each may call, what each may spend, and what happens when a rule is broken — and that makes the configuration binding by sitting in, or being consulted by, the path those agents’ requests take. The name is borrowed from networking, where the control plane decides what should happen and the data plane carries the traffic that does it; the distinction matters here because a product can implement either half and still be sold under the same label.
Inline enforcement
Inline enforcement means the decision to allow, refuse, alter or hold an action is taken in the path the action must travel, before it takes effect, by a component the acting system cannot bypass or overrule. The alternatives — a rule the agent is asked to follow, a check inside the agent’s own framework, an alert raised afterwards — are advisory rather than enforcing, because in each case the party being governed is also the party enforcing.
Fail-closed
Fail-closed describes a control that denies the action it governs whenever it cannot complete its own check — because a dependency is unavailable, a required piece of evidence is missing or stale, a value cannot be evaluated, or the control itself is down. The opposite arrangement, fail-open, permits the action in those same circumstances, which turns every outage in the control into a temporary and silent absence of the control.
Where Token Observe does this
The definition above is the field's, not the product's. This is the part of the product that implements it, for a reader who wants to see one.
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Nothing verifies a provider’s ZDR or training claim
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
One billable egress: no retry, no failover
The terms next to this one
What the field is called, what its members have in common, and the one property that separates them from each other.
Definitions are the easy part.
The glossary is written to be useful whether or not you ever buy anything. If you have got to the point of deciding how to implement one of these in your own estate, say what your agents do and you will get a straight answer about what it would actually take.
no form · no qualification step · no sales desk · the other three ways in