---
title: "Glossary: LLM gateway"
canonical_url: "https://tokenobserve.com/glossary/llm-gateway"
published: 2026-09-02
last_updated: 2026-10-21
---

# Glossary: LLM gateway

An LLM gateway is a proxy that sits between applications and one or more model providers, presenting a single endpoint and a single credential while handling provider routing, failover, rate limiting, caching, key custody and usage accounting on their behalf. Applications adopt one by changing a base URL rather than by rewriting code, which is why it is usually the first piece of shared AI infrastructure an organisation deploys.

## Also called

- AI proxy
- LLM proxy
- model gateway
- LLM router
- AI model gateway

## In practice

The work a gateway does is unglamorous and valuable. It normalises provider dialects, so an application written against one vendor’s request shape can reach several. It holds the provider keys, so no application ever sees one and rotating a key is one operation rather than a deployment per service. It routes — by model name, by cost, by availability — and it fails over when a provider is unavailable. It applies rate limits and quotas per caller. It caches, where the traffic tolerates it. And it attributes usage, which is the part most teams buy it for, because a single provider bill with one line on it cannot answer which team, which application or which agent spent the money.

The adoption property is what makes the category so large. Because integration is a base URL and a credential, a gateway can be introduced into an estate without asking any application team to change code, which means it can be introduced quickly and removed quickly. That is a real architectural advantage and it is also the reason the slot is contested: the same one-line change is how a governance layer, a security inspection layer or a cost-control layer arrives, so several quite different products compete for the same environment variable. Kong Agent Gateway, Agent Router (formerly Envoy AI Gateway), Portkey (now Prisma AIRS AI Gateway), LiteLLM, Cloudflare AI Gateway and MuleSoft Omni Gateway are the names that come up most; how each behaves is a question for its own documentation, and nothing here has been tested against any of them.

Three places leak, and they are the places to press a vendor on. The first is money. A quota expressed in requests bounds volume, not spend, because what a call costs depends on which model actually served it after failover and on how that provider counts cached, reasoning and batch tokens (providers differ), and a gateway that estimates cost from the caller’s request rather than from what the route could reach will under-count exactly when it matters. The second is streaming. A streamed response has no moment at which the whole response exists, no way to recall a byte already written, and no status line left to answer with once the first byte is out; response-side controls that work on a buffered reply have to be rebuilt for it, and a gateway that quietly disables them for streams has disabled them for most production traffic. The third is failover semantics, which is a governance property masquerading as an availability one: a 429, a timeout and an upstream 5xx are worth retrying elsewhere, while a content-policy refusal, an authentication failure, an invalid request and an over-long context fail identically at every provider, so failing over on them either pays a second vendor for the same error or launders a refusal into a success, and the trace then records a clean, completed call that the first provider declined to perform.

What a gateway does not settle is authority. Carrying a request is not deciding it, and the questions that decide it are about the caller rather than the traffic: is this agent permitted this action, was that permission delegated by somebody with the standing to delegate it, and is anyone other than the agent’s own operator able to check afterwards what it did. A gateway can be extended to answer those, and several are being extended in that direction; the distinction matters because it determines whether an outage in the component is an availability incident or a governance one. Token Observe occupies this slot deliberately, with the same base-URL change and the same routing and quota table stakes, and treats the connectivity features as the price of entry rather than the reason to install it.

A gateway also sits on the critical path of every model call, so its availability becomes the estate’s availability for anything model-backed. That argues for running it close to the agents and deciding in advance whether it should refuse or step aside when it cannot do its job. If it is carrying traffic, stepping aside may be defensible. If it is deciding, stepping aside means the control was absent and nothing recorded that it was.

## Example: What the first week of a gateway usually shows

A platform team points twelve services at a gateway by changing one base URL and one key in each. Nothing about the applications changes and, at first, neither does the bill. What changes is the arithmetic they can do. Within a week they can see that two services account for four fifths of the spend, that one of them is calling the largest available model for a classification task a small one would do, that a retry loop in a third service is issuing three identical requests for every user action, and that a fourth is calling a provider nobody remembers approving. None of those is a governance decision and all four are the input to one. This is the case for a gateway: it does not decide anything, and until it exists nobody in the organisation can see enough to decide.

## Routinely confused with

AI gateway: In common use, an LLM gateway is defined by what it carries — model calls — and an AI gateway by what it decides, usually across model calls, tool traffic and agent-to-agent traffic together. Vendors use the two names interchangeably, so the label is not evidence; ask which traffic types are terminated and whether the component can refuse one.

API gateway: The same architectural position, different failure modes. LLM traffic brings token-based cost that is not knowable from the request alone, streaming responses that cannot be inspected after the fact, non-deterministic outputs that break response caching assumptions, and dialect translation between providers. A general API gateway handles none of those without extension.

AI control plane: The gateway carries and the control plane decides. A gateway that also holds the agent registry, the permission model and the policy set is doing both jobs, which is common and fine, but separate the two when evaluating, because a strong gateway with a weak record of authority fails silently rather than loudly.