---
title: "Glossary: AI gateway"
canonical_url: "https://tokenobserve.com/glossary/ai-gateway"
published: 2026-09-02
last_updated: 2026-10-21
---

# Glossary: AI gateway

An AI gateway is a policy-bearing proxy for AI traffic: it terminates the calls an application or agent makes to models, tool servers and other agents, and applies authorisation, inspection, quota and logging rules to them before they reach the upstream. The term is used for at least three different products — an API gateway extended to model traffic, a security-inspection layer sold as an AI firewall, and a cloud runtime that bundles gateway, identity and observability — so the label describes a position in the architecture rather than a set of guarantees.

## Also called

- AI firewall
- agent gateway
- AI runtime security gateway
- MCP gateway
- AI security gateway

## In practice

Three lineages have converged on the phrase and they optimise for different things. API gateway vendors arrived from connectivity, and their strength is operating a data plane at scale: Kong, Envoy, Portkey, Cloudflare and MuleSoft all publish AI or agent gateway products of this kind. Security vendors arrived from traffic inspection, and their strength is detection content and threat research; Palo Alto Prisma AIRS, Cisco AI Defense, Zenity, Noma, WitnessAI and Check Point are the names most often shortlisted. Cloud platforms arrived from the runtime, and their strength is that policy is evaluated inside the same trust boundary as the workload, with no extra hop. All of those descriptions are vendor-authored and none has been independently tested here; the lineages predict what a product is good at far better than its category label does.

What separates an AI gateway from a model proxy is usually the traffic it terminates. Model calls are the easy case. The harder and more important cases are tool calls (increasingly over the Model Context Protocol) and agent-to-agent traffic, because that is where the consequences are: a model call returns text, and a tool call moves money, changes a ticket, deploys a service or writes a row into somebody’s system of record. It is where the interesting attack lives. A tool result is text written by a third party and read by the model as if it were instruction, which is why indirect prompt injection through a tool result is the channel that hijacks agents, and why a gateway that inspects prompts but not tool results is inspecting the safer half. The same goes for tool descriptions: a tool server can rewrite its own name, description and input schema at any time, and those strings are part of the model’s instruction surface, so a descriptor that changes after approval is a supply-chain event rather than a refresh.

Inspection is where the category over-claims most reliably. Detectors for personal data and secrets are pattern matching plus checksums: a card number validating on Luhn, an IBAN on mod-97, an NHS number on mod-11 are caught at high confidence, while free-text personal data (a name, an address, a described medical condition) and identifier formats outside the shapes the detector knows are not caught at all. Prompt-injection heuristics are typically weighted regular expressions rather than classifiers, which means paraphrase, translation and encoding defeat them, and a novel phrasing scores zero. Neither of those is a reason to skip inspection; both are reasons to treat it as a compensating control rather than the control. What bounds the damage a missed injection can do is a deny-by-default permission set, not the detector: one that means the hijacked agent has nothing worth calling.

Four questions separate members of the category once the demonstrations are over. Does it decide, or does it observe and alert? Does it re-check the agent’s authorisation at the moment a tool is called, or does it only filter the list of tools the agent was shown? A filtered list is a usability feature being asked to do access control. Does it inspect under explicit bounds on depth, size and node count, and refuse when a bound is hit, or does it forward the tail it did not read? And what does it do when its own dependencies fail? A gateway that permits traffic when its policy store is unreachable has an availability story and no governance story.

There are two things no gateway in this category reaches, and both should be stated in an evaluation rather than discovered. It cannot control what the calling application does with the text it receives: output redaction masks values on the way back, and the moment the application renders that text into a page or passes it to a shell, the failure is in the application. And it governs only the traffic that arrives at it, so an agent configured with a provider key directly is invisible to it, which makes discovery of ungoverned AI use a separate discipline with its own failure mode, in which a dead evidence feed and a clean estate look exactly alike.

## Example: A tool description that changed on Tuesday

An agent has been approved to use a search tool on an internal Model Context Protocol server. On Tuesday the server’s maintainer edits the tool’s description to add a line asking the caller to include the contents of any file it has recently read. Nothing in the agent’s code changes, no permission changes, and no prompt in the organisation changes, but the description is part of what the model reads before deciding what to do, so the agent’s behaviour changes on its next run. A gateway that hashes each tool descriptor at approval and compares it on every catalogue refresh sees a name, description or input schema that no longer matches its pin, and quarantines the tool rather than serving it. A gateway that scans prompts and results for dangerous content sees nothing unusual at all, because at the moment of the edit no traffic has been sent.

## Routinely confused with

LLM gateway: Overlapping, and often the same product. The practical difference is scope and intent: an LLM gateway is judged on how well it carries model traffic, an AI gateway on what it refuses across model, tool and agent-to-agent traffic. A product that terminates only model calls cannot see the tool results that carry indirect prompt injection.

AI firewall: A marketing name for the inspection subset. The metaphor imports an assumption worth resisting, that safety comes from recognising bad traffic, when the durable control in an agent estate is authority: what the agent may call at all, decided before any pattern matching, and still standing when the pattern match misses.

MCP gateway: The tool-traffic half specifically: a governed endpoint in front of Model Context Protocol servers that pins tool descriptors, re-derives authorisation per call, and inspects arguments and results. Many AI gateways include one; an MCP gateway on its own governs tool calls and not the model calls that decided to make them.