AI gateway

Three lineages have converged on the phrase and they optimise for different things. API gateway vendors arrived from connectivity, and their strength is operating a data plane at scale — Kong, Envoy, Portkey, Cloudflare and MuleSoft all publish AI or agent gateway products of this kind. Security vendors arrived from traffic inspection, and their strength is detection content and threat research; Palo Alto Prisma AIRS, Cisco AI Defense, Zenity, Noma, WitnessAI and Check Point are the names most often shortlisted. Cloud platforms arrived from the runtime, and their strength is that policy is evaluated inside the same trust boundary as the workload, with no extra hop. All of those descriptions are vendor-authored and none has been independently tested here; the lineages predict what a product is good at far better than its category label does.

What separates an AI gateway from a model proxy is usually the traffic it terminates. Model calls are the easy case. The harder and more important cases are tool calls — increasingly over the Model Context Protocol — and agent-to-agent traffic, because that is where the consequences are: a model call returns text, and a tool call moves money, changes a ticket, deploys a service or writes a row into somebody’s system of record. It is where the interesting attack lives. A tool result is text written by a third party and read by the model as if it were instruction, which is why indirect prompt injection through a tool result is the channel that actually hijacks agents, and why a gateway that inspects prompts but not tool results is inspecting the safer half. The same goes for tool descriptions: a tool server can rewrite its own name, description and input schema at any time, and those strings are part of the model’s instruction surface, so a descriptor that changes after approval is a supply-chain event rather than a refresh.

It is worth being precise about what inspection is worth, because this is where the category over-claims most reliably. Detectors for personal data and secrets are pattern matching plus checksums: a card number validating on Luhn, an IBAN on mod-97, an NHS number on mod-11 are caught at high confidence, while free-text personal data — a name, an address, a described medical condition — and identifier formats outside the shapes the detector knows are not caught at all. Prompt-injection heuristics are typically weighted regular expressions rather than classifiers, which means paraphrase, translation and encoding defeat them, and a novel phrasing scores zero. Neither of those is a reason to skip inspection; both are reasons to treat it as a compensating control rather than the control. What actually bounds the damage a missed injection can do is not the detector — it is a deny-by-default permission set that means the hijacked agent has nothing worth calling.

Four questions separate members of the category once the demonstrations are over. Does it decide, or does it observe and alert? Does it re-check the agent’s authorisation at the moment a tool is called, or does it only filter the list of tools the agent was shown — a filtered list is a usability feature being asked to do access control. Does it inspect under explicit bounds on depth, size and node count, and refuse when a bound is hit, or does it forward the tail it did not read? And what does it do when its own dependencies fail? A gateway that permits traffic when its policy store is unreachable has an availability story and no governance story.

There are two things no gateway in this category reaches, and both should be stated in an evaluation rather than discovered. It cannot control what the calling application does with the text it receives: output redaction masks values on the way back, and the moment the application renders that text into a page or passes it to a shell, the failure is in the application. And it governs only the traffic that arrives at it, so an agent configured with a provider key directly is invisible to it — which makes discovery of ungoverned AI use a separate discipline with its own failure mode, in which a dead evidence feed and a clean estate look exactly alike.