LLM gateway architecture

Token Observe’s request path is eleven steps and the order is encoded in the evaluator rather than being a documentation artefact. It is worth walking because each position is an argument.

Authenticate first, obviously. Resolve the subject second — the agent, its roles, the active kill switches and its recent spend window — from the same row the registry holds, so there is no exported copy to drift. Open the trace third, before anything can refuse the request, so that a refusal is recorded rather than missing; the identifier returns on a response header on every response including a blocked one. Sanitise fourth, so that every later reader sees the text the model would have received rather than the text as it was transmitted. Scan fifth, over the sanitised text, per fragment and under each fragment’s own source.

Govern sixth. Inside that step the order is again fixed: engaged kill switches, then the lifecycle state, then deny-by-default permissions including the delegation intersection, then the ceilings, then the policies whose scope selects this subject in priority order. A halt has to beat a lifecycle check because a halt is the control somebody engages under pressure. Permissions have to beat ceilings because an agent that may not perform an action at all should be refused for that reason rather than for being over budget. And the mask that narrows an agent’s authority to the named human’s runs after the verdict and before the approval branch — after, because an already-blocked request gains nothing from a second reason, and before, because nothing should ask a human to approve something the intersection forbids.

Enact seventh: a typed error and a trace closed as blocked, or an approval record and a 403 carrying its identifier, or the redaction plan applied to the outbound payload. Route eighth, honouring the agent’s data policy across the primary and every fallback. Call upstream ninth, with an explicit timeout, retries only on idempotent failures, and a per-provider circuit breaker. Govern the response tenth — if the model proposes a tool call, evaluate it against tool-call rules before returning it, which is what makes a rule such as refunds over a threshold need approval bind an agent that executes tools outside your gateway. Meter and record eleventh, including when step ten refused the proposal, because the tokens were spent either way.

Step ten deserves its caveat stated where the claim is: it is defence in depth, not a guarantee. A gateway can only refuse a proposal it is shown, and an agent that never routes its tool calls anywhere near you is a discovery problem rather than a policy one.

A buffered response has a moment where the whole answer is in hand and a decision can still be taken. A stream has no such moment: the status line is spent on the first byte, and bytes already written cannot be recalled. Every guarantee a gateway offers on the buffered path has to be re-argued for the streaming one, and most implementations do not.

Three rules make it work. Resolve the response-side plan before the first byte, computed from the policies that could apply rather than from the classes that turn out to be present, because there is no later point at which to widen it. Hold back enough of the stream that a value split across two chunks cannot escape half-masked, with a separate buffer per tool-call argument channel, and pull the cut back off anything it would split — both off a match that straddles it and out of the middle of an unbroken run of value characters, since a token that matches nothing until its third segment arrives will otherwise have its head emitted before the detectors have seen it whole. Token Observe’s hold-back floor is 64 characters, which exceeds every kind that states a maximum length, and its per-channel run bound is 4,096 characters, beyond which it emits one irreversible marker and suppresses the run through its delimiter rather than releasing either half of an ambiguous value.

The refusal shape has to change too. A blocking rule cannot answer with a status code once the stream has started, so it ends the stream with an in-band error frame the instant its class is seen, masked either way so the value never reaches the client. A rule running in observation mode uses an observe-only scan over the same boundaries and appends its evidence without changing the response, which is what keeps a dry run distinguishable from an outage.

Two smaller streaming details save a lot of debugging. Always ask the upstream for usage and suppress the extra chunk downstream when the client did not ask for it, so metering does not depend on client behaviour. And close a client disconnect as an error with what was received so far, rather than as success — a partial trace is still evidence, and Token Observe shipped the opposite behaviour once and records it as a defect.

A gateway is in the path of every agent call, so the discipline that matters most is about what is allowed to make a request slower or make it fail. Four things belong outside it in Token Observe, and the reasoning generalises.

Event delivery is best-effort and happens on a later tick, so a webhook receiver that is down cannot delay or fail the governed request that produced the event. Licensing never reaches the request path at all: being over a ceiling is reported rather than retroactively enforced, and deleting the licence file returns the install to an unlimited fallback with every control still enforcing — because a licence problem that degraded a customer’s safety controls is the one failure a governance product cannot have. Retention runs as an hourly pass in small batches, each its own short transaction, so ageing traces out interleaves with traffic instead of stalling the fail-closed path. And discovery is scheduled from a due-time column written when a run finishes rather than from an in-process timer, so a process redeployed more often than the interval still scans.

The inverse discipline is that anything genuinely inline has to be bounded. Inspection of attacker-controlled text runs synchronously, and a JavaScript runtime cannot interrupt a running regular expression, so every pattern has to be linear in the length of its input and every walk needs caps on depth, node count and string length that fail closed. Token Observe found two super-linear patterns in its own scanner, rewrote them, added the scan cap and published the episode.

Finally, be explicit about the topology the whole design assumes. Token Observe is one process with one write path; PostgreSQL exists behind the store ports as an evaluation alternative rather than as a supported high-availability topology, and there is no multi-replica, clustering or vendor-operated uptime claim. That is a real constraint on where this architecture fits, and a gateway inline in every agent call is the last place to discover an availability assumption you had not made explicit.