What happens between your agent and the model.

1. 01

   Authenticate

   The agent presents its gateway-minted key as a bearer token, or on x-api-key for the Anthropic dialect and x-goog-api-key for native Gemini. Token Observe takes the SHA-256 of what was presented, looks the key up by that digest, and re-compares the stored and computed digests in constant time.

   It is first because everything after it is scoped to a subject: the role set, the budget window, the policy scope, the spend ledger and the trace all hang off an agent id, and none of them can be gathered for a caller who has not been named. The constant-time re-comparison is there so that a storage layer which ever answered on a prefix could not be turned into a byte-at-a-time oracle against a live credential. Unknown, revoked and expired keys are three different operational events and are logged as three, but the caller receives one identical message, because telling somebody their key merely expired confirms that it was once valid.

2. 02

   Resolve the agent

   Loads the agent record, its roles, the kill switches currently in force that select it, and its recent spend window. All four are read live from the store on every request: on this path there is no cached copy, no sync job and no propagation step. The one deliberate exception is elsewhere — the signed bundle a developer seat hook decides against, which is a snapshot by design and is bounded by a freshness window rather than read live.

   It sits between authentication and the decision because the decision is a pure function that performs no input or output of its own, so everything it will read has to be gathered ahead of it. Reading live rather than from a cache is what makes an edit or a suspension take effect on the agent’s next call instead of after a redeploy — and it is why the registry cannot drift from what is actually running, because there is only one list.

3. 03

   Open the trace

   A trace id is minted and its row written before anything can reject the request, and a metadata-only opening event is appended: the requested model, message and tool counts, tags, session id and the names of any forwarded headers. The id comes back on x-acp-trace-id on every outcome, blocked ones included, and on both vendors’ own request-id headers as well, so an SDK’s logging correlates with the flight recorder without being configured to.

   Third, and specifically before sanitisation, scanning and the verdict, because a request that vanished from the flight recorder is indistinguishable from one that was never made. It cannot be first because a trace belongs to an agent. And the opening event carries counts and names rather than content for a reason that is really a rule about ordering: at this point nothing has been sanitised, scanned or redacted, so no payload text may be persisted yet.

4. 04

   Sanitise

   All text content is normalised to a fixpoint — the loop repeats because removing one layer can reveal another — stripping the Unicode Tags block, zero-width characters, bidirectional embeddings, overrides and isolates, the invisible formatting characters, the supplementary private-use planes, and lone halves of a surrogate pair. Where anything is removed, a trace event records how many characters and which categories went — never the characters themselves.

   Before the scanners, because they have to see what the model will actually read. The Tags block encodes a complete invisible ASCII alphabet, so an instruction written in it is unreadable to a human reviewer and to a naive pattern match while remaining perfectly legible to the model; a scanner run first would score the visible text and miss the payload entirely. The zero-width joiner is deliberately left in place, because emoji sequences need it, and a sanitiser that mangles ordinary text gets switched off.

5. 05

   Scan

   The personal-data and secret detectors run over the prompt content, and a heuristic injection scan scores it. Content that arrived as a tool result is scored at 1.25 times, because the author of a tool result is data rather than a principal — a directive found there is indirect injection by definition, where the same words from the user might be a request. Both detectors are pattern-based, and their limits are published rather than implied: matching is regular expressions plus checksums, so a card number, an IBAN or an NHS number validates at high confidence while free-text personal data — a name, an address, a described condition — is not detected at all, and neither are identifier formats outside the UK and US shapes the detectors know. The injection heuristics are regular expressions too, so paraphrase, translation and encoding defeat them; the product’s own threat model records that as an accepted false-negative rate requiring a named person’s sign-off. Confidence scores are exposed so a policy can set its own threshold, and the honest description in the source is a compensating control rather than a complete data-loss-prevention system.

   Before the verdict, because the policy triggers read its output: a data-class rule has nothing to fire on until a class has been detected, and an injection threshold has nothing to compare against. It is also the last step before the decision that touches the payload at all, which is what allows the decision itself to be pure — it receives findings, not text. And because the detectors are heuristic, the position matters more than the detection rate: an injection finding is one input to a policy rather than the control itself, and what actually bounds the damage a missed injection can do is the deny-by-default role check, the tool scoping and the approval branch that all sit downstream of it.

6. 06

   Govern

   One function returns one verdict — allow, block or require approval — plus a redaction plan. Inside it the order is fixed and the first hard failure wins: engaged kill switches, then agent lifecycle status, then deny-by-default role checks including every link of an agent-to-agent delegation chain, then rate limits — the USD ceiling is deliberately deferred for any agent that has one configured, because money cannot be decided honestly until routing has fixed which upstream will actually serve the call — then the policies whose scope selects this subject, by priority, with block beating require-approval beating redact and warn.

   This is the single decision point, and it is the reason the rest of the path can be read at all: there is one place where a request is allowed or refused rather than a scattering of checks. It is pure and has no database of its own, which is what lets the identical bytes of policy logic decide a request at the gateway and a tool call on a developer’s laptop instead of a second evaluator drifting away from the first. The internal order runs most absolute first — a kill switch is an operator’s emergency stop and must not be outranked by anything, and there is no sense evaluating a policy against an agent that is already suspended.

7. 6b

   Intersect with the named human

   When on-behalf-of enforcement is switched on, the roles mapped from the named principal’s identity-provider groups are appended as the last link of the delegation chain and run through the same delegated evaluator that agent-to-agent delegation uses — a function whose entire contract is that every link must allow. The intersection can therefore only narrow what step 6 allowed: a human whose group grants everything is a no-op rather than an escalation. Only one of the three settings actually refuses anything, and the threat model says so in as many words: off, the default, resolves no principal at all, and shadow resolves everything and records what it would have refused while letting the request through. An install that has not reached enforce should not describe the intersection as a mitigation it holds.

   It runs after the verdict because an already-blocked request gains nothing from a second reason, and the trace should not carry intersection noise for a call that never happened. It runs before the approval branch because asking a named human to approve something the intersection forbids spends their attention on a request that has to be refused either way. And when it is off — which is the default — it reads nothing at all, not one store call, so an install that has never turned it on behaves exactly as it did before the feature existed.

8. 07

   Enact

   Shadow matches are recorded first, whatever the verdict. On block, a typed error goes back and the trace closes as blocked; on require-approval, an approval record bound to a hash of the request payload is created, an approval-requested event is published, and the caller receives a 403 carrying the approval id; on allow, the redaction plan is applied to the outbound payload and a redaction event records kinds and counts, never values.

   Enactment is separated from decision because the decision is pure and enactment is not — every write, publication and status change lives here, which is what makes the verdict reviewable and testable without a database. Shadow matches are written before the branch on purpose: a shadow policy nobody can see is not a dry run, it is a policy that does nothing. Two cases are refused rather than rewritten here, and the reason is the same both times — renaming a JSON object key changes which argument a tool receives, and text redaction cannot reach the pixels of an image, so a redaction that could not be applied honestly becomes a refusal instead.

9. 08

   Route

   A concrete provider and model are resolved, honouring the agent’s data policy — zero data retention, no training on payloads and a required serving region are three independent constraints rather than one flag — with a typed fallback chain built behind the chosen target. For an agent under a budget, the primary target and every fallback still standing behind it are priced against the resolved model and provider kind rather than the name the caller typed; a reachable target with no active price row is refused with a 409 before egress; and the conservative estimate — each leg taken at the highest rate in the reachable candidate set — is tested against the hour, day and month windows and reserved in one per-agent transaction, so concurrent calls cannot each pass a ceiling one of them breaches.

   After the verdict and after redaction, because what gets routed is the redacted payload, and because where bytes may go is a governance constraint attached to the agent rather than a transport detail. Three separate data-policy booleans rather than one retention flag because providers genuinely differ on each — a provider may retain but not train, or train but not retain, and region pinning is orthogonal to both — so a single flag would overpromise. This is also where the money verdict is taken rather than at step 6, because it is the first point at which route rules, tier selection, compatibility filtering and every failover behind the chosen target are all fixed: a ceiling tested against the model the caller named would be tested against a price the call may never pay. Routing is also the last step that can still refuse locally: after it, the request leaves the building.

10. 09

    Call upstream

    One explicit timeout, at most two retries beyond the first attempt per provider with full-jitter exponential backoff from 250 ms capped at four seconds, an upstream Retry-After honoured up to a ten-second ceiling, and a per-provider circuit breaker. Failover is typed: a timeout, a rate limit and an upstream server error move to the next candidate in the chain, while a content-policy refusal, an authentication failure, an invalid request and an over-long context do not.

    The typed rule is a governance property rather than an availability one. Those four classes fail identically at every provider, so failing over on them either pays a second vendor to return the same error or quietly launders a refusal into a success — and a fallback chain that launders refusals is worse than no fallback chain, because the trace then records a clean call. A request under a hard spend ceiling gets exactly one network attempt, because a timeout is ambiguous: the vendor may have completed and billed the call, and one admission reservation must not end up covering several independently billable attempts.

11. 10

    Govern the response

    If the model proposes a tool call, that proposal is evaluated against tool-call policies before it is handed to the client. On a stream the frames are held per index until the arguments parse as complete JSON, governed, and only then released.

    Without this step, a rule such as refunds over £200 need approval would bind only the agents that route execution through Token Observe’s own MCP gateway, and the policy author has no way to express that distinction in the rule they wrote. Holding the frames is what makes it real rather than nominal: a tool call’s name arrives in the opening frame while its arguments stream in afterwards, so evaluating at the start can only enforce name-based rules and evaluating after the deltas are on the wire enforces nothing — and the interesting rule is always an argument rule. It is defence in depth and not a guarantee, and the source says so plainly: Token Observe can only refuse a proposal it is shown. An agent that never routes tools through it at all is caught by the shadow-AI radar, not here.

12. 11

    Meter and record

    Provider usage is normalised into one shape whose cache buckets are mutually exclusive by construction, priced, written to the ledger, appended to the trace as events, and the trace is closed with a terminal status before webhook events are emitted.

    Last, because it is the only step that knows what actually happened, and it runs on every terminal path including the ones nobody would think to instrument. Metering happens even when step 10 refuses the proposed tool call, and even when the client disconnected halfway through a stream: the tokens were spent either way, and a partial trace is still evidence. Mutually exclusive cache buckets matter for the same reason — a normalisation that let a cached-read token be counted twice would produce a spend figure nobody can reconcile against the vendor’s invoice.

The floor: 64 characters

Longer than any value the detectors match at a stated maximum length. It guarantees that a card number, an IBAN, a national insurance number or a private-key header cannot be split across the boundary — and it guarantees nothing at all about the kinds that declare no length.

The run rule

The cut is pulled out of the middle of an unbroken run of value characters, because a value that has not finished arriving matches nothing yet. The alphabet is the base64url set every token-shaped credential is built from, plus the punctuation the unbounded detectors allow; whitespace is deliberately excluded, which is why ordinary prose is held back by less than a word more than the floor.

The cap: 4,096 characters

The most one channel will hold and classify. Past it, one irreversible marker is emitted and the run is suppressed through its delimiter — the kind is no longer claimed, and neither half of the value is released. Memory stays bounded by failing closed rather than by cutting.

One buffer per channel

Tool-call arguments stream as their own delta channel, one per index, and a credential can straddle two chunks there exactly as it can in text. Each channel forks its own hold-back under the same plan and the same placeholder map, so one value keeps one placeholder for the whole conversation.

Held tool proposals

Frames are buffered per index and released only once the arguments parse as complete JSON and the proposal has been governed. Argument text is capped at 256 KB, past which the stream is refused rather than released ungoverned — a tool call whose arguments do not parse inside that bound is not one this gateway can evaluate, and could not evaluate must never resolve to allowed on the path whose whole purpose is refusing what a policy forbids.

Your reverse proxy

Streaming breaks under response buffering, so the terminator in front of Token Observe must not buffer responses — for nginx that is proxy_buffering off, and Token Observe also sends x-accel-buffering: no. It must preserve the authorization and api-key headers, set the forwarded-for and forwarded-proto headers, and use a read timeout at least as long as your longest expected model response; 600 seconds is the documented safe default.

delta 1   here is the token: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJhZ2VudCIs
delta 2   ImV4cCI6MTc2NzIyNTYwMH0.5Kq3nP2wR9tYb1cZ0aLmX7vQd8sEjHkGf4u

With a 64-character hold-back alone, everything more than 64 characters
behind the tail is released. By delta 2 that is the head of the token,
and no detector has objected, because a JWT matches nothing at all
until its third segment arrives.

With the run rule, the cut is walked back to the start of the unbroken
run, so nothing is released until the value has arrived whole:

emitted   here is the token: [REDACTED:JWT]

And a refusal on a stream cannot be a status code, so it is a frame:

data: {"error":{"message":"policy \"no card numbers in output\" refused
  the model's response: model response contained credit_card",
  "type":"acp_policy_blocked","code":"ACP_POLICY_BLOCKED",
  "acp":{"traceId":"trc_7Qk2Zpv4"}}}

data: [DONE]

Wrong runtime

The host ran Node 20 while the release image and the required continuous integration run Node 24. The number is informative about the code path, not evidence for the exact release runtime.

Not a soak

Thirty seconds does not expose multi-hour write-ahead-log growth, thermal throttling, disk exhaustion, long-run memory behaviour or recovery under overload.

Mock upstream

No provider latency, rate limits, streaming, retries, failover or internet failure. It measures Token Observe’s inline work, not end-user response time — which in practice is dominated by the model.

Narrow request mix

No tool calls, approvals, large prompts, images, streaming, embeddings, MCP traffic or concurrent dashboard and report reads.

Fresh database, one host

It does not model partner-sized trace search, audit, approval or radar tables, and it is one run on developer hardware rather than a controlled fleet.

Not the release container

Docker was not installed on the measuring host, so the run is not from the release image. Continuous integration separately requires a packaged-image boot, backup and restore smoke test.

Duration                          30.143 s
Concurrency                       16
Completed / succeeded / failed    6,216 / 6,216 / 0
Successful throughput             206.2 requests/s
Latency p50 / p95 / p99           71.2 / 163.8 / 223.4 ms
Database growth                   19,538,664 bytes (3,143.3 per request)
Scope                             single-node governed path, in-process mock upstream

Deployment model

Self-hosted, bring-your-own-key

Runtime

Node.js 24 LTS, or the published container image

Data store

SQLite in WAL mode, one file

Shape

A modular monolith: one process, one database, five surfaces

Vendor telemetry

None. No phone-home, no product analytics, no vendor-side copy of your traces

Network egress

Governed payloads leave your network only for the model and tool providers you configure, after policy and redaction

The ingress surface

A gateway that lists “OpenAI support” and leaves an engineer to read the source for the path is not answering the question they arrived with. These are the paths.

/v1/chat/completions

OpenAI Chat Completions, streaming and buffered

/v1/responses

OpenAI Responses

/v1/messages

Anthropic Messages

/v1/embeddings

Embeddings

/v1beta/models/{model}:generateContent

Native Gemini, streaming on :streamGenerateContent; the same two methods are also accepted under /v1/models

/v1/models

Model listing, filtered to what the agent may reach; the same prefix also carries the Gemini generateContent methods

/mcp

MCP gateway over Streamable HTTP

/api/*

Control-plane REST API

/scim/v2/*

Bounded SCIM Users provisioning