Your rules. Applied to every governed call.
One checkpoint, on hardware you run, between your agents and the model providers they call.
OPENAI_BASE_URL="https://tokenobserve-cloud-production.up.railway.app/w/YOUR_WORKSPACE_ID/v1"
OPENAI_API_KEY="<issued-agent-key>"
# Or use the Anthropic dialect:
ANTHROPIC_BASE_URL="https://tokenobserve-cloud-production.up.railway.app/w/YOUR_WORKSPACE_ID"
ANTHROPIC_API_KEY="<issued-agent-key>"Replace YOUR_WORKSPACE_ID with your workspace ID. Customer-hosted deployments use their own hostname. A base-URL change alone does not create credentials or grant permissions. Check your SDK and version against the compatibility guide, then test one request before expanding the rollout.
Every governed request follows the same path.
Follow a request through five stages. Open the eleven steps underneath to see how the order of the checks supports each control.
Sample data in the real Token Observe interface, recorded from a local evaluation with a temporary database, a mock provider and one request. No live provider key was used.
- beat 1 of 5
Name the caller
Nothing is decided for an anonymous caller. The key resolves to one registered agent with a named owner, and its roles, its ceilings and any stop switch in force are read live from the store on every request, not from a cached copy, so a suspension takes effect on the next call instead of after a redeploy.
- 01
Authenticate
The agent presents its gateway-minted key as a bearer token, or on x-api-key for the Anthropic dialect and x-goog-api-key for native Gemini. Token Observe takes the SHA-256 of what was presented, looks the key up by that digest, and re-compares the stored and computed digests in constant time.
Why this step comes here
It is first because everything after it is scoped to a subject: the role set, the budget window, the policy scope, the spend ledger and the trace all hang off an agent id, and none of them can be gathered for a caller who has not been named. The constant-time re-comparison is there so that a storage layer which ever answered on a prefix could not be turned into a byte-at-a-time oracle against a live credential. Unknown, revoked and expired keys are three different operational events and are logged as three, but the caller receives one identical message, because telling somebody their key merely expired confirms that it was once valid.
- 02
Resolve the agent
Loads the agent record, its roles, the kill switches currently in force that select it, and its recent spend window. All four are read live from the store on every request: on this path there is no cached copy, no sync job and no propagation step. The one deliberate exception is elsewhere — the signed bundle a developer seat hook decides against, which is a snapshot by design and is bounded by a freshness window rather than read live.
Why this step comes here
It sits between authentication and the decision because the decision is a pure function that performs no input or output of its own, so everything it will read has to be gathered ahead of it. Reading live rather than from a cache is what makes an edit or a suspension take effect on the agent’s next call instead of after a redeploy. It is also why the registry cannot drift from what is running: there is only one list.
- 01
- beat 2 of 5
Start the record, then read what is being sent
The trace is opened before anything can refuse the request, which is why a blocked call is recorded as thoroughly as one that went through. Only then is the text normalised and scanned, in that order, so the detectors see what the model will read rather than what a human reviewer can see.
- 03
Open the trace
A trace id is minted and its row written before anything can reject the request, and a metadata-only opening event is appended: the requested model, message and tool counts, tags, session id and the names of any forwarded headers. The id comes back on x-acp-trace-id on every outcome, blocked ones included, and on both vendors’ own request-id headers as well, so an SDK’s logging correlates with the flight recorder without being configured to.
Why this step comes here
Third, and specifically before sanitisation, scanning and the verdict, because a request that vanished from the flight recorder is indistinguishable from one that was never made. It cannot be first because a trace belongs to an agent. And the opening event carries counts and names, not content, for a reason that is a rule about ordering: at this point nothing has been sanitised, scanned or redacted, so no payload text may be persisted yet.
- 04
Sanitise
All text content is normalised to a fixpoint — the loop repeats because removing one layer can reveal another — stripping the Unicode Tags block, zero-width characters, bidirectional embeddings, overrides and isolates, the invisible formatting characters, the supplementary private-use planes, and lone halves of a surrogate pair. Where anything is removed, a trace event records how many characters and which categories went — never the characters themselves.
Why this step comes here
Before the scanners, because they have to see what the model will read. The Tags block encodes a complete invisible ASCII alphabet, so an instruction written in it is unreadable to a human reviewer and to a naive pattern match while remaining perfectly legible to the model; a scanner run first would score the visible text and miss the payload. The zero-width joiner is deliberately left in place, because emoji sequences need it, and a sanitiser that mangles ordinary text gets switched off.
- 05
Scan
The personal-data and secret detectors run over the prompt content, and a heuristic injection scan scores it. Content that arrived as a tool result is scored at 1.25 times, because the author of a tool result is data rather than a principal — a directive found there is indirect injection by definition, where the same words from the user might be a request. Confidence scores are exposed, so a policy sets its own threshold rather than inheriting one.
Why this step comes here
Before the verdict, because the policy triggers read its output: a data-class rule has nothing to fire on until a class has been detected, and an injection threshold has nothing to compare against. It is also the last step before the decision that touches the payload at all, which is what allows the decision itself to be pure — it receives findings, not text. And because the detectors are heuristic, the position matters more than the detection rate: an injection finding is one input to a policy rather than the control itself, and what bounds the damage a missed injection can do is the deny-by-default role check, the tool scoping and the approval branch that all sit downstream of it. Heuristic is not exhaustive, and the limits are published. Matching is regular expressions plus checksums, so a card number, an IBAN or an NHS number validates at high confidence while free-text personal data — a name, an address, a described condition — is not detected at all, and neither are identifier formats outside the UK and US shapes the detectors know. The injection heuristics are regular expressions too, so paraphrase, translation and encoding defeat them, and the product’s own threat model records that as an accepted false-negative rate requiring a named person’s written sign-off. The source’s own description is a compensating control rather than a complete data-loss-prevention system.
- 03
- beat 3 of 5
Decide once
One function returns one verdict: allow, block, or hold it for a named person to approve. Every check lives in that one place rather than scattered along the path, which is what makes it possible to read your rules and know what they do.
- 06
Govern
One function returns one verdict (allow, block or require approval) plus a redaction plan. Inside it the order is fixed and the first hard failure wins: engaged kill switches, then agent lifecycle status, then deny-by-default role checks including every link of an agent-to-agent delegation chain, then rate limits — the USD ceiling is deliberately deferred for any agent that has one configured, because money cannot be decided accurately until routing has fixed which upstream will serve the call — then the policies whose scope selects this subject, by priority, with block beating require-approval beating redact and warn.
Why this step comes here
This is the single decision point, and it is the reason the rest of the path can be read at all: there is one place where a request is allowed or refused rather than a scattering of checks. It is pure and has no database of its own, which is what lets the identical bytes of policy logic decide a request at the gateway and a tool call on a developer’s laptop instead of a second evaluator drifting away from the first. The internal order runs most absolute first — a kill switch is an operator’s emergency stop and must not be outranked by anything, and there is no sense evaluating a policy against an agent that is already suspended.
- 6b
Intersect with the named human
When on-behalf-of enforcement is switched on, the roles mapped from the named principal’s identity-provider groups are appended as the last link of the delegation chain and run through the same delegated evaluator that agent-to-agent delegation uses — a function whose entire contract is that every link must allow. The intersection can therefore only narrow what step 6 allowed: a human whose group grants everything is a no-op rather than an escalation. Only one of the three settings refuses anything, and the threat model says so: off, the default, resolves no principal at all, and shadow resolves everything and records what it would have refused while letting the request through. An install that has not reached enforce should not describe the intersection as a mitigation it holds.
Why this step comes here
It runs after the verdict because an already-blocked request gains nothing from a second reason, and the trace should not carry intersection noise for a call that never happened. It runs before the approval branch because asking a named human to approve something the intersection forbids spends their attention on a request that has to be refused either way. And when it is off — which is the default — it reads nothing at all, not one store call, so an install that has never turned it on behaves exactly as it did before the feature existed.
- 07
Enact
Shadow matches are recorded first, whatever the verdict. On block, a typed error goes back and the trace closes as blocked; on require-approval, an approval record bound to a hash of the request payload is created, an approval-requested event is published, and the caller receives a 403 carrying the approval id; on allow, the redaction plan is applied to the outbound payload and a redaction event records kinds and counts, never values.
Why this step comes here
Enactment is separated from decision because the decision is pure and enactment is not — every write, publication and status change lives here, which is what makes the verdict reviewable and testable without a database. Shadow matches are written before the branch on purpose: a shadow policy nobody can see is not a dry run, it is a policy that does nothing. Two cases are refused rather than rewritten here, and the reason is the same both times — renaming a JSON object key changes which argument a tool receives, and text redaction cannot reach the pixels of an image, so a redaction that could not be applied faithfully becomes a refusal instead.
- 06
- beat 4 of 5
Send it, and only then spend the money
Routing fixes which provider and model will serve the call, so where an agent has a USD ceiling it is tested against the price you will pay, not the one the caller typed. Routing is also the last step that can still refuse locally: after it, the request has left your network.
- 08
Route
A concrete provider and model are resolved, honouring the agent’s data policy — zero data retention, no training on payloads and a required serving region are three independent constraints rather than one flag — with a typed fallback chain built behind the chosen target. For an agent under a budget, the primary target and every fallback still standing behind it are priced against the resolved model and provider kind rather than the name the caller typed; a reachable target with no active price row is refused with a 409 before egress; and the conservative estimate — each leg taken at the highest rate in the reachable candidate set — is tested against the hour, day and month windows and reserved in one per-agent transaction, so concurrent calls cannot each pass a ceiling one of them breaches.
Why this step comes here
After the verdict and after redaction, because what gets routed is the redacted payload, and because where bytes may go is a governance constraint attached to the agent rather than a transport detail. Three separate data-policy booleans rather than one retention flag because providers differ on each — a provider may retain but not train, or train but not retain, and region pinning is orthogonal to both — so a single flag would overpromise. This is also where the money verdict is taken rather than at step 6, because it is the first point at which route rules, tier selection, compatibility filtering and every failover behind the chosen target are all fixed: a ceiling tested against the model the caller named would be tested against a price the call may never pay. Routing is also the last step that can still refuse locally: after it, the request leaves the building.
- 09
Call upstream
One explicit timeout, at most two retries beyond the first attempt per provider with full-jitter exponential backoff from 250 ms capped at four seconds, an upstream Retry-After honoured up to a ten-second ceiling, and a per-provider circuit breaker. Failover is typed: a timeout, a rate limit and an upstream server error move to the next candidate in the chain, while a content-policy refusal, an authentication failure, an invalid request and an over-long context do not.
Why this step comes here
The typed rule is a governance property rather than an availability one. Those four classes fail identically at every provider, so failing over on them either pays a second vendor to return the same error or launders a refusal into a success — and a fallback chain that launders refusals is worse than no fallback chain, because the trace then records a clean call. A request under a hard spend ceiling gets exactly one network attempt, because a timeout is ambiguous: the vendor may have completed and billed the call, and one admission reservation must not end up covering several independently billable attempts.
- 08
- beat 5 of 5
Govern the answer, then count the cost
A tool call the model proposes is judged before it reaches your agent. Metering runs last and on every ending, including the refusals and the streams a client abandoned halfway, because the tokens were spent either way and a partial trace is still evidence.
- 10
Govern the response
If the model proposes a tool call, that proposal is evaluated against tool-call policies before it is handed to the client. On a stream the frames are held per index until the arguments parse as complete JSON, governed, and only then released.
Why this step comes here
Without this step, a rule such as refunds over £200 need approval would bind only the agents that route execution through Token Observe’s own MCP gateway, and the policy author has no way to express that distinction in the rule they wrote. Holding the frames is what makes it real rather than nominal: a tool call’s name arrives in the opening frame while its arguments stream in afterwards, so evaluating at the start can only enforce name-based rules and evaluating after the deltas are on the wire enforces nothing — and the interesting rule is always an argument rule. It is defence in depth and not a guarantee, and the source says so plainly: Token Observe can only refuse a proposal it is shown. An agent that never routes tools through it at all is caught by the shadow-AI radar, not here.
- 11
Meter and record
Provider usage is normalised into one shape whose cache buckets are mutually exclusive by construction, priced, written to the ledger, appended to the trace as events, and the trace is closed with a terminal status before webhook events are emitted.
Why this step comes here
Last, because it is the only step that knows what happened, and it runs on every terminal path including the ones nobody would think to instrument. Metering happens even when step 10 refuses the proposed tool call, and even when the client disconnected halfway through a stream: the tokens were spent either way, and a partial trace is still evidence. Mutually exclusive cache buckets matter for the same reason — a normalisation that let a cached-read token be counted twice would produce a spend figure nobody can reconcile against the vendor’s invoice.
- 10
For traffic routed through the gateway, configured policies can allow, redact, hold or block a request. Detection is heuristic and can miss sensitive data or misclassify a payload.
The request passes the configured checks and proceeds to its provider. Its trace records the policy decision.
A configured rule matches a value and masks or tokenises it before forwarding the payload to the provider. Unmatched values can still pass through.
A policy requires human approval. The approval binds to this payload, expires and can be used once; the caller receives a typed refusal with the approval identifier.
The gateway denies the routed request and records the blocking policy and outcome in its trace.
Shadow mode records policy decisions without changing the request. Review false positives and missed detections before enforcing the policy on live traffic.
Choose the right first workload.
The mechanism above is worth your afternoon or it is not, and the fastest way to tell is the second column.
Built for
One IT team that owns the agents, the provider accounts and the AI line on the card statement. The documented pilot envelope for this release is roughly five to fifty API-key agents, one or two model providers and a bounded set of pinned MCP tools.
Agents you can point at a new base URL. For the OpenAI-compatible, Anthropic and Gemini dialects that is normally a base-URL change and a credential swap rather than an application refactor.
A business that would rather run one container on one volume than buy a hosted control plane. One Node process, one SQLite file, your own provider keys, and no vendor-operated component anywhere in the request path.
Anyone who has to show afterwards what an agent was asked to do and what it was refused. Every governed outcome returns a trace id, refusals included, and a blocked request is recorded as thoroughly as one that went through.
Wrong for
Anyone who cannot accept that if Token Observe is unavailable, the agents it governs stop calling models. Governance is inline and fails closed, on one node and one SQLite process, with no high availability, no proven recovery objectives, no availability service level and no service credits.
Work whose outage would harm your customers. The documentation’s own instruction is to keep production-critical workloads outside this release.
Anyone who needs a throughput or concurrency commitment. There is none, and the only published figure is a 30-second laboratory baseline against a mock upstream, which the source explicitly refuses to let become a limit.
Traffic that never reaches the gateway. An agent pointed straight at a provider is invisible to this path; that is the shadow-AI radar’s job, and the radar sees the evidence you feed it and no more.
Anyone who needs the detectors to be complete. Personal-data and injection matching is heuristic rather than exhaustive: free-text personal data is not detected at all, and paraphrase, translation and encoding defeat the injection heuristics.
Procurement that requires SOC 2, ISO 27001, ISO 42001 or an independent penetration test. There are none, and you will be told so on the first call.
A base URL and a credential. No SDK to swap.
Adoption is a base-URL change and a credential swap; for supported OpenAI-compatible, Anthropic and Gemini ingress the product documents this as normally a base-URL change rather than an application refactor.
Point the agent at Token Observe and give it a gateway-minted agent key where the vendor key used to sit. There is no SDK to swap, no library to import and, for the request shapes listed below, no code change; the agent keeps speaking the dialect it already speaks. From that moment it has an identity, a budget, a permission set and a searchable record of each governed request. The shapes that are not on that list — media parts, the hosted Responses features, OpenRouter’s routing options — are refused rather than passed through, and the section on refusals below is the one to read before you plan a rollout rather than after.
One asymmetric detail catches people, so check it before you edit a config file: the OpenAI dialect takes a trailing /v1 on the base URL and the Anthropic dialect does not. The onboarding troubleshooting names that as one of the top three causes of an invalid-key 401.
There is also a commercial consequence, and the documentation puts it in the imperative: tell your developers before you do this. Once a subscription-based coding client is given a gateway credential it stops using that developer’s own subscription, and the work is billed per token to whichever provider account your install uses. For a governed company fleet that is the purpose, but people notice it on the day it happens, not in the rollout plan.
What Token Observe still does not do is accept a subscription as a credential. A signed-in client pointed at the gateway without an agent key is rejected and recorded on the shadow-AI radar as an unrecognised caller, because relaying a consumer subscription is prohibited by Anthropic’s terms, and Anthropic began blocking subscription OAuth in third-party clients in January 2026. That is a limit of the vendor’s terms rather than of the gateway, and it is stated rather than worked around.
The exact request shapes on each dialect, and what is refused rather than passed through
- OpenAI dialect
- POST /v1/chat/completions and POST /v1/responses. The chat response is byte-faithful to OpenAI including the cached-token detail, and streaming is data-only server-sent events terminated by a done sentinel. The Responses adapter is a compatibility subset rather than a hosted response store: previous-response references, conversations, item references, background jobs, prompt templates, opaque file ids and hosted tools are refused rather than passed through, so governance always sees the complete payload.
- Anthropic dialect
- POST /v1/messages and POST /v1/messages/count_tokens. A max-tokens value is required, the anthropic-version and anthropic-beta headers are forwarded verbatim, and the stream is named-event SSE from message start to message stop. The token counter doubles as the pre-flight budget primitive, which is why it is a governed endpoint rather than a passthrough.
- Native Gemini
- The generateContent and streamGenerateContent methods under /v1beta/models, also accepted under /v1/models. One candidate per call is governed, so a candidate count other than one is refused rather than partly evaluated; cached content, opaque file references, built-in Google tools, thought state and non-text response modalities are refused for the same reason.
- Model list
- GET /v1/models returns only the models the calling agent’s roles permit. That is where deny-by-default role scoping becomes visible inside a client SDK rather than only in a console — the agent’s own tooling shows it a smaller world.
- Embeddings
- POST /v1/embeddings carries the same request governance and usage metering as completions, for OpenAI and OpenRouter provider rows only. Anthropic, Google, Azure and Bedrock routes fail closed before credential resolution rather than attempting a call that would not be governed the same way.
- Tools
- POST, GET and DELETE on /mcp speak streamable-HTTP MCP. Initialisation negotiates the protocol version and issues a session id, the tool list returns the namespaced union filtered to the agent’s grants, and a tool call re-enforces grants and argument policy independently — filtering the list alone is a known anti-pattern, because the list is a convenience and the call is the control.
# OpenAI SDK, or anything OpenAI-compatible
OPENAI_BASE_URL="https://gateway.example.com/v1" # was https://api.openai.com/v1
OPENAI_API_KEY="<gateway-minted agent key>"
# Anthropic SDK — note the deliberate absence of a trailing /v1
ANTHROPIC_BASE_URL="https://gateway.example.com" # was https://api.anthropic.com
ANTHROPIC_API_KEY="<gateway-minted agent key>"
# Every governed outcome comes back with its trace id, blocked ones included
x-acp-trace-id: trc_7Qk2Zpv4 # both vendors' request-id headers carry it tooStreaming is where a gateway’s guarantees are easiest to lose.
A streamed response gets the same redaction, the same response-side policy, the same proposed-tool governance and the same metering as a buffered one, and it takes four separate mechanisms to make that true — because a stream has no moment at which the whole response exists, and a byte already on the wire cannot be recalled. Two differences survive those mechanisms and are stated below rather than smoothed over: a stream cannot answer 403, and its redaction plan is settled before the first byte by policies that may not end up firing.
Start with metering, because it is the guarantee most easily lost by accident. Token Observe always asks the upstream for usage on the OpenAI dialect, on every stream, regardless of what the client asked for, and then suppresses the extra usage chunk on the way out when the client did not want it. The alternative is metering that depends on client behaviour, which is to say a spend ledger an agent can opt out of by omitting a field.
The four mechanisms in full: the 64-character floor, the run rule, the 4,096-character cap, and why a refusal on a stream is a frame rather than a status code
Then redaction. Outbound text passes through a hold-back buffer, and every tool-call argument channel gets its own. Without it, a card number split across two chunks is invisible to a per-chunk scan and escapes output redaction intact; sharing one buffer between the text stream and the argument channels would interleave unrelated content and corrupt both. The buffer only ever emits text that sits at least a hold-back behind the newest character, and the tail is re-scanned on every push.
The interesting part is where the cut is made. A proposed cut is walked back off anything it would split, and there are two ways it can split a value. It can fall inside a value the detectors have already matched, in which case redaction would replace half of it and emit the other half verbatim. Or it can fall inside an unbroken run of value characters, which may be a value that has not finished arriving and therefore matches nothing yet. The two interleave — moving the cut off one can land it inside the other — so the walk iterates rather than applying each rule once, and it is bounded so that it always terminates.
Sixty-four characters is the floor, and the reason is precise. It exceeds every kind whose pattern states a maximum length — the longest being a spaced 34-character IBAN, then the private-key header — so no such value can straddle the boundary and escape the scan. It is not enough on its own. The JWT and API-key detectors state no maximum at all, and a JWT matches nothing whatsoever until its third segment arrives, so a fixed width alone shipped the head of a 256-character token to the client 64 characters at a time and then reported that it had masked nothing. That is the failure the run rule exists to fix, and it is why both rules are needed rather than either: the fixed width covers the kinds that declare a length, the run rule covers the kinds that do not.
Holding an unbroken run cannot be unbounded, so it is capped at 4,096 characters — an order of magnitude longer than any fixed credential shape the detectors know, and longer than the enterprise single-sign-on tokens that motivated the cap. A run that crosses it is not cut in half. Token Observe emits one irreversible API-key marker and suppresses the rest of the run through to its delimiter, giving up the claim to name the kind exactly rather than releasing either half of an ambiguous value. Bounded memory therefore fails closed. A buffered scan sees the whole text at once and has no such fallback, and the source states that difference between the two paths.
Response-side policy is the fourth mechanism, and the one that changes shape entirely. The buffered path decides from a finished response: it can look at what the model returned and widen its plan to cover the classes that are present. A stream never has that moment, and evaluating the accumulated text in a finaliser would decide correctly and far too late, because every byte it was deciding about has already been written. So the plan is computed before the first byte, from the policies that could bear on the response rather than from the classes that turn out to be in it. Widening speculatively costs nothing in effect, since a kind that never appears is never matched — but it is not free in fidelity, and the cost is named: the redaction mode is settled by policies that may not end up firing, which can make a placeholder irreversible where the buffered path would have left it reversible. That is the one place the two paths differ.
A block policy cannot be pre-empted that way, because whether it fires depends on what the model says. Its classes go into the mask set so the value never reaches the client either way, and the stream is ended the instant one of them is seen. What a stream cannot do is answer 403: the reply is hijacked before the first byte, so the status line is long spent by the time there is anything to refuse. The refusal therefore arrives in band, as a typed error frame carrying the same error code a buffered refusal would have carried, which is the thing a client can branch on. The same constraint is why the cost of a streamed call rides in an HTTP trailer rather than a header — it is not known until the stream ends.
Shadow-mode classes are scanned through the same boundary-safe hold-back as enforcing ones and recorded as policy-decision evidence without changing a byte of the response, so you can learn your false-positive rate on streamed traffic before you turn anything on. And a client disconnect aborts the upstream request rather than letting an abandoned stream keep costing money: the trace closes as an error with whatever arrived, metered, because a partial trace is still evidence.
- The floor: 64 characters
- Longer than any value the detectors match at a stated maximum length. It guarantees that a card number, an IBAN, a national insurance number or a private-key header cannot be split across the boundary — and it guarantees nothing at all about the kinds that declare no length.
- The run rule
- The cut is pulled out of the middle of an unbroken run of value characters, because a value that has not finished arriving matches nothing yet. The alphabet is the base64url set every token-shaped credential is built from, plus the punctuation the unbounded detectors allow; whitespace is deliberately excluded, which is why ordinary prose is held back by less than a word more than the floor.
- The cap: 4,096 characters
- The most one channel will hold and classify. Past it, one irreversible marker is emitted and the run is suppressed through its delimiter — the kind is no longer claimed, and neither half of the value is released. Memory stays bounded by failing closed rather than by cutting.
- One buffer per channel
- Tool-call arguments stream as their own delta channel, one per index, and a credential can straddle two chunks there exactly as it can in text. Each channel forks its own hold-back under the same plan and the same placeholder map, so one value keeps one placeholder for the whole conversation.
- Held tool proposals
- Frames are buffered per index and released only once the arguments parse as complete JSON and the proposal has been governed. Argument text is capped at 256 KB, past which the stream is refused rather than released ungoverned — a tool call whose arguments do not parse inside that bound is not one this gateway can evaluate, and could not evaluate must never resolve to allowed on the path whose whole purpose is refusing what a policy forbids.
- Your reverse proxy
- Streaming breaks under response buffering, so the terminator in front of Token Observe must not buffer responses — for nginx that is proxy_buffering off, and Token Observe also sends x-accel-buffering: no. It must preserve the authorization and api-key headers, set the forwarded-for and forwarded-proto headers, and use a read timeout at least as long as your longest expected model response; 600 seconds is the documented safe default.
delta 1 here is the token: eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJhZ2VudCIs
delta 2 ImV4cCI6MTc2NzIyNTYwMH0.5Kq3nP2wR9tYb1cZ0aLmX7vQd8sEjHkGf4u
With a 64-character hold-back alone, everything more than 64 characters
behind the tail is released. By delta 2 that is the head of the token,
and no detector has objected, because a JWT matches nothing at all
until its third segment arrives.
With the run rule, the cut is walked back to the start of the unbroken
run, so nothing is released until the value has arrived whole:
emitted here is the token: [REDACTED:JWT]
And a refusal on a stream cannot be a status code, so it is a frame:
data: {"error":{"message":"policy \"no card numbers in output\" refused
the model's response: model response contained credit_card",
"type":"acp_policy_blocked","code":"ACP_POLICY_BLOCKED",
"acp":{"traceId":"trc_7Qk2Zpv4"}}}
data: [DONE]Cloud workspace or customer-hosted runtime.
The cloud pilot runs one replica with a parent process and separate workspace children, using separate directory and workspace databases. It holds cloud evidence and supplied credentials. The customer-hosted runtime below is one Node process with one SQLite file in write-ahead-logging mode, supporting one active process on one node whichever store is selected.
It is a modular monolith by decision rather than by drift. Service extraction is not justified yet — there is no independent scaling, deployment or team-autonomy driver — and the request path benefits from every governance decision happening in-process with zero network hops, which is the difference between a governance decision and a governance round trip. The surfaces on that one listener are the model gateway, the MCP endpoint, the control-plane API, bounded SCIM user provisioning and the dashboard.
For customer-hosted installations, two things a security reviewer asks early. The metrics endpoint is unauthenticated and shares the main listener, and metrics include agent identifiers and spend — which is why the reference Compose file publishes to loopback rather than to every interface, and why an ingress that exposes it is a defect rather than a convenience. And Token Observe is self-hosted and bring-your-own-key: the vendor receives no product telemetry, no phone-home data, no prompts, no keys and no trace database, and governed payloads leave your network only for the model and tool providers you configure, after policy and redaction. The one optional built-in outbound path with a fixed public destination is the price-catalogue sync, which you can leave switched off.
The dependency rule, the capped runtime shape, the five ways to run it, and the storage and recovery limits stated in the same register
The dependency direction is the rule that makes the rest reviewable. The domain package is pure: no web framework, no database driver, no fetch, no node crypto. Everything it needs from the outside world is an interface — a clock, an id generator, a hasher, a signer, a password hasher and the store ports — and the server package is the composition root that implements them. The signer is the clearest illustration of why the rule is kept. The domain decides what an anchor statement is: its serialisation, its domain separation, its chaining, and the requirement that verification names an external trusted key. The Ed25519 itself lives in the server. The private half never crosses the port, so no domain code, no route and no log line can reach it, and the standalone verification script can then check an anchor holding only the domain package and node crypto — which is only possible because the rule held.
The runtime shape is deliberately small and deliberately capped. Node 24 only, about 200 MB of disk, and a 1 GB memory limit in both the shipped Compose file and the systemd unit the deployment guide publishes, each carrying the same comment: the gateway is in the request path for every agent, so cap its blast radius. The container is a multi-stage image pinned by digest, running as a non-root system user on port 4100 with separate data and backup volumes, an init process that reaps zombies and forwards the termination signal so shutdown stays graceful, and a boot-time assertion that proves the native SQLite binding loads in the exact shipped layout rather than in a developer’s.
The limits belong beside that, in the same register. Customer-hosted trace retention is unset by default and unset means keep forever; cloud activity retention is configured to 30 days, which is a deliberate choice (retention has to be decided, not inherited, and an upgrade that silently began deleting a customer’s evidence would be the worse failure), but it is also the reason a volume sized on the per-request storage figure below has to be sized against a decided window rather than against a month. Set the window and an hourly pass ages traces out in bounded batches. SQLite serialises writes, so under sustained heavy write load the bottleneck is trace-event insertion rather than the governance decision, and two instances must never point at one SQLite file. PostgreSQL is implemented behind the same store ports with dual-backend continuous integration for the store and concurrency invariants — but as an evaluation alternative, not a supported high-availability topology, and the onboarding endpoint keeps its technical-readiness flag false whenever a PostgreSQL URL is set. Backup is a full-snapshot vacuum into a single compacted file, consistent against a live database and taken with the source opened read-only; it is explicitly not point-in-time recovery, and no scheduled backup job, no retention enforcement and no backup-freshness alarm ship with it. Restoring leaves a quarantine marker that blocks a normal boot and is never cleared automatically.
- Customer-hosted runtime
- One Compose command, SQLite on a mounted volume. It suits a single control plane serving a whole organisation at the scale this release targets, because the write path is one process by design.
- Hybrid VPC
- In customer-hosted mode, the same image inside your own network behind your own TLS terminator. Payloads never leave your network except to the upstream model providers you explicitly enable.
- Air-gapped and on-premises
- A prebuilt image has no runtime dependency on a vendor service or a CDN, and the dashboard bundles its assets. The price catalogue ships and loads on every boot so metering works offline — and there is no price-creation endpoint, so correcting a price in an air-gapped install means updating the stored row directly.
- Platform-as-a-service
- Single node with someone else’s volume and edge, carrying three defects configuration cannot fix: an unauthenticated metrics endpoint on a public hostname, deploy-time-only healthchecks that make the readiness gate inert, and platform volume snapshots that bypass the restore script. A readiness-path healthcheck also makes the audit-key ceremonies impossible, so those deploys must use the liveness path instead.
- Kubernetes and Terraform
- A Kustomize base and a Terraform module ship, and both deliberately deploy one process with SQLite and separate read-write-once data and backup claims. Neither turns this release into a multi-replica or highly available topology, and both require an immutable image digest, an externally managed secret, an ingress that does not publish metrics, and a real restore drill on the chosen storage class.
Node >=24.0.0 <25.0.0 what the engines field, the container base and CI all pin
Process one active process, one node, whichever store is selected
Store SQLite in WAL mode supported; PostgreSQL is an evaluation alternative
Disk about 200 MB, plus roughly 3.1 KB per governed request of trace and audit
Retention trace retention is unset by default, and unset means keep forever
Memory 1 GB cap in the shipped Compose file and the documented systemd unit
In flight 256 concurrent external requests by default; 0 is explicitly unbounded
Container port 4100, non-root uid 10001, /data and /backups volumes, digest-pinned
Backup full-snapshot VACUUM INTO — not point-in-time recovery71 ms at the median, 206 requests a second, on one measured run.
206.2 requests per second and a 71.2 ms median on a single node, from one 30-second laboratory run against an in-process mock upstream on an Apple M1 Max under Node 20 — a reproducible baseline, not a capacity promise and not a service-level agreement.
The full figures, from an immutable commit measured on 14 August 2026: 6,216 requests completed, 6,216 succeeded, none failed, at concurrency 16 over 30.143 seconds; latency of 71.2 ms at the median, 163.8 ms at the 95th percentile and 223.4 ms at the 99th; and 19,538,664 bytes of database growth, which is 3,143.3 bytes per completed request. Seven non-rate policies were enabled during the run. The environment was a ten-core M1 Max MacBook Pro with 64 GB of memory, on a clean repository rebuilt immediately before the run.
It is reproducible by anyone holding the repository — build, then run the bench script with the same duration and concurrency — and the script exits non-zero if any warm-up or measured request fails, so a run full of fast refusals cannot be recorded as passing capacity evidence. The storage figure is the one most useful to a buyer sizing a volume: roughly 3.1 KB of trace and audit growth per governed request, from a fresh database with that particular request mix — and it compounds without limit unless a retention window is set, because the default is to keep every trace forever.
Six caveats travel with those numbers, and dropping any of them misrepresents the document that produced them. The document’s own closing instruction is unambiguous: until the partner-shaped test passes with agreed error, latency, storage and recovery thresholds, do not turn this baseline into a concurrency limit or a throughput commitment.
Duration 30.143 s
Concurrency 16
Completed / succeeded / failed 6,216 / 6,216 / 0
Successful throughput 206.2 requests/s
Latency p50 / p95 / p99 71.2 / 163.8 / 223.4 ms
Database growth 19,538,664 bytes (3,143.3 per request)
Scope single-node governed path, in-process mock upstreamThe full method, the six things this number does not prove, and what the soak harness does and does not close
The soak harness that would close the remaining gap ships, and its evidence does not. It runs only the compiled artefact on loopback against a fresh temporary database and the in-process mock provider, supplies all of its own settings so live configuration cannot leak into the child process, and has no deployment-URL option at all. Duration is mandatory and bounded from one minute to 24 hours; the request mix is deterministic at 60 per cent allowed model calls, 20 per cent policy blocks, 10 per cent approval journeys and 10 per cent authenticated reads; and the defaults fail the run below 95 per cent of planned journeys, above 1 per cent unexpected outcomes, 0.1 per cent unexpected refusals, 1,000 ms at the 95th percentile, 2,000 ms at the 99th, 256 MiB of memory growth, 32 KiB of database growth per attempt, or 2.5 times latency degradation. Missing samples, early exit and interruption all fail closed. The documentation draws the distinction: the harness closes the automation gap and does not retroactively close the evidence gap, and no completed multi-hour result is asserted.
- Wrong runtime
- The host ran Node 20 while the release image and the required continuous integration run Node 24. The number is informative about the code path, not evidence for the exact release runtime.
- Not a soak
- Thirty seconds does not expose multi-hour write-ahead-log growth, thermal throttling, disk exhaustion, long-run memory behaviour or recovery under overload.
- Mock upstream
- No provider latency, rate limits, streaming, retries, failover or internet failure. It measures Token Observe’s inline work, not end-user response time — which in practice is dominated by the model.
- Narrow request mix
- No tool calls, approvals, large prompts, images, streaming, embeddings, MCP traffic or concurrent dashboard and report reads.
- Fresh database, one host
- It does not model partner-sized trace search, audit, approval or radar tables, and it is one run on developer hardware rather than a controlled fleet.
- Not the release container
- Docker was not installed on the measuring host, so the run is not from the release image. Continuous integration separately requires a packaged-image boot, backup and restore smoke test.
What it cannot inspect, it refuses rather than forwards.
Several request shapes are refused with a typed error before any upstream call rather than forwarded ungoverned, because a gateway that silently passes on what it cannot inspect is worse than no gateway: the trace then says the request was governed.
Media is the largest of these and the one most likely to matter to a multimodal agent. Read the default install first: with no inspector configured — which is how it ships — every image is refused on every dialect, and audio and PDF parts are refused outright whatever is configured, because the request translator accepts text and inline image parts and nothing else. That is a real scope limit rather than a rough edge, and the product’s own API reference still describes the older, blanket position in which all media was rejected. What the code now adds is a governed path for images only, and it is worth reading because of what it refuses to assume. An inspector has to be explicitly configured; then inline bytes are decoded canonically, checked against their file signature and dimensions, and sent for inspection, and the inspector’s verdict must repeat the SHA-256 digest of the exact bytes it judged — otherwise the response cannot authorise those bytes. A caller-supplied MIME type is never treated as proof that opaque bytes are safe. Remote image URLs are deliberately outside the contract altogether, because fetching them at the gateway would create a server-side request forgery surface and fetching them at the provider would bypass inspection through time-of-check to time-of-use drift.
- Refused before egress, not after
- These limits fail with a typed error before an upstream call rather than after one, so a refusal costs nothing and cannot be mistaken for a provider outage in the trace.
- Unpriced routes
- For an agent under a budget, a resolved target or fallback with no price is refused before egress. Spending against a ceiling nobody can compute is how a budget stops being one.
- Ungoverned tool paths
- Token Observe can only refuse a tool proposal it is shown. An agent that executes tools without routing them through the gateway is not caught here at all; it is caught, if at all, by the shadow-AI radar — and that means the billing, egress, service-account and IDE evidence you feed it, plus what Token Observe can read of its own tables. Four of those five sources are operator-fed, so the radar sees what you give it and no more.
The other three families it refuses at the door: what is rewritten rather than refused, who chooses the model, and the tool result it will not show you
Two cases are refused rather than rewritten even when a redaction policy would otherwise cover them, and the reasoning is the same both times. Sensitive data in a JSON object key is refused because renaming an executable or schema key changes which argument a tool receives — a redaction that silently alters a contract is not a redaction. Sensitive data found by inspection inside an image is refused because text redaction cannot reach pixels, and a partly-redacted response would be a false assurance rather than a smaller leak.
The third family is about who chooses. On the OpenRouter dialect, the model list, provider selection, route, plugins, transforms and web-search options are refused, along with retained route suffixes that pin a provider or a price band, because each of them delegates model choice, provider choice, processing, search egress or charges outside governance. Configure fallbacks and provider selection on route rules instead, where the decision is recorded and auditable. The same instinct governs failover across dialects: vendor-specific options stay within the wire family that defines them, incompatible fallbacks are removed before egress, and the call is refused if none remains rather than silently dropping or reinterpreting the behaviour the caller asked for.
The last one is the most uncomfortable. When an MCP tool returns image, audio, blob, resource or URI content, that content is withheld. The refusal is terminal, and it says explicitly that the tool already ran. The side effect happened; the gateway is telling you it cannot show you the result. That sentence exists because the alternative reads as though nothing occurred.
What happens when it is unavailable.
If Token Observe is unavailable, governed agents cannot call models. It fails closed to prevent ungoverned traffic during an outage. Plan for this dependency before routing an agent through it.
The full availability position, including everything that is not committed: no high availability, no recovery objectives, no service level and no service credits
The available recovery approach has clear limits. This release supports exactly one active process on one node, whichever store is selected. The documented answer is platform-level restart, the full-snapshot backup and restore path, and a readiness endpoint that reports unready until migrations have applied so traffic never reaches a half-migrated process. It is not an active-active topology, there is no measured recovery-point or recovery-time objective — the planning figures that circulate are unqualified objectives, not evidence and not a property of the shipped scripts — and there is no availability service-level agreement — no uptime percentage is offered, on the stated grounds that a vendor who does not operate your deployment, your network, your disk or your restart policy could not measure one. The service-level terms are published as a template, and the ones that bind are whatever a signed agreement says; the template’s own position is that there are no service credits, because there is no availability agreement to credit against. If active-active availability or PostgreSQL-native recovery is a requirement, the documentation’s own advice is to keep production-critical workloads outside this release.
Admission fails closed at the front door too. A process-wide in-flight ceiling, 256 concurrent external requests by default, returns a typed overload error with a Retry-After when the process is full or draining, while health, readiness and metrics probes stay reachable — so a saturated gateway remains legible to whatever is watching it rather than going dark. Size that ceiling for the whole process, not per agent and not per route.
There is one place where calling home would be the unsafe choice. A tool hook that fails open on timeout can bypass policy when it cannot reach a central server. An outage, slow VPN or DNS failure can trigger that timeout. So the seat hook on a developer’s machine never asks Token Observe anything. It decides locally against a signed policy bundle it was given in advance, and it is fail-closed by construction: no bundle, a malformed or unsigned one, a signature from a key the device does not trust, a bundle issued for a different seat or a different install, one past its expiry or older than its freshness bound, or the hook’s own deadline elapsing — every one of those is a deny. There are costs. A snapshot is not live state, so a kill switch engaged after issuance does not reach the laptop until the next bundle, and the freshness bound is the whole extent of that exposure. Every bound is measured on the device’s own clock, which the governed party administers. And the seats surface ships as a preview that is not production-eligible: an install with any active seat cannot report itself technically ready.
The kill switch is global and immediate, and it is checked per request — so streaming responses already in flight run to completion. Stopping those means terminating the process. Include that distinction in your incident plan.
Inside the path, the same instinct governs the smaller decisions, which is what makes the posture consistent rather than a slogan. A tool call whose arguments do not parse inside the held-argument bound is refused rather than released ungoverned. An unbroken run of value characters longer than the streaming cap is masked whole rather than cut in half. An unpriced route target is refused before egress for a budgeted agent. An image with no configured inspector is refused rather than forwarded. In each case the failure mode chosen is the one that stops work, because the alternative failure mode is the one that lets work through and records it as governed.
One process. No network hop between the checks.
Governance decisions run in-process with no network hop between checks. Use these runtime details and measurements to plan your evaluation.
- Deployment model
- Cloud pilot by default; optional customer-hosted deployment
- Runtime
- Hosted runtime; no customer server installation for cloud onboarding
- Data store
- Cloud registry database and a separate evidence database per workspace
- Shape
- Cloud supervisor and workspace processes on a shared managed host
- Vendor telemetry
- Cloud stores workspace evidence and encrypted credentials; no-vendor-data applies only to customer hosting
- Network egress
- Cloud traffic reaches Token Observe infrastructure; configured policy and redaction run before provider egress
- Workspace configuration, registry, policies and decisions
- Billing evidence from the connections and imports you supply
- Supplied connection credentials, encrypted at rest
- Request metadata for traffic routed through the workspace
- Bounded, redacted prompt and reply text when capture is enabled
- Billing queries using the reporting permissions you supply
- Routed model payloads after the applicable policy checks
- Model-assisted search using a separately configured inference key
- Authorized evidence reads and CSV exports to your client
The cloud service processes and stores workspace evidence and supplied credentials. Supabase handles account authentication; Railway runs the cloud pilot. Cloud hosting does not keep all data inside your network.
Billing cannot recover historical conversations. Activity covers routed traffic or supported evidence supplied to the workspace. Text capture is off by default, and redaction is heuristic.
Customer hosting is an optional, separate deployment choice. Its infrastructure and data boundary differ from the cloud pilot.
Supported API paths
Use the path that matches your agent’s provider protocol.
- /v1/chat/completions
- OpenAI Chat Completions, streaming and buffered
- /v1/responses
- OpenAI Responses
- /v1/messages
- Anthropic Messages
- /v1/embeddings
- Embeddings
- /v1beta/models/{model}:generateContent
- Native Gemini, streaming on :streamGenerateContent; the same two methods are also accepted under /v1/models
- /v1/models
- Model listing, filtered to what the agent may reach; the same prefix also carries the Gemini generateContent methods
- /mcp
- MCP gateway over Streamable HTTP
- /api/*
- Control-plane REST API
- /scim/v2/*
- Bounded SCIM Users provisioning
- OpenAIChat Completions, Responses and embeddings
- AnthropicMessages, including the x-api-key dialect
- Google GeminiNative Gemini ingress
- OpenRouterWith caller-controlled route selectors refused before egress
- Amazon BedrockFor estates that will not egress to a model vendor directly
- Azure OpenAIDeployment-name routing under the same policies
- Your own endpointAny OpenAI-compatible endpoint you host
Cloud setup offers OpenAI, Anthropic, xAI and Google. Use separate billing and inference credentials.
- p50
- 71ms
- p95
- 164ms
- p99
- 223ms
- throughput
- 206 requests/second
14 August 2026. 30 seconds at concurrency 16, 6,216 requests, zero failures, on an Apple M1 Max with a mock upstream provider. Thirty seconds is not a soak, a mock upstream excludes provider latency and failover, and a fresh database does not model a partner-sized trace corpus. Treat it as evidence about the code path, not as a throughput commitment. Reproduce it with node scripts/bench.mjs --duration-seconds 30 --concurrency 16 --json.
Test your request mix, corpus size and providers in the evaluation; the laboratory benchmark does not measure them.
The deployment questions in fullWhy can the eleven steps not be reordered?
Because each position buys a specific guarantee, and moving a step spends it. Sanitisation runs before scanning so the detectors see what the model will read: an instruction written in the Unicode Tags block is invisible to a scanner that runs first and perfectly legible to the model. The trace is opened before anything can reject the request, so a refusal is still evidence rather than an absence. The on-behalf-of intersection runs after the verdict, because an already-blocked request gains nothing from a second reason, and before the approval branch, so no human is asked to approve something the intersection forbids. The proposed tool call is governed before it reaches the client rather than after. And metering runs last, on every terminal path, because it is the only step that knows what happened. The order is encoded in the evaluator and the pipeline around it, and the architecture note says it must not be reordered casually.
Does a streamed response get the same governance as a buffered one?
For redaction, response-side policy, proposed-tool governance and metering, yes — and the mechanisms differ because a stream has no moment at which the whole response exists. Outbound text passes through a hold-back buffer with a 64-character floor plus a run rule for the kinds that declare no maximum length, with a separate buffer per tool-call argument channel; tool proposals are held per index until their arguments parse and have been governed; and usage is always requested from the upstream so metering never depends on what the client asked for. Two differences are real and stated rather than hidden. A stream cannot answer 403, because the status line is spent on the first byte, so a refusal arrives as an in-band typed error frame carrying the same error code. And the response-side plan is computed before the first byte from the policies that could apply rather than from the classes that turn out to be present, which can make a placeholder irreversible where the buffered path would have left it reversible.
How much latency does the governance path add?
The only measured figure is 71.2 ms at the median, 163.8 ms at the 95th percentile and 223.4 ms at the 99th, at 206.2 requests per second and concurrency 16, from a single 30-second run against an in-process mock upstream on an M1 Max under Node 20 with a fresh database. That is a reproducible laboratory baseline and explicitly not a throughput commitment or a service-level agreement. Read it carefully in one respect above all: a mock upstream excludes provider latency, rate limits, streaming, retries and failover, so the number describes Token Observe’s inline work rather than end-user response time, which in practice is dominated by the model. Six caveats travel with it — wrong Node major version, no soak, mock upstream, a narrow request mix, a fresh database on developer hardware, and not the release container — and the source document’s closing instruction is not to turn the baseline into a concurrency limit until a partner-shaped test has passed against agreed thresholds.
What happens to my agents if Token Observe goes down?
They cannot call models, and that is the design. Token Observe is inline, this release supports exactly one active process on one node, and the answer it ships is platform-level restart plus a documented full-snapshot backup and restore path — not active-active, not point-in-time recovery, and not an availability agreement; the published service-level terms are a template offering no service credits, on the grounds that there is no availability agreement to credit against, and what binds is a signed one. When the process is full or draining it returns a typed overload error with a Retry-After while health, readiness and metrics probes stay reachable, so saturation is visible rather than silent. The one deliberate exception is the developer seat hook, which never calls home at all: every vendor’s hook fails open on timeout, so a hook that round-trips to a server would convert an outage into a silent organisation-wide bypass. It decides locally against a signed bundle and denies on anything it cannot verify, at the cost that a kill switch engaged after issuance does not reach that laptop until the next bundle.
Can an agent get round the path by calling a different endpoint?
Not by picking a different surface on the same install: authentication realms are route-scoped and not interchangeable. The model gateway takes an agent bearer token, the MCP and telemetry endpoints take an agent or a seat credential, seat bundles take a seat credential only, the control-plane API takes a human session cookie and has no admin bearer mode at all — which is what makes every control-plane action attributable to a named person — SCIM takes its own bearer token, and radar ingest takes a scoped, revocable ingest token. A credential from one realm does not authenticate in another. What no gateway can do is govern traffic that never reaches it: an agent pointed straight at a provider is invisible to this path. That is why every authentication rejection at the door is folded into an hourly roll-up the shadow-AI radar reads, recording the reason and, where the credential resolved, the key id — never the presented token and never a digest of it, because a digest of a live secret is an offline oracle against that secret.
What does the on-behalf-of intersection at step 6b change?
It masks the agent’s authority with the roles mapped from the named human’s identity-provider groups, appended as the last link of the delegation chain and evaluated by a function whose contract is that every link must allow — so it can only ever narrow what step 6 permitted, and a human whose group grants everything is a no-op rather than an escalation. Rollout is staged deliberately: off is the default and reads nothing at all, shadow resolves everything and records what it would have refused, and enforce refuses, because switching straight to enforce against an empty mapping table would deny every on-behalf-of request on the install. The limits are worth reading before you turn it on. The intersection is against the person’s groups as of their last sign-in, never live, because Token Observe holds no refresh token and requests no offline scope on purpose; a configurable maximum claim age bounds how long a capture may stand in, past which the request is refused rather than decided on stale evidence. Entra stops emitting the groups claim past roughly 200 groups and sends a directory link instead, which is not followed, so those principals capture no groups and are denied until an admin narrows the claim. Google Workspace emits no group claim at all, so the feature is unavailable there whatever is configured. And the header itself is unauthenticated: forging it can only narrow authority, but an agent can still omit it, which is why an agent record can be set to require it.
Cloud custody and optional customer-hosted controls are described separately. The 24 limits we publish, and the 9 risks a named person has to accept in writing, are on one page.
What we do not claim →Prefer to ask a person? Write to James
Start with one agent.
Tell us what your agent does. We can help you choose a non-critical evaluation workload, start policies in shadow mode and review the traces together.