How it works, answered in full.

An agent normally starts routing through Token Observe by changing one environment variable. For the supported OpenAI-compatible, Anthropic and Gemini ingress, pointing OPENAI_BASE_URL or ANTHROPIC_BASE_URL at your Token Observe host is a base-URL change rather than an application refactor, and from that moment the agent has an identity, a permission set, a budget and a searchable record of each governed request. The ingress surface is explicit rather than implied: /v1/chat/completions, /v1/responses, /v1/messages, /v1/embeddings, the native Gemini generateContent and streamGenerateContent paths, /v1/models filtered to what that agent may reach, and /mcp over Streamable HTTP. Whether your own SDK and version behave that way is the first thing a proof of concept settles, and it is the honest limit on the claim. The offline path needs no provider key at all, because a built-in mock provider answers — but it fabricates text, so it proves the plumbing and nothing else.

Every request Token Observe governs goes through the same eleven steps in a fixed order, all inside one process with no network hop between them. Authenticate the agent’s bearer token by digest lookup and timing-safe comparison. Resolve the agent, its roles, any matching kill switches and its recent spend window. Open a trace immediately, so even a blocked request is recorded, and return its id on the response. Sanitise unicode, stripping smuggled invisible characters before anything else reads the payload. Scan for personal data and prompt injection. Govern — one decision point returning allow, block or require approval plus a redaction plan, optionally intersected with the named human the call is made on behalf of. Enact: return a typed error, park an approval, or apply the redaction plan to the outbound payload. Route, honouring the agent’s data policy. Call upstream under an explicit timeout. Govern any tool call the model proposes. Meter, price and record.

The only measured figure Token Observe publishes is a laboratory baseline, and it travels with the conditions it was measured under: 206 requests per second, p50 71ms, p95 164ms and p99 223ms, over 30 seconds at concurrency 16, 6,216 requests, zero failures, on an Apple M1 Max against a mock upstream provider, on 14 August 2026. That measures the product’s own inline work rather than end-user response time. Thirty seconds is not a soak; a mock upstream excludes provider latency, streaming, retries and failover; a fresh database does not model a partner-sized trace corpus; and the run used Node 20 while the release image uses Node 24. No throughput or concurrency commitment is derived from it, because the partner-shaped sustained-load result that would justify one has not been run. The command that reproduces the figure ships in the repository, so you can measure your own.

Yes — Token Observe governs a streamed response on the same terms as a buffered one, and the mechanism is worth knowing, because streaming is where gateways usually lose their guarantees quietly. Outbound streams pass through a hold-back buffer with a 64-character floor and a separate buffer for each tool-call argument channel, so a card number split across two chunks cannot escape output redaction; the cut is pulled back off anything it would split, so a detected value is never half-masked and a value still arriving never has its head emitted. An unbroken run beyond 4,096 characters is replaced with one irreversible redaction marker and suppressed through its delimiter, which keeps memory bounded rather than releasing half of an ambiguous value. Response-side data-class decisions are taken before the first byte, because a stream has no later enforcement point, and a block ends the stream with an in-band ACP_POLICY_BLOCKED frame — a stream cannot answer 403, because the status line is already spent.

If Token Observe is unavailable, the agents it governs stop calling models, and that is the design rather than a defect. Governance is inline and fails closed: the evaluator returns allow only when every gate passes, and missing, unresolvable or erroring state denies the request. A control you can bypass by turning it off is not a control — which makes Token Observe’s own availability a governance property of your environment rather than somebody else’s problem. Plan for it explicitly: run it close to the agents, watch /readyz, and decide in advance what you do if it stops, because the design-partner gate requires that emergency decision to be named and owned before traffic arrives. Two related behaviours: process admission is global and applied before the body is read, returning 503 ACP_OVERLOADED with a Retry-After when the ceiling is reached, while health, readiness and metrics probes stay reachable.

What Token Observe does next depends on the type of failure, and that distinction is the feature. A 429 or a timeout fails over to the next provider in the chain; a content-policy refusal, an auth failure, an invalid request or a context-length error does not — otherwise the fallback chain quietly launders a refusal into a success, and the trace records a completed action that the first provider declined to perform. Retries with exponential backoff and jitter apply to idempotent failures only, behind a per-provider circuit breaker. The agent’s data policy — zero data retention, no training on payloads, serving region — is enforced on the fallback chain as well as the primary route, and when no route satisfies it the request fails closed with ACP_ZDR_UNAVAILABLE rather than downgrading silently. The limit beside that: those provider data-policy flags are operator assertions, recorded and audited but not verified.