---
title: "Questions: How it works"
canonical_url: "https://tokenobserve.com/faq/how-it-works"
published: 2026-09-02
last_updated: 2026-10-15
---

# Questions: How it works

The request path step by step: how an agent is onboarded, what happens inline, what streaming and failover do, what happens when something is unavailable, and what you do when a policy blocks work it should not have.

Q: How does an agent start routing through Token Observe?
A: An agent normally starts routing through Token Observe by changing its base URL and its credential. For the supported OpenAI-compatible, Anthropic and Gemini ingress, pointing OPENAI_BASE_URL or ANTHROPIC_BASE_URL at your Token Observe host is a base-URL change rather than an application refactor, and from that moment the agent has an identity, a permission set, a budget and a searchable record of each governed request. The ingress surface is explicit: /v1/chat/completions, /v1/responses, /v1/messages, /v1/embeddings, the native Gemini generateContent and streamGenerateContent paths, /v1/models filtered to what that agent may reach, and /mcp over Streamable HTTP. Whether your own SDK and version behave that way is the first thing a proof of concept settles, and it is the limit on the claim. The offline path needs no provider key at all, because a built-in mock provider answers, but it fabricates text, so it proves the plumbing and nothing else.
Cite: https://tokenobserve.com/faq/how-it-works#how-does-an-agent-start-routing-through-token-observe

Q: What happens on one governed request?
A: Every request Token Observe governs goes through the same eleven steps in a fixed order, all inside one process with no network hop between them. Authenticate the agent’s bearer token by digest lookup and timing-safe comparison. Resolve the agent, its roles, any matching kill switches and its recent spend window. Open a trace immediately, so even a blocked request is recorded, and return its id on the response. Sanitise unicode, stripping smuggled invisible characters before anything else reads the payload. Scan for personal data and prompt injection. Govern: one decision point returning allow, block or require approval plus a redaction plan, optionally intersected with the named human the call is made on behalf of. Enact: return a typed error, park an approval, or apply the redaction plan to the outbound payload. Route, honouring the agent’s data policy. Call upstream under an explicit timeout. Govern any tool call the model proposes. Meter, price and record.
Cite: https://tokenobserve.com/faq/how-it-works#what-happens-on-one-governed-request

Q: How much latency does Token Observe add?
A: The only measured figure Token Observe publishes is a laboratory baseline, and it travels with the conditions it was measured under: 206 requests per second, p50 71ms, p95 164ms and p99 223ms, over 30 seconds at concurrency 16, 6,216 requests, zero failures, on an Apple M1 Max against a mock upstream provider, on 14 August 2026. That measures the product’s own inline work rather than end-user response time. Thirty seconds is not a soak; a mock upstream excludes provider latency, streaming, retries and failover; a fresh database does not model a partner-sized trace corpus; and the run used Node 20 while the release image uses Node 24. No throughput or concurrency commitment is derived from it, because the partner-shaped sustained-load result that would justify one has not been run. The command that reproduces the figure ships in the repository, so you can measure your own.
Cite: https://tokenobserve.com/faq/how-it-works#how-much-latency-does-token-observe-add

Q: Does streaming work, and do policies still hold on a stream?
A: Yes. Token Observe governs a streamed response on the same terms as a buffered one. Outbound streams pass through a hold-back buffer with a 64-character floor and a separate buffer for each tool-call argument channel, so a card number split across two chunks cannot escape output redaction; the cut is pulled back off anything it would split, so a detected value is never half-masked and a value still arriving never has its head emitted. An unbroken run beyond 4,096 characters is replaced with one irreversible redaction marker and suppressed through its delimiter, which keeps memory bounded rather than releasing half of an ambiguous value. Response-side data-class decisions are taken before the first byte, because a stream has no later enforcement point, and a block ends the stream with an in-band ACP_POLICY_BLOCKED frame; a stream cannot answer 403, because the status line is already spent.
Cite: https://tokenobserve.com/faq/how-it-works#does-streaming-work-and-do-policies-still-hold-on-a-stream

Q: What happens to our agents if Token Observe is down?
A: If Token Observe is unavailable, the agents it governs stop calling models, by design. Governance is inline and fails closed: the evaluator returns allow only when every gate passes, and missing, unresolvable or erroring state denies the request. A control you can bypass by turning it off is not a control, which makes Token Observe’s own availability a governance property of your environment. Plan for it explicitly: run it close to the agents, watch /readyz, and decide in advance what you do if it stops, because the design-partner gate requires that emergency decision to be named and owned before traffic arrives. Two related behaviours: process admission is global and applied before the body is read, returning 503 ACP_OVERLOADED with a Retry-After when the ceiling is reached, while health, readiness and metrics probes stay reachable.
Cite: https://tokenobserve.com/faq/how-it-works#what-happens-to-our-agents-if-token-observe-is-down

Q: What happens when Token Observe blocks something it should not have?
A: You change the rule, and the next request goes through. Token Observe reads its enabled policy set from the store on every governed request, so switching a policy to shadow, narrowing its scope or disabling it takes effect on the following call rather than at the next restart or deployment, and an operator can do it without an engineer. The promotion gate that an install can require (a named person acknowledging a backtest of that exact rule before it enters enforcement) deliberately does not fire when you edit a rule that is already enforcing, because a gate that stops somebody fixing a live rule at three in the morning is a gate they turn off permanently. What the blocked caller received is built to make that possible: a typed error carrying the policy’s own reason rather than a generic refusal, with the trace id on the response, so the engineer holding the error and the operator holding the console are looking at the same record. That record is the same shape as one for a request that succeeded (which policy matched, which shadow policies would have matched, the injection score, the personal-data kinds), and every change you then make is itself audited with the mode, enablement and priority before and after it. Three things to plan for, because none of them is automatic. There is no per-request bypass and no break-glass allow: this fails closed, so between the block and the fix the work does not happen, which is why the design-partner gate asks for that emergency decision and its owner to be named before traffic arrives rather than found during an incident. A kill switch is the emergency stop, not the emergency unblock. And the way to have fewer of these is upstream: start every policy in shadow mode, where it records what it would have done against your real traffic, and send a rule back to shadow rather than deleting it while you work out why it fired.
Cite: https://tokenobserve.com/faq/how-it-works#what-happens-when-token-observe-blocks-something-it-should-not-h

Q: What happens when a model provider fails?
A: What Token Observe does next depends on the type of failure. A 429 or a timeout fails over to the next provider in the chain; a content-policy refusal, an auth failure, an invalid request or a context-length error does not; otherwise the fallback chain quietly launders a refusal into a success, and the trace records a completed action that the first provider declined to perform. Retries with exponential backoff and jitter apply to idempotent failures only, behind a per-provider circuit breaker. The agent’s data policy (zero data retention, no training on payloads, serving region) is enforced on the fallback chain as well as the primary route, and when no route satisfies it the request fails closed with ACP_ZDR_UNAVAILABLE rather than downgrading silently. The limit beside that: those provider data-policy flags are operator assertions, recorded and audited but not verified.
Cite: https://tokenobserve.com/faq/how-it-works#what-happens-when-a-model-provider-fails

Q: How are tools and MCP servers governed?
A: Token Observe governs tools at two enforcement points, because agents reach tools two ways. Through the MCP gateway, a tools/call is re-checked against the agent’s grants independently of tools/list, and each tool descriptor is pinned by a hash over its canonicalised name, description and input schema, re-verified at a 60-second catalogue TTL; drift quarantines the tool and emits an event, though a change may be served from the trusted snapshot for up to that minute. Tool results are treated as the higher-risk channel and injection findings in them are weighted more heavily than findings in prompts, because indirect injection arrives in results rather than in the user’s message. Separately, when a model proposes a tool call on the request path, that proposal is evaluated against tool-call policies before it is returned. The stated limit: Token Observe can only refuse a proposal it is shown. An agent that never routes tools through it is the radar’s problem.
Cite: https://tokenobserve.com/faq/how-it-works#how-are-tools-and-mcp-servers-governed