Platform engineering
One base URL changes. The fail-closed trade is the thing to decide before you change it.
On this page
What is already on you
- 01
The agents are already in production and you did not write them
Governance that requires every team to refactor is governance that does not happen. What you can realistically change is one environment variable in someone else’s service, which is why the ingress dialects and the error shapes matter more than the console does. - 02
You are the one paged when a control blocks real work
A rule promoted straight to enforce is an outage with a compliance justification. Shadow mode exists so the first day produces findings rather than incidents, and an optional gate can refuse the transition into enforcement until a backtest of that exact rule digest has been run and acknowledged by a named person. - 03
Spend is a reliability problem before it is a finance problem
Hard per-agent circuit breakers by request, hour, day and month are the reason a looping agent stops at a ceiling rather than at an invoice. The sharp edge is metering: a model with no price row meters at zero, so where an agent carries any budget field, an unpriced target is refused with a typed error before provider egress instead of running free — an unmetered estate and an idle one otherwise look identical on every spend surface. - 04
Somebody has to own the answer to what happens when it stops
The design-partner gate requires named incident contacts, a severity path, an upgrade window, a backup owner, restore authority, secret rotation, disk alerting and an explicit emergency decision about the fail-closed gateway being unavailable. That is a runbook you write before the pilot, not after it.
The objections, in the words they are raised in
Not a marketing FAQ. These are the sentences this reader says out loud in a first call, quoted rather than softened, and several of the answers below concede that the limitation is real and name the person who has to accept it.
- 01· asked
If this is down, my agents stop. Why would I put it in the path?
answeredBecause a control you can bypass by turning it off is not a control — and yes, that means Token Observe’s availability becomes a governance property of your environment rather than a vendor’s. There is no fail-open setting to reach for during an incident, deliberately. What replaces it is a decision made in advance: run it close to the agents, watch the readiness endpoint, and write down what you do if it stops, because the design-partner gate does not close until a named person owns that answer. Note also that the escape hatch is real but visible — pointing a base URL back at the provider produces ungoverned traffic that the shadow-AI radar later reconciles from your vendor bill and egress logs.
- 02· asked
What does it add to request latency?
answeredThere is one published number and it is a laboratory baseline, not a capacity commitment: 206.2 requests per second at concurrency 16 over 30.1 seconds, p50 71.2 ms, p95 163.8 ms, p99 223.4 ms, measured from an immutable commit against a mock upstream on a single MacBook Pro on Node 20, while the release image and CI use Node 24. It measures the inline governance work and excludes provider latency, streaming, retries, failover, tool calls, approvals and a partner-sized database, and thirty seconds is not a soak. No throughput or latency commitment is offered on the back of it, and none should be accepted until a partner-shaped sustained-load result exists. The honest answer for your estate is that gateway latency overhead is a row in the design-partner success review, with a target you set and a figure measured on your traffic — and that the underlying constraint, a single writer on a single host whose governance latency degrades under sustained write load, is named in the threat model as a residual risk the engineering lead has to accept in writing.
- 03· asked
Will it break my agents?
answeredA block returns in the shape your client already parses. On OpenAI-compatible ingress that is the OpenAI error envelope with a typed code such as ACP_POLICY_BLOCKED, and where the policy asked for a human rather than a refusal it carries the trace id, the approval id and the header to resume with — single-use, and bound to the same payload. Anthropic and Gemini get their native error shapes with the same machine-readable detail. Beyond that: shadow mode first, and typed failover so a 429 or timeout fails over while a content-policy refusal does not. The limit worth knowing before you migrate is that image, audio, PDF and opaque file inputs are rejected before provider egress across all dialects rather than passed through ungoverned.
- 04· asked
Can we roll back a release?
answeredUsually, with one condition that removes the ordinary path. If an effect-required tool pin has ever been activated, a plain N-1 rollback is not an available recovery step: it needs drained traffic, every affected pin quarantined through an effect-aware binary, every replica stopped, and the pins kept quarantined until an effect-aware fleet returns. Rolling back two releases is not tested. The reason is the point of the feature — an older binary that does not understand a durable effect run would either dispatch the action twice or lose it, and both are worse outcomes than a slower rollback. Put that in the upgrade window before you enable effect contracts, not after.
If the objection that would actually decide this for you is not one of the ones above, that is the more useful question, and it is the one worth sending.
Talk it throughThe first hour, honestly timed
About twenty minutes of the first hour are mechanical and bounded by npm, the TypeScript build and SQLite: install, configure, boot, one quickstart call, mint a key, redirect one base URL, watch the first trace appear. The two steps that take the rest are judgements rather than commands — deciding who owns this agent and what it is for, and deciding which rule you are willing to have block production traffic at three in the morning. That is why the published answer is an hour rather than four minutes.
You do not have to take the claim on faith. The install reports its own local configuration as eight items, each computed from live state rather than from a setup-finished flag, so it goes red again the day someone revokes the last agent key or rotates a provider credential out of the environment. The same response carries a deliberate boundary: technically ready is not a production certification, because the endpoint cannot evaluate release provenance, an independent penetration test, your exact integration matrix, workload-shaped soak and restore evidence, legal approval, or a production reference.
The offline demo runs against a built-in mock provider and needs no API keys, which makes it a good way to prove the plumbing to a colleague in fifteen minutes. It is also the one thing to switch off deliberately before production, because without a live provider the mock answers fabricated text and nothing downstream can tell.
What sits in the request path, and the order it refuses in
The kill switch is evaluated first and beats everything else — scoped to one agent, a team or the whole estate, and requiring an attributable actor and a reason, because it is the literal implementation of a stop capability somebody will later have to evidence. After that the gateway resolves the agent record itself rather than an exported copy, which is why an edit or a suspension applies on that agent’s next governed request with no propagation step and no redeploy.
The general rule is that missing, unresolvable or erroring state denies rather than permits. Policy failures fail closed. Data-policy requirements — zero retention, no training on payloads, a serving region — are enforced on the fallback chain as well as the primary route, and a request that can find no route satisfying them is refused with a typed error rather than quietly downgraded to a provider that does not meet them. An intrinsic audit-integrity failure latches the process unready and refuses governed writes, and the incident row that records it is monotonic — a restart does not clear it and no endpoint can, so recovery is a restore from a trusted pre-incident database or a documented offline maintenance ceremony.
Two operational surfaces are worth wiring into your existing tooling on day one. Webhook events carry identifiers, counts, policy names and a one-line summary into your ITSM or SIEM, and a webhook can never delay or fail the governed request that produced it. Prometheus metrics use only the matched route pattern as a label, so a caller cannot write arbitrary text into your scrape output and cardinality is bounded by construction — and the endpoint is unauthenticated by convention while exposing agent identifiers and month-to-date spend, so it belongs on the monitoring network and nowhere else.
- Shadow first, then a backtest gate
- Every policy can run in shadow mode so you learn your false-positive rate before blocking. An optional setting refuses the transition into enforcement until a backtest of that exact rule digest has been acknowledged by a named person, whose name and accepted figures are copied into the audit entry. It is off by default and it is a process control, not a technical one.
- Typed failover
- A 429 or a timeout fails over. An authentication error, an invalid request, a context-length error and a content-policy refusal do not — otherwise the fallback chain quietly launders a refusal into a success.
- Natural-language trace search
- Translated into a validated filter object, never into SQL, shown back as editable chips, with the signed-in user’s team predicate applied after translation where the model cannot supply or remove it. It degrades to a deterministic keyword parser whenever the translation does not come back — no model configured, a timeout, an unreachable provider, or output that fails validation — so a failed translator narrows the search rather than breaking it.
Deployment modes, proxy requirements and the traps worth knowing first
Five modes are documented: single node on a mounted volume, the same image inside your VPC, air-gapped or on-premise, a platform-as-a-service container, and Kubernetes with GitOps and Terraform. Nothing about the artifact changes between them — only where it runs and which egress it is allowed.
The reverse proxy has five requirements and they are the usual source of a bad first day. It must terminate TLS, because the process speaks plain HTTP and manages no certificates. It must not buffer responses, or streaming breaks. It must preserve the authorisation and API-key headers. It must set the forwarded-for and forwarded-proto headers, and be the only thing that can. And its read timeout must exceed your longest expected model response. Behind a platform-as-a-service edge there are three defects configuration cannot fix, including an unauthenticated metrics endpoint on a public hostname and platform volume snapshots that bypass the supported restore path.
The trap that costs an afternoon is the readiness probe. The first-key seal boot and every rotation seal boot are designed to stay unready for their entire life, so a platform that gates deploys on readiness will mark those ceremony deploys failed and roll them back. Switch the healthcheck to liveness for the ceremony and switch it back afterwards.
Finally, back the database up like evidence rather than like state. There is no vendor-side copy — the same property that removes the processor relationship — so if the file is lost and there is no backup, the traces and the audit chain are gone. Recovery is from a retained full snapshot; there is no point-in-time recovery claim, and the planning objectives that exist are objectives rather than vendor commitments. After any restore, verify the chain before anything else.
The parts of the platform this leans on
In the order they matter to this reader rather than the order the platform argues them, and each one carries its own stated limit on the page it links to.
Policy engine
One deterministic verdict on every governed request: allow, block, redact, or park it for a human.
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Agent registry
One record per agent, and it is the record the gateway enforces against.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
How many agents and which shape of workload is a pilot scoped to?
One self-hosted deployment in your own VPC, roughly five to fifty API-key agents owned by a single platform team, a small named console population using your own single sign-on, one or two model providers and a bounded set of pinned tools. Shadow policies first, enforcement only after an acknowledged backtest, and a non-production-critical evaluation before any workload whose outage would harm customers or regulated operations. That boundary is written into the readiness record rather than negotiated later, because the honest scope of what has been proven is smaller than the scope of what has been built.
Is there an uptime SLA?
No, and the reasons are given in order of weight rather than hidden. The vendor does not operate your deployment, cannot observe it and cannot restart it, so an uptime number from that party would be unmeasurable by either side. No partner-shaped sustained-load result has been retained yet, and publishing a capacity figure before that would be a guess wearing a number. Recovery automation exists but proven recovery-point and recovery-time evidence does not. There are no service credits, because there is no availability commitment to credit against. Asking for the date on which one becomes possible is a fair thing to put in an agreement.
SQLite in production?
It is a single-writer process on one host by design at this scale, and the engineering lead is named as the role that has to accept that: sustained write load degrades governance latency and there is no replica. The rationale offered is that the swap remains mechanical behind the store ports, and a PostgreSQL backend exists — but it is evaluation-only in the current release and always leaves the technical launch gate red. If your pilot’s write profile is heavier than a bounded agent fleet, that is a sizing conversation to have before the pilot rather than a surprise to find inside it.
What does it do to streaming responses?
Streaming is supported and redaction covers it, which is the part worth checking against your proxy. A hold-back buffer runs on the response stream with a separate buffer for each tool-call argument channel, and the response-side data-class decision is taken before the first byte because a stream has no later enforcement point. That means your reverse proxy must not buffer responses — the product also sends the header that asks nginx not to — and your read timeout has to exceed the longest model response you expect rather than the default.
Can we keep using our existing gateway, tracing and identity stack?
Yes, and that is the intended shape. Provider routing, retries, caching, quotas and cost dashboards are explicitly table stakes rather than the lead story, so Token Observe integrates above or beside a gateway rather than competing on connectivity. Telemetry arrives over OTLP as bounded JSON or protobuf and becomes recorded flight-recorder evidence rather than replacing your tracing product. Identity comes from your own OIDC provider, and no group claim from it can promote anyone in the console — directory-mapped roles are a permission mask on the agent path alone.
Bring the objection that is not on this page.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in