Observability and governance
An agent that routes through neither, and the model’s hidden reasoning
every questionOn this page
The two questions, and why one tool rarely answers both well
Observability grew out of debugging distributed systems and inherits that shape: it is optimised for reconstructing what a system did across many hops, attaching scores and latencies, and letting an engineer find the one bad case among thousands. Its unit is the span, its consumer is a developer, and its bias is towards completeness — keep everything, sample if you must, make it searchable.
Governance grew out of access control and audit and inherits that shape: it is optimised for taking a decision before an action, refusing when the answer is no, and producing a record whose integrity survives contact with the people who run the system. Its unit is the decision, its consumer is a reviewer, and its bias is towards refusal and attribution — record the attempt, name the person, keep the chain intact.
Those biases pull in opposite directions on almost every design question. Observability wants the prompt and the answer stored so an engineer can read them; governance wants the smallest bounded excerpt that still evidences the decision, taken after redaction, because a complete copy of every prompt your organisation has ever sent is itself a liability. Observability wants sampling under load; governance cannot sample, because the request you dropped is the one that will be asked about. Observability wants the agent to emit its own spans; governance cannot accept that as evidence, for the same reason a defendant’s own account of events is not the record.
That is why a product built for one and extended into the other tends to be shallow at the extension. The useful question when evaluating a tool is not which category it is in but which bias it was built with, because the bias shows up exactly where it hurts.
Why a trace tree is weak evidence and a refusal is strong
Instrumentation lives inside the thing being instrumented. A framework callback that records a tool call records it because the framework chose to; a span is missing when the code path that emits it was not taken; and the whole record can be turned off, sampled down or redeployed by the same team whose behaviour is under review. That is not an accusation, it is a structural property, and it is why an auditor treats an application log differently from an access log.
A governance record is produced by the component that made the decision, and the important consequence is that it includes the things that did not happen. Token Observe opens the trace at step 3 of its request path — before sanitisation, before the scanners and before the verdict — so a request refused a millisecond later is a row rather than an absence, and the identifier comes back on a response header even on the refusal. An agent working through tool names it was never granted is visible afterwards precisely because the refusals were recorded; the same behaviour is invisible in a system that only records what succeeded.
The second property is that a governance record covers the decision as well as the outcome. Every rule that matched writes a decision event naming the policy, its mode, its action and why it matched — including a rule running in observation mode, which is what lets a reader distinguish a control that was quiet from a control that was off. That distinction is the whole reason the record exists: an estate that blocked nothing and an estate whose rules were all in shadow emit identical outcomes.
Then there is what the governance record deliberately does not hold, which should be stated to an auditor rather than discovered by one. Token Observe stores the prompt as a bounded post-redaction excerpt and does not store the model’s answer text, so a trace evidences what was decided, what was routed where and what it cost, rather than being a transcript you could replay. It never sees hidden model reasoning at all. If your reason for wanting the prompt and the answer is quality rather than evidence, that is an observability requirement and it should be met by an observability tool.
What observability does that no governance layer will do for you
Quality is the obvious one and it is not a small exception. A governance layer can tell you a request was allowed, routed to a particular model and cost a particular amount. It cannot tell you the answer was wrong, that a retrieval step returned the wrong document, that a prompt change regressed a class of question, or that the agent took nine turns to do what it used to do in three. Those are the questions that decide whether an agent is worth running, and they need traces with content, evaluation scoring and experiment tooling.
Debugging across the whole agent, rather than at the boundary, is the second. A gateway sees requests and responses at one point. It does not see the loop the framework ran, the retry the SDK swallowed, the memory the agent wrote and read back, or the code path that chose one tool over another. When an agent misbehaves for a reason that is not a governance decision, the gateway record narrows the problem and the application trace solves it.
The third is the one people forget: latency and cost attribution as an engineering discipline rather than a control. Knowing which prompt template is responsible for two-thirds of the token spend is an optimisation question, not a governance one, and the answer usually lives in the shape of the traffic rather than in the ledger.
Where the two genuinely meet is telemetry ingestion, and it is worth being precise about what ingestion is and is not. Token Observe accepts telemetry over the OpenTelemetry protocol as a way of getting agent-side and endpoint-side activity into the same flight recorder, attributes every write from the credential that sent it rather than from the resource attributes in the payload, and marks what arrives as recorded rather than inline-enforced. That distinction is the entire point: telemetry that arrives after the fact is a description, and describing it as governed would mean claiming a control over something that has already happened.
Where to wire them together, and where to keep them apart
Share the inventory. One record per agent with a named human owner and a declared purpose, resolved by the enforcement point on every call, and referenced by the observability tool rather than duplicated in it. Two inventories diverge from the first week, and the one that is not enforced against is the one that will be wrong when it matters.
Share the identifiers. A trace identifier returned on every gateway response — including refusals — is what lets an engineer pivot from an application trace to the governance record and back. Without it, correlating the two is a timestamp-and-hope exercise.
Push governance events into what the team already reads rather than into a second console. Token Observe publishes typed events — a policy block, a flagged match, a budget warning or breach, an agent suspension, a kill switch engaged or released, a discovery finding, a tool descriptor drifting from its approved hash — with identifiers, counts, policy names and a one-line summary, signed when a secret is configured, and delivered best-effort on a later tick so a receiver being down cannot delay a governed request. A dashboard nobody has alerting on is not monitoring.
Keep the evidence chain apart. The administrative record — who created an agent, who widened a role, who promoted a policy to enforcing, who approved what, who engaged the kill switch — belongs in a hash-chained log with its own integrity story, not in the same store as application telemetry, because its threat model includes the people who administer that store. Token Observe keeps it in a separate chained table that a trace purge never touches, which is what lets the record of a deletion outlive the deleted data.
And keep the retrospective analysis honest about what it read. Token Observe’s policy backtesting replays a candidate rule against recorded traffic using only the decision inputs the pipeline already writes onto each call — the requested model, the estimated input tokens, the estimated cost, the personal-data kinds, the injection score and the heuristic names — and never the prompt text. That is a deliberate limit on how much of the observability corpus the governance layer is allowed to reach into, and it is the right default.
- One inventory, read by both
- The record the gateway resolves on every call is the one the observability tool should reference. A second list is a list that is wrong.
- One identifier, on every response
- Returned on refusals too, so a blocked request can be found from the application side rather than existing only as a gap.
- Events out, not a second console
- Blocks, budget breaches, suspensions, kill switches, discovery findings and descriptor drift into your service-management or security tooling.
- Evidence in its own chain
- Administrative changes in a hash-chained log a payload purge never touches, because its threat model includes whoever administers the telemetry store.
How to tell which one a tool actually is
Four questions separate them quickly, and none of them is about the feature list.
Ask what happens when a rule fires. If the answer is that the request is annotated, scored or flagged, you are looking at observability with a policy vocabulary. If the answer is that the request is refused, parked on a named human, or has its payload rewritten before egress, you are looking at a control. Then ask the follow-up that matters: can a rule be staged in observation mode first, with its would-be decision recorded, so the false-positive rate is learned before somebody’s work stops.
Ask who produces the record. If the record comes from an SDK the application imports, it is a description. If it comes from a component in the path that the application cannot decline, it is evidence. A hybrid is fine, and common, but the two halves have different standing and should be presented that way rather than merged into one timeline that implies uniform provenance.
Ask what the empty state means. A findings list with nothing in it, a spend figure of zero, an estate with no policy blocks — each of those has two causes, and only one of them is good news. A tool that cannot distinguish a dead evidence feed from a clean estate has a failure mode identical to being switched off, and the failure is silent. Token Observe treats that as the cardinal rule of its discovery surface and reports per-source coverage beside the findings, separating a connector that is alive from a connector that is delivering rows.
Ask what survives someone with database access. For observability the honest answer is usually nothing, and that is acceptable — quality data does not need to resist an insider. For governance it is the whole question, and the honest answers are hash chaining, keyed digests and an off-box signed anchor, in increasing order of what they cost and what they buy.
How to put observability and governance into practice
- 01
Write down the question each tool is for
One sentence each: what is this for when an engineer is debugging, and what is this for when a reviewer is asking whether a control held. If the same tool is named twice, check it can refuse. - 02
Pick one inventory and make it the enforced one
The record the gateway resolves on every call. Everything else references it rather than maintaining a parallel list that drifts from what is running. - 03
Return one identifier everywhere, including on refusals
So an application trace and a governance decision can be pivoted between. Without it the correlation is a timestamp guess and refusals are invisible from the application side. - 04
Route governance events into what the team already reads
Blocks, budget breaches, suspensions, kill switches, discovery findings and descriptor drift into your incident tooling, with alerting attached. A second console gets read for a fortnight. - 05
Keep the administrative record in its own chain
Hash-chained, ideally keyed and anchored off the box, in a store a payload purge never touches, so the record that a deletion happened outlives the deleted data. - 06
State what the governance record does not hold
No answer text, a bounded post-redaction prompt excerpt, no hidden model reasoning. Tell the auditor before they find out, and meet the quality need with the observability tool instead.
Where this argument meets an implementation
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
Policy engine
One deterministic verdict on every governed request: allow, block, redact, or park it for a human.
Audit chain
Every administrative act hash-chained; seal it under a key held off the box, and anchor it with a signature your auditor can check alone.
Shadow AI radar
Five evidence sources for AI activity that never touched the gateway, and a coverage model that refuses to call a dead feed a clean estate.
Those pages are one product's implementation of what this guide argues for; describe what your agents actually do and you will get a straight answer about whether you need any of it, including when the answer is no.
Talk it throughCan an observability platform be used as governance evidence?
For the descriptive part, sometimes; for the decisive part, no. The record is produced by the system being described and can be sampled, disabled or redeployed by the team under review, and it records what happened rather than what was refused. The two properties an auditor tests are whether the refusals are in the record and whether the record resists someone with write access to its store, and an instrumentation-based trace usually fails both. Use it for quality, debugging and cost attribution, and produce the decision record from the enforcement point.
Is a policy engine that only warns worth having?
Yes, as a stage, and no, as a destination. Every rule that reaches production has to answer whose work does this stop, and the only way to answer that is to run it in observation mode over real traffic first — evaluated exactly as an enforcing rule, its match recorded on the trace with the policy, its action and why it matched, then skipped so the request proceeds. What makes that a stage rather than a permanent state is a promotion that is a separate, attributable act. A rulebook that is entirely in shadow six months later is a reporting product with a policy vocabulary.
Do we need both if the gateway already records traces?
You need the quality half from somewhere, and a gateway is a poor place to get it. A gateway sees the boundary: the request, the decision, the routing, the cost and the response metadata. It does not see the loop the framework ran, the retrieval that returned the wrong document, the retry the SDK swallowed, or whether the answer was any good — and Token Observe deliberately does not store answer text at all. If your agents are in development or their output quality is contested, that is an observability requirement and it will not be met by a control.
How do the two share data without duplicating the inventory?
Make the enforced record the authoritative one and reference it. One agent record with a named owner, a team, a declared purpose and a lifecycle status, resolved by the gateway on every call so it cannot drift from what is running, with the observability tool joining on that identifier. Then carry one trace identifier across both, returned on every gateway response including refusals. Everything else — events, dashboards, alerting — is a push from the control into the tools your team already watches rather than a second place to look.
What does ingesting agent telemetry into a governance tool actually buy?
One searchable record, and nothing more than that, which is why the labelling matters. Token Observe accepts logs and spans over the OpenTelemetry protocol, attributes every write from the authenticated credential that sent it rather than from resource attributes a sender can set, applies irreversible redaction on ingest, and marks what arrives as recorded rather than inline-enforced. Metrics are accepted and not stored, with every data point returned as rejected so nobody assumes otherwise. The value is that endpoint and framework activity sits beside gateway decisions in one place; the limit is that arriving after the fact means it was never a decision anybody could have taken.
Prefer to ask a person? Write to us →
Bring us the question this guide did not answer.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in