Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
On this page
Evidence nobody can question is not evidence
The record exists and the question cannot be asked. An agent estate writes its history into application logs, provider dashboards and whatever the framework happened to print, and the question a compliance officer actually arrives with — did any agent move more than £500 without a human looking at it, in the last quarter — is not answerable from any of them without a join across four systems and somebody who can write SQL. Evidence that is present and unusable is, in an audit, the same as evidence that is absent.
The obvious fix is the one that cannot ship. Point a model at the trace database and let it write the query, and you have built exactly the pattern the product’s own engineering handbook forbids: an interpreter input generated from untrusted text. Trace content is attacker-influenced by construction — the events table holds prompts, tool arguments and tool results, some of them written by an external party who wanted them read — so a read-only database user and a SQL parser in front of the query narrow the blast radius without closing it. What they leave is cross-agent and cross-team disclosure driven by a prompt-injection payload already sitting in the corpus being searched.
The other obvious fix is worse product. Structured filters and no natural language has no injection surface at all, and it obliges every compliance officer to learn a query syntax in order to ask the question the product exists to answer. That option was rejected on product grounds rather than security grounds, and the third one was taken: the model’s only output is a filter object, the server validates it against a closed field set, and the query builder emits parameterised SQL from known keys and typed values.
Reading the evidence is itself an act, which is the part most audit trails miss. Pulling up one named person’s prompt history is a privileged read of a personal-data store the customer did not have before they deployed agents, so the read is recorded with the actor, the filter and the result count. That is not decoration: a buying-committee review found the erasure preview — the query that counts one subject’s traces before anything is deleted — going unaudited, and it is audited now for the same reason the deletion is.
How it actually works
Steps 3 and 11 of the request path. The order is load-bearing rather than incidental: it is encoded in the evaluator, and the reason each step sits where it does is the reason the guarantee holds.
- 01
The trace opens before anything is decided
A trace id is minted at step 3 of the eleven-step request path, before unicode sanitisation, before the personal-data and injection scanners and before the policy verdict, so a request blocked a millisecond later is recorded rather than missing. The id comes back on a response header on every request, refusals included, so a caller can quote the trace for a request that never reached a provider. - 02
Each stage appends its own event
Governance, the provider call and the metering pass all append to the same trace, so sequence numbers are assigned by the store inside the transaction that does the insert rather than by the caller. A caller-supplied sequence would collide under concurrency and the uniqueness constraint would fail the governed request it was meant to be recording. - 03
Search columns are derived as events land
Tool names, detected personal-data kinds, whether a policy matched, whether a human approval was involved and the largest amount seen on a tool call are merged onto the trace row as each event is appended, so the common filters never walk a payload at query time. The walk that derives them is bounded to 2,000 nodes and twelve levels of nesting, because it runs on the governed hot path. - 04
A question becomes a filter object
The translation model’s only permitted output is minified JSON matching the trace filter schema. The server validates it against fourteen allow-listed keys before it reaches the query builder, and an unknown key is a hard failure rather than something to drop, because it means the model produced a shape this code was not written against. Strings are capped at 500 characters, arrays at 50 items, statuses and personal-data kinds at their closed sets. - 05
Paging stays the server’s decision
The page size and offset are deliberately absent from the allow-list, so a translated filter cannot ask for a larger page than the endpoint permits. The team boundary the server applies is stripped from the filter echoed back to the browser for the matching reason: it is an authorisation decision, never accepted from a client and never presented as one of the operator’s own chips. - 06
The interpretation is shown back, editable
The response carries the filter, a plain-English explanation naming every field that was set, and whether a model or the keyword parser produced it. The console renders each field as a chip you can edit in place or remove, and writes the chips into the URL, so an interpreted filter is a link you can hand to somebody else. - 07
Reads and exports are recorded
Listing traces, searching them, opening one and exporting one each append an audit entry naming the actor and the thing acted on. The list and search entries carry the interpreted filter, the teams the account was effectively authorised for and how many rows came back; the search entry adds whether a model or the keyword parser produced that filter. The read and export entries name the trace and its agent with the counts, and the export carries the digest of the bundle it issued. An export bundles the traces, their events, the approvals that gated them, the audit entries that account for them and a full chain verification, then seals the result with a SHA-256 digest over its canonical JSON.
Natural language becomes a filter object, never SQL
The model never emits SQL, table names, column names or operators. Its entire output is a JSON object matching the trace filter type — every field a known key with a typed value — and the server validates that object against a closed field set before it reaches a query builder that emits parameterised SQL only. This is a written decision with its alternatives recorded beside it rather than something you would have to infer from the code, and the reason it went this way is a residual risk rather than a preference.
That residual is what makes the decision specific. Trace content is attacker-influenced by construction: the events table holds prompts, tool arguments and tool results, some of them authored by an external party. A read-only database user and a SQL parser narrow what a generated query can do; what they leave is cross-agent and cross-team disclosure driven by a payload already sitting in the corpus being searched. A filter object closes that by construction, because there is no interpreter for a payload to reach.
The translation call itself runs through Token Observe’s own gateway rather than a hardcoded provider client, so it inherits the same route rules, failover chain, circuit breakers and retry policy as governed traffic. It runs at temperature zero with a 512-token output cap, a 500-character limit on the question and a hard fifteen-second ceiling independent of the provider’s own timeout. It deliberately opens no trace, because a compliance officer’s search must not add rows to the corpus they are searching, and the question text is never written to the logs, because it is an operator’s question about production data and may quote the personal data being asked about.
What this buys is bounded, and the cost is named in the same place the benefit is. Misinterpretation replaces injection as the main failure mode: the model will confidently return a filter meaning something slightly different from the question, and the explanation shown beside the results is the only mitigation and is advisory. The expressiveness ceiling is low and deliberate, so a new question shape needs a schema field, a validator change and a query-builder change. That is a code change rather than a prompt change, and it is the trade the product would rather make than ship a query engine that can be talked into things.
- The closed field set
- Agent ids, agent name, team, statuses, tool name, free text, minimum amount, minimum cost, since, until, whether a human approved, whether a policy matched, personal-data kinds and the human principal. Fourteen keys; an unknown one fails the whole translation.
- Bounded values, not just bounded keys
- Strings 500 characters, arrays 50 items, statuses and personal-data kinds checked against their closed unions, timestamps parsed rather than trusted. A value failing any of these is a rejection, not a silent drop.
- Free text is still not free
- The full-text term is split into bounded terms — quoted phrases on the SQLite backend, normalised tokens bound as parameters to a phrase query on Postgres — capped at 24 terms of 64 characters each, so operator text cannot become match syntax on either supported database.
- The interpretation is legible
- The explanation string describes every field the filter set — agent name contains, tool amount at least, with a human approval, containing credit_card — because misinterpretation is the named failure mode and this string is the only thing standing between it and a wrong conclusion.
POST /api/traces/search
{ "query": "refunds over £200 approved by a human last week" }
-- the model's entire output, validated before it is used --
{ "minAmount": 200,
"hasApproval": true,
"since": "2026-08-25T00:00:00.000Z",
"textQuery": "refunds" }
-- rejected, and the keyword parser answers instead --
any key outside the fourteen a string over 500 chars
an array over 50 items a status outside the set
a timestamp that will not parse page size or offsetThe search still works when no model does
With no translation model configured, the search runs anyway. A deterministic keyword parser extracts the same filter fields — relative time windows and the two day boundaries it recognises, monetary comparisons, approval and policy markers, personal-data kinds, tool names, agent names and the human principal — and everything it does not recognise becomes full-text terms. The response says which path answered and, where a model did, which model, so an operator can tell a model interpretation from a keyword one without asking anybody.
The fallback is mandatory rather than optional, because search quality would otherwise depend on an upstream provider being reachable. Any failure takes that route: no model configured, a timeout, an unreachable provider, every provider in the chain sitting behind an open circuit breaker, output that is not JSON, or output that passes as JSON and fails validation. The search degrades; it does not fail.
It is also materially worse, and it degrades quietly, which is the sentence a buyer should hold onto. Phrasing the parser does not recognise falls through to an over-broad full-text term rather than raising an error, so the failure mode returns too much rather than nothing — the safer of the two, and still a failure. Two of its rules exist because the alternative was worse in a way somebody found out the hard way.
The first is that it refuses to guess an agent name. A bare word after “agent” is usually a verb — an agent accessed, an agent sent — so a name is taken only from a quoted string or from a token shaped like an identifier, one carrying a hyphen, an underscore or a digit. Guessing wrong is worse than not guessing, because it silently filters a compliance query down to zero results and an empty result reads like a clean estate. The second is a word-boundary assertion in the money rule: written across the whole alternation it never matched a bare greater-than sign, because the position before a symbol sits between two non-word characters, so the words and the symbols are matched separately and the comment says why.
- Time
- Last 3 days, last week, today and yesterday become a since, or a half-open since and until pair, computed from the server’s clock rather than the browser’s.
- Money
- Over, above, more than, greater than, exceeding, and the two comparison symbols all set a minimum amount; a leading pound, dollar or euro sign is stripped before the number is read.
- Oversight
- Approved by a human, a manager or a person sets the approval flag; blocked, denied by policy and policy violation set the policy-match flag; awaiting approval and failed map onto trace statuses.
- Personal data
- Credit cards, SSNs, National Insurance numbers, NHS numbers, IBANs and email addresses map onto the detected-kind filter, which matches on the kind the redactor recorded rather than on any stored value.
- Everything else
- Residual words are stripped of filler — show, find, all, traces, where — and become the full-text term. This is exactly the over-broad case the design record names as the fallback’s known weakness.
A timeline that says what happened, in words
Every event in a trace carries a plain sentence, not only a type and a status. A request event says the call came through the gateway, so every check below it ran before anything left the network. A tool-result event says tool output is treated as untrusted input and scanned for injection. A redaction event says the kind of data found was recorded and the value never was. The colour band and the sentence are produced by two functions that share their readers, deliberately, and the narration has a test suite of its own, because it is the only part of the flight recorder a reader who cannot read a raw event table will actually rely on.
Shadow mode is answered before the verdict, and that ordering is load-bearing. A dry-run policy still records the action it would have taken, so asking what a policy did before asking whether it was live narrates a shadow match as an enforced block — the exact inversion shadow mode exists to let an operator avoid. It went wrong in precisely that way once: a generic status check ran first and painted a red Blocked by policy on a decision whose recorded verdict was require_approval, one row above an amber Paused for human approval on the same event, and in doing so made the shadow branch unreachable for any writer that stamps a shadow match as blocked.
One sentence was rewritten because it claimed more than its event could support. The prompt-cleared event is written before the upstream call and never settled — trace events are append-only by design — so a sentence reading that the prompt was forwarded to a named model at a named provider was printed for calls that never left the process, including one where the provider’s credential was missing entirely. It now says what the event genuinely records: the policy checks passed, this is what was routed, and whether the provider accepted it is the next step.
Three parts of the pipeline write a policy decision in three different shapes, and the timeline reads all of them. The policy matcher writes an action beside a mode, the refusal paths write a verdict, and the on-behalf-of intersection writes one in the past tense; reading only one of them is how the timeline came to narrate every policy step as raising no objection. Redaction events are rendered from their metadata alone, because dumping the payload into the detail panel would put the values the redactor removed back on the screen.
- What each step shows
- Sequence number, model, provider, input and output tokens with the cached share called out separately, cost in USD, elapsed time, and the offset from the start of the trace.
- What opens by default
- Policy decisions, approval requests and resolutions, tool calls, errors and anything blocked are expanded on load; ordinary steps stay collapsed behind their sentence.
- Errors that name their fix
- A provider credential refusal, a provider rate limit and an upstream timeout each get their own sentence, because the generic one is right for an unexpected fault and useless for the failures a customer actually hits in week one, each of which has a fix the operator can act on without opening a ticket.
- An empty timeline means something
- A trace with no recorded steps is a request rejected at authentication, before the pipeline started, and the page says so rather than showing a blank list.
What a trace holds, and what it deliberately does not
The trace row holds metadata and the events hanging off it hold the payload record, which is bounded on purpose. The prompt is kept as a post-redaction excerpt capped at 4,000 characters, read back off the outbound payload rather than off the original, so the search index cannot contain what the redactor has just removed. Tool arguments and results arriving through the tool gateway are capped at 16,000 characters, because a tool result is evidence of an action rather than a conversation.
The model’s answer is not stored. The response event records the stop reason, the upstream request id, the token counts split by cache bucket, the number of redaction placeholders and the types of the content blocks — not the text. Redaction events record the kinds of data found and the counts, never the values. That is what makes a trace safe to keep for as long as an audit requires, and it is also why a trace is not a transcript: you cannot replay a conversation from one.
One payload-adjacent exception is worth stating plainly, because a reviewer will find it. Where a policy requires approval for a tool call the model proposed, the approval record’s summary is the tool name plus up to 160 characters of that proposal’s arguments, built before egress redaction runs on the response — so an approval record can carry up to 160 characters of unredacted model-generated text. It cannot carry unredacted prompt text, because the request was redacted before it reached the provider, and approvals raised on the tool gateway and on the request path carry no arguments at all.
Not everything in the recorder was decided by Token Observe, and the parts that were not are marked as such. A subscription seat’s local activity — the shell and file tools its client runs, model traffic on the vendor’s own connection — reaches the recorder only through an authenticated telemetry receiver, after the same redaction the gateway performs, with policy deliberately not evaluated because there is nothing left to refuse about an action that has already happened. The receiver writes a marker as the first event of every trace it ingests, and the seat census reads that marker to classify those traces as recorded rather than enforced. Attribution comes from the presented credential and never from a resource attribute in the payload, because a receiver that believed one would let any seat write another seat’s history.
- Denormalised for search, not for display
- Tool names, personal-data kinds, the policy-match and approval flags and the largest tool-call amount are copied onto the trace row as events land, so the common filters never open a payload. Most events change none of them, and an event that changes nothing writes nothing.
- The amount column is a number, not money
- A leading pound, dollar or euro sign is stripped and the value stored unconverted, so a filter for amounts over 200 matches 200 of any currency. The console names the hazard beside the search box — amounts are matched against tool-call arguments, and currency symbols are read as a number and compared in USD — rather than leaving it to be discovered.
- Absurd values are clamped, not rejected
- A crafted tool payload claiming an amount of 1e300 would overflow the column and fail the event append — content that stops the recorder recording, on the governed hot path. It is clamped instead, so an over-claimed amount stays visible at the top of exactly the range an investigator would search.
- A failed run is recorded as failed
- An interrupted stream closes the trace as an error with an error event, and its partial usage is still metered. A failed run recorded as a success is worse than no record at all, because it is a record that lies.
Who may read it, what an export proves, and how it is deleted
Evidence reads are scoped by team, and the scope is derived from the signed-in account rather than accepted from the request. Every human account carries an explicit list of team scopes; an organisation-wide scope is a single wildcard entry, an empty list denies everything, and that deny is written as a visible clause in the query rather than left to emerge from a set operation, because it is an authorisation boundary. The same check is repeated on the detail page and on the export, since scoping the list endpoint and leaving the detail URL open is the usual way this goes wrong. A trace outside your scope returns forbidden rather than not-found, so the denial becomes an attributable event instead of an ambiguous missing id.
Listing, searching, opening and exporting each append an audit entry naming the actor and the thing acted on, and the four entries do not carry the same detail. The list and search entries carry the filter as interpreted, the teams the account was effectively authorised for and how many rows came back, and the search entry adds whether a model or the keyword parser produced that filter. The read and export entries name the trace and its agent with the event, approval and audit-entry counts, and the export carries the digest of the bundle it issued, so a bundle in circulation can be tied back to the read that produced it. Organisation-wide surfaces — the audit ledger itself, the retention controls, subject erasure — return forbidden to a team-scoped account rather than a narrower answer, because projecting those joins onto one team would produce a misleading result rather than a smaller one.
An export bundles the traces, their events, the approvals that gated them, the audit entries that account for them and a verification of the whole audit chain, then seals it with a SHA-256 digest over the canonical JSON of the bundle body. The wording matters: the bundle is digest-sealed and explicitly not signed. Recomputing the digest detects an accidental or deliberate edit after issue; it does not prove who issued the file, and provenance comes from authenticated delivery rather than from the bundle. The embedded chain verification carries its protection level for the same reason — on a default install the chain is unkeyed, and a valid result there means nothing was altered without recomputing, which is a weaker claim than it looks. Bundles are bounded: the organisation-wide compliance bundle takes at most 200 traces, 5,000 events, 1,000 approvals and 2,000 audit entries, and a single-trace export collects up to 500 approvals and the same 2,000 entries. Three of those bounds set a truncation flag the recipient can read — traces, events and audit entries. The approvals cap sets none, so a bundle that reached it looks complete, and that is said here rather than left for a recipient to work out.
Retention is unset by default, and unset means keep forever. That default is deliberate in both directions: retention has to be decided rather than inherited, and an upgrade that silently began deleting a customer’s evidence would be the worse failure. Set a window and an hourly pass ages traces out in batches of 250, each batch its own short transaction, so a purge interleaves with gateway traffic instead of stalling the inline request path. The status endpoint reports the window, the exact cutoff instant, how many traces are currently eligible and the outcome of the last pass, so the purge is configured and the purge is actually running stay separately checkable.
One data subject’s traces can be erased on demand, matched on the human principal, the session id or both, with a dry run that returns the count before anything goes. Both the preview and the deletion are audited, and the audit entry carries a SHA-256 digest of the subject identifier rather than the identifier, alongside the ids of the traces removed — enough to tie the entry to the export that preceded it without the erasure record becoming a fresh copy of the thing just erased. Traces and the audit chain are separate tables with no foreign key between them, so chain verification still passes after a purge and the record that a deletion happened outlives the deleted data. The limit is the one a data protection officer will ask about first, and it is stated here rather than in a footnote: this acts on the live primary database only.
- Backups sit outside the boundary
- Restoring a pre-erasure snapshot can resurrect erased traces and can lose the audit row that recorded the erasure. Token Observe maintains no external deletion-tombstone ledger and does not reapply deletions automatically after a recovery, so erased from the live primary is the precise claim.
- Approvals are not purged
- An approval record can hold up to 160 characters of model-proposed tool arguments, and it is the one place payload-adjacent text survives a purge that removed the trace it came from.
- Nor are radar findings or webhook deliveries
- Findings can carry staff usernames and workstation hostnames from evidence you supplied; delivery rows keep the exact bytes posted to each receiver, kept so a signature dispute can be settled.
- The rejected-caller roll-up ages out on its own window
- Ninety days by default, on the same hourly tick, whether or not a trace window is set — so quoting the trace retention period for it is wrong in both directions.
GET /api/traces/:id/export auditor role, team-scoped
{
kind, generatedAt, generatedBy { id, email, role },
subject, range, counts, truncated, page,
traces [ { trace, events } ],
approvals [ ... ],
auditEntries [ ... ],
chainVerification { valid, entriesChecked, brokenAtSeq,
reason, protection, checkpoint }
}
digest {
algorithm: "sha256",
value: "<hex>",
covers: "sha256 over the canonical JSON of the bundle"
}What this does not do
Stated here rather than discovered during an evaluation. Every line below closes off a reasonable assumption a reader would otherwise carry into a proof of concept.
- A trace is not a transcript. The model’s answer text is not stored and the prompt is kept only as a 4,000-character post-redaction excerpt, so a conversation cannot be replayed from the record.
- The search cannot aggregate, group or correlate across traces. Which agents used the same card number twice is not expressible, and adding a question shape is a schema, validator and query-builder change rather than a prompt change.
- The interpreted filter is not guaranteed to mean what you asked. Misinterpretation replaces injection as the main failure mode, and the explanation shown beside the results is advisory rather than a proof.
- Evidence bundles are digest-sealed, not signed. Recomputing the digest detects an edit after issue; it does not establish who issued the file.
- Erasure and retention act on the live primary database only, and reach neither approvals, radar findings nor webhook deliveries. A restored pre-erasure backup can resurrect what was erased.
If one of those limits is the thing that decides it for you, say so and you will get a straight answer about whether it is on the roadmap or out of scope.
Talk it throughWhat this leans on
Audit chain
Every administrative act hash-chained; seal it under a key held off the box, and anchor it with a signature your auditor can check alone.
Policy engine
One deterministic verdict on every governed request: allow, block, redact, or park it for a human.
Endpoint seats
Policy enforced inside each vendor’s own administrator hook, decided offline against a signed bundle, because a hook that phones home fails open.
Does the trace search send my prompts to a model?
No. Only the operator’s question and the current time are sent to the translation model; it never sees a trace, a result set or the database. The call is routed through Token Observe’s own gateway, so it inherits the route rules, failover chain and circuit breakers that govern ordinary traffic, and it deliberately opens no trace of its own, because a compliance officer’s search must not add rows to the corpus they are searching. The question text is never written to the logs either — it is a question about production data and may quote the personal data being asked about. With no translation model configured, nothing leaves the process at all.
What happens when the translation model is unavailable?
The deterministic keyword parser answers instead, and the response says which path ran. Any failure takes that route: no model configured, a timeout against the fifteen-second ceiling, an unreachable provider, every provider in the chain behind an open circuit breaker, output that is not JSON, or output that parses and then fails validation. The fallback is mandatory rather than optional, because search quality would otherwise depend on an upstream provider being reachable. It is also materially worse and it degrades quietly: phrasing it does not recognise falls through to an over-broad full-text term rather than raising an error, so the failure returns too much rather than nothing.
Can a compliance officer read another team’s traces?
Only if their account carries the scope for it. Each human account holds an explicit list of team scopes, the server derives the query predicate from the signed-in user, and a caller cannot widen it with a query parameter. The same check is repeated on search, on the detail page, on the single-trace export and on the compliance bundle, because scoping a list endpoint and leaving the detail URL open is the usual way this fails. An empty scope list matches nothing, written as an explicit clause rather than left to emerge from a set operation. Organisation-wide surfaces — the audit ledger, retention, subject erasure — return forbidden to a scoped account rather than a partial answer.
Is an evidence export signed?
No, and the distinction is deliberate. An export is digest-sealed: a SHA-256 taken over the canonical JSON of the bundle body, which a recipient recomputes to prove the file was not edited after it was issued. It does not prove origin — an unkeyed digest detects an edit and attributes nothing — so provenance comes from authenticated delivery rather than from the file itself. The bundle also embeds a verification of the audit chain and names that chain’s protection level, because on a default install the chain is unkeyed, and a valid result there means nothing was altered without recomputing rather than nothing was altered.
How long are traces kept, and can one person’s be erased?
Retention is unset by default and unset means keep forever, so a deployment with a storage-limitation duty has to set it. Once set, an hourly pass ages traces out in batches of 250, each in its own short transaction so a purge interleaves with gateway traffic rather than stalling the fail-closed request path. One subject’s traces can be erased on demand, matched on the human principal, the session id or both, with a dry run that returns the count first. Both the preview and the deletion are audited, and the entry carries a digest of the subject identifier rather than the identifier. This acts on the live primary database only — a restored pre-erasure backup can resurrect what was erased.
Can I ask aggregate questions, such as which agent was blocked most often?
Not through this search. The filter has no aggregation, grouping or cross-trace correlation, so which agents used the same card number twice is not expressible, and a new question shape needs a schema field, a validator change and a query-builder change — a code change rather than a prompt change. That ceiling is the price of never handing a model an interpreter, and it was accepted with the reason written down rather than discovered later. What the recorder does answer is per-trace: filter to the population you care about, read the totals on each trace, and export the set as a sealed bundle.
Prefer to ask a person? Write to us →
Bring us the agent you are least comfortable with.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in