Produce audit evidence for AI agents that an auditor accepts
One bundle with a verification verdict inside it, and the protection level stated beside the verdict.
On this page
The folder of CSVs, and the word tamper-proof
The usual first artefact is a set of exports assembled the week before the audit: application logs from the agent framework, a provider usage report, a spreadsheet of who approved what, and a screenshot of the policy configuration as it stands today. Every one of those is produced by, or about, the party under examination, and none of them says anything about whether it was edited between the event and the export. An auditor is not being difficult when they ask how you would know; that is the question the artefact exists to answer.
The second attempt is an append-only table, enforced by database permissions. That is genuinely better and it fails against the threat model that matters, which has to include the operator: an administrator who can relax a policy, run an agent against it, then delete the row that says so. Database permissions do not help when the whole database is a file on that operator’s disk, and immutability asserted by configuration is not evidence.
The third is a vendor claim of a tamper-proof log, and this is where a careful buyer should slow down. A per-record hash chain catches any alteration that does not also recompute every downstream digest — which is exactly the alteration somebody with write access to the database will not make. Token Observe ships a forgery test asserting that a rewrite-and-recompute passes verification on a default install, and asserting that it fails once a key is configured, so neither half of the claim can drift away from the code. The word tamper-proof does not appear anywhere in the product, and a vendor who uses it about a database they also write to has not thought about it.
The fourth is a restore drill whose success criterion was a passing verification, which is how a restore that had lost ten of ninety entries passed. Lopping entries off the end leaves every remaining link honest, so a shortened chain verifies clean and nothing inside the database can fix that — the length has to be attested somewhere the same operator cannot edit. The head endpoint exists for that, and the deployment guide now says to record the head and the entry count outside the deployment before the drill rather than after it.
How to actually do it
Each step is something you can go and do. Where a step depends on a decision somebody has to make rather than a setting somebody has to change, it says so.
- 01
Configure the audit MAC key while your history is still short
Inject it from a secret manager your database administrators cannot read; a key in a file beside the database buys nothing. Entry digests become HMAC-SHA256 under it and the head is sealed at every boot by a checkpoint MAC. The guarantee is forward-looking — entries written before the key existed cannot be re-MAC’d, because that is the act being prevented — so every month you wait is a month of permanently weaker prefix. - 02
Turn on anchoring and send the anchors somewhere you do not administer
With an Ed25519 signing key set, Token Observe signs a canonical statement of the chain head daily and on demand for an admin, chains anchors to one another, and publishes each one to a file or HTTP sink. The claim an anchor buys is narrow and it is the only one made: any copy you retained off the box beats any rewrite made after you took it. If every sink is administered by the party that runs the database, deleting the anchors and the rows is one action rather than two. - 03
Record the attested head out of band, and use it on the way back
The head endpoint returns the attested head sequence and entry count. Write them down outside the deployment, then pass them back to the verification endpoint as an expected sequence and hash after any restore. The result then says what it was checked against — an operator-supplied head, a stored one, or none — and a bare passing result stops being the success criterion of a drill. - 04
Decide a retention period rather than inheriting one
Trace retention is unset by default and unset means keep forever, which over-satisfies a minimum-retention duty and satisfies no storage-limitation duty at all. Set a window and an hourly pass ages traces out in batches of 250, each in its own short transaction so a purge interleaves with gateway traffic. The status endpoint reports the window, the exact cutoff, how many traces are currently eligible and the outcome of the last pass, so configured and actually running stay separately checkable. - 05
Produce the bundle for the period and read the truncation flags before you send it
Export over the dates the auditor asked for, then open the file. Traces, events and audit entries each set a truncation flag when their cap bites; the approvals cap sets none, so for approvals you have to read the count in the bundle against the period you requested. Nobody opens a sealed JSON file to check before forwarding it, which is why the caps belong in your covering note rather than in the file alone. - 06
Give the auditor the public key out of band and the offline verifier
The standalone verifier takes a JSONL anchor export and an Ed25519 public key and prints trusted or not trusted, with no install, no database, no network and no configuration. The key is a required argument and is never defaulted from the file being checked — that single rule is what separates anchoring from theatre, since a key defaulted from the material under verification lets an attacker ship their own key beside their own rewritten anchors.
What one bundle contains, what the seal covers, and where it stops
The bundle is assembled from four sources and carries a fifth thing that distinguishes it from an exported log file. The four are the traces and their events for the period, the approvals with approver identity, timestamp and rationale, the audit entries covering the governance-plane changes in that window and within the requester’s evidence scope, and the counts and range that describe what was asked for. The fifth is the chain verification block: whether the chain is valid, how many entries were checked, the first broken sequence number if there is one, the protection level, and the checkpoint the run verified against.
That protection level is the field to read first, and naming it is the difference between an auditor being informed and being misled. A bare valid flag invites the reader to assume more than an unkeyed chain offers: it means nothing was altered without recomputing, not that nothing was altered. Under a keyed configuration the digests are HMACs under a key held outside the database and the head is sealed at every boot, so a rewrite needs the key as well as database access. Both states are labelled in the export, and the label is what you present.
The seal is a SHA-256 over the canonical JSON of the bundle body, with the algorithm and what it covers stated inside the file. It detects an accidental or deliberate edit made after issue, and it attributes nothing — provenance comes from authenticated delivery rather than from the file. There is no Ed25519 signature over an export anywhere in the product; the signature lives on the anchors, which is the only place an off-box copy makes it worth something.
One more thing belongs in the covering note. A trace is not a transcript. The prompt survives as a post-redaction excerpt capped at 4,000 characters and the model’s answer text is not stored at all, so the bundle is evidence of what was decided, what was refused and what it cost rather than a conversation somebody could replay. Say that to the auditor rather than letting them discover it, because the alternative is a request for something that was never collected.
- Refusals are in it
- The trace opens at step 3 of the request path — before sanitisation, before the scanners and before the verdict — so a request blocked a millisecond later is recorded rather than missing, and the trace id comes back on a response header even on a refusal.
- Shadow decisions are in it
- Every rule that matched writes a decision event naming the policy, its mode, its action and why it matched, so a rule running in observation mode leaves evidence that it was running rather than being indistinguishable from a rule nobody wrote.
- The read is in it too
- Listing, searching, opening and exporting traces each append an audit entry naming the actor, the interpreted filter, the teams the account was authorised for and the counts — and the export entry carries the digest of the bundle it issued, so a bundle in circulation can be tied back to the read that produced it.
- Not in it
- Hidden model reasoning, the model’s answer text, training-data provenance, model cards, and anything at all about an agent that never routed through Token Observe.
Three layers of integrity, and what each one moves the target to
The bare hash chain is tamper-evidence and is the default. Each entry stores the previous entry’s hash and its own digest over that hash plus the canonical JSON of its content, with the row’s own hash and its prevHash excluded from that JSON so a later recomputation covers exactly what the original covered. A verification walk finds three distinct classes of failure and names which one it found: a link mismatch, a content mismatch, or a sequence gap — including a missing prefix, because a writer who deletes entry 1 and recomputes from genesis would otherwise leave a chain that begins at 2 and checks out link by link.
Keying moves the target from whoever can write the database to whoever holds the key. Entries written before the key existed cannot be re-MAC’d, so they are covered by the checkpoint over the head they reached, and altering any of them moves that head. Unkeyed hashes are tolerated only as an unbroken prefix inside the checkpointed range: once an entry has verified under the key, an unkeyed successor is the forgery rather than a legacy row. An ordinary keyed boot refuses a missing checkpoint schedule even when the surviving rows form a perfectly valid plain chain, which is what stops a database writer deleting every checkpoint, recomputing unkeyed, and asking the product to seal the downgrade as first enablement.
Anchoring moves it again, and it exists because a MAC has a structural problem as evidence: the key that verifies the chain is the key that can forge it, so handing it to an auditor hands them the power to fabricate the record they were given it to check. Ed25519 splits those powers — the install signs, everybody else verifies. An anchor is a short statement, canonicalised and signed: install id, anchor sequence, previous anchor hash, the head being attested, entries covered, the protection level at signing time, the checkpoint sequence when keyed, the creation time, the key id, the algorithm and the cadence. Every field is inside the signature. Anchoring refuses to sign at all when the chain does not verify, which is the feature rather than a limitation, because signing over a forged head would launder the rewrite under a key the auditor was told to trust.
One small mechanism deserves naming because it is where a governance product can insult its own customer. A missing key and an altered entry produce the same bytes and completely different events: one is a configuration mistake somebody made five minutes ago, the other is an accusation of tampering. When verification fails at or after a sequence a checkpoint seals and no key is configured in the process, the result says the key is suspected missing and labels that a diagnosis rather than proof. The console previously reported the second for the first, which meant the product accused a customer’s own operators of the exact act it exists to detect, over a dropped environment variable.
What the bundle evidences, and what remains the deploying organisation’s duty
A governance layer that sits between agents and providers evidences runtime facts about traffic that passed through it, and nothing else. Which agent made which call, under which permissions and delegation chain; which policies matched and what they did, including the ones that only observed; which human approved what, when, and with what stated reason; what it cost; and every administrative change to the governing configuration in a chain where an edit breaks verification at a known point. That set maps usefully onto an evidence request under the EU AI Act, an ISO/IEC 42001 management system or a NIST AI Risk Management Framework programme.
It does not evidence training-data provenance, model cards, or bias and fairness testing — those are provider obligations and separate work. It does not record hidden model reasoning, because the gateway sees the request and the response and not the deliberation between them. And it says nothing whatsoever about an agent that never routed through it, which is why a coverage claim needs a number on both sides of it rather than a percentage on one.
The obligations that stay with you are worth listing plainly, because this is where mapping tables overreach. Deciding your risk classification is yours. Running the fundamental-rights impact assessment where Article 27 requires one of a narrower set of deployers is yours, and it is a different obligation in a different article from the data protection impact assessment Article 26(9) points at. Choosing a retention period that satisfies both a minimum-retention duty and a storage-limitation duty is yours, and counsel has to decide which apply. Notifying an authority is yours.
And no tool confers a certificate on the organisation that installs it. Token Observe holds no SOC 2 report, no ISO 27001 certificate, no ISO/IEC 42001 certificate and no independent penetration test, and its published licence is a template pending counsel. A control helps you evidence a clause; anything that says otherwise is selling something it does not hold.
What this still does not solve
Doing everything above leaves a residue. It is smaller than what you started with and it is not nothing, and knowing its shape in advance is the difference between a control you trust correctly and one you trust too much.
- There is no Merkle tree and there are no inclusion proofs, so you cannot hand a regulator a verifiable subset of the log without handing over the rest of it. The architecture decision defers that deliberately rather than having overlooked it.
- There is no trusted timestamp. An anchor’s creation time is inside the signed bytes and cannot be edited afterwards, but it is asserted by the signer at the moment of signing, and only a timestamp authority or a public ledger proves when.
- Nothing survives host compromise. Anyone who reads the audit MAC key out of the process environment, or the anchor private key out of the key management service, can produce a history that verifies perfectly — no in-database scheme survives that, and Token Observe does not claim to.
- An audit entry cannot be edited to remove something without breaking verification from that sequence onward, which is the design working. That makes the governance-plane log the wrong place to put anything you may later be required to erase, and it is a planning decision rather than a setting.
If one of those residues is the thing that actually worries you, that is the conversation worth having rather than the one about the steps above it.
Talk it throughThe parts of the platform this uses
Audit chain
Every administrative act hash-chained; seal it under a key held off the box, and anchor it with a signature your auditor can check alone.
Unkeyed, a rewrite that re-hashes everything verifies clean
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
The filter cannot group, count or correlate across traces
Human approvals
One human decision, bound to one exact payload, spendable once.
An approval takes effect only when the agent retries
Agent registry
One record per agent, and it is the record the gateway enforces against.
An overdue review never suspends the agent itself
The other jobs
Is the audit log tamper-proof?
No, and Token Observe will not use the word. Unkeyed, which is the default, the chain is tamper-evident: any edit or deletion that does not also recompute every downstream hash breaks verification at a named sequence number. An operator with write access to the database file can recompute it, and the repository’s own forgery test asserts that the forged chain passes. Configure an audit MAC key from a secret manager the database administrator cannot read and a rewrite needs the key as well. Retain an Ed25519 anchor off the box and any rewrite made after you took that copy is contradicted by it. That is as far as it goes.
Are compliance exports signed?
No. A bundle is digest-sealed with a SHA-256 over the canonical JSON of its body, generated at a recorded time, and it carries the chain verification verdict — valid or not, entries checked, the first broken sequence if any, the protection level and the checkpoint it was verified against. The digest lets a recipient confirm the file has not changed since someone told them what the digest was; it is not a signature, and whoever can rewrite the bundle can recompute it. Durable origin evidence comes from the keyed chain plus an anchor you kept off the box, which is where the Ed25519 signature actually lives.
What does an auditor need in order to check an anchor independently?
Two things, and the second must arrive out of band. First the anchor export — the JSONL sink file, or the anchors endpoint. Second the current Ed25519 public key and, after any rotation, every retired public key in newest-to-oldest order, obtained from a key ceremony, a published fingerprint or an earlier export, never from the anchor file itself. A pass says the holders of those keys signed every anchor, that each key change was dual-signed by adjacent keys, that the anchors are contiguous and correctly linked, and that attested heads only move forward. It does not say the history beneath the first anchor was honest.
How much of the period does one bundle actually cover?
As much as fits inside its caps: at most 200 traces, 5,000 events, 1,000 approvals and 2,000 audit entries over a default 30-day window. Traces, events and audit entries each set a truncation flag inside the file when their cap bites. The approvals cap does not, so a bundle that reached it looks complete and the count has to be read against the period you asked for. Narrow the dates until the flags are clear, and put the caps in the covering note rather than relying on a recipient to open a sealed file before forwarding it.
Does erasing one person’s data break the evidence?
No. Traces and the audit log are separate tables and the chain covers governance-plane changes rather than payloads, so a retention pass or a data-subject erasure leaves verification passing, and the record that the deletion happened survives the deletion — with the erasure entry carrying a digest of the subject identifier rather than the identifier, so the record does not become a fresh copy of what was erased. The limit a data protection officer will ask about first is that this acts on the live primary database only: a restored pre-erasure backup can resurrect erased traces and can lose the audit row that recorded the erasure.
Describe the version of this you actually have.
The steps above are the general shape. Which of them matter, and in what order, depends on what your agents do and which of them worries you — say that and you will get a straight answer, including when the answer is that Token Observe is not what you need for it.
no form · no qualification step · no sales desk · the other three ways in