Audit evidence for agent actions
Unkeyed, a rewrite that recomputes every hash verifies clean
every questionOn this page
The question evidence has to survive
In an audit, the value of a record is that it answers who changed this, when, and whether anybody edited the answer afterwards. The third part is what makes it evidence rather than documentation, and it means the threat model has to include the operator: an administrator who can relax a policy, run an agent against the relaxed version, then delete the row that says so.
Database permissions do not help when the database is a file on that operator’s disk, and immutability asserted by configuration is not evidence — a setting that can be changed by the party under examination is a statement of intent. So the useful question to ask of any audit capability is not is it append-only but what specifically would have to be true for a rewrite to go undetected, and who would have to be involved.
The alternatives are worth knowing because they are the ones you will be offered. An external write-once store makes tampering hard at the infrastructure layer and is a genuine complementary control, but it proves nothing cryptographically, cannot be verified offline, and adds a cloud dependency to something that often has to run on-premises or air-gapped. A Merkle tree with signed tree heads and a trusted timestamp authority is strictly better than a plain chain — inclusion proofs would let you hand a regulator a verifiable subset without exposing the whole log — and it costs a network dependency on a path that has to stay local. A per-record hash chain is the option that works everywhere, and it should be chosen knowing exactly what it does not do.
There is one more axis that gets forgotten until an incident. Evidence has to distinguish a control that was quiet from a control that was off. A policy running in observation mode has to leave a record saying it was running in observation mode; a discovery source that has not been fed has to be distinguishable from a source that found nothing; a verification that could not check something has to say not checked rather than passed. Absence of evidence is not evidence of absence, and a record that cannot express the difference will be read as the reassuring one.
Hash chaining: what it covers and the three things it cannot see
The construction is simple and the details are where implementations go wrong. Each entry’s digest covers the previous entry’s hash plus the canonical serialisation of that entry’s own content, with the entry’s own hash field excluded — excluded not for tidiness but because a recomputation that included it would cover a field the original write never covered, and every verification would fail. Canonical means keys sorted recursively and no incidental whitespace, so one record has exactly one encoding and anybody holding the row can reproduce the digest.
Appending has to take the chain tip inside the same transaction as the write, or two concurrent writers fork the chain and you have two valid histories.
A verification walk should distinguish three classes of failure and name which one it found, because they mean different things operationally. A link mismatch means an entry no longer carries its predecessor’s hash — something was inserted, reordered or replaced. A content mismatch means the entry itself was altered. A sequence gap means a row was removed, and this includes a missing prefix: a writer who deletes entry 1 and recomputes an unkeyed remainder from genesis would otherwise leave a chain that begins at 2 and checks out link by link.
What no walk can see is truncation by somebody willing to edit the sequence high-water mark as well. Lopping entries off the end leaves every remaining link honest, so a shortened chain verifies clean, and nothing inside the database can fix that — the length has to be attested somewhere the same operator cannot edit. This is why a head endpoint that returns the attested head and entry count, with instructions to record them outside the deployment, is not a nicety: it is the only way a restore that lost its tail is distinguishable from a restore that did not. Token Observe’s own deployment guide once made a bare passing verification the success criterion of its restore drill, which is how a restore that had lost 10 of 90 entries passed it.
One more distinction is worth building in. A caller-supplied expected head that does not match is diagnostic evidence, not proof of corruption, and it must not be able to latch a control plane into unavailability — a mistyped hash should be reported, while an intrinsic failure like a content mismatch or a sequence gap is the one that latches.
hash = digest( prevHash + separator + canonicalJson(entry, minus its own hash) )
genesis prevHash = 000...000 (64 zeros)
digest = SHA-256, or HMAC-SHA256 when an audit key is configured
verify with the head you wrote down BEFORE the restore:
valid false
entriesChecked 80
brokenAtSeq 90
reason chain is truncated: it no longer reaches sequence 90,
which was previously attested
protection keyed
attestedAgainst operatorKeyed epochs: moving the target from database access to key possession
Set an audit key and the entry digests become HMACs under it, with the chain head sealed at every boot by a checkpoint MAC. Rewriting an entry then requires the key as well as database access: without it an attacker can only recompute with a plain hash, which no longer matches. This is the single highest-value configuration change available in this area, and it is the one most deployments have not made, because the default is off and the default is what most installs run.
State the guarantee with its qualifications, all of which are real. It holds only if the key is injected from a secret manager the database administrator cannot read — a key in a file beside the database buys nothing. An attacker with host access reads the key out of the process environment and is therefore out of scope; that is a host compromise and no in-database scheme survives one. The checkpoint vouches for pre-key history only from the moment the key was introduced, and cannot say whether that history was already honest. And rotation needs the complete ordered ring of predecessor keys, because losing a historical key makes that epoch permanently unverifiable.
Entries written before the key existed cannot be re-MAC’d — doing that is precisely the act being prevented — so they are covered instead by the checkpoint over the head they reached, and altering any of them moves that head. Tolerate unkeyed hashes only as an unbroken prefix inside the checkpointed range: once an entry has verified under the key, an unkeyed successor is a forgery rather than a legacy row.
First enablement should be an external two-boot ceremony rather than something the software can decide for itself, and the reason is instructive. Using the database file alone, the process cannot distinguish legitimate empty or pre-key history from a file somebody reset or recomputed just before the ceremony. So the flag that authorises the first seal comes from outside, the sealing boot is read-only, and a second boot with the flag removed is required before writes are admitted. From then on an ordinary keyed boot should refuse a missing checkpoint schedule even when the surviving rows form a perfectly valid plain-hash chain — otherwise a database writer deletes every checkpoint, recomputes unkeyed, and asks the software to seal the downgrade as if it were first enablement.
One small piece of this deserves naming on its own because it is a lesson about how integrity tooling fails people. A missing key and an altered entry produce the same bytes — the digests do not match — and completely different events: one is a configuration mistake somebody made five minutes ago, the other is an accusation of tampering. A verification result should say when it suspects the former, explicitly labelled as a diagnosis rather than proof. Token Observe’s console previously reported the second for the first, which means the product accused the customer’s own operators of the exact act it exists to detect, over a dropped environment variable.
unkeyed keyed
edit a row, do not re-hash detected detected
delete a row from the middle detected detected
delete the tail, keep logging sequence gap sequence gap
delete the tail, stop logging off-box head off-box head
rewrite and recompute every hash UNDETECTED detected
rewrite an entry written before the key UNDETECTED detected
delete the checkpoints n/a boot refuses
read the key out of the host UNDETECTED UNDETECTEDAnchoring: evidence a regulator can check without being handed a forging key
A MAC has a structural problem as evidence: the key that verifies the chain is the key that can forge it. Hand it to an auditor and you have handed them the power to fabricate the very record they were given it to check — so they cannot verify independently, and you cannot prove you did not.
An asymmetric signature splits those powers. The install signs a statement of the chain head with a private key; everybody else verifies with the public half. Token Observe periodically signs such a statement — install id, anchor sequence, previous anchor hash, the head sequence and hash, entries covered, the protection level the chain had at signing time, the newest checkpoint, the creation time, the key id, the algorithm and the cadence — chains anchors to one another, and publishes each one to a file or an HTTP sink, because an anchor that never leaves the database it attests is only as durable as that database.
The exact claim is narrow and it is the whole thing: any copy of the anchors you kept off the box beats any rewrite made after you took it. It does not say the history beneath the first anchor was honest, that the signer’s clock was truthful, or that somebody holding the signing key could not have produced the same file.
Two design decisions in anchoring are worth copying. The first is that the public key must be a required input to verification and must never be defaulted from the artefact being verified — not from the anchor file, not from the install’s own status endpoint. If a key could be defaulted from the material being checked, an attacker ships their own key beside their own rewritten anchors and every check reports success. The second is that anchoring refuses to sign when the chain does not verify, when the head has moved backwards, or when an already-anchored entry no longer matches. That is the opposite of a retry and it is the feature: signing over a forged head would launder the rewrite under a key the auditor was told to trust, while not signing leaves the previous anchor standing — and that anchor still contradicts the rewrite.
The residuals should be printed on every run rather than buried in a manual. Key theft signs anything. History before the first anchor is covered by no anchor. A sink administered by the same party that runs the database is not independent — put one somewhere they cannot rewrite. The creation time is asserted by the signer, so only a timestamp authority proves when, and Token Observe does not build one. There is no Merkle tree, so partial-log proofs are not available: you cannot hand a regulator a verifiable subset without handing over the rest.
- What a verification pass says
- Every anchor in the file was signed by the holder of one of the keys you supplied, each key change was dual-signed by adjacent keys, the anchors are contiguous and correctly linked, and attested heads only ever move forward.
- What it does not say
- That the history beneath the first anchor was honest, that the signing clock was truthful, or that nobody stole the key. A verifier that does not print these limits invites a pass to be read as more than it is.
- Why the endpoint and the offline tool are both needed
- The endpoint holds the database and walks the entry chain first, so it can confirm today’s history is intact — and it is served by the party under audit. The offline tool runs against a copy taken months ago and confirms today’s history is the same one you were shown then.
- The gap that gets printed rather than hidden
- At a daily cadence a live chain has normally grown past its newest anchor, and in that state a supplied head cannot speak: an entry beneath the anchor could have been rewritten and two ordinary entries appended over it. Closing it needs the chain’s hash at the anchored sequence, which only a caller holding the database can supply.
The bundle you actually hand over, and the words on it
What gets handed to an auditor is usually an export rather than a database, so the export’s properties matter as much as the chain’s. A useful bundle covers a period and contains the traces and their events, the approvals with approver identity, timestamp and rationale, the audit entries covering every governance-plane change, and the chain verification result.
Be precise about the seal. Token Observe’s export carries a SHA-256 digest over the canonical JSON of the bundle body, generated at a recorded time. That is a seal and not a signature: it lets a recipient confirm the file is byte-for-byte the one whose digest they were given through another channel, and whoever can rewrite the bundle can recompute it. There is no signature over an export. Durable origin evidence comes from the keyed chain plus an anchor you retained off the box, which is where the signature actually lives. Anyone who describes an export as signed when it is digest-sealed has given an auditor a false impression of what they are holding.
The verdict travelling inside the bundle is what makes it more than an exported log file, and the field to read first is the protection level rather than the valid flag. A bare valid invites the reader to assume more than an unkeyed chain offers: it means nothing was altered without recomputing, not that nothing was altered. Naming the protection level beside the verdict is the difference between an auditor being informed and being misled.
Watch the caps. A bundle with limits on how many traces, events, approvals and audit entries it will carry needs to set a truncation flag inside the file and to state those limits where the person downloading it will see them, because nobody opens a sealed JSON file to check before forwarding it to a regulator. Token Observe caps at 200 traces, 5,000 events, 1,000 approvals and 2,000 audit entries over a default 30-day window. Its console repeats the trace cap beside the download button, with an instruction to narrow the dates and check the truncation flag before forwarding the file; the other three caps are documented rather than shown, which is worth knowing if you are the person doing the forwarding.
Finally, plan the interaction with erasure before you need it. Traces and the governance-plane log should be separate tables, with the chain covering administrative changes rather than payloads, so a retention pass or a data-subject erasure leaves verification passing and the record that a deletion happened survives the deletion — with the erasure entry carrying a digest of the subject identifier rather than the identifier itself, so the erasure record does not become a new copy of the thing that was erased. The corollary is a rule to live by: an audit entry cannot be edited to remove something without breaking verification from that sequence onward, so never put anything into the governance-plane log that you may later be required to erase.
How to put audit evidence for agent actions into practice
- 01
Decide what belongs in the governance-plane log
Administrative acts — an agent created, a role widened, a policy promoted to enforcing, an approval decided, the kill switch engaged — and deliberately not payloads. Keeping personal data out of the chain is what lets erasure and integrity coexist. - 02
Chain the entries and append inside the transaction that reads the tip
Each digest covers the previous hash plus the canonical content of the entry with its own hash excluded. Taking the tip in the same transaction as the append is what stops concurrent writers forking the history. - 03
Configure the audit key from a secret manager your DBAs cannot read
Do it early, because the guarantee is forward-looking: entries written before the key existed are covered only by the checkpoint over the head they reached. A key stored beside the database buys nothing at all. - 04
Turn on anchoring and publish off the box
Sign a statement of the head on a schedule, chain the anchors, and deliver them to a destination administered by somebody other than the party that runs the database. An anchor that never leaves that database is only as durable as it is. - 05
Distribute the public key out of band, and only the public key
Through a key ceremony, a published fingerprint or an earlier export — never from the anchor file itself. A signature checked against a key read out of the artefact being verified proves nothing. - 06
Record the head off-box before every restore drill
Write down the attested head sequence and hash, restore, then verify against the recorded value rather than against whatever the restored file says. A bare passing verification after a restore does not detect a lost tail. - 07
Rehearse the handover
Produce the bundle for a past month, read the verification verdict and its protection level as an auditor would, check whether any cap truncated it, and run the offline verifier against an anchor copy on a machine with no access to the install.
Where this argument meets an implementation
Audit chain
Every administrative act hash-chained; seal it under a key held off the box, and anchor it with a signature your auditor can check alone.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
Human approvals
One human decision, bound to one exact payload, spendable once.
Agent registry
One record per agent, and it is the record the gateway enforces against.
Those pages are one product's implementation of what this guide argues for; describe what your agents actually do and you will get a straight answer about whether you need any of it, including when the answer is no.
Talk it throughIs a hash-chained audit log tamper-proof?
No, and the word to use is tamper-evident. A hash chain catches any alteration that does not also recompute every downstream digest, and locates it at a named sequence number — which is genuinely useful and is exactly the alteration a determined database writer will not make. Token Observe’s repository ships a forgery test asserting that a rewrite-and-recompute passes verification on a default unkeyed install, and that it fails once a key is configured, so neither half of the claim can drift. Keying moves the requirement to possession of a key; an off-box signature moves it again. None of those steps reaches tamper-proof, and a product using that word about a database it also writes to has not thought about it.
What does an auditor need in order to verify an anchor?
Two things, and the second must arrive out of band. First, the anchor export — a file of signed statements, or the equivalent endpoint. Second, the current public key and, after any rotation, every retired public key in newest-to-oldest order, obtained from a key ceremony, a published fingerprint or an earlier compliance export, and never from the anchor file. A pass then says the holders of those keys signed every anchor, that each key change was dual-signed by adjacent keys, that the anchors are contiguous and correctly linked, and that attested heads only move forward. It does not say the history beneath the first anchor was honest.
Are compliance exports signed?
In Token Observe, no — and the distinction is the one an informed auditor will test. A bundle is sealed with a SHA-256 digest over the canonical JSON of its body, generated at a recorded time, and it carries the chain verification verdict: valid or not, entries checked, the first broken sequence if there is one, the protection level, and the checkpoint it was verified against. The digest lets a recipient confirm the file has not changed since somebody told them what the digest was; it is not a signature, and whoever can rewrite the bundle can recompute it. The signature lives on the anchors, which is where an off-box copy makes it worth something.
Does deleting a customer’s data break the audit chain?
It should not, if the design keeps payloads out of the chain. Traces and the governance-plane log belong in separate tables with the chain covering administrative changes rather than payloads, so a retention pass or a data-subject erasure leaves verification passing — and the record that the deletion happened survives the deletion, carrying a digest of the subject identifier rather than the identifier itself so the erasure record does not become a new copy of what was erased. The reverse is emphatically true: an audit entry cannot be edited to remove something without breaking verification from that point onward, so the governance-plane log is the wrong place for anything erasable.
How do you detect somebody deleting the end of the log?
Not from inside the database, which is the uncomfortable answer. Lopping entries off the end leaves every remaining link honest, so an ordinary tail deletion is caught only because the sequence high-water mark does not fall on a delete — and an administrator who edits that too leaves a chain that verifies clean. The only real defences are external: a head sequence and hash you recorded outside the deployment before the event, or an off-box signed anchor. This is also why a restore drill whose success criterion is a bare passing verification is not a drill — Token Observe’s own guide made that mistake, and a restore that had lost 10 of 90 entries passed it.
Prefer to ask a person? Write to us →
Bring us the question this guide did not answer.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in