Audit chain

The value of an audit log in an audit is that it answers who widened this policy, when, and whether anyone edited the record afterwards. That means the threat model has to include the operator: an administrator who can relax a policy, run an agent against it, then delete the row that says so. Database permissions do not help when the whole database is a file on that operator’s disk, and immutability asserted by configuration is not evidence.

The alternatives were considered and rejected for stated reasons. An external write-once store makes tampering hard at the infrastructure layer and is a genuine complementary control, but it proves nothing cryptographically, cannot be verified offline, and adds a cloud dependency to a product that gets deployed on-premises and sometimes air-gapped. A Merkle tree with signed tree heads and a trusted timestamp authority is strictly better — inclusion proofs would let you hand a regulator a verifiable subset without exposing the whole log — and it costs a network dependency on a path that has to stay local, for a capability nobody has yet asked for. The per-record hash chain was chosen knowing what it does not do.

What it does not do is stated in the product’s own documentation rather than discovered during your evaluation. The chain alone is only evidence against an attacker who cannot recompute it; whoever can write the database can rewrite an entry and re-hash everything downstream, and a test in the repository asserts exactly that — the forged chain passes verification on a default install, and the same test asserts that it fails once a key is configured, so neither half of the claim can drift.

Everything above the bare chain exists to move the target. Keying the digests moves it from whoever can write the database to whoever holds the key. Signing a head with Ed25519 and publishing it off the box moves it again, to whoever holds the signing key and can also reach every copy you took. None of those steps reaches tamper-proof, and Token Observe does not use the word.

The digest input is the previous entry’s hash, a separator, and the canonical JSON of the entry with both its own hash field and its prevHash removed — the prevHash is already the prefix, so it is covered once rather than twice. Canonical means object keys sorted recursively and no whitespace, so one record has exactly one encoding and the digest is reproducible by anyone holding the row. Removing the row’s own hash is not tidiness: a recomputation that included it would cover a field the original write never covered, and every verification would fail.

A verification walk finds three distinct classes of failure and names which one it found. A link mismatch means an entry no longer carries its predecessor’s hash. A content mismatch means the entry was altered. A sequence gap means a row was removed — including a missing prefix, because a database writer who deletes entry 1 and recomputes an unkeyed remainder from genesis would otherwise leave a chain that begins at 2 and checks out link by link. Sequence numbers come from an auto-increment high-water mark that an ordinary delete does not lower, so entries removed from the middle — or from the end, once anything is logged afterwards — leave a hole that no amount of re-hashing can close.

What no walk can see is truncation where nothing is logged after it, or truncation by someone willing to edit that high-water row too. Lopping entries off the end leaves every remaining link honest, so a shortened chain verifies clean, and nothing inside the database can fix that — the length would have to be attested somewhere the same operator cannot edit. That is what the head endpoint is for: it returns the attested head and the entry count with a note telling you to record them outside the deployment before a restore drill, and the verification endpoint takes them back as an expected sequence and hash. The result then says what it was checked against — an operator-supplied head, a stored one, or none — and the deployment guide used to make a bare passing result the success criterion of the restore drill, which is how a restore that had lost 10 of 90 entries passed it.

A caller-supplied expectation is treated as diagnostic evidence rather than trusted runtime state. A mistyped hash must not be able to latch the whole control plane unavailable, so an expectation mismatch is reported and does not trip the corruption latch that an intrinsic failure does.

A MAC has a structural problem as evidence: the key that verifies the chain is the key that can forge it. Hand it to an auditor and you have handed them the power to fabricate the very record they were given it to check, so they cannot verify independently and you cannot prove you did not. Ed25519 splits those two powers — the install signs, everybody else verifies — and it ships in Node’s own crypto module, so it adds no runtime dependency to a product that gets deployed air-gapped.

An anchor is a short statement, canonicalised and then signed: install id, anchor sequence, previous anchor hash, the chain head being attested, entries covered, the protection level the chain itself had at signing time, the newest checkpoint sequence when the chain is keyed, the creation time, the key id, the algorithm and the cadence. Every field is inside the signature; there is no envelope the signer did not commit to. The stored columns a dashboard or a SQL query displays are checked against the signed statement, and a disagreement makes the anchor untrusted rather than being quietly rendered — if they differ, the displayed values are unsigned. The creation time being inside the signed bytes is a deliberate contrast with the checkpoint, whose timestamp sits beside the MAC and is therefore editable without invalidating anything. The link hash between anchors is deliberately plain SHA-256 rather than the keyed audit digest, because an outside verifier holds the public key and nothing else: the signature is what makes each link unforgeable, the hash only has to be reproducible.

Anchors are chained to one another and numbered contiguously, which is what turns a deleted anchor into a visible break rather than a re-numberable hole. Two validly-signed anchors sharing a sequence with different heads are a self-contradiction the install signed itself into — both bear its key, so it cannot disown either — and that is checked first and reported as equivocation, because reporting it as a broken link would send an auditor looking for a deletion that did not happen. Later anchors may not attest a shorter chain or fewer entries than earlier ones. Anchoring also refuses to sign at all when the chain does not verify, when the head has moved backwards, or when an already-anchored entry no longer matches the hash that was attested. That is the opposite of a retry, and it is the feature: signing over a forged head would launder the rewrite under a key the auditor was told to trust, while not signing leaves the previous anchor standing — and that anchor still contradicts the rewrite. Every refusal is logged at error level and returned by the manual endpoint as a 200 carrying the status and the reason, because an operator pressing the button on a broken chain needs to be told exactly why it refused, not handed a 500.

Changing the anchor signing key produces one dual-signed transition. The first anchor under the new key embeds a canonical transition statement, under its own independent signing domain, that both the retiring and the incoming key sign; the incoming key then signs the containing anchor. It binds the unchanged install id, the sequence and previous-anchor link, the attested head, entries covered, protection and checkpoint state, the asserted time, the cadence and both key ids, so the hand-off is independently checkable without rewriting an old anchor or trusting a key carried beside the evidence. Verification treats the current key plus the supplied previous keys as one ordered current-to-oldest ring: a complete ledger must begin under the oldest key, move one position at each dual-signed transition, never return to a retired key, and finish under the current one. The retiring private key exists only for that one-shot transition, and the product refuses to start if it is left configured afterwards.

Publication is restart-safe and at-least-once. A durable per-destination cursor, keyed by a one-way digest of the destination rather than its raw path or URL, advances only after that sink has acknowledged that exact anchor, so a crash between acknowledgement and cursor commit replays one anchor and never skips one — a missing transition is replayed before any later anchor. The verifier accepts an exact duplicate and rejects a same-sequence duplicate whose displayed envelope was altered. HTTP receivers get the install id and anchor sequence as an idempotency key, and the contract is easy to get wrong in one specific way: a receiver must answer 2xx to an anchor it already holds, because a 4xx is treated as permanent, so a receiver that returns 409 on a duplicate wedges the pipeline at that sequence and every later anchor silently stops publishing while the local ledger keeps growing and looks healthy.

A compliance export is one bundle containing the traces and their events for the period, the approvals with approver identity, timestamp and rationale, the audit entries covering the governance-plane changes in that window and within the requester’s evidence scope, the chain verification result, and a SHA-256 digest over the canonical JSON of the bundle body, generated at a recorded time. Be precise about what that digest is: it is a seal, not a signature. It lets a recipient confirm the file is byte-for-byte the one whose digest they were given through some other channel, and anyone who can rewrite the bundle can recompute it. Durable origin evidence comes from the keyed chain plus an anchor you retained off the box, not from the export.

What makes the bundle more than an exported log file is the verdict travelling inside it. The chain verification block carries whether the chain is valid, how many entries were checked, the first broken sequence number if there is one, the protection level and the checkpoint the run verified against. A bare valid flag invites the reader to assume more than an unkeyed chain offers — it means nothing was altered without recomputing, not that nothing was altered — so naming the protection level is the difference between an auditor being informed and being misled.

The bundle is capped: at most 200 traces, 5,000 events, 1,000 approvals and 2,000 audit entries, over a default 30-day window. Traces, events and audit entries each set a truncation flag inside the file; the approvals cap is the one that does not, so for approvals the count in the bundle is what you have to read against the period you asked for. The console repeats the trace cap beside the download button and tells you to check the truncation flags before you send the file on, on the reasoning that nobody opens a sealed JSON file to check before forwarding it to a regulator.

Verification remains O(n) over the chain, which is a real cost and is managed rather than hidden. Request-triggered verification reads at most 512 primary-key-ordered rows per chunk and yields to the event loop between them, so gateway, readiness and kill-switch work still progresses and audit-row memory stays bounded independently of history size. One process-wide in-flight walk, a ten-second snapshot cache keyed on the storage engine’s cross-connection commit counter, and a ten-second completion-based cooldown stop a privileged caller turning the evidence check into overlapping event-loop work; throttled callers get a retry-after header. A commit by another connection during a walk invalidates the verdict rather than letting a mixed snapshot be cached or signed, and boot performs the full streamed walk separately, before the server listens. If that walk finds intrinsic corruption, the detecting request receives the finding — or the one evidence bundle carrying it — and Token Observe does not append an audit row onto the chain it has just proved unsafe; the verdict then latches, so readiness, later audit and domain writes and later governed requests fail with a 503 until the process is restarted, while authenticated audit diagnostics stay readable.

Erasure and retention do not break any of this. Traces and the audit log are separate tables and the chain covers governance-plane changes rather than payloads, so a retention pass or a data-subject erasure leaves verification passing, and the record that the deletion happened survives the deletion — with the erasure entry carrying a digest of the subject identifier rather than the identifier itself, so the erasure record does not become a new copy of the thing that was erased.