Policy as code for AI

A rule in Token Observe is five fields. A trigger says what it fires on. An action says what happens. A scope says which subjects it selects. A mode says whether it enforces or observes. A priority orders evaluation, lower first.

There are seven triggers and they are worth knowing individually because they are the vocabulary. A tool call, optionally filtered by a tool name pattern and by matchers over the argument tree — a dot path, one of eight comparison operators, and a value — which is what lets a rule say refunds over a threshold rather than refunds. A model request, optionally filtered by a model pattern or by an estimated input-token count. Spend over a threshold in a window. A rate over requests, tool calls or tokens in a window. A data class, naming the kinds of personal data or secret it fires on and the direction. An injection score above a minimum confidence, optionally restricted to the sources the finding came from. And a time window in which the agent may operate.

There are five actions: block, park on a named human, redact, warn, and suspend the agent. Having more than one verb matters more than it sounds: a single-verb engine forces every rule to be an outage or a log line, and teams respond to that by writing no rules at all.

Two details are the ones that silently disable a rule. The first is direction, on a data-class trigger, which is stated from the enterprise’s point of view: outbound means data leaving your control, which is the direction that matters for preventing disclosure, and inbound means data arriving back, which is the direction that matters for catching an injected instruction. Getting it backwards produces a rule that never fires and reports nothing, and the mistake is invisible in the console. The second is scope: an empty scope means global, and a scope that names an agent id, a team or a tag selects on any of them — with team matched case-insensitively and tag matched case-sensitively. That asymmetry is deliberate and worth knowing before you name things, because to a kill switch two spellings of a team are the same team, while to a policy two spellings of a tag are two different tags.

Three properties decide whether a replay report means anything, and each one is a place where a naive rendering turns an unmeasured result into a reassuring one.

Coverage is full, partial or none, and where it is none the counters come back as null rather than zero. The distinction is the whole point: null means the recorded traffic never carried the input this rule triggers on, so nothing was measured, while zero means it did and the rule would have changed nothing. Rendering the first as the second reports an unmeasured rule as a safe one, which is precisely the mistake a replay exists to prevent.

The denominator is the count of requests evaluated, not the count scanned. Requests refused before policy evaluation — a bad credential, a permission denial, an engaged kill switch, a delegation refusal — are counted separately as unreachable and excluded, because a rule cannot be credited with stopping a request that never reached it. A report that quotes the scanned figure as its base will understate the rule’s hit rate on the traffic it actually governs.

The earliest trace actually seen is reported, which is the only place a retention-trimmed window is visible. Asking for ninety days when retention keeps thirty produces a report about thirty days, and without that field it reads as a report about ninety.

One more property is worth checking in any implementation: whether the replay reads decision inputs or re-runs detection over stored payloads. The second is more accurate and much more invasive, and it changes the answer to who can run a replay from an operator to somebody who should have a reason to read the prompt corpus.

The last mile of policy as code is promotion between environments, and the interesting requirement is that a review pipeline should be able to see the change without being given authority to make it.

Token Observe’s shape is an export that emits a locked bundle carrying one hash per resource plus a digest over the whole bundle; a plan that imports a bundle and compares declared, locked and deployed state, optionally against an observed inventory in a portable software-bill-of-materials format; and a promote that emits a target-environment artefact only if the reviewed source digest still matches. Promotion never applies changes implicitly. That separation is what lets a review process read the artefact without a build job holding write authority over the live registry.

Two properties of that comparison are worth copying. Import rejects duplicate resources, changed locks, unknown kinds and credential-shaped fields, and keeps URL paths, query strings and credential-shaped metadata write-only — so a bundle is a state and drift artefact rather than a backup of your secrets. And a component in the observed inventory whose identity matches but which carries no configuration hash is reported as unverified rather than in sync, because identity alone is not configuration evidence. That single rule is the difference between drift detection and a green tick.

Compiling to another engine deserves a caveat in the same register. Token Observe can emit a policy-matching artefact with witnesses for an external evaluator, and it lists every source policy it rejected or could not represent — which is the honest half. The caveat is that the emitted artefact covers policy scope and trigger matching only. It is not a replacement for the permission model, the ceilings, the approval workflow, the kill switch or the effect authority, all of which live in the enforcement point rather than in the rule. A compiled rulebook that is treated as the whole control is a rulebook that has quietly shed four-fifths of the enforcement it was part of.