Shadow AI radar

Shadow AI discovery is sold as a detection problem and it is mostly a reporting-integrity problem. Any tool can produce a list of findings; the hard part is the sentence underneath the list when it is empty. A console that renders no findings as a green banner has made a claim about your estate, and on most days it has no idea whether it is entitled to. The export could have stopped three weeks ago. The credential it arrives on could have been revoked this morning. The exporter could be posting on schedule into a mapping that reads none of it. All four situations emit exactly the same bytes as an estate where nobody is bypassing the gateway.

That failure is not hypothetical and the product records the version of it that it shipped. Arrival used to be recorded as a batch arriving, which meant an exporter delivering an empty page every hour held its source at fresh indefinitely — with a row count of zero stored beside the claim, where no state machine read it. A wrong window, a page that came back empty, a permission quietly downgraded to one that returns nothing: those are the commonest ways a real exporter fails, and each of them is worse than having no feed at all, because coverage was affirmatively asserting freshness over the top.

The second half of the problem is that freshness alone is always late. The bound a feed may go quiet for has to be measured in cycles of its own export — a vendor master bill is published once a calendar month, so forty-five days of silence is the first gap that unambiguously means somebody stopped exporting. Which means a credential revoked this morning leaves its source reading fresh until September. The connector’s own verdict is available today. Reporting the later of two true answers when the earlier one is already in hand is the same class of lie as the green all-clear, just slower.

So Token Observe separates the three questions and reports them side by side rather than folding them into one number. Was anything found. Has this source been fed, and how recently. Is anything knocking, and what happens when it does. A surface showing only the first is a findings list; a surface showing only the first two is a findings list with a slower disguise.

Coverage records two facts per source — when evidence last arrived, and how long that source may plausibly go quiet — and computes one of five states from them. Only connected_fresh entitles a surface to render an unqualified all-clear. The other four each describe a different thing being wrong, and folding any of them into a neighbour would either produce the lie the model exists to remove or send an operator hunting a connector that is working perfectly.

The distinction the whole model turns on is between a delivery and a row. A delivery proves the connector is alive. A delivery carrying at least one row proves the estate was observed. Those are different facts about the world and they are kept in different columns, because they were once the same column and the consequence was a source held at fresh indefinitely by an exporter delivering nothing. An empty delivery is not meaningless either — a genuinely quiet hour on an egress feed is a real answer, and calling a working exporter silent is how a status surface gets muted, after which it reports nothing at all. So an empty hour keeps the connector alive and moves no observation clock, and connected_empty says both halves out loud.

The bounds are a product judgement rather than a constant somebody picked, and the rule behind every number is roughly two expected cycles, rounded up so a jittery cron does not flap. A vendor master bill and a per-user seat export are published once a calendar month, so the natural gap is already about thirty-one days at its widest; forty-five days is that plus a fortnight, which is the smallest gap that unambiguously means somebody stopped exporting. Egress is six hours rather than twenty-four for one reason: it keeps the detection window inside a working day, and it is the only source that can catch an exfiltration channel while it is still open. One bound across both would either scream about a healthy billing feed every month or stay silent about an egress feed that died at breakfast.

Two smaller decisions are worth knowing because they change what you read. A timestamp that cannot be parsed is reported as stale rather than fresh — corruption makes somebody look, it does not reassure them. And an arrival dated in the future, from a connector with a skewed clock or a restored database, is treated as age zero rather than as a negative age: the alternative arithmetic makes a badly skewed feed look arbitrarily stale and sends an operator hunting a connector that is working.

The vendor master bill runs first because it is ground truth for what was spent, and because no other detector can find usage that left no network, IAM or endpoint trace at all. Vendor lines for a calendar month are compared against the gateway’s own metered spend for the same month, with two tolerances applied together — a gap must exceed both one dollar and five per cent of the larger side — because rounding, currency conversion and mid-month proration make small gaps meaningless. Severity is taken as the worse of a ratio reading and an absolute reading, so a ninety per cent gap on twelve dollars and a six per cent gap on forty thousand are both reportable and neither hides behind the other. Over-metering is reported and capped: it is a price-table problem that should reach an engineer, not the risk register.

That reconciliation refuses to guess a join, and the refusal is the most interesting decision in the detector. Vendor lines are keyed by invoice name and account; metered lines are keyed by the provider name an operator registered, which is free text. Matching them by string would report every provider whose registered name differs from its invoice name as unmetered spend — the most severe finding the product emits — for money that was metered perfectly. So per-provider attribution happens only where you have declared the binding, an account claimed by two provider rows falls back to the month-level aggregate rather than being resolved by list order, and a bound provider under which nothing was metered at all is reported as unverifiable at medium rather than as an unmetered critical, because a renamed provider row is the usual cause and historic traces keep the name they were served under.

Per-user seat spend reads the same kind of evidence one named person at a time, and it is the first radar shape whose subject is an employee, so what travels where is deliberate. The finding detail carries the address, the months, the amounts and the internal user id where one matches, because there is no version of this feature that withholds them — somebody spending four thousand dollars a month outside the gateway is not actionable otherwise. The summary and the dedup key carry no address at all: both leave the process into an alert, and an alert lands in a chat channel with a far wider readership than the console, so the key is the first twelve hex characters of a digest of the address, which is stable, unique per person and meaningless to a recipient. Severity reads the largest single month rather than the total, so a twelve-month export and a one-month export rate the same person the same way.

Egress, keys and IDE telemetry each answer a narrower question. Egress matches destinations against a fixed list of vendor API hostnames by exact match or subdomain, plus a shape match for regional Bedrock endpoints — substring matching would happily accept a hostname ending in a lookalike domain — and flows whose source is the gateway itself are excluded, since those are what the gateway is supposed to produce. The key audit reports three shapes that make a bypass durable: keys past the rotation window, keys the gateway has no record of using, and keys idle long enough to revoke outright, with the key itself as the identity — the provider, the account and a twelve-character digest of the key id, never the raw value, which is masked wherever it is stored — so that one credential ageing through three thresholds advances one finding instead of opening three. IDE telemetry recognises six coding agents by name and flags any whose configured base URL is not this gateway; an unset base URL means the tool falls back to the vendor’s own endpoint, which is the most common bypass there is and the one nobody notices because nothing looks misconfigured.

The fifth source needs no export from you and cannot go stale between them, because it reads the gateway’s own tables. Its strongest finding is the unpriced model: cost is computed at zero when no price row matches, so a model nobody registered is metered at zero dollars while the vendor bills for it in full, and that finding names by key the exact billing-gap findings its months explain rather than merely correlating with them. It is the one case where governed traffic is invisible to governance. The predicate is narrower than zero cost alone, because a response served from the gateway’s own cache is also metered at zero on purpose: requiring input plus output tokens above zero excludes every cache hit without parsing a byte of the payload, and the test suite drives real cached responses through the gateway to assert they never raise a finding. The same source also reports credentials in daily use on agents nobody activated, active agents with no usable credential, and sustained pressure from callers presenting credentials the gateway refuses — a revoked key still arriving every day being a decommissioned integration that did not stop when its access did.

A finding is a lead to investigate rather than proof, and every one carries the evidence it was raised from. Its identity is the source plus a stable dedup key, and that pair names a condition rather than an observation, so a re-scan over a fresh export advances the existing finding instead of minting a second one beside it. Each pass is a set difference against what was true last time, which produces the transitions an operator actually acts on: new, recurring, escalated, de-escalated, regressed and cleared. Recurring is the overwhelming majority and is silent by design.

Closing a finding by absence is the operation that has to be hardest, because a wrongly resolved finding is worse than a stale one — it is a condition that was true, is still true, and is now filed as handled. Four things must hold. The run for that source reported completed, since a failed or timed-out run produced a partial picture, and “I did not see it” is not “it is not there”. The source is one where absence is evidence, which is only the self source: it reads the gateway’s own tables and is authoritative, while a scan of an operator-supplied export sees exactly the rows somebody posted, and an export that omits last month’s shadow key has not remediated it. The source actually ran in this pass, so that switching a detector off cannot quietly close everything it ever reported. And the detection side saw its whole population, because a condition that fell past a row ceiling is missing from the observed set for exactly the same reason a remediated one is.

Read access is split by what the payload carries rather than by importance. The findings list is auditor rank, because a finding carries workstation hostnames, staff usernames, service-account identifiers and, on a seat row, an employee’s email address — and that rank is a deliberate raise to match a promise the threat model had been making to buyers while the code said otherwise. The coverage report is viewer rank, carrying a typed error code and never a run’s message text, because it is the payload a console has to render beside nothing found, and a coverage report only privileged users can read is a coverage report the all-clear gets rendered without. Run history is auditor, because a parse failure quotes the row value that failed and a row value can be a person’s address; pull-connector runs are auditor for the neighbouring reason, that a vendor’s own text travels in a run’s message. The connector ledger is operator, because it carries a rejection reason with quoted values struck out. Rank is not the only gate on any of them: every radar surface also demands an organisation-wide evidence scope, so an account scoped to a single team is refused outright rather than served a partial estate.

Unattended scanning is off by default, and the schedule lives in the database rather than in a timer. The obvious implementation resets its clock on every restart, so a process redeployed every forty minutes with an hourly interval never scans at all and nothing anywhere reports it — the schedule looks enabled, the logs look healthy, and the radar is simply never asked a question. Instead the next due time is a column written when a run finishes, so a restart changes nothing. A wedged detector is closed as timed out against a per-source ceiling and cannot stop the other four; a run abandoned by a dead process is reaped rather than waited on forever; a source that keeps failing backs off exponentially with jitter, capped so that it still retries roughly hourly and a fleet that failed together does not synchronise its retries.