Shadow AI discovery
On this page
What you are actually looking for
Shadow AI is not one thing, and the reason discovery programmes stall is that they start by looking for the most visible version of it. The employee pasting a customer record into a consumer chatbot is real and is mostly a data-loss and acceptable-use question. The version that matters for governance is quieter: model traffic and agent activity inside your own systems that never touches the controls you built.
It arrives in four recognisable shapes. A team that stood up an agent with a provider key straight from the vendor console, because that was faster than getting a gateway credential. A service account created for a proof of concept two quarters ago that is still running in production because the proof of concept worked. A coding assistant on a developer laptop pointed at the vendor default, which is what every one of those tools does out of the box. And a governed agent calling a model your price table does not know, which is the strangest case of the four: the traffic is going through your controls, and your controls are metering it at nothing.
The reason to care is not tidiness. Every governance claim you make has a denominator, and ungoverned usage is what makes the denominator unknown. An estate where 40 agents route through the gateway and an unknown number do not is not governed; it is partly governed by an unmeasured fraction, and the honest form of that sentence needs a number on both sides. Discovery findings belong in a risk register for that reason rather than in a backlog.
The discipline is reconciliation rather than detection. Nobody is going to give a governance layer a network tap, and it should not ask for one — a system that also held read access to the finance system, the flow logs and cloud identity would be a far more attractive target than the thing it protects. So in Token Observe every detector is a pure function of rows an operator supplies or a scoped feed delivers, and by default the layer holds no credential into any other system. State the exception rather than rounding it off: two optional pull connectors do hold a vendor credential — an organisation-scoped GitHub app for coding-assistant seat spend, and a reporting-read API key for a DNS and proxy activity feed — and both are off until somebody turns them on. The default connection points inwards, which is both the shorter security review and the smaller blast radius. The consequence is honest and belongs in the interface: a finding is only as current as the export it was computed from.
The five sources, and why they are in this order
They are ordered by signal rather than by convenience, and the order matters when you are deciding what to wire up first with limited time.
The vendor’s master bill runs first because it is ground truth for what was actually spent. Anything it shows that your gateway never metered is usage your gateway never saw, and no other source can find spend that left no network, identity or endpoint trace at all. Reconciliation needs a tolerance — rounding, currency conversion and mid-month proration make small gaps meaningless, so Token Observe ignores gaps under 5 per cent or one dollar — and the severity should scale with the size of the gap in both proportional and absolute terms, because a 40 per cent gap on a small account and a 5,000-dollar gap on a large one are different conversations.
Network egress runs second because a workstation or a service talking directly to a model API is bypassing the gateway by construction. This is exact-hostname matching against the known vendor endpoints, plus a pattern for regional endpoints that carry a region in the hostname, and it has to exclude the hosts that are your gateway — including whatever load balancer or terminator name it is actually reached under, which the process cannot know on its own and which an operator has to name.
Provider key listings run third because long-lived, unattributed keys are the mechanism that makes bypass possible in the first place. A key that is unknown to your governance layer and was used this week is an active shadow channel; a key created 400 days ago and never rotated is a standing risk; a key issued and never used is either forgotten or leaked. These are slow-moving findings — nothing here is urgent in hours — but they are the ones that explain how the other findings happened.
Endpoint telemetry about coding assistants runs fourth because it catches the most common bypass and the easiest to fix. Those tools ship pointing at the vendor endpoint, so an unset base URL on a developer machine is not misconfiguration so much as the default state of the world, and the remedy is a settings change rather than an investigation.
The fifth source is your own tables, and it is different in kind: it needs no export, cannot go stale between deliveries, and is authoritative for the estate you do control. It is also where the most interesting single finding in the whole system lives — a governed model call that your price table cannot price, which is metered at zero while the vendor bills for it in full. That is not another sighting of a symptom; it is the mechanism behind a billing gap, which is why it is worth running after billing so the finding it explains already exists when somebody follows the pointer.
- Vendor billing
- One line per provider per calendar month. Compare against what you metered, ignore gaps under 5 per cent or a dollar, and scale severity by both ratio and absolute dollars.
- Per-user seat spend
- A separate feed with its own cadence — seat-billed vendors publish a figure per named person — reported under the billing source. A licence is tens of dollars a month; anything materially above that is metered usage on top of it, which is the part that can run away.
- Network egress
- Flow logs, proxy logs or a VPC export, aggregated. Match destinations exactly or as a subdomain of a known vendor host, never as a substring, or a hostname ending in your vendor’s name will match something that is not it.
- Provider key listings
- The vendor’s own account of its API keys. Age, last use and whether your governance layer issued it are the three fields that matter; unknown plus used-this-week is the combination that means a live channel.
- Endpoint telemetry
- Which coding assistants are installed on which workstations and what base URL each is configured with. An empty base URL is not missing data — it is the vendor default, which is the finding.
- Your own tables
- Unpriced models at the gateway, credentials in use for agents nobody activated, active agents with no key at all, and callers presenting credentials you reject. No export, no staleness, and authoritative.
Coverage is the headline, because an empty list has two meanings
The findings list cannot tell the difference between the egress export not having arrived since July and nobody talking directly to a model API. Both are an empty list, and an empty list is exactly what a console renders as a green all-clear. So coverage has to travel with every surface that can report a clean result — the findings query, the scan response and the executive summary alike — and no caller should be able to read zero findings from the API without also being handed which sources were silent when it said so.
Recording coverage properly needs two facts per source rather than one, and the second is the one everybody misses. A delivery proves the connector is alive. A delivery carrying rows proves the estate was observed. Those are different facts, and collapsing them produces a specific failure: an exporter delivering an empty page every hour holds its source at fresh indefinitely, with a row count of zero recorded beside the claim where no state machine reads it. That is the commonest way a real exporter fails — a wrong window, a page that came back empty, a permission quietly downgraded to one that returns nothing — and it is worse than no feed at all, because coverage is now affirmatively asserting freshness over it.
How long a source may go quiet is a product judgement rather than a constant somebody picked, and one bound across all sources is wrong in both directions. A vendor bill arrives monthly and is not late at forty days; a flow-log export runs on a schedule measured in minutes, and six hours of silence is many missed cycles and a whole working morning nobody was watching. The rule Token Observe uses is roughly two expected cycles, rounded up so a jittery scheduled job does not flap — because a status surface that cries wolf on a healthy feed gets muted, after which it reports nothing at all, which is the failure it was built to prevent.
Feed health is a separate input and it can only ever take a reassuring answer away, never hand one back. A connector that has stopped delivering makes its source not covered even while the last evidence is still inside its bound, because the connector’s own verdict is available on the day it breaks rather than at the end of the window. That gap can be enormous: a seat feed may go 45 days before silence crosses its bound, so a credential revoked this morning would leave that source reading fresh for six weeks while nothing at all was being collected. Read the two together or you will read one of them wrong.
Conversely, a feed nobody has configured is not a feed that has stopped. A source may be fed perfectly well by an operator exporting a file by hand, and demoting it because no automated connector exists would cry wolf at an estate doing the right thing the slow way. Report it beside the entry and change no state.
- connected_fresh
- Evidence carrying rows arrived inside the bound. The only state that entitles a surface to render an unqualified all-clear.
- connected_empty
- Deliveries are still arriving and none of them recently carried a row. The connector is alive and the estate has not been observed — two different facts, and this is the state that says both.
- feed_failing
- The evidence is still inside its bound and the thing that fetches it has stopped working. The earlier of two true answers, and the honest one to report.
- connected_stale
- Evidence has arrived before, and not recently enough. The dangerous state, because everything still looks configured: the connector exists, the credential exists, and the last scan succeeded over evidence that is now weeks old.
- never_connected
- Nothing has ever arrived. A clean result from this source is not a result at all.
billing 45 days one whole missed invoice cycle; a monthly bill is
not late at forty days
seats 45 days same cadence, separate feed, separate way of dying
egress 6 hours several missed cycles, and it keeps detection
inside a working day: this is the only source
that catches an open channel while it is open
service 10 days a key listing changes slowly; one missed weekly
accounts export plus slack
ide 48 hours endpoint inventories collect daily, and a fleet
export is exactly the thing that skips a weekend
self twice the scan interval — what makes it stale is the
scheduler stopping, not a connectorThe source that needs no export: your own tables
Four of the five detectors are reconciliation against somebody else’s system. The fifth reads what your governance layer already holds, which makes it the only one that cannot go stale between exports and the only one that can answer the shadow-AI question about the estate you are authoritative for.
It answers three questions. The first is whether governed traffic is invisible to governance: a model call that no price row matches is metered at zero while the vendor bills for it in full, so a model nobody registered is spending real money inside your own gateway. This is the strongest signal in the whole system, because it is the mechanism behind a billing gap rather than another sighting of its symptom.
The second is whether the registry and the estate agree: a credential in daily use belonging to an agent nobody activated or that somebody retired, an active agent with no key at all, a key issued and never used. None of those is by itself traffic that escaped the gateway, and a detector that scored them as though they were would flood the queue — so they are scored as gaps between the record and reality, which is what they are.
The third is who is knocking. Callers presenting credentials the gateway rejects are ordinary noise at low volume — a typo, a restarted client, a stale browser tab — and a signal at volume and over time. Token Observe requires at least five attempts spread across at least three distinct hours before it treats rejections as a condition rather than an accident, because a process is what distinguishes an integration that is still running somewhere from somebody fat-fingering a token. A revoked key still arriving every day is a decommissioned integration that has lost its access, which means the work it was doing either stopped silently or moved somewhere you cannot see.
One implementation detail from this detector generalises. A predicate of cost equals zero would sweep in every response served from the gateway’s own cache, which is metered at zero on purpose because no tokens were bought — so the predicate requires tokens as well, since the cache path writes zero tokens for the same reason. It is a small thing, and it is the difference between a detector that produces findings and one that produces a wall of false positives on the first busy day.
The write side of that third question needs a cap, and the reason is worth copying. Recording rejected callers is the only path in a discovery system that an unauthenticated caller can reach, so every rejected request would otherwise drive a database write — turning the detection table into an amplification target and making the write load a function of the attack rather than of the estate. Capping writes per time bucket means a capped bucket’s count becomes a floor rather than a total, and that is a limit worth stating rather than hiding.
Turning findings into work, without the queue eating itself
A finding should identify a condition rather than an observation. Give each one a stable key derived from what it is about, so re-running a scan against a fresh export folds into the finding that already exists — advancing its counters and leaving whoever is triaging it alone — instead of creating a duplicate. A condition that had been resolved and has come back should reopen the existing finding, because that is news.
Only clear a finding when the run that would have re-found it actually completed. A failed, timed-out or truncated pass has to leave every existing finding standing, or a scheduled scan that fails quietly starts silently resolving your estate. This is the same rule as the coverage rule wearing different clothes: absence of a re-detection is not evidence that the condition is gone.
Keep the scheduling state in the data rather than in a timer. If the next due time is a column written when a run finishes, a process that is redeployed every forty minutes with an hourly interval still scans; if it is an in-memory timer, that deployment cadence means the scan never runs at all and nothing anywhere reports it.
Think about who can read what. Discovery findings join evidence that often has no reliable team key — billing accounts, egress rows, workstations, service accounts — so scoping them to a team produces a partial view presented as an estate result. Token Observe refuses that: all human reads and mutations of the discovery surface require an explicit organisation-wide evidence scope, and it returns a refusal rather than a misleading fraction.
Machine ingest should be a separate, narrow realm. A scoped, revocable token that can submit only the streams named on it and cannot read findings, call a model or call a tool means an exporter running on some machine in your estate holds the least authority that will do the job. Show the token exactly once, list keys by identity and lifecycle rather than by secret, and make revocation a single action.
Then feed it with whatever you already have. Vendor bills exist in your finance system today; flow logs or proxy logs exist in your security stack today; a provider key listing is an export from a console somebody already administers; endpoint inventory exists in whatever manages your laptops. Two vendor connectors ship in Token Observe and both are off by default — a coding-assistant seat report and a DNS and proxy activity feed — but the operator-supplied path is the primary one, deliberately, because it does not require the governance layer to hold a credential into another system.
How to put shadow AI discovery into practice
- 01
Start with the bill, because it is ground truth
Export one line per provider per month from finance and reconcile it against what your gateway metered. Anything on the bill that your controls never saw is usage your controls never saw, and no other source can find it. - 02
Add egress, and tell it what your gateway is called
Feed a flow-log or proxy export and name every host and base URL that addresses your gateway, including load balancers and terminators. Without that, your own governed traffic reads as a bypass and the report is noise. - 03
Export the provider key listing
From each vendor console, with creation date, last use and description. Mark the keys your governance layer issued so the audit can tell governed keys from shadow ones; unknown plus used this week is the combination worth a same-day answer. - 04
Collect endpoint telemetry for coding assistants
Which tools are installed, on which machines, configured with which base URL. Treat an empty base URL as the finding rather than as missing data, because it means the vendor default. - 05
Read coverage before you read findings
For each source, check whether evidence carrying rows arrived inside its bound, and check the feed’s own health separately. A clean findings list from a source that has never delivered is not a result, and a fresh-looking source behind a broken connector will stay fresh for the whole bound. - 06
Triage by condition, not by observation
Work the findings as stable conditions that recur and reopen rather than as a stream of events, and confirm that a failed scan leaves existing findings standing rather than clearing them. - 07
Close the loop by governing what you found
Register the agent, revoke the shadow key, point the coding assistant at the gateway, add the missing price row. Then re-run and watch the finding clear for the right reason — a completed run that no longer detects it.
Where this argument meets an implementation
Shadow AI radar
Five evidence sources for AI activity that never touched the gateway, and a coverage model that refuses to call a dead feed a clean estate.
Endpoint seats
Policy enforced inside each vendor’s own administrator hook, decided offline against a signed bundle, because a hook that phones home fails open.
Agent registry
One record per agent, and it is the record the gateway enforces against.
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
Those pages are one product's implementation of what this guide argues for; describe what your agents actually do and you will get a straight answer about whether you need any of it, including when the answer is no.
Talk it throughWhy is an empty findings list not good news?
Because it has two causes that look identical. The egress export has not arrived since July, and nobody in your estate is talking directly to a model API, both render as no findings — and a console renders no findings as a green all-clear. That is why coverage has to travel with every surface that can report a clean result, and why exactly one coverage state entitles a surface to an unqualified all-clear: evidence carrying rows arrived inside the bound for that source. Absence of evidence is not evidence of absence, and a discovery tool that cannot express the difference will always be read as the reassuring one.
What evidence do you actually need to feed a discovery programme?
Four exports you almost certainly already have, plus your own gateway data. A vendor master bill per provider per month, from finance. A network egress, proxy or flow-log export, from your security stack. The vendor’s own listing of its API keys, from the console somebody already administers. And endpoint or device-management telemetry about which coding assistants are installed and what base URL each is configured with. None of that requires the governance layer to hold a credential into another system, which is deliberate: something that also held read access to finance and the flow logs would be a more attractive target than the thing it protects.
How stale can each evidence source be before it stops counting?
Roughly two expected cycles, rounded up so a jittery scheduled job does not flap, which means the bounds differ by source rather than sharing one number. In Token Observe: 45 days for a monthly vendor bill and for a per-user seat export, because a bill is not late at forty days; 6 hours for egress, which is several missed cycles and keeps detection inside a working day, since that is the only source that can catch an open channel while it is still open; 10 days for a key listing, which changes slowly; 48 hours for endpoint telemetry, because a fleet export is exactly the thing that skips a weekend. The self source is bounded by twice the scan interval instead, because what makes it stale is the scheduler stopping.
Can a healthy-looking coverage report still be wrong?
Yes, in one specific way, which is why feed health is reported separately. Coverage measures when evidence last arrived; it cannot see that the connector fetching it broke this morning. Inside the bound, a source with a dead connector still reads as fresh — and for a seat feed that bound is 45 days, so a credential revoked today would leave the source reading fresh for six weeks while nothing was being collected. The connector’s own verdict is available on the day it breaks, so a broken feed should take the reassuring answer away immediately. Read both, and never let a feed report hand back a state the arrival clocks did not support.
What is the most valuable finding a discovery scan produces?
Usually the one from your own tables: a governed model call that your price table cannot price. It is metered at zero while the vendor bills in full, which means the traffic is going through your controls and your controls cannot see it — and it is the mechanism behind a billing-gap finding rather than another sighting of the same symptom, so it turns a discrepancy into a fix. Second place goes to a provider key that your governance layer does not know about and that was used this week, because that is an active channel rather than a historical artefact, and the remedy is one revocation.
Prefer to ask a person? Write to us →
Bring us the question this guide did not answer.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in