LLM data residency
No control can prove where a vendor actually served a request
every questionOn this page
Residency is three questions, and they have different answers
The word residency gets used for three separate problems, and a programme that answers only one of them will be surprised by the other two.
The first is processing: which company, in which jurisdiction, on which infrastructure, sees the prompt and the tool arguments. For agents this is almost entirely a routing question, because the payload leaves your network the moment a model call is made, and it leaves to whichever provider and region the routing decision chose.
The second is storage: where the record of that traffic lives afterwards. This is a deployment question rather than a routing one. A control that stores traces, approvals and an audit chain is holding a searchable corpus of prompts and tool arguments, bounded and redacted or not, and the jurisdiction of that store is the jurisdiction of that corpus.
The third is evidence custody: where the proof about the record lives. It is the smallest of the three in volume and often the most awkward legally, because the whole point of an off-box copy of an audit attestation is that it sits somewhere the party under examination cannot rewrite — which usually means somewhere else, and sometimes somewhere else geographically.
Answer all three explicitly. A statement that says data stays in the European Union, without saying which of the three it refers to, will be read as all three by whoever relies on it.
Where the payload goes: three independent constraints, applied before egress
Token Observe attaches three independent booleans to each agent rather than a single residency flag, because providers genuinely differ on each and a single flag overpromises. Zero data retention is one contractual posture. Not training on submitted payloads is another, and a provider may do either without the other. A serving region is orthogonal to both. Collapsing them into one setting means a request that needed only region pinning is refused for want of a retention agreement, or worse, a request that needed a retention agreement is served by a provider that merely happens to be in the right region.
The check itself is a filter applied wherever a provider might be chosen: a provider that does not satisfy the agent’s data policy is not usable, and unusable providers are removed from the primary target and from the fallback list before any of them is selected. That last clause is the one that most implementations get wrong, and Token Observe shipped the defect once — failover discarding the narrowed route — and records it. A fallback chain that forgets the constraint is a constraint that stops binding at exactly the moment the primary is failing.
The same filter applies to any substitution. Where cost-based routing is enabled and permitted for that agent, the substituted model is still resolved through the ordinary route rules, provider selection and data policy, because an optimisation that skipped them would be a hole in the control rather than a saving. And where the substituted model has nowhere to go, the caller gets the model they asked for rather than an outage — a cost optimisation must never take a request down.
When nothing satisfies the policy, the request is refused before egress with a typed error that distinguishes no provider satisfies this agent’s data policy from no enabled provider can serve this model at all. Two different problems with two different fixes, and conflating them sends an engineer to the wrong place.
agent dataPolicy: { requireNoTraining: true, requireRegion: "eu-west-2" }
candidate providers
prv_a region: eu-west-2 noTraining: true -> usable
prv_b region: us-east-1 noTraining: true -> filtered out
prv_c region: eu-west-2 noTraining: false -> filtered out
route rule matched: primary prv_b, fallbacks [prv_c, prv_a]
primary unusable; fallbacks filtered to [prv_a]
-> served by prv_a
if the filtered set were empty
-> ACP_ZDR_UNAVAILABLE before egress
(distinct from ACP_PROVIDER_UNAVAILABLE, which means
no enabled provider serves this model at all)The limit that matters: the flag is an assertion, and it is matched as a string
Setting a region or a retention posture on a provider row records your assertion about your contract with that vendor. Nothing in the software verifies it, and nothing could: a gateway sees an endpoint and a response, and a vendor’s internal routing is not observable from the outside. Token Observe states this as a residual risk requiring dated acceptance rather than burying it, and the mitigation it does offer is attribution — the assertion appears in the audit record with the person who set it, so the claim has a name against it.
The matching is deliberately literal. A region is compared case-insensitively for equality against what the provider row declares, which means the value you write has to be the value that actually constrains egress. Token Observe’s guidance for a cloud model service is to use the exact regional identifier because the declaration is bound to a specific endpoint and its signing scope, rather than a broad geographic label — a provider row saying europe proves nothing about where a request went, while one naming an exact region at least corresponds to an endpoint somebody can point at.
Two practical consequences follow. Choose your vocabulary once and write it down, because two spellings of the same region are two different constraints and an agent pinned to one of them will be refused by a provider declaring the other. And treat the provider row as a compliance artefact rather than a configuration detail: the person who edits it is making a claim on the organisation’s behalf, so the edit belongs behind the same approval discipline as any other control change.
The stronger version of residency, where it is available to you, removes the assertion entirely by removing the third party. A provider row can point at an inference endpoint you run — a self-hosted model server, or a cloud model service reached through a private endpoint inside your own network — in which case the region question is answered by your own infrastructure rather than by a contract clause. That is the only arrangement where the answer is checkable rather than asserted, and it is worth the comparison even where you conclude it is not worth the cost.
Where the record lives, and what each surface actually holds
Running the control in your own network settles the storage question by construction: the traces, the approvals, the audit chain and the discovery findings sit in a database file on your disk, and there is no vendor-operated component in the path. That is worth verifying rather than accepting, and the method is in the guide on self-hosting — enumerate the outbound call sites, enumerate the hard-coded hosts, then run the deployment with egress allowed only to your own providers and confirm nothing breaks.
Within that boundary, know what each surface holds, because they have different sensitivities and different retention.
Traces hold the decision record and a bounded excerpt of the prompt taken after redaction; the model’s answer text is not stored and the search index cannot contain what the redactor removed. Approvals hold approver identity, timestamp and rationale, and a short excerpt of the model-proposed tool arguments that is not redacted — which makes approvals the one surface where payload-adjacent text survives a trace purge. Discovery findings may carry staff usernames and workstation hostnames from evidence somebody supplied. Webhook delivery records keep the exact bytes that were posted, including summary text that embeds agent and policy names.
Two surfaces leave the database in ways worth naming in a residency assessment. Application logs go to standard output as structured JSON carrying method, route pattern, status, duration and identifiers, with no query string, no body, no headers and no prompt content at any log level — but they go wherever your log shipper sends them. And the metrics endpoint is unauthenticated by convention and exposes agent identifiers and month-to-date spend, which is why the reference deployment publishes on loopback and the guidance is to keep that endpoint inside the monitoring boundary rather than on a public hostname.
The complete egress list, and how to bound it
A residency claim is only as good as the list of places data can go, so the list has to be short enough to state. For Token Observe it is: the model providers you registered, the tool servers you registered, the webhook receivers you registered, your identity provider if single sign-on is configured, your anchor sink if anchoring is configured, and one public price catalogue that is fetched only when an administrator invokes it and carries no prompt, trace or identifier. That is the whole of it, and an install where none of the optional features is enabled reaches only the first three.
Two allowlists turn that from a description into a control, and both are worth narrowing on day one. The first names which hosts a provider may point at, defaulting to the public vendor endpoints — set it to your own internal endpoints for a self-hosted or offline deployment. The second names which environment-variable prefixes a credential may be read from, keeping namespaces disjoint so no integration can name another’s secret and none can name the session secret. Neither list may be empty, which matters more than it sounds: the natural reading of an unset list is any, and any is the vulnerability. Unguarded, a provider row naming a URL and an environment variable is a primitive that reads any secret in the process and posts it anywhere, in the one process that deliberately concentrates every provider key in the estate.
Then bound the client side of each destination. Refuse redirects rather than following them, so an upstream cannot bounce a credentialed request elsewhere. Put an explicit timeout on every call. Return stored URLs origin-only so a credential smuggled into a path or query cannot be read back out through the API that stored it. And treat webhook receivers as being inside your data boundary, because their payloads embed agent and policy names and, on one path, a bounded excerpt — an internal receiver is a better default than a public chat service for anything where the summary could carry business content.
The anchor sink deserves a specific note in a residency assessment because it is the one destination that exists to be outside your control. What it receives is a signed statement about the shape of the audit chain — the head sequence and hash, the entries covered, the previous anchor’s hash, the key id and the time. It contains no payload, no identifier and no personal data, which is what makes it publishable to a destination the database administrator cannot rewrite without that destination becoming a data transfer.
- Model providers
- The governed request after sanitisation and after the redaction plan is applied, with your key read from the process environment. Constrained by the host allowlist and by the agent’s data policy.
- Tool servers
- The tool name and arguments after policy evaluation and inbound redaction, with the per-server credential read from the environment. Registered destinations only, with a loopback-only default.
- Webhook receivers
- Identifiers, counts, policy names and a one-line summary, signed when a secret is configured. Inside your boundary, because summaries can embed business content.
- Anchor sink
- A signed statement about the chain head. No payload, no identifier, no personal data — which is precisely why it can be published somewhere you do not control.
Retention is the other half of residency, and its default is keep-forever
A jurisdiction question is rarely only about geography. Once the record is in the right place, the next question is how long it stays there, and this is where a residency programme usually discovers it has inherited a decision rather than made one.
Token Observe’s trace retention is unset by default, and unset means keep forever. That over-satisfies a minimum-retention duty of the kind the EU AI Act places on deployers and satisfies no storage-limitation duty at all, so a deployment handling personal data has to set it — and somebody with a legal qualification has to decide whether the obligation applies and what period satisfies both directions. The default is deliberate in both directions: retention has to be decided rather than assumed, and an upgrade that silently started deleting a customer’s evidence would be the worse failure.
Once set, an hourly pass ages traces out in small batches, each its own short transaction so it interleaves with gateway traffic rather than stalling the request path, and a status endpoint reports the window, the exact cutoff instant, how many traces are currently eligible and the outcome of the last pass — so the purge is configured and the purge is running stay separately checkable.
Now the sentence that makes a retention statement accurate. The window acts on traces and everything hanging off them: their events, the search index and their scores. It does not touch the audit chain, by design, which is what lets the record of a deletion outlive the deleted data. It does not reach approvals, which are not purged and cannot be erased by subject. It does not reach discovery findings or webhook delivery records. Identity group snapshots age out on their own authority-aligned window, and gateway caller sightings on another window again, neither inheriting the trace setting. A policy that says we delete after ninety days without naming those is inaccurate, and a data protection officer will ask.
The last boundary is backups, and it is the one that most often turns a confident statement into a qualified one. Retention and erasure act on the live database. A retained snapshot preserves whatever existed when it was taken, so restoring a pre-erasure snapshot can resurrect erased records and can lose the audit row that recorded the erasure. The precise claim is erased from the live primary, and a complete answer needs the separately agreed backup retention window, destruction evidence when it closes, and a controlled procedure for reconciling deletions in a lost interval before traffic returns.
How to put LLM data residency into practice
- 01
Write down which of the three questions you are answering
Processing, storage and evidence custody have different answers. A statement that says the data stays in one jurisdiction, without saying which, will be read as all three. - 02
Declare the constraints per agent, not per estate
Retention posture, training posture and serving region as three independent settings, because providers differ on each and a single flag either over-refuses or under-constrains. - 03
Check that the constraint survives failover and substitution
Filter the fallback chain and any cost-based substitution with the same rule as the primary. A constraint that stops binding when the primary fails is a constraint that is absent during the incident. - 04
Confirm the refusal is typed and distinct
No provider satisfies this agent’s policy is a different problem from no enabled provider serves this model. Conflating them sends the engineer who reads the error to the wrong place. - 05
Fix the region vocabulary and put a name against each assertion
Exact regional identifiers rather than broad labels, one spelling across the estate, and an audited edit so every declared posture has an accountable person behind it. - 06
Narrow both egress allowlists to your own endpoints
Which hosts a provider may name, and which environment-variable prefixes a credential may come from. Neither may be empty, and narrowing them is what stops a registry write becoming a read of every secret in the process. - 07
Decide the retention period, and enumerate what it does not cover
Set the window, then write down the surfaces outside it — approvals, discovery findings, webhook deliveries, identity snapshots, caller sightings — and the backup boundary, before anyone quotes a number.
Where this argument meets an implementation
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Agent registry
One record per agent, and it is the record the gateway enforces against.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
Audit chain
Every administrative act hash-chained; seal it under a key held off the box, and anchor it with a signature your auditor can check alone.
Those pages are one product's implementation of what this guide argues for; describe what your agents actually do and you will get a straight answer about whether you need any of it, including when the answer is no.
Talk it throughWhy three separate flags instead of one residency setting?
Because providers differ on each of them independently. A vendor may contractually retain payloads without training on them, or train without retaining, and a regional serving commitment is orthogonal to both. A single combined flag either refuses a request that only needed region pinning, or admits one to a provider that happens to be in the right region without the retention agreement the agent actually required. Token Observe keeps zero data retention, no training on payloads and a serving region as three booleans on the agent and matches each against the corresponding declaration on the provider.
What happens when no provider satisfies an agent’s data policy?
The request is refused before egress with a typed error that specifically means no provider satisfying this agent’s data policy can serve this model, distinct from the error meaning no enabled provider can serve the model at all. It fails closed rather than downgrading quietly to a provider that does not satisfy the constraint, which is the property that makes the setting a control rather than a preference. The two errors are separated because the fixes are different: one needs a provider row with the right declarations, the other needs a provider that serves the model.
Can the gateway prove that a provider actually served from the declared region?
No, and no gateway can. The region, the retention posture and the training posture are operator-declared assertions about a contract with that vendor; the software enforces them as a routing constraint and cannot verify that the vendor honoured them. Token Observe records this as a residual risk requiring dated acceptance, and offers attribution rather than verification: the assertion appears in the audit record with the person who made it. The only arrangement where the answer is checkable rather than asserted is pointing the provider row at an inference endpoint you operate.
Does an anchor published outside our network create a data transfer?
The anchor itself carries no payload, no identifier and no personal data — it is a signed statement of the chain head, the entries covered, the previous anchor’s hash, the key id and the time. That is deliberate, because the whole value of the destination is that it is somewhere the party running the database cannot rewrite, which usually means somewhere outside the deployment. A file sink writes the same statement to local disk and makes no network call at all, which is the option where an off-box copy has to stay inside a boundary; the trade is that a local file is only as independent as the volume it sits on.
What does a retention period actually cover?
Traces and everything hanging off them: their events, the full-text search index and their scores. It does not touch the audit chain, deliberately, so the record that a deletion happened outlives the deleted data. It does not reach approvals, discovery findings or webhook delivery records, none of which are purged. Identity group snapshots and gateway caller sightings each age out on their own separate windows. And none of it reaches retained backups, so the accurate phrase is erased from the live primary rather than gone from every copy. A retention statement that omits those is not a retention statement a data protection officer will accept.
Prefer to ask a person? Write to us →
Bring us the question this guide did not answer.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in