The AI vendor questionnaire
A questionnaire is a filter; it does not replace testing the product yourself
every questionOn this page
How to use this without turning it into a paperwork exercise
Three habits make the difference between a questionnaire that finds things and one that generates a folder nobody reads.
Ask for the artefact rather than the adjective. Encrypted, monitored, hardened and audited are claims with no failure mode; a report number, a defect list with dates, a threat model, a data-flow diagram and a licence file are objects that can be wrong in checkable ways. Where a vendor answers with an adjective, the follow-up is what would I look at to confirm that.
Ask what each control does when its input is missing, because that is where governance products fail silently. A budget with no price for the model that was served admits everything. A discovery tool whose evidence feed died in July shows an empty findings list that renders as a clean estate. A policy engine where every rule is in observation mode produces the same outcomes as one with no rules at all. In each case the console looks identical to the healthy state, which is why the question has to be asked directly rather than inferred from a demonstration.
Ask for the defect list before you ask for the roadmap. A vendor who publishes their own outstanding issues — naming the file, the concrete cost and the attack that works — is showing you the same list their engineers work from, which makes the review about substance rather than about what was left out. Token Observe publishes one and states the trade plainly: it hands an attacker a starting point, and the corollary is that anything not on it is genuinely not known to them. That is a stronger claim than a clean assertion, and it is falsifiable, which is the point.
One framing to hold onto throughout. A control helps you evidence a clause; it does not make you compliant with it, and installing software confers no certificate on the installer. Every mapping table in this market is a list of the first kind being read as the second.
Section 1 — Assurance and certification (six questions)
This section exists to establish what has been independently checked, which for most AI vendors right now is very little. That is not disqualifying; presenting it as more than it is, is.
Token Observe’s answers, for the record: no SOC 2 report, no ISO 27001 certificate, no ISO/IEC 42001 certificate, and no independent penetration test, red-team engagement or third-party code audit of any kind. What exists instead is internal: a structured adversarial review, a STRIDE threat model, targeted testing of specific attacks including a deliberately forged audit chain and adversarial inputs against the scanner, and an automated release gate. That is a genuine amount of work and it is not an external test, so it is not presented as one. A pre-purchase penetration test by a prospective customer is expressly permitted by the licence, and the published defect list and threat model are offered as the place to start.
- 1. Which certifications do you hold, and may I see the report rather than the badge?
- A good answer names the standard, the scope, the auditor, the period and the exceptions. A badge on a website is not a report, and a report scoped to a corporate function is not a report about the product.
- 2. Has an independent party tested this product, and may I read the summary?
- Ask specifically about the product rather than the company. If the answer is no, that is workable — ask whether your own test is contractually permitted, and whether findings may be published.
- 3. Where is your outstanding defect list, and when was it last updated?
- The most informative question in the set. Look for named files, concrete costs and the attacks that work, including findings that were investigated and refuted. A sanitised list is worse than none, because it is indistinguishable from a complete one.
- 4. What is your severity scale and your patch target for each level?
- Look for targets measured from a triage verdict rather than from a report, and for a scale that rates a bypass of the control itself above what a generic score would suggest. Ask what the supported-version policy actually is; pre-1.0 products often support only the latest published minor.
- 5. What ships with a release so I can verify rather than trust?
- A software bill of materials, checksums over every artefact, an immutable image digest, and the full test gate re-run on the release tag. Ask whether the published artefact is the one that was tested, and how you would confirm that.
- 6. What is your disclosure policy, and is there a gag clause?
- Look for a stated deadline from triage, credit at the reporter’s choice, and an explicit right to publish assessment and benchmark results. A licence that requires pre-approval of security findings is a licence that decides what you may tell your own board.
Section 2 — Does the control actually refuse (seven questions)
Most products in this market can describe. The question is whether anything refuses, whether the refusal is in the request path, and whether you can find out what a rule will do before it does it.
Token Observe’s answers: the decision is a single pure function evaluated inline before egress, returning allow, block or require-approval plus a redaction plan; the order — halt, lifecycle, permissions, ceilings, policy — is encoded rather than documented; every rule can run in observation mode with its would-be decision recorded on the trace; and a candidate rule can be replayed against recorded traffic to report what it would have changed. What it cannot do is refuse a proposal it is never shown: an agent that routes neither its model calls nor its tool calls through the gateway is a discovery problem rather than a policy one.
- 7. Is the decision taken before the request leaves my network, or after?
- Inline and synchronous, or asynchronous and advisory. A control that annotates after the fact is reporting. Ask where in the path the decision sits and what happens to the request while it is being taken.
- 8. Can it refuse one specific action for one specific agent?
- Action-level rather than system-level, denying by default. A product that can only allow or deny a whole integration will hand over the refund endpoint along with the order lookup.
- 9. What are the possible outcomes besides block?
- Block, park on a named human, redact, warn, stop the agent. A single-verb engine forces every rule to be an outage or a log line, and teams respond by writing no rules at all.
- 10. How do I learn a rule’s false-positive rate before it stops someone’s work?
- An observation mode that evaluates exactly as enforcement would and records the decision, plus retrospective replay against recorded traffic. Ask what the replay reads — prompt text or only decision inputs — because the answer is a privacy question as well as an accuracy one.
- 11. Is promotion to enforcement a separate, attributable act?
- Look for a named person, an audit entry, and optionally a gate requiring an acknowledged replay of that exact rule. Ask whether editing an already-enforcing rule is blocked by the gate; if it is, operators will turn the gate off.
- 12. Does delegation between agents intersect or accumulate?
- The single highest-value question about a multi-agent product. Union across a chain is privilege escalation by architecture — a low-privileged agent obtains a grant by asking a higher-privileged one. Ask what happens to a hop that cannot be resolved.
- 13. What is a human approval bound to?
- The exact payload and its execution context, single-use and expiring — or a category, in which case it is a standing licence for everything the agent proposes afterwards. Ask what invalidates it and what does not, because a binding that includes transport identifiers forces re-approval on every reconnect.
Section 3 — Evidence integrity (seven questions)
This is the section where language does the most damage, because the words are close together and mean very different things. Tamper-evident, tamper-proof, signed, sealed, immutable and append-only are routinely used interchangeably by people who have not thought about which adversary they are talking about.
Token Observe’s answers, stated in the same register as its claims: the audit log is hash-chained, so any edit or deletion that does not recompute every downstream digest breaks verification at a named sequence number. Under the default configuration those digests are plain SHA-256, so an operator with write access to the database can rewrite an entry, recompute the chain and have verification report valid — there is a test in the repository that does exactly this on a default install, so the limit is a tested fact rather than a caveat. Keying the digests requires a key held outside the database and a two-boot ceremony; anchoring signs the head with Ed25519 and publishes it off-box. Evidence exports are digest-sealed, not signed. None of this is tamper-proof, and the product does not use that word.
- 14. Who is the adversary your integrity story defends against?
- If the answer is not somebody with write access to your own database, the story is about accidents rather than insiders. Ask them to say which alterations are detected and which are not.
- 15. Is verification keyed, and what is it by default?
- An unkeyed hash chain is tamper-evidence against alteration that does not recompute the chain. Ask what the default is, whether the export says which state it is in, and what enabling the key requires operationally.
- 16. Is an export signed, or sealed?
- A digest lets a recipient confirm the file matches a value they were given through another channel; anyone who can rewrite the file can recompute it. A signature proves origin. Vendors say signed for both, and the distinction is the difference between informing an auditor and misleading one.
- 17. Can an auditor verify without being handed the ability to forge?
- A shared key that verifies is a key that forges. Ask whether there is an asymmetric attestation, whether the public key is distributed out of band, and whether the verifier will refuse to print a pass without one.
- 18. What does the record cover before you turned the protection on?
- Both keying and anchoring are forward-looking. Ask what happens to history written earlier, and beware an answer that claims to retroactively authenticate it — that is the act the mechanism is supposed to prevent.
- 19. What does the export contain, and does it have caps?
- Ask for the caps by number, ask whether a truncation flag is set inside the file, and ask whether every cap is visible on screen at the moment of download. A partial bundle forwarded as complete is a procurement problem you can prevent in one question.
- 20. Does the evidence survive a restore, and how would I know?
- A restore that lost its tail also lost the attestation of that tail and verifies clean against its own stored state. Ask whether verification tells you what it compared against, and whether an externally recorded head is supported.
Section 4 — Data handling (seven questions)
For an inline control this section is unusually consequential, because the product sees every prompt and every tool argument before anything has decided whether they were allowed to leave.
Token Observe’s answers: the product is self-hosted and bring-your-own-key, with no telemetry, no phone-home, no licence callback and no hosted component, and that claim is offered as something to verify in about five minutes rather than to accept. Prompts survive as a bounded post-redaction excerpt; model answer text is not stored. Detection is regular expressions plus checksums for a fixed set of kinds, so free-text personal data and identifier formats outside the shipped UK and US set are not detected at all. Trace retention is unset by default and unset means keep forever. Erasure reaches traces and not approvals, discovery findings or webhook deliveries, and reaches the live database rather than backups.
- 21. What does the product send to you, the vendor, and how would I check?
- Ask for the list of outbound call sites and hard-coded hosts, then run it with egress restricted and see what breaks. An answer that cannot survive that test has a destination it did not mention.
- 22. What is stored from the payload, and for how long by default?
- Ask whether prompts and answers are stored whole, excerpted or not at all, whether the excerpt is taken before or after redaction, and what the default retention is. Keep-forever and thirty days are both defensible; not knowing which you have is not.
- 23. What exactly does the detector recognise?
- Ask for the list of kinds and the method. Pattern-and-checksum detection does not see free-text personal data and does not know identifier formats it was not written for. A vendor describing this as data-loss prevention rather than a compensating control is overclaiming.
- 24. What happens on a streamed response?
- The hardest case, and the one most implementations get wrong. Ask when the response-side decision is taken, how deep the hold-back buffer is, what happens to a value split across chunks, and how a blocking rule refuses once the status line is spent.
- 25. What does erasure actually reach?
- Ask for the tables by name. Approvals, discovery findings, delivery records and identity snapshots are commonly outside the retention window, and backups are almost always outside erasure. The accurate phrase is erased from the live primary.
- 26. Which sub-processors are involved in the runtime?
- For a self-hosted product the honest answer is usually none in the runtime and a separate question for support and professional services. Ask them to distinguish the two rather than answering only the flattering half.
- 27. What leaves the boundary that I might not expect?
- Metrics endpoints that are unauthenticated by convention and expose identifiers and spend; webhook summaries that embed business content; log shipping. Ask what each carries rather than whether it exists.
Section 5 — Failure, availability and operations (six questions)
An inline control is a dependency of every agent behind it, which makes this section a reliability review rather than a formality.
Token Observe’s answers: it fails closed on every governance-bearing failure, including refusing governed requests with a typed unavailable error and latching readiness when it detects corruption in its own evidence chain. It is one process with one write path and its own database file; a PostgreSQL adapter exists as an evaluation alternative and is explicitly not a supported high-availability topology; there is no multi-replica claim, no point-in-time recovery claim and no vendor-operated service level. A request under a hard spend ceiling gets at most one potentially billable network attempt, which means no retry and no failover on that call.
- 28. Does it fail open or fail closed, and can I choose?
- Fail-open is a control an attacker can switch off by making it unavailable, and it is off during the incident it was bought for. If a vendor offers the choice, ask what the default is and what the console shows while it is open.
- 29. What is the supported topology, in one sentence?
- Single process, replica set, managed service. Then ask what state is process-local — circuit breakers, throttles, sessions — because that is what decides whether the second replica is safe.
- 30. What is the recovery story, precisely?
- Full snapshot or point-in-time. Ask for the drill: what is restored, what is compared against, and what the success criterion is. The process starting is not the criterion that matters.
- 31. What availability commitment exists, and from whom?
- For a self-hosted product the answer is usually none, and the operational consequence is yours. Ask them to say so plainly rather than pointing at a support response target as though it were an uptime commitment.
- 32. What happens to my agents when the control plane is upgraded?
- Ask about the upgrade path, whether it is in-place, what a rollback looks like, and whether any migration is one-way. Then ask who owns the written answer to what happens to traffic while it is down.
- 33. What does the product do that could make a request slower?
- Inline scanning on attacker-controlled text is the usual answer. Ask what bounds it, whether exceeding a bound fails closed or forwards the uninspected remainder, and whether they have measured adversarial inputs rather than typical ones.
Section 6 — Commercial, legal and the compliance table (eight questions)
The last section is where the overclaiming concentrates, because a mapping table is cheap to produce and expensive to check.
Token Observe’s answers: the published licence is a template drafted by the engineering team rather than by a lawyer, is marked as requiring review and approval by counsel before it is relied upon, and carries unfilled placeholders and a liability cap that has to be checked against insurance cover. It expressly permits inspection, testing, fuzzing, penetration testing and reverse engineering on infrastructure the licensee controls, permits publication of benchmark results identifying the version and configuration, and permits publication of security findings after coordinated disclosure — with the stated intention of being consistent with the vendor’s own practice of publishing its defects rather than suppressing yours.
- 34. Has your licence been reviewed by a qualified lawyer?
- An unusual question and a fair one for an early-stage vendor. A template honestly labelled as pending review is a workable starting point; a template presented as executed terms is a procurement risk your own counsel will find later.
- 35. May I test the product, and may I publish what I find?
- Look for an explicit right to test on infrastructure you control, an explicit right to commission a third party, and no pre-approval requirement for benchmark or assessment results.
- 36. What is your compliance table claiming?
- Read every row as this feature helps evidence that clause and check the vendor agrees. A row that reads as makes you compliant is the overclaim to push back on, and the honest tables say so at the top.
- 37. Which obligations remain mine?
- Risk classification, impact assessments, choosing and defending a retention period, informing workers, notifying authorities. A vendor who cannot list these has not read the regulation they are mapping to.
- 38. What is measured, and what is modelled?
- Cost figures computed from provider-reported usage against price rows you hold will not match a vendor invoice, and savings figures are retrospective models. Ask which numbers are which, and whether the modelled ones carry their assumptions in the payload.
- 39. Which features are preview, and what does preview mean here?
- Ask for the list and the specific limits of each. Token Observe publishes its subscription-seat surface as preview with four named gaps, including an unsigned executable and a freshness bound that depends on the device clock.
- 40. What happens to my data and my evidence at the end of the term?
- For a self-hosted product the data was always yours, and the question becomes what you must stop using and what you may keep. Read the survival clause and check that your own records are not caught by a deletion obligation.
- 41. What would you tell me not to use this for?
- The single best closing question. A vendor with a real answer has thought about their boundary; a vendor with no answer has either not thought about it or has decided not to tell you.
The four answers that should end the conversation
Tamper-proof, said about a database the vendor also writes to. There is no such property. Hash chaining catches alteration that does not recompute the chain; keying moves the requirement to a key; an off-box signature lets somebody else hold the proof. Each is real and each is narrow, and a vendor who collapses them into one word has either not thought about the adversary or is hoping you have not.
A compliance mapping table with no caveat at the top. Every honest table in this market opens by saying that these entries mean this feature helps evidence that clause, and that deployer obligations remain the deploying organisation’s. A table without that sentence is being written to be quoted in a procurement document.
We have no known issues. Either the list exists and is not being shown, or nobody is looking. Both are worse than a long list with dates against it, because an unrecorded gap is indistinguishable from one nobody found, and the second kind is the one that surprises everybody in production.
Any number without its method. A detection rate, a saving, a latency figure or a coverage percentage that arrives without the corpus, the configuration, the date and the assumptions is not a measurement. The follow-up is always the same: what would I run to reproduce that, and what would make it come out differently.
How to put the AI vendor questionnaire into practice
- 01
Send the sections that match your risk, not all forty-one
A shorter questionnaire that gets answered carefully beats a complete one that gets a template response. Pick the sections where a wrong answer would actually change your decision. - 02
Ask for the defect list and the threat model before the demo
Read both first. They tell you what the product is for and where it breaks, and they turn the demonstration into a set of specific questions rather than a presentation. - 03
Test the empty states yourself
Remove a price row and see whether the budget still admits requests. Stop an evidence feed and see whether the console still shows a clean estate. Put every rule in observation mode and see whether anything says so. - 04
Read the export rather than the export button
Take a bundle, recompute its digest, check the truncation flag, and read the field that says what the verification was compared against. Most of the honest answers in this market are inside the artefact. - 05
Have counsel read the licence, not the datasheet
Check the testing and publication rights, the liability position, the survival clause, and whether the document has been reviewed by a lawyer at all. - 06
Write down the residual risks and get them accepted by name
No certification, no independent test, single-node topology, preview features, unkeyed defaults. An accepted risk with a name and a date is a decision; an unmentioned one is a surprise waiting for an incident.
Where this argument meets an implementation
Audit chain
Every administrative act hash-chained; seal it under a key held off the box, and anchor it with a signature your auditor can check alone.
Agent registry
One record per agent, and it is the record the gateway enforces against.
Policy engine
One deterministic verdict on every governed request: allow, block, redact, or park it for a human.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
Those pages are one product's implementation of what this guide argues for; describe what your agents actually do and you will get a straight answer about whether you need any of it, including when the answer is no.
Talk it throughIs a vendor without SOC 2 or ISO 27001 automatically disqualified?
Not automatically, and treating it that way selects for vendors who can afford an audit rather than for products that work. What matters is whether the absence is stated plainly or papered over, and what exists in its place: a published threat model, a published defect list with dates and named files, a documented data flow, a security policy with severity targets, and a licence that permits you to test the product yourself. Token Observe holds none of those certifications and no independent penetration test, says so in its own documentation, and expressly permits a pre-purchase test — which is a different posture from implying an assurance that does not exist.
What single question separates a control from a dashboard?
Show me the last thing this refused. Not a demonstration of a rule firing on a hand-typed sample, but a real refusal in a real deployment, with the record it produced. The follow-up is nearly as good: how would I know if every rule in here were in observation mode. If the console renders that state identically to an enforcing one, the product cannot distinguish a control that is quiet from a control that is off, and that is the failure mode a governance layer cannot have.
How should a compliance mapping table be read?
As a list of features that help you evidence clauses, never as a claim about your compliance. The honest tables say this at the top, in the same words: a control is not a certification, and deployer obligations remain the deploying organisation’s. Then check the article numbers, because this is where specialist readers find errors — obligations addressed to the provider that built a system are routinely listed as though they were duties of the organisation deploying it, and conflating a data protection impact assessment with a fundamental-rights impact assessment is the commonest mistake of all.
What should I ask about a vendor’s own numbers?
The method, the corpus, the configuration and the date, every time. A detection rate with no corpus is not a measurement. A cost saving is a retrospective model and should carry its assumptions and a confidence split in the payload rather than in a footnote. A latency figure taken on typical inputs says nothing about adversarial ones. And a coverage percentage needs a denominator: an estate where forty agents route through the gateway and an unknown number do not is not fully governed, and the honest sentence needs a number on both sides.
Can I use this questionnaire against Token Observe?
That is what it is for, and the answers are in the sections above rather than in a footnote. The short version of where it comes off badly: no SOC 2, no ISO 27001, no ISO/IEC 42001, no independent penetration test; a licence that is an engineering-drafted template pending counsel; one process with one write path, no high-availability topology, no point-in-time recovery and no vendor-operated service level; an unkeyed audit chain by default, tamper-evident rather than tamper-proof, with a repository test that forges one successfully; exports that are digest-sealed rather than signed; heuristic detection with false negatives by construction and no media inspection at all; erasure that does not reach approvals, discovery findings or backups; a keep-forever retention default; and a subscription-seat surface published as preview with four named gaps.
Prefer to ask a person? Write to us →
Bring us the question this guide did not answer.
Write to hello@tenhaw.com with what your agents do, which providers they call and what would have to be true for you to put something in front of them. James Rooney replies. You will get a straight answer about whether Token Observe fits, including when it does not.
no form · no qualification step · no sales desk · the other three ways in