Token Observe vs LLM observability
Match the tool to the question: what the provider charged, what an agent did, or what it may do next.
For the loop a developer works in, the tracing tools are the right tool.
The daily job of improving an agent is iterative and offline: run a dataset, compare two prompts, look at the trace tree of the run that regressed, score outputs against a rubric. LangSmith, Langfuse and Arize each publish material describing that loop, and Datadog publishes LLM Observability inside a platform your operations team may already run. Token Observe does none of it: evaluation harnesses and session replay were designed for in the data model and deliberately not implemented.
The second advantage is architectural, and it is an argument against buying Token Observe. A tracing SDK is out of band: if the exporter fails, the agent keeps working and you lose visibility. Token Observe is in band and fails closed, so its failures are your agents’ failures. An observability tool can therefore be adopted with no operational conversation, and this cannot. Where a row below reads as an absence in one of those products, treat it as a question for that vendor rather than a finding.
Token Observe and LLM observability, row by row
Where these two differ, one row at a time.
Ask which provider accounts, currencies and traffic paths the existing tool covers, and retain its source evidence.
Connect supported organization billing sources. Read collection freshness and reported costs separately from invoices and gateway estimates.
After the response. The record is the product. LangSmith’s LLM Gateway, documented as in beta, and Datadog’s AI Guard, in Preview, are separate products that check inline.
Before the routed request reaches the provider. The hosted gateway records the decision on Token Observe infrastructure; customer hosting has a separate custody model.
On the record of a call that has already completed: a span attribute, a score, an alert.
A typed refusal. The trace closes as blocked, the caller receives ACP_POLICY_BLOCKED, and nothing reached the provider.
An out-of-band exporter: telemetry loss degrades visibility and the agent keeps running.
Fail-closed. A chain found corrupt latches readiness and audit writes unavailable, and governed requests receive ACP_AUDIT_UNAVAILABLE 503. The latch survives a restart deliberately.
Query languages, aggregation, dashboards, retention tiers.
Plain English translated into a validated filter object over fourteen allow-listed fields, shown back as editable chips. No grouping, no counting, no correlation across traces.
Telemetry, usually readable by the whole team that owns the service.
Evidence: hash-chained, anchored off-box, exported under a SHA-256 seal carrying the chain verdict. Sealed is not signed, and tamper-evident is not tamper-proof.
- beat 01Name the caller
- 1Authenticate
- 2Resolve the agent
Nothing is decided for an anonymous caller. The key resolves to one registered agent with a named owner, and its roles, its ceilings and any stop switch in force are read live from the store on every request, not from a cached copy, so a suspension takes effect on the next call instead of after a redeploy.
- beat 02Start the record, then read what is being sent
- 3Open the trace
- 4Sanitise
- 5Scan
The trace is opened before anything can refuse the request, which is why a blocked call is recorded as thoroughly as one that went through. Only then is the text normalised and scanned, in that order, so the detectors see what the model will read rather than what a human reviewer can see.
- beat 03Decide once
- 6Govern
- 6bIntersect with the named human
- 7Enact
One function returns one verdict: allow, block, or hold it for a named person to approve. Every check lives in that one place rather than scattered along the path, which is what makes it possible to read your rules and know what they do.
- beat 04Send it, and only then spend the money
- 8Route
- 9Call upstream
Routing fixes which provider and model will serve the call, so where an agent has a USD ceiling it is tested against the price you will pay, not the one the caller typed. Routing is also the last step that can still refuse locally: after it, the request has left your network.
- beat 05Govern the answer, then count the cost
- 10Govern the response
- 11Meter and record
A tool call the model proposes is judged before it reaches your agent. Metering runs last and on every ending, including the refusals and the streams a client abandoned halfway, because the tokens were spent either way and a partial trace is still evidence.
Step names and their grouping are the product’s own, and the order is encoded in the evaluator rather than drawn for the page.
Decide and refuse inline, or record and score after.
The two products answer questions that sound similar and are not. A tracing tool answers what happened, in enough detail to debug it. Token Observe answers whether this may happen, at the moment it is proposed, and keeps the answer in a form that survives being asked about a year later. A scoring pipeline that flags a violation after the fact has produced a finding; an inline verdict has produced a refusal, and the difference is whether the card number left the building.
That timing is why the request path is ordered the way it is. Unicode sanitisation runs before anything reads the payload, so smuggled invisible characters cannot slip past a detector reading a different string from the one the model will read. Detection runs before the verdict, and the verdict is a single point rather than a set of middlewares each deciding something different. A streamed response cannot be re-decided once bytes are on the wire, so response-side policy is resolved before the first byte.
What OpenTelemetry as an input buys you.
Token Observe runs an OTLP over HTTP receiver that accepts bounded JSON and protobuf for logs, traces and metrics and attributes every write to the credential that presented it. That is a real receiver, and deliberately not a claim to be your tracing backend. What it is for is the observed view: the product holds four views of one agent — declared in the registry, locked in a manifest, deployed as a runtime artefact, and observed — and the difference between them is the finding.
- What it does not do
- It is not a trace viewer for application spans, it runs no evaluations, and it holds no dataset or prompt versioning. Those are named strategic non-goals, not a backlog.
- What it does not attest
- It does not attest the device or exporter that produced the telemetry, and protobuf interoperability has repository tests rather than a live collector matrix. Collection trust is an open gap, named as one.
Which of the two to put in.
Read the left one first. If two of those describe you, buy the other thing. If the right column describes you, the next step is thirty days with the source on your own hardware.
When to choose LLM observability
- The question is why is this chain slow, why did quality drop, or which prompt version regressed. Token Observe records governed requests, not your application’s reasoning.
- You need datasets, evaluation runs, prompt experiments or a judge harness. Those were designed for and deliberately not built, and building another one is a stated non-goal.
- You cannot accept a fail-closed component in the request path, which is legitimate for an estate whose agents draft text a human reads before anything happens.
- Your instrumentation is already framework-deep and out of band, and what you want is more of that, not a decision point.
When to choose Token Observe
- You need a routed payload checked before provider egress, with the hosted or customer-hosted custody model agreed for that workload.
- The reader of the record is an auditor or a compliance officer, and reads of that record need to be attributable to a named person.
- You want the decision and its evidence in one place: the policy that fired, the human who approved it, the tokens spent, and a chain verdict that names any break.
- You already run a tracing tool and the gap you have found is enforcement, not visibility.
If the left-hand column describes you, that is still worth an email: a short reply costs both of us less than an evaluation that ends in the same place.
See the thirty-day evaluationThe 5 named products in this category
This page argues about the category. If you have already shortlisted a product, these go head to head with it.
LangSmith
LangSmith’s callback handler watches the call from beside it, and their newer LLM Gateway now stands in it too. The differences that remain are narrower than they were.
Langfuse
Keep Langfuse for your tracing and evaluation workflow. Compare Token Observe for the billing evidence and routed controls your IT team needs.
Arize
Arize is instrumented into your application and reads what it did. Token Observe is a hop your agents call through and decides what they may do.
Datadog LLM Observability
Datadog puts LLM spans beside the rest of your telemetry. Token Observe puts a verdict in front of the call. Datadog also sells a verdict, and the comparison turns on where it is made.
Braintrust
Braintrust is in the request path too. What it does there is deliver the call and record it; what it blocks is the release that would have made the call worse.
The other comparisons
The other 6 categories, each opening with the case for the alternative.
Do we have to replace Langfuse or LangSmith to use Token Observe?
No, and you should not. The roadmap names another standalone tracing and evaluation product as a strategic non-goal, so the intended shape is both: your tracing tool keeps answering developer questions, and Token Observe holds the decision and the evidence. The overlap is real. It records governed requests, their tool calls, their policy decisions and their cost, but it has no prompt playground, no dataset management, no experiment runs and no evaluation harness, and it is not going to grow them.
Can Token Observe’s search answer aggregate questions?
No. Search translates a plain-English question into a validated filter object over fourteen allow-listed fields and never into SQL, because trace content is attacker-influenced and an interpreter reached from it would be an injection surface. That filter has no grouping and no cross-trace correlation, so which agents used the same card number twice cannot be asked. Aggregate reporting is what the cost ledger and the compliance export cover.
What happens to governed traffic if the evidence layer fails?
It stops, and that is deliberate. The boot sequence walks the whole audit chain before the process listens. If that walk finds corruption, the verdict latches readiness and audit writes unavailable and governed requests receive a 503 carrying ACP_AUDIT_UNAVAILABLE. Restarting does not clear it: there is intentionally no online clear-incident endpoint, so recovery means restoring a database whose chain and independently retained head verify. An observability tool would degrade quietly; a product that exists to produce the record cannot serve traffic it has stopped recording.
Tell us which way you are leaning, and why.
Create a cloud account and workspace. No card or server installation. Model calls and Google BigQuery scans can incur separate provider charges.