Token Observe vs LiteLLM

The objection is real, and it deserves the concession first: LiteLLM sits in the same slot Token Observe asks for, the base URL is already swapped, the keys are already central, and the spend is already counted against a virtual key. Pulling it out to install something else is work with a migration attached and no governance outcome attached, and nothing on this page recommends it.

The question that decides whether you need anything more is about what a governed call can end in. If every call ends in text that a human reads before anything happens (drafting, summarising, chat, code suggestions someone reviews), then a routing proxy with a spend counter and a guardrail plugin is a proportionate control, and it is the one to keep. If a call can end in a refund being issued, a pull request being merged, an email leaving the building, a ticket transitioning or a row being written, the interesting question stops being what did it cost and becomes whether the thing that was authorised is the thing that happened. That is a different product rather than a larger configuration of the same one.

Put three questions in writing to whichever proxy you run. They are questions rather than findings, drawn from failure patterns recorded in Token Observe’s own engineering notes rather than from anything observed in LiteLLM. Does the fallback chain treat a provider’s content-policy refusal as a retryable error, so the next provider’s answer returns as a success and nothing in the record says a refusal happened? Does cache-token accounting add Anthropic’s cache read and write buckets to a total that already includes them, or fail to add OpenAI’s, and produce a figure the invoice disagrees with? Does an approval, where one exists, authorise an action type rather than an exact payload, so a retry with one argument changed is still allowed? The answers are specific and checkable, and a good product will have them to hand.

“We already have LiteLLM” usually means the MIT proxy, and the features a governance buyer is shopping for are mostly on the other side of LiteLLM’s published line. Their pricing panel puts virtual keys, budgets and teams, load balancing, RPM and TPM limits, guardrails and the logging integrations in the $0 tier. Their enterprise page and enterprise documentation put SSO for the admin UI across Okta, Azure AD, Google Workspace and OIDC or SAML, SCIM, JWT-based authentication, role-based access control, IP-based access lists, key rotation, secret-manager integration and audit logs with retention policies under Enterprise, deployed self-hosted with a licence key. One qualification is theirs, and it moves the line: their enterprise documentation states that “SSO is free for up to 5 users. Beyond that, an enterprise license is required.” That is not a criticism of the split — it is a sensible place to draw it, and the free tier is unusually generous by the standards of the category — but it changes the comparison a buyer is making.

The practical consequence is that a shortlist which reads “Token Observe, or the free thing we already run” is comparing against something narrower than the enterprise pages describe. If the requirement that started the evaluation is single sign-on, an audit trail of who changed which key, or role-based separation between the platform team and the application teams, then the comparison is between two commercial purchases and the cost line on the LiteLLM side is a quote rather than zero. Ask for it early; their documentation says pricing is based on usage.

Read the licence boundary for yourself. The repository LICENSE says content under the enterprise/ directory is licensed under enterprise/LICENSE and everything outside it is MIT, and the enterprise documentation describes a self-hosted Docker deployment activated by a licence key. Which specific behaviours that key turns on in a given release is a question for the vendor and for your counsel, and it is the sort of question that is much cheaper to answer before a deployment than after one.

Start with what LiteLLM states plainly about itself, because it is a strength in its own frame: no database write sits in the request path, and logging, rate-limit accounting and spend tracking run as asynchronous background tasks once the provider has answered. A carrying proxy should be built that way. Budgets do not wait for those writes: their budgets page describes budget reservation, on by default, which estimates a request’s maximum cost from the request body and the model’s pricing, reserves it against the applicable budget, and rejects the request before sending it to the provider if the reservation would exceed it. The page does not say whether that estimate covers a fallback deployment the router might try. The hot-path figure is a cross-pod counter in Redis with the database reconciled in the background, and fail_closed_budget_enforcement makes every budgeted request validate spend against the database first. For routes without token pricing, their page says, LiteLLM cannot reserve a cost and enforces the budget using recorded spend; and a section headed “Budgets require a database” says that without one “none of them cap anything”, with the global check failing open rather than erroring. Both products now reserve before egress, so the comparison is about what each reservation prices and what it does when there is no price.

Token Observe defers the money verdict until the route is resolved. Permissions, policy and rate limits are decided first, then the route is resolved, then every provider and fallback in that route is priced and the most expensive candidate rates, applied to the request’s max_tokens or to a 4,096-token output cap stamped onto the outbound request, are projected and reserved against the agent’s rolling hour, UTC day and UTC month inside a single per-agent database transaction, before the call goes out. A budgeted agent whose resolved route contains a target with no price row is refused with a 409 rather than admitted at an assumed zero. The cost of that design: a hard ceiling buys you one billable egress, with no retry and no failover behind it, and the ceiling is conservative by construction.

The approval gate is the second half of the same argument, and LiteLLM’s side comes first. LiteLLM documents an approval path of its own for MCP tool calls: omit require_approval, or set it to anything but “never”, and their MCP page says the tool calls are returned to the client for you to review and execute manually, with the Responses API carrying that round trip as an mcp_approval_request and a matching mcp_approval_response. The decision there lives in the client’s next request. In Token Observe, a policy whose action is require_approval refuses the request with a 403, mints an approval bound to the SHA-256 of the canonicalised action plus the execution context it was proposed in, and holds the trace open until a named person decides. That approval authorises that payload and nothing else: change one argument and the hash no longer matches, so the retry is refused as a mismatch rather than allowed as near enough. It is spendable once by compare-and-set so two concurrent retries cannot both execute, and it expires at 60 minutes by default. The limit sits beside it: approving pushes nothing to the agent, because Token Observe has no way to call an agent back, so the approval takes effect only when the agent repeats the identical request carrying its id.