Token Observe for OpenAI
One base URL, one credential, and every Chat Completions call becomes a governed request.
On this page
What moves in a OpenAI client
OPENAI_BASE_URL="https://gateway.example.com/v1" # was https://api.openai.com/v1
OPENAI_API_KEY="acp_agent_…" # the agent key, not the OpenAI key
# CrewAI and several older libraries read the legacy name as well
OPENAI_API_BASE="https://gateway.example.com/v1"
# Or per client, which is better when one process runs several agents
client = OpenAI(
base_url="https://gateway.example.com/v1",
api_key=os.environ["ACP_AGENT_KEY"],
default_headers={"x-acp-session-id": run_id, "x-acp-tags": "support,tier1"},
)Where that traffic lands
- POST /v1/chat/completions
- The main surface. Accepts model, messages with string or multi-part content, tools, tool_choice, both max_tokens and max_completion_tokens, temperature, top_p, stop, seed, response_format, stream, stream_options and user, plus unknown top-level fields that are governed and then forwarded on a compatible OpenAI-family route. The response is byte-faithful to OpenAI, including the cached-token detail.
- POST /v1/responses
- A compatibility subset for inline, stateless requests: string or item-array input, instructions, function tools and tool choice, function_call and function_call_output items, text and JSON response formats, token and sampling controls, and buffered or typed-event streaming. It enters the same role check, budget check, policy, redaction, routing, metering and trace path as Chat Completions.
- GET /v1/models
- Returns only the models the calling agent’s roles permit. This is where deny-by-default scoping first becomes visible inside a client SDK rather than only in a console — and it is what a client like Cursor calls to validate a key, so the agent’s own tooling shows it a smaller world.
- POST /v1/embeddings
- Proxied with the same request governance and usage metering as completions, for openai and openrouter provider rows only. The adapter speaks the OpenAI-compatible POST /embeddings and bearer contract, and Anthropic, Google, Azure and Bedrock routes fail closed before credential resolution rather than being sent a request in the wrong dialect.
What is true of OpenAI and not of the others
Every provider in this list behaves differently somewhere that matters, and those differences are the reason a single general integration page is not enough.
- Cached tokens are inclusive
- prompt_tokens already contains prompt_tokens_details.cached_tokens, so Token Observe subtracts the cached figure to get the uncached bucket. A payload claiming more cached tokens than prompt tokens is refused with a range error rather than producing a negative bucket that would credit a budget.
- max_completion_tokens, not max_tokens
- The direct OpenAI route sends the newer field name by default. OpenRouter and Azure rows are composed from the same client with the field overridden to max_tokens, because that is what those two document and the newer name is refused on Azure’s pinned GA contract.
- Usage is requested on every stream
- stream_options.include_usage is sent upstream whatever the client asked for, because metering must not depend on client behaviour, and the extra usage chunk is suppressed on the way out when the client did not want it. A spend ledger an agent can opt out of by omitting a field is not a ledger.
- Three headers are forwarded, and no more
- openai-organization, openai-project and openai-beta reach the upstream; everything else an inbound client sends is dropped. Reflecting arbitrary inbound headers at an upstream you hold credentials for is a request-smuggling primitive, so the list is an allowlist rather than a denylist.
- Tool arguments that do not parse are fatal
- An unparseable arguments string is a hard failure rather than an empty object. Policy argument matchers read those values, and substituting an empty object would walk an unparseable call straight past a rule written to stop it.
- Redirects are refused
- The upstream fetch is issued with redirect handling set to error. A 307 or 308 would replay your prompt and the gateway’s provider credential at the redirect target, which is precisely the destination the base-URL allowlist exists to constrain.
The change, and the two things it does not change
Adoption is a base-URL change and a credential swap. There is no library to import, no wrapper to construct and, for the request shapes listed above, no code change: the agent keeps speaking the dialect it already speaks and Token Observe answers in it. The credential swap is the part people skip. The agent key is the agent’s identity, it is minted by an administrator against a registry record, and it will not work against api.openai.com — that is the point, because a credential that works in both places tells you nothing about which path a call took.
The base-URL change is normal for the supported dialects, and the first thing to check when it does not work is your own SDK and its version. Client libraries disagree about which variable wins, about whether a constructor argument overrides the environment, and about whether the path they append expects the trailing /v1 to already be there. The asymmetry that catches people most often is between dialects rather than within one: the OpenAI dialect takes a trailing /v1 on the base URL and the Anthropic dialect does not, and the product’s own troubleshooting names that as one of the top three causes of an invalid-key 401.
The first thing that does not change is your prompt handling. Redaction masks values on the way back through the gateway, and the moment your application renders that text into an HTML page or hands it to a shell, the failure is in the application. The second is availability arithmetic: Token Observe is inline, in the request path, with no network hop between the governance decision and the call. That is what makes the decision a decision rather than a report, and it is also why an outage in the gateway is an outage for governed agents. Both facts belong in the rollout plan rather than in the retrospective.
One commercial consequence is worth saying out loud before the change ships, because the documentation puts it in the imperative: tell your developers first. A subscription-based coding client that is given a gateway credential stops using that developer’s own subscription, and the work is then billed per token to whichever provider account the install uses. For a governed company fleet that is exactly the intent, and it is still a change people notice on the day it happens.
How an OpenAI response is priced, and where the cache count goes
Every provider adapter normalises usage into four mutually exclusive buckets before any cost arithmetic runs: uncached input, cache reads, cache writes and output. No bucket contains another, and the invariant is asserted at every point a figure enters pricing, reporting or persistence. That normalisation exists because providers genuinely disagree about what a prompt total means, and a ledger that took each vendor’s numbers at face value would be wrong by a different amount per vendor.
OpenAI’s convention is inclusive. prompt_tokens is the whole prompt and prompt_tokens_details.cached_tokens is the part of it that was served from cache, so the uncached bucket is the subtraction of the second from the first. Anthropic’s convention is the opposite — cache reads and cache writes are reported alongside input_tokens rather than inside it — and Gemini follows OpenAI’s. That single difference is why the adapters normalise rather than the ledger branching on vendor, and getting it backwards misprices cache-heavy agent traffic by between half and nine tenths.
Pricing is then a lookup against your own price rows, matched on provider kind and model with exact rows preferred over wildcards and the longest pattern winning among equals. Where a row states no explicit cache rates, cache reads bill at the full input rate and cache writes at the input rate — conservative by construction, so the ledger never under-bills relative to the invoice. Figures are rounded to eight decimal places so repeated addition across a month stays stable.
The case worth planning for is the model your price table does not know. If a USD budget is configured and any model or provider candidate on the resolved route has no active price row, the request is refused before egress with ACP_BUDGET_UNPRICED as a 409 rather than being priced at zero. That is a conflict rather than a bad request: no ceiling has been exceeded, and the fix is to add the price row or to remove the USD ceilings from an agent you genuinely meant to leave unbudgeted. An unpriced model quietly metered at zero would disarm every ceiling above it while the console still displayed the ceiling, which is the failure mode a spend control cannot have.
- Streaming counters
- Provider stream counters are cumulative, and some endpoints emit more than one usage frame where a later frame omits or regresses a bucket. The merge keeps the greatest validated value seen per bucket, which never credits a budget and never mistakes a repeated total for an increment.
- Pre-flight estimation
- A crude four-characters-per-token estimate exists for pre-flight budget checks only and is never used for billing. Actual usage always comes from the provider response, and on the Anthropic dialect POST /v1/messages/count_tokens is the pre-flight primitive.
- Untrusted wire data
- Provider usage figures are validated as non-negative safe integers before they reach arithmetic, even though the interface describes them as numbers. Anything that could subtract from a budget, lose precision in SQLite, or turn cost into NaN or Infinity is refused rather than absorbed.
- What is on the response
- x-acp-trace-id on every governed outcome including refusals, plus x-acp-cost-usd, x-acp-provider, x-acp-model, x-acp-policy-matches, x-acp-redactions (kinds, never values) and x-acp-cache. On a stream the cost rides in a trailer rather than a header, because it is not known until the stream ends.
What fails over to another provider, and what deliberately does not
A route rule carries an ordered fallback chain, and whether a failure walks down it is decided by the class of the failure rather than by a retry count. Seven classes exist. Three of them fail over: timeout, rate_limited and server_error. Four of them do not: context_too_long, content_policy, auth and invalid_request. The switch is exhaustive with no default branch, so adding a class is a compile error until somebody decides its behaviour.
The reasoning is specific to a governance product rather than to availability. A content-policy refusal is a signal: one vendor’s safety system declined, and sending the same payload to the next vendor is a second attempt at the same action, with the trace recording a success while the objection disappears. An authentication failure means the gateway’s provider key is wrong, revoked or scope-limited, which fails identically wherever that credential is used, so failing over masks a broken key behind a more expensive provider until the invoice arrives. And an over-long context is a property of the payload, not the provider, so failing over pays a full input-token charge to receive the same error.
The cost of that decision is published rather than hidden. Requests classed content_policy, auth, invalid_request or context_too_long fail where a dumber gateway would have succeeded on a second provider, and the product’s own architecture note says that is intended and will be reported as a bug. Two further honesty notes travel with it: classification is a lossy mapping from heterogeneous vendor error shapes onto seven classes and will get cases wrong, and a provider that returns 5xx for what is really a refusal will be failed over, producing exactly the laundering the design prevents everywhere else.
Above the chain sits one circuit breaker per provider: five consecutive failures opens it, and it stays open for thirty seconds before a probe is allowed through. Breakers are keyed by provider id in a map that configuration refreshes never touch, because rebuilding a breaker on every config reload hands a flapping upstream a clean slate and is precisely how a breaker stops working. The metrics scrape and the request path read the same breaker instance, so the gauge reports the state that is actually admitting or refusing traffic.
- Failure surfaces
- ACP_UPSTREAM_TIMEOUT as 504, ACP_PROVIDER_UNAVAILABLE as 502, ACP_INVALID_REQUEST as 400. Within a provider, retries use capped backoff with jitter on idempotent calls only.
- OpenAI status mapping
- 408 is a timeout, 429 is rate-limited, anything 5xx is a server error, 401 and 403 are auth. A 400 whose message names a context length is context_too_long; a message matching the content-policy pattern is a policy refusal rather than a bad request; everything else is an invalid request.
- Once bytes are on the wire
- A stream that has already opened cannot fail over. Before the first byte the error is thrown so the router can still try the next provider; afterwards it becomes an in-band error event, because the partial stream is already evidence and the consumer needs to close its trace with what it received.
- Failover changes the metering target
- A failed-over request is priced against the provider that actually served it, on both the buffered and the streamed paths. Metering against the originally resolved route was a real defect: it priced against the wrong provider kind and attributed the spend to a provider that never ran the call.
What the OpenAI ingress refuses, and why each refusal is there
The refusals are worth reading before a rollout rather than after, because they are the only part of the change that can require an application edit. Each of them has the same shape of reason: governance sees the request, so anything that moves part of the request outside what the gateway can inspect is refused rather than forwarded.
Media parts are the broadest. Image, audio and PDF inputs are rejected on every dialect — Chat image_url, Anthropic image, Responses input_image and Gemini inlineData alike — because there is not yet bounded media decoding and OCR under the data-loss and injection policies, and a caller-supplied MIME type is not proof that opaque bytes are safe. On the Responses adapter, previous_response_id, conversation, item references, background jobs, prompt templates, opaque file ids, hosted tools and unsupported content parts are refused with ACP_INVALID_REQUEST: the adapter is a compatibility subset rather than a hosted response store, and inlining or resolving those inputs before the call is what lets governance see the complete payload.
OpenRouter’s routing and processing controls are rejected on this ingress too, not only on the OpenRouter one. models, provider, route, plugins, transforms and web_search_options each delegate a governed decision — model choice, provider choice, processing, search egress or charges — to the vendor, where model permissions, data policy and the price ceiling cannot reach. Retained :online, :nitro, :floor and :exacto model suffixes are rejected for the same reason. Fallbacks and provider selection are configured on route rules instead, where they are audited.
Unknown top-level fields are the interesting middle case, because they are neither forwarded blindly nor refused. They are walked with the same bounded walker used for tool-call arguments, so their strings go through sanitisation, scanning, redaction, policy and recording like any other text, and each bag carries the wire dialect it arrived on. Routing then removes any failover target that would drop or reinterpret those fields, and if no compatible target remains the request is refused before egress rather than served with the vendor extension silently missing.
The parts of the platform this uses
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
Agent permissions
Deny by default, explicit deny wins, and delegation intersects — so an agent cannot borrow authority it was never granted.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
The same policies apply identically whichever provider serves the request, and that equivalence is enforced by a test over every provider kind rather than asserted.
See the request pathThe rest of the upstreams
Do I have to change my code, or only my environment?
For the shapes listed above, only the environment: OPENAI_BASE_URL and OPENAI_API_KEY, or the equivalent constructor arguments. The one piece of new code most teams write is a branch on the governance responses — a typed error carrying ACP_APPROVAL_REQUIRED gives you an approval id to re-send the identical request with once a human decides, and ACP_BUDGET_EXCEEDED means the agent is out of budget for the window rather than that the model failed. A base-URL change is normal for the supported dialects, and if it does not take effect the first thing to check is your own SDK and version: libraries disagree about which variable wins, whether a constructor argument overrides the environment, and whether they append the /v1 you already put in the URL.
Why does my cached-token cost look different from the vendor console?
Because the buckets are mutually exclusive here and the vendor reports one bucket inside another. OpenAI counts cached prompt tokens inside prompt_tokens, so Token Observe subtracts prompt_tokens_details.cached_tokens to get the uncached figure and prices the two at separate rates; the operator-facing total input figure adds them back. If your price row states no explicit cached rate, cache reads bill at the full input rate, which is deliberately conservative and will read slightly high against an invoice that discounts them. The place this genuinely diverges is a provider using the other convention: Anthropic reports cache tokens beside input_tokens rather than inside, and reading either one with the other’s assumption misprices cache-heavy traffic by between half and nine tenths.
Will a rate limit on OpenAI move my traffic to another provider?
Yes, if you have configured a fallback on the route rule that matched. A 429 is classed rate_limited, which is one of the three classes that fail over, along with timeout and server_error. A content-policy refusal, an authentication failure, an over-long context and a malformed request are the four that do not, because each fails the same way at the next vendor and failing over would either waste budget or hide the real cause — and in the content-policy case would record a success where a vendor had objected. After five consecutive failures the provider’s circuit breaker opens for thirty seconds, so a genuinely dead upstream stops absorbing and charging for traffic rather than being retried indefinitely.
Does the response still look exactly like OpenAI’s?
Yes on Chat Completions, which is byte-faithful including usage.prompt_tokens_details.cached_tokens, with streaming as data-only server-sent events terminated by data: [DONE]. Extra information travels in headers rather than in the body — the trace id, the cost, the provider and model actually served, the policy matches, the kinds redacted and whether the response cache served it — so a strict SDK parser sees the shape it expects. The one deliberate divergence is that the model served is not always the model asked for: with smart routing switched on and the agent’s routing.allowDowngrade set, a request may be served by a cheaper tier, and x-acp-model reports what actually ran while the decision and its reasons sit on the trace.
What happens to a request that uses a feature the gateway refuses?
It is refused before provider egress with a typed error, not silently stripped. Image, audio and PDF parts are rejected on every dialect; the hosted Responses features — previous-response references, conversations, item references, background jobs, prompt templates, opaque file ids and built-in tools — are rejected because the adapter is a compatibility subset rather than a hosted store; and OpenRouter’s routing and processing controls are rejected because they delegate a governed decision to the vendor. Refusing rather than stripping is the deliberate choice: a stripped field changes the meaning of a request that then succeeds, and nobody reads the trace of a call that worked.
Name the SDK and the version.
A base-URL change is the normal case for the OpenAI dialect, and whether your own client library and version behave that way is the first thing worth checking. Say which you use and you will get a straight answer.
no form · no qualification step · no sales desk · the other three ways in