Token Observe for OpenAI-compatible endpoints
vLLM, a proxy, a colleague’s inference server — registered as a row, bounded by two allowlists.
On this page
What moves in a OpenAI-compatible endpoints client
# On the gateway, before the provider row will build at all
ACP_ALLOWED_PROVIDER_HOSTS="llm.internal.example.com"
ACP_PROVIDER_VLLM_API_KEY="…"
# The row itself
POST /api/providers
{ "kind": "openai",
"baseUrl": "https://llm.internal.example.com/v1",
"apiKeyEnvVar": "ACP_PROVIDER_VLLM_API_KEY" }
# On the agent — unchanged from the OpenAI case
OPENAI_BASE_URL="https://gateway.example.com/v1"
OPENAI_API_KEY="acp_agent_…"Where that traffic lands
- POST /v1/chat/completions
- The ingress. Your agent speaks the OpenAI dialect at the gateway; where the call goes is decided by the route rule that matched the model, so moving a model between a hosted vendor and your own endpoint is a route-rule change rather than a fleet reconfiguration.
- {baseUrl}/chat/completions
- The egress, with Authorization: Bearer carrying the credential named on the row. Redirect handling is set to error, because a 307 or 308 would replay your prompt and that credential at the redirect target and step outside the host allowlist entirely.
- {baseUrl}/models
- What GET /v1/models resolves against for this row, filtered afterwards to the models the calling agent’s roles permit. Entries without a usable string id are dropped rather than surfaced.
- POST /v1/embeddings
- Available on openai and openrouter rows, which includes a compatible endpoint registered under the openai kind. The adapter speaks the OpenAI-compatible POST /embeddings and bearer contract, and persisted rows are rechecked against both the credential and destination allowlists before any environment value is read.
What is true of OpenAI-compatible endpoints and not of the others
Every provider in this list behaves differently somewhere that matters, and those differences are the reason a single general integration page is not enough.
- An unset host policy is not permission
- With ACP_ALLOWED_PROVIDER_HOSTS unset, the API and the live registry permit only the shipped vendor hosts. Regional and self-hosted endpoints must be named explicitly, and a wildcard entry stands for exactly one label and never spans a dot — so a lookalike domain ending in the same characters does not match.
- Persisted rows are rechecked, not grandfathered
- Both the credential-prefix and the destination checks run again when the registry rebuilds, so a row written by an older version is skipped before its environment value is read. An upgraded deployment with a harmless legacy variable name has to rename it under an allowed prefix and update the row; it is not silently carried through a credential-egress boundary.
- max_completion_tokens is sent on this kind
- The shared client’s default field name is the newer one, and it is overridden only for OpenRouter and Azure rows. A compatible server that accepts only max_tokens will refuse the request, which is the first thing to check when a self-hosted endpoint 400s a body that works against a hosted vendor.
- A key is required even when the endpoint ignores it
- The client refuses to build when the named variable is unset or empty, naming the variable rather than echoing any value. A local server that accepts anything still needs a variable holding something, and a missing key is a skip with a named log line rather than a crash — routing then sees the provider as unavailable and fails over.
- Repeated usage frames are merged conservatively
- Compatible endpoints sometimes emit more than one usage frame in a stream, and a later frame can omit or regress a bucket. The merge keeps the greatest validated value seen for each mutually exclusive bucket, which never credits a budget and does not mistake a repeated total for an incremental delta.
- Three headers are forwarded by default
- This kind inherits the OpenAI allowlist — openai-organization, openai-project and openai-beta — and nothing else an inbound client sends reaches the upstream, because reflecting arbitrary headers at an endpoint you hold credentials for is a request-smuggling primitive.
- The credential namespaces are disjoint by test
- A provider row cannot name a tool-server, webhook or vendor-admin credential, and none of the four can name the session secret, either audit key, the retired key ring or the effect key. The reserved list is the fixture of a test rather than a statement of intent, and an ADMIN segment in a variable name is refused even under a matching vendor prefix.
Registering the endpoint, and the two allowlists that gate it
A self-hosted or third-party OpenAI-compatible endpoint is a provider row with kind openai and your own base URL. Everything downstream of that treats it like any other provider: route rules select it, the agent’s data policy filters it, prices are matched on its kind and model, its failures are classified into the same seven classes, and it gets its own circuit breaker. There is no separate compatible mode, which is the point — the OpenAI client is the reusable core the vendor adapters are all composed from.
Two allowlists decide whether the row will build at all, and both fail closed in a way worth stating precisely, because the failure is silent from the caller’s point of view. With no host policy configured, an unset value does not mean no restriction: it means the shipped vendor hosts only, so a self-hosted endpoint has to be named in the host policy explicitly. And the variable holding the credential has to sit under an allowed prefix, of which ACP_PROVIDER_ is the general-purpose one that ships for exactly this case. A row failing either check is skipped with a named reason and the policy that refused it, and routing then treats the provider as unavailable.
The reason those limits exist is worth reading once rather than working around. A provider row names an environment variable whose value the process will send to a URL in the same row. Without the two bounds, a registry write is not an integration at all but a read primitive over the whole process environment with an operator-chosen sink — and it matters most here, because this design deliberately concentrates every provider key in the estate into one process. The four integration namespaces are disjoint by test rather than by convention, so no integration can name another’s secret, and none can name the deployment’s own secrets.
Both checks run again when the live registry rebuilds, not only when a row is written. That second boundary is what stops a row persisted by an older version from surviving an upgrade as an exfiltration path: a disallowed name is skipped before its environment value is read. An upgraded deployment holding a harmless legacy variable name outside the allowlist has to rename it under an allowed prefix and update the row, which is deliberate friction at a credential-egress boundary rather than an oversight.
- Wildcards never span a dot
- A wildcard entry stands for exactly one label, so an entry admitting a customer’s resource host admits nothing outside that domain. It cannot be satisfied by a path that merely contains the pattern, by a subdomain of a lookalike, or by a host that ends in the same characters.
- One bad row does not take the gateway down
- A row whose client cannot be constructed — a malformed base URL, say — is logged with a named reason and skipped while the other providers load. The same is true of a missing credential, and of the mock provider, which is not built at all when any enabled row has a different kind.
- Redirects are refused
- The upstream fetch sets redirect handling to error on every call. A 307 or 308 would replay your prompt and the gateway’s credential at the redirect target, which is a way around the host allowlist rather than a transport detail.
Where compatible stops being compatible
The OpenAI dialect is a family rather than a specification, and the differences between members of that family are the practical content of this page. The gateway’s client is written against the real OpenAI contract, so an endpoint that diverges from it diverges from what will be sent.
The field name is the most common surprise. This provider kind sends max_completion_tokens, the newer OpenAI name; the override to the older max_tokens is applied for OpenRouter and Azure rows specifically, because those two document it. A compatible server that accepts only max_tokens will refuse a body that works perfectly against a hosted vendor, and that is the first thing to check when a self-hosted endpoint returns a 400 on a request the same agent sends successfully elsewhere.
Usage reporting is the second. The client asks for usage on every stream regardless of what the caller requested, because metering must not depend on client behaviour, and it then merges cumulative counters conservatively — keeping the greatest validated value seen per bucket — precisely because compatible endpoints emit more than one usage frame and later frames sometimes omit or regress a bucket. An endpoint that reports no usage at all is a harder problem: there is nothing to meter, and nothing to price.
That is where the honest limit sits. If a USD budget is configured and any model or provider candidate on the resolved route has no active price row, the request is refused before egress with ACP_BUDGET_UNPRICED as a 409 — a conflict rather than a bad request, because no ceiling has been exceeded and the fix is to add a price row or to remove the USD ceilings from an agent you meant to leave unbudgeted. An unpriced candidate quietly metered at zero would disarm every ceiling above it while the console still showed the ceiling, which is the failure mode a spend control cannot have. For a self-hosted endpoint whose marginal cost is genuinely near zero, the honest configuration is a price row stating that, not the absence of one.
Tool-call arguments that do not parse are a hard failure here as everywhere, and it is worth naming because a compatible server’s tool-calling implementation is the most common source of malformed argument strings. Policy argument matchers read those values, and substituting an empty object would let an unparseable call walk past a rule written to stop it.
What changes on the agent, which is nothing new
The agent-side change is the OpenAI one: point OPENAI_BASE_URL at the gateway with the trailing /v1 and put an agent key where the previous key sat. Whether the call ends at a hosted vendor or at your own inference server is decided by the route rule that matched the model, which means moving a model between the two is a route-rule change rather than a fleet reconfiguration — and it is recorded, so somebody can answer later where that model was actually served in March.
A base-URL change is the normal way in for the supported dialects, and the first thing to check when it does not take effect is your own SDK and its version. That advice applies with more force here than anywhere else, because the tools most likely to be pointed at a compatible endpoint are the ones with the most configuration surface: a self-hosted client library, an agent framework with its own provider abstraction, an IDE extension with a provider dropdown. Several of them read more than one variable and disagree about which wins.
Two rollout notes travel with that. Where a client offers an OpenAI-compatible provider option, use it rather than a vendor-specific one, because the vendor-specific option usually builds a vendor-specific URL the gateway does not route. And roll the settings out through device management rather than asking people to set them: an opt-in redirect is the single largest source of shadow-AI findings, for the ordinary reason that a control somebody has to remember to switch on is a control most people will not switch on.
What the change buys is the same on this provider as on any other: a resolved identity, deny-by-default permissions, budget and rate ceilings decided before egress, a policy verdict, redaction on the way out, and a trace of every governed request including the refused ones. What it costs is that the gateway is inline, so if it is down governed agents cannot call models — which is the difference between a decision and a report, stated as a cost rather than left to be discovered.
Failure classification against an endpoint you control
The seven failure classes apply unchanged: timeout, rate_limited and server_error fail over to the next target in a route rule’s chain, while context_too_long, content_policy, auth and invalid_request do not. The classification is drawn from the HTTP status and the error body — 408 a timeout, 429 a rate limit, 5xx a server error, 401 and 403 auth, a 400 naming a context length an over-long context, a message matching the content-policy pattern a policy refusal, and everything else an invalid request.
Two properties of that mapping matter more for an endpoint you operate than for one you do not, because you can do something about them. Classification is a lossy mapping from heterogeneous error shapes onto seven classes and will get cases wrong: a content refusal returned as a bare 400 is indistinguishable from a malformed request, and although both correctly do not fail over, the reason recorded in the trace will be wrong. And an endpoint that returns 5xx for what is really a refusal will be failed over, producing exactly the laundering that typed classes exist to prevent. Both are cheap to fix in a server you control, and neither can be fixed from this side.
The circuit breaker is per provider — five consecutive failures to open, thirty seconds before a probe — and it survives configuration refreshes deliberately. That is worth knowing on a self-hosted endpoint because those rows get edited more often than vendor rows do, and rebuilding a breaker on every edit would hand a flapping upstream a clean slate each time somebody changed a timeout.
One deployment note completes the picture. Streaming breaks under response buffering, so any reverse proxy in front of either the gateway or your own endpoint must not buffer responses, must preserve the authorization and api-key headers, and must use a read timeout at least as long as your longest expected model response — six hundred seconds is the documented safe default.
The parts of the platform this uses
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
Shadow AI radar
Five evidence sources for AI activity that never touched the gateway, and a coverage model that refuses to call a dead feed a clean estate.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
The same policies apply identically whichever provider serves the request, and that equivalence is enforced by a test over every provider kind rather than asserted.
See the request pathThe rest of the upstreams
Why was my self-hosted provider row skipped?
Almost always one of the two allowlists, and the log line names which. With ACP_ALLOWED_PROVIDER_HOSTS unset, only the shipped vendor hosts are permitted — an unset value means no policy has been stated, not that there is no restriction — so a self-hosted host must be named explicitly. And the credential variable has to sit under an allowed prefix, of which ACP_PROVIDER_ is the general-purpose one for this case; an ADMIN segment in the name is refused even under a matching vendor prefix, and the deployment’s own secrets are reserved outright. Both checks run again when the live registry rebuilds rather than only at write time, so a row created by an older version is skipped before its environment value is read. A skip is a named log line and an unavailable provider, never a boot failure.
My endpoint returns a 400 on a body that works against OpenAI. Why?
Check the output-cap field first. This provider kind sends max_completion_tokens, the newer OpenAI name; the override to the older max_tokens is applied only for OpenRouter and Azure rows, because those two document it. A compatible server implemented against the older contract will refuse the newer field. The next things to check are the header allowlist — openai-organization, openai-project and openai-beta are forwarded and nothing else is — and your tool-call argument encoding, since arguments that do not parse are a hard failure here rather than being substituted with an empty object, because policy matchers read those values.
How is a self-hosted model priced?
From a price row you write, matched on provider kind and model with exact rows preferred over wildcards and the longest pattern winning among equals. If a USD budget is configured and any candidate on the resolved route has no active price row, the request is refused before egress with ACP_BUDGET_UNPRICED as a 409 rather than being metered at zero — an unpriced candidate treated as free would disarm every ceiling above it while the console still displayed the ceiling. For an endpoint whose marginal cost is genuinely near zero, write a price row that says so rather than leaving the row absent: the first is a statement, the second is a gap, and only one of them survives an audit of why a budget did not bind.
Does the gateway work with an endpoint that does not report usage?
It will call it, and it cannot meter it. Usage always comes from the provider response — the crude four-characters-per-token estimate in the product exists for pre-flight budget checks only and is never used for billing — so an endpoint reporting nothing produces no token counts and therefore no cost. Where an endpoint reports usage inconsistently, which is common, the streaming merge is conservative: it keeps the greatest validated value seen for each mutually exclusive bucket, which never credits a budget and does not mistake a repeated cumulative total for an incremental delta. If metering matters for that traffic, treat usage reporting as a requirement of the endpoint rather than as a property of the gateway.
Can I put a whole third-party proxy behind one provider row?
Yes, and the same rules apply to it as to any other destination: name its host in the host policy, name its credential under an allowed prefix, and accept that its errors are classified from the shapes it returns. Two consequences are worth thinking about before you do. The gateway can only govern what it is shown, so a proxy that itself fans out to several vendors makes the provider dimension of your record less specific — the trace names the row that served, and the row is the proxy. And prices are matched on provider kind and model, so a proxy with a blended price cannot be priced accurately from a per-model table, which is the same limitation that makes an aggregator’s own reported cost the authoritative figure where one exists.
Name the SDK and the version.
A base-URL change is the normal case for the OpenAI-compatible endpoints dialect, and whether your own client library and version behave that way is the first thing worth checking. Say which you use and you will get a straight answer.
no form · no qualification step · no sales desk · the other three ways in