Token Observe for Google Gemini
The Generative Language API spoken natively, not through a compatibility shim.
On this page
What moves in a Google Gemini client
# Python, google-genai
client = genai.Client(
api_key=os.environ["ACP_AGENT_KEY"], # the agent key, not a Google key
http_options={"base_url": "https://gateway.example.com"},
)
# What that produces on the wire
POST https://gateway.example.com/v1beta/models/gemini-3.6-flash:generateContent
x-goog-api-key: acp_agent_…
# Streaming is a different method, not a body flag
POST .../v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sseWhere that traffic lands
- POST /v1beta/models/{model}:generateContent
- The buffered native method. Supports contents, systemInstruction, text parts, function calls and responses, function declarations and function-calling configuration, plus the common generationConfig fields. The model and the method are parsed out of the path, and a model name longer than the bound is refused rather than forwarded.
- POST /v1beta/models/{model}:streamGenerateContent
- The streaming method, emitting native data-only Gemini server-sent events. It traverses the same governance path as the buffered one. Upstream, ?alt=sse is added explicitly, because without it the method answers with a JSON array streamed as one document rather than as frames.
- POST /v1/models/{model}:generateContent
- The same two methods under the other API prefix. Both are accepted because deployed clients use both, and the body is translated into the same canonical request as every other dialect either way.
- GET /v1/models
- Registry-filtered to the models the calling agent’s roles permit. A single model fetched by a namespaced id containing a slash is matched by a wildcard route, since one path parameter cannot hold a slash.
What is true of Google Gemini and not of the others
Every provider in this list behaves differently somewhere that matters, and those differences are the reason a single general integration page is not enough.
- Cached tokens are inclusive
- cachedContentTokenCount is part of promptTokenCount, following OpenAI’s convention rather than Anthropic’s, so the uncached bucket is the subtraction of one from the other. Anthropic reports cache tokens beside the input total instead, and the difference between the two conventions is the difference between a right and a badly wrong cost figure on cache-heavy traffic.
- Thinking tokens count as output
- thoughtsTokenCount is added to candidatesTokenCount to form the output figure before normalisation. Dropping it would under-report the most expensive half of a reasoning call, and the gateway does not see the deliberation itself in any case — only the token count for it.
- A refused prompt is a 200
- promptFeedback.blockReason with no candidates is a safety refusal wearing a success status. It is classified as content_policy, which is one of the four classes that never fail over, so the router cannot replay a refused prompt at another vendor and record a success where a safety system objected. Only the enum names travel into the message; the blocked text does not.
- A bad credential is a 400
- Google answers an invalid key with 400 INVALID_ARGUMENT rather than 401. A 400 that would otherwise be classed invalid_request is reclassified as auth when its message names an API key, so a wrong provider key is reported as a credential problem instead of being blamed on the caller’s body.
- Tool schemas are stripped, not passed
- Gemini’s function declarations take an OpenAPI subset and reject the whole request on a keyword they do not know, so $schema, $id, $ref, $defs, $comment, definitions, additionalProperties and patternProperties are removed before egress. The walk is bounded at 500 nodes and 12 levels, because a caller-supplied schema’s depth, size and shape — including a reference cycle — are attacker-influenced rather than trustworthy.
- No system role and no assistant role
- The system prompt is a top-level systemInstruction, the assistant is called model, and consecutive same-role turns are rejected upstream, so turns are merged on the way out. A conversation that round-trips cleanly through an OpenAI-shaped proxy will not necessarily round-trip through a naive Gemini translation, which is why this dialect is spoken natively.
- Passthrough is bounded to five fields
- safetySettings, toolConfig, generationConfig, labels and serviceTier may be set by the caller. Everything the gateway governs — contents, systemInstruction, tools — is written after those and cannot be overridden from a request body.
The change, and why there is no single variable to set
Point the client at your gateway and give it an agent key where a Google key would go. The credential travels on x-goog-api-key, which is where the Google SDKs already put one — at the gateway that header carries the agent key, never the provider key, and the upstream client injects Google’s own credential separately. Authorization: Bearer works as well, and takes precedence when both are present.
Unlike the OpenAI and Anthropic dialects, there is no ambient environment variable that every Gemini client agrees on for the endpoint. The Google SDKs take it through their HTTP options, so the change is usually a constructor argument rather than a line in an environment file. A base-URL change is still the normal way in for this dialect, but check your own SDK and its version before assuming a variable exists: the client libraries here have moved faster than the others, the older and newer Python packages differ, and the two API prefixes are both in live use — which is why the gateway accepts /v1beta/models and /v1/models rather than picking one and being right for half the fleet.
What you get in return is the same as on every other dialect. The agent has an identity resolved from the registry on each request, a deny-by-default permission set, budget and rate ceilings decided before egress, a policy verdict, redaction on the way out and a trace that exists whether the request succeeded or was refused. Errors come back in the Gemini envelope with the typed governance code carried in a details entry, so a native client sees a shape it can parse rather than an OpenAI error wearing Google’s status codes.
Two limits belong beside that. Token Observe is inline, so if it is down governed agents cannot call models — that is the cost of a decision point rather than a report. And the gateway sees the request and the response, not the deliberation between them: thinking tokens are counted and priced, and the reasoning itself is not recorded because it never arrives.
Five structural differences, and why a compatibility shim would lose them
It would be cheaper to translate Gemini into the OpenAI dialect and reuse one adapter. That approach loses information at five separate points, each of which shows up as an error the caller cannot diagnose, so this dialect is spoken natively instead.
The model lives in the URL rather than the body, and streaming is a different method rather than a flag — so a proxy that reads the model from a body field finds nothing, and one that sets stream: true gets a buffered answer. There is no system role and no assistant role: the system prompt is a top-level field and the assistant is called model. Consecutive same-role turns are rejected upstream, so they are merged on the way out rather than forwarded and refused. Tool schemas are an OpenAPI subset rather than JSON Schema, and a keyword Gemini does not know fails the entire request, so definitions authored for another vendor are stripped before egress under a bounded walk. And a safety refusal arrives as a 200 with a block reason and no candidates, which any status-based classifier reads as a success.
That last one is the difference that matters most to governance rather than to compatibility. A refusal read as a success would be recorded as a success, and if the router then failed over, the same payload would be tried at the next vendor with the objection nowhere in the record. Mapping it to content_policy puts it in the class that never fails over, which is the whole reason the failure classes are typed rather than counted.
The bounded walk over tool schemas deserves the same treatment. The schema is caller-supplied, so its depth, size and shape are attacker-influenced: a reference cycle in a tool definition would be an unbounded traversal inside the request path. The walk is capped at 500 nodes and 12 levels and the reference machinery is stripped wholesale — $defs, definitions and $id go alongside $ref, because they exist only to serve it and keeping them would ship a payload of dead schema to be re-tokenised on every turn.
- One candidate per call
- candidateCount values other than 1 are rejected, because the gateway governs one candidate and partly evaluating a set of alternatives would mean redaction and policy applying to some of what the caller receives.
- Cached content is not forwarded
- cachedContent, opaque file references and built-in Google tools are refused rather than passed outside policy inspection. Inline a cached turn as text in contents instead — which costs tokens, and is stated rather than hidden.
- Thinking and non-text modalities
- Refused until the opaque state and the media can be governed end to end. Provider-specific text generation and safety settings still require a route to a provider that understands them.
- Model listing is paged and capped
- The upstream catalogue is fetched 200 entries at a time and pagination is followed for at most five pages, because every extra page is another network call inside one deadline and a catalogue larger than that is a provider change worth noticing rather than a loop worth running.
Token accounting on Gemini, and what the numbers mean
Gemini’s usageMetadata follows OpenAI’s convention: cachedContentTokenCount is part of promptTokenCount rather than sitting beside it. The adapter therefore hands the figures to the same inclusive normaliser the OpenAI adapter uses, which subtracts the cached count to produce the uncached bucket, and the result is the same four mutually exclusive buckets every other provider produces — uncached input, cache reads, cache writes and output. Anthropic is the exception in the other direction, reporting cache tokens alongside the input total, and that difference between the two conventions is the single most common source of a cost figure that cannot be reconciled against an invoice.
The output side needs one addition that the other dialects do not. candidatesTokenCount alone under-reports a reasoning call, because thinking tokens are billed and are counted separately, so thoughtsTokenCount is added to it before normalisation. There is no equivalent adjustment on the input side, and no attempt to reconstruct what the thinking contained: the gateway sees the request and the response, and the deliberation between them never arrives.
Everything downstream of that is provider-agnostic. Prices are matched on provider kind and model, exact rows beat wildcards, and the longest pattern wins among equals. Where a price row states no explicit cached-read rate, cache reads bill at the full input rate, which reads slightly high against a discounting invoice and never under-bills. And if a USD budget is configured while any candidate on the resolved route has no active price row, the request is refused before egress with ACP_BUDGET_UNPRICED as a 409 rather than being priced at zero, because a candidate treated as free disables the budget above it.
On a stream, the counters are cumulative and some endpoints emit more than one usage frame. The merge keeps the greatest validated value seen for each bucket, which is conservative — it never credits a budget, and it does not mistake a repeated total for an increment.
What fails over from Gemini, and the 200 that must not
Three failure classes walk down a route rule’s fallback chain: timeout, rate_limited and server_error. Four do not: context_too_long, content_policy, auth and invalid_request. On this provider two of those mappings need provider-specific handling, and both are worth knowing because they change which of your requests survive an upstream problem.
The first is the safety refusal that arrives as a 200 with promptFeedback.blockReason and no candidates. It is raised as a content_policy failure so the router cannot retry it elsewhere. The alternative — treating it as a success because the status said so — would record a completed call with no content and no objection, and would let a second vendor be asked the same question with the first vendor’s refusal invisible. Only the enum names travel into the message, because the blocked text is the caller’s payload and belongs on the trace rather than in an error string.
The second is the invalid credential. Google answers a bad key with 400 INVALID_ARGUMENT rather than 401, which the shared status classifier would file as a malformed request from the caller. A 400 whose message names an API key is reclassified as auth, which both attributes the fault correctly and keeps it in the no-failover set: a wrong provider key fails identically at every provider that shares it, and failing over would mask a broken key behind a more expensive upstream until the invoice arrives.
Above all of that, the per-provider circuit breaker opens after five consecutive failures and stays open for thirty seconds. It is keyed by provider id in a map that configuration refreshes never rebuild, so a flapping upstream is not handed a clean slate every time somebody edits a row, and the metrics gauge reads the same instance the request path consults rather than a second breaker that would report healthy during an outage.
The parts of the platform this uses
Model routing
Six upstreams behind one set of policies, and a fallback chain that will not launder a refusal.
Policy engine
One deterministic verdict on every governed request: allow, block, redact, or park it for a human.
Spend controls
Hard USD ceilings, per-minute rate limits and a kill switch, all decided before the request leaves your network.
Flight recorder
Every governed request in a timeline a compliance officer can read, and a search box that never writes SQL.
The same policies apply identically whichever provider serves the request, and that equivalence is enforced by a test over every provider kind rather than asserted.
See the request pathThe rest of the upstreams
Which endpoint variable do I set for the Google SDK?
There is not one that every Gemini client agrees on, which is the honest answer and the reason this page shows a constructor argument instead. The Google GenAI SDKs take the endpoint through their HTTP options, so the change is usually a client construction argument rather than a line in an environment file, and your own SDK and its version is the first thing to check before assuming a variable exists. What is fixed is what arrives at the gateway: POST to /v1beta/models/{model}:generateContent or :streamGenerateContent with the agent key on x-goog-api-key. Both API prefixes are accepted — /v1beta/models and /v1/models — because deployed clients use both.
Why is a blocked prompt not an error status?
Because Gemini returns it as a 200 carrying promptFeedback.blockReason and no candidates, and the gateway treats that as a content-policy failure rather than a success. That matters for two separate reasons. A refusal recorded as a success is a hole in the evidence — the call looks completed and the objection is nowhere. And content_policy is one of the four classes that never fail over, so the router cannot send the same payload to the next vendor in the chain and record a success where a safety system had declined. Only the enum names reach the error message; the blocked text stays in the trace where it belongs.
Why were my tool definitions changed on the way to Gemini?
Because Gemini’s function declarations take an OpenAPI subset rather than JSON Schema, and reject the entire request on a keyword they do not recognise. A tool definition authored for OpenAI or Anthropic is therefore stripped before egress: $schema, $id, $ref, $defs, $comment, definitions, additionalProperties and patternProperties are removed, with the reference machinery going wholesale because $defs and definitions exist only to serve $ref and keeping them would ship dead schema to be re-tokenised every turn. The walk that does the stripping is capped at 500 nodes and 12 levels, because a caller-supplied schema’s shape — including a reference cycle — is attacker-influenced rather than trusted to terminate.
Are Gemini’s cached tokens counted the same way as Anthropic’s?
No, and this is the difference most likely to produce a cost figure you cannot reconcile. Gemini follows OpenAI’s convention: cachedContentTokenCount is inside promptTokenCount, so the uncached figure is the subtraction of one from the other. Anthropic reports cache reads and cache writes alongside input_tokens rather than inside it. Both are normalised into the same four mutually exclusive buckets before any arithmetic, which is exactly why the normalisation lives in each provider’s adapter rather than in the ledger. One Gemini-specific addition: thinking tokens are added to the candidate count to form the output figure, because they are billed and dropping them would under-report the expensive half of a reasoning call.
Can I use Gemini’s context caching through the gateway?
Not through cachedContent, which is refused before egress along with opaque file references, built-in Google tools, thought state and non-text response modalities. The reason is uniform: governance evaluates the payload it is shown, and an opaque handle to content stored at the vendor is content the policy layer, the redaction pass and the injection detectors never see. Inline the cached turn as text in contents instead. That costs tokens, and it is stated here rather than presented as a limitation of your client — the trade is between a cheaper call the gateway cannot inspect and a more expensive one it can.
Name the SDK and the version.
A base-URL change is the normal case for the Google Gemini dialect, and whether your own client library and version behave that way is the first thing worth checking. Say which you use and you will get a straight answer.
no form · no qualification step · no sales desk · the other three ways in