Circuit breaker

There are three states and the transitions between them are the whole design. Closed is normal: traffic flows and consecutive failures are counted. Open means every request is refused locally, with no network call at all — this is the state that buys back the latency. Half-open is the recovery test: exactly one request is allowed through, and its result decides everything. Success closes the breaker and resets the counter; failure re-opens it and restarts the cooldown from scratch. It is worth deriving half-open from the clock rather than storing it as a separate state — open plus elapsed cooldown reads as half-open — because a breaker that has to be woken by a timer is a breaker that can get stuck open when the timer is lost.

What counts as a failure is the decision that separates a useful breaker from one that takes healthy providers out of rotation. An HTTP 400 caused by a malformed request says nothing whatsoever about the provider’s health; if it counts, one client sending bad JSON in a loop can open the circuit for everybody else. So only genuinely transient classes should count — a timeout, a rate limit, a server error — and everything that is a property of the request or the credential should not. This has a pleasant consequence: an expired or revoked vendor key produces a fast, typed authentication error on every call rather than an open circuit, which points the operator at the environment variable instead of at an apparent outage. Threshold and cooldown are ordinary tuning; five consecutive transient failures to open and thirty seconds to a probe is a defensible starting point for a model provider, where an outage is usually minutes rather than milliseconds.

Scope matters as much as the thresholds. The unit is the dependency — one breaker per upstream provider and credential, not one per model and not one for the estate — because a provider is what actually goes down. Beyond that, there must be exactly one breaker object per dependency in the process: if the request path holds one map of breakers and the metrics endpoint builds another, the dashboard will report healthy while traffic is being refused, and the operator will spend the outage arguing with the graph. For the same reason, reloading provider configuration must not construct fresh breakers, or every configuration change silently forgives the outage history that was about to protect you.

A breaker sits between two other mechanisms and is often confused with both. Inside a provider, a capped retry with jittered backoff handles the single failed attempt. Above it, a fallback policy decides which provider to try next. The breaker decides only whether a given candidate is worth calling at all: an open circuit means that candidate is skipped and the chain continues to the next one, so a request usually still succeeds, just not there. That ordering is why a breaker is not a substitute for a failover policy. It answers whether to call; the class of the failure answers where to go instead.

The honest limits are three. First, a threshold on consecutive failures never trips for a provider that is failing forty per cent of calls — nothing reaches five in a row — so a breaker tuned this way protects against outages and not against degradation; a failure-rate window catches that case but is harder to reason about and slower to react. Second, counting a timeout as a failure is right for availability and ambiguous for money, because a timeout cannot prove the upstream did not execute and bill the request. Third, a breaker measures transport, not truth: a provider returning prompt 200s full of nonsense is perfectly healthy as far as the state machine is concerned, and detecting that is the job of evaluation, not of a breaker.