Fail-closed

The choice is not about reliability but about which error you would rather have. A fail-closed control produces availability incidents: loud, attributable, and fixed by the team that owns them. A fail-open control produces governance incidents: silent by construction, because no error is returned, no alert fires, and the record afterwards shows a clean, successful call. The asymmetry matters most for the failures nobody notices at the time — a policy store unreachable for ninety seconds, a price lookup returning nothing for a model added last week, an audit sink quietly rejecting writes since an upgrade. In each case the fail-open system keeps working and stops governing, and the only signal that anything changed is an absence in a log nobody reads.

The decision recurs at a surprising number of specific points, and each one has a concrete failure attached. An unpriced model estimated at zero disarms every spend ceiling above it while the console keeps displaying the ceiling, so a price that cannot be resolved has to refuse rather than assume. A required data-policy constraint — zero retention, no training on payloads, a named serving region — that cannot be satisfied by any candidate on the fallback chain has to be a typed refusal rather than a quiet downgrade to a provider that does not meet it. An inspection bound on depth, node count or string length that is exceeded has to refuse rather than forward the tail it did not read, because a payload that skips the part nobody walked is shaped exactly like the thing an attacker would build. A delegation hop that cannot be resolved has to contribute nothing rather than be dropped from the chain. And an audit record that cannot be written should stop the write it was supposed to describe, since an action with no record is the outcome the record exists to prevent.

Fail-closed is not usefully applied at one granularity to everything, and getting the boundary wrong turns a control into an outage generator. The distinction that matters is between an intrinsic failure and a diagnostic one. A content hash that does not match, or a gap in a sequence, is intrinsic evidence that the system’s own state is wrong, and latching the install into a refusing state until somebody looks is proportionate. A caller-supplied expectation that does not match — a mistyped hash in a verification request — is a statement by an outside party about what it believed, and treating it as proof of corruption hands any caller the ability to take the control down with a typo. Design the latch for the first and report the second.

Two details decide whether a fail-closed refusal is usable. The first is what the caller receives: a typed error code the client can branch on, rather than a generic 500 that is indistinguishable from a crash, means an agent can retry a rate limit and stop on a policy refusal instead of treating both as transient. The second is streaming. Once the first byte of a streamed response is on the wire the status line is spent, so a refusal reached mid-stream cannot be an HTTP status at all and has to arrive in band as a typed error frame carrying the same code the buffered path would have returned. A system that fails closed on buffered replies and silently completes streamed ones has a control that is off for most production traffic.

The cost belongs in the same breath as the property, because it is the objection every reviewer will raise and it is a fair one. A fail-closed control in the request path is a single point of failure for everything it governs; if it stops, the agents it governs stop. That is the trade being bought, and the right way to buy it is with a named owner for the question of what happens when the control is unavailable, decided in advance rather than during the incident. What should not be bought is a fail-open switch to reach for under pressure. A control that can be turned off during an incident will be turned off during an incident, and the incident is exactly when the actions it governs are least well considered.