Fail Closed, Not Fast
When a protective dependency disappears, a system can continue without it or stop the protected operation. For a VPN, authorization service, budget check, or output validator, continuing can be the dangerous choice. Those paths should fail closed.
A VPN-dependent service made the failure visible
I run a service on my home network whose entire purpose depends on its traffic going through a VPN tunnel. If the tunnel is down, that traffic must not go out at all — that’s the whole point of the setup.
In the naïve configuration, the tunnel could drop while the service continued over the normal connection. Setup testing would not expose that path because the tunnel was available during the test. The relevant case was an unattended failure hours later.
Four pieces made the network path fail closed
Building it to fail closed is a handful of unglamorous details, and every one of them is a spot people forget:
- Pin DNS so name lookups can’t leak to your ISP’s resolver even for a moment — bind the resolver, override it, don’t just inherit the host’s.
- Make the tunnel self-terminate on silence — if it goes quiet for a minute, kill it, rather than let it hang half-open pretending to be connected.
- Probe liveness against a known host, so “the tunnel is up” is something you verify on a schedule, not something you assume.
- Carve out the local network explicitly, so you can still reach the box to manage it while everything else is forced through the tunnel.
None of this changed the application. It defined the network behaviour when the dependency was unavailable.
The same choice appears in authorization and validation
A budget check that fails open lets spend through when the checker is down. A permission system that fails open grants access when the auth service hiccups. An AI guardrail that fails open ships the unvalidated output when the validator times out — which means it was never really a guardrail, just a thing that happened to be in the way when everything was working. In every one of these, the demo passes because the dependency is up, and reality punishes you because the dependency is precisely what was down when it mattered.
The rule generalizes cleanly: for anything protective, if the thing that makes it safe is unavailable, stop. A check that keeps letting things through while it’s broken is worse than no check, because it also gave you the false confidence of thinking you had one.
Safety is less convenient
Fail-closed behaviour makes every dependency interruption visible to users and operators, including brief ones. Fail-open behaviour appears smoother because it hides those interruptions, but the protected action may continue without the protection that justified the design.
For each protective dependency, the design needs an explicit answer for its unavailable state. In this network service, that answer is to stop external traffic while retaining a narrow local management path.
Next in “Day two, when the demo has to run” How Twelve Rejected Files Became Unreloadable →