There’s a small configuration decision that separates systems that survive reality from systems that merely demo well, and it comes down to one question: when a dependency dies, what does your system do by default?

Most things are built to fail open or fail fast — when the thing they rely on goes away, they either keep running without it or error out quickly and carry on. For anything where the dependency was there for a reason, that default is exactly backwards. You want to fail closed: when the thing you depend on is gone, you stop. You don’t proceed.

The case that made it concrete

I run a service on my home network whose entire purpose depends on its traffic going through a VPN tunnel. If the tunnel is down, that traffic must not go out at all — that’s the whole point of the setup.

The naive configuration fails open. The tunnel drops, the service keeps running, and now it’s happily sending traffic straight out the normal connection — doing the exact thing the tunnel existed to prevent, silently, until you happen to notice. And you won’t notice, because the demo works: when you set it up, the tunnel is up and everything looks perfect. Reality is the 3am moment the tunnel drops and nothing tells you, and by then the thing you were protecting against has been happening for hours.

What fail-closed actually takes

Building it to fail closed is a handful of unglamorous details, and every one of them is a spot people forget:

• Pin DNS so name lookups can’t leak to your ISP’s resolver even for a moment — bind the resolver, override it, don’t just inherit the host’s.
• Make the tunnel self-terminate on silence — if it goes quiet for a minute, kill it, rather than let it hang half-open pretending to be connected.
• Probe liveness against a known host, so “the tunnel is up” is something you verify on a schedule, not something you assume.
• Carve out the local network explicitly, so you can still reach the box to manage it while everything else is forced through the tunnel.

Notice that none of that is the application. It’s all the failure-mode plumbing around the application — the part that decides what happens on the bad day. Which is the only day that matters, because the good day takes care of itself.

It’s a universal shape, not a networking trick

Once you see fail-open-versus-fail-closed, it’s everywhere, and it’s almost always the same trap.

A budget check that fails open lets spend through when the checker is down. A permission system that fails open grants access when the auth service hiccups. An AI guardrail that fails open ships the unvalidated output when the validator times out — which means it was never really a guardrail, just a thing that happened to be in the way when everything was working. In every one of these, the demo passes because the dependency is up, and reality punishes you because the dependency is precisely what was down when it mattered.

The rule generalizes cleanly: for anything protective, if the thing that makes it safe is unavailable, stop. A check that keeps letting things through while it’s broken is worse than no check, because it also gave you the false confidence of thinking you had one.

Why it’s the harder default

Fail-closed is more annoying, and that’s exactly why people don’t reach for it. It stops you when the dependency flakes, even briefly — you feel every hiccup, and hiccups are common. Fail-open feels smooth right up until the one time smoothness is catastrophic. Choosing the annoying-but-safe default over the smooth-but-treacherous one, again and again, is most of what reliability engineering actually is once you strip the jargon off it.

The demo is fail-open’s best friend, because in a demo the dependency is always up. The question that turns a demo into a system is the one the demo never asks: what happens when the thing you’re relying on isn’t there? Build it so the answer is “it stops” — not “it silently keeps doing the dangerous thing.” Fail closed, not fast.