It's easy to say the gates are the important part. It's tougher to build one you'd actually bet on. A gate you've never watched do its job isn't really a safety mechanism. It's just an optimistic construct that helps you sleep better at night.
I've been thinking about this a lot lately because more and more work is moving to systems that run without someone sitting there and watching. When that happens, the thing you author stops being the implementation and starts being the gates, specs, and budgets around it. So the gates had better be real.
What you're actually building
When you delegate work to a system or a person what you miss out on isn't necessarily accuracy or quality. In fact, it's inarguably to your benefit to hand off work to someone (or something) that can do it better than you can. The concept of specialization has been a cornerstone of modern economics for exactly this reason.
What you miss out on is the firsthand exposure to what happened and why.
Anyone who's been in a position where delegation mattered as much as execution learns this fast. The bulk of the work you're responsible for trickling up from your reports, your team, or your agents isn't something you can keep under a magnifying glass. You can't know every fine-grained detail. You familiarize yourself with strengths and weaknesses, calibrate trust accordingly, and if you fail to delegate, you either crash and burn or drive yourself insane trying to be one person doing the work of many.
So when a stakeholder asks how one of your reports implemented a feature, the answer isn't coming up with how you would have done it. It's circling back. Checking your report's notes, asking the relevant follow up questions, and taking the time to understand the thought process that led them to the solution they landed on.
Now, some of you see where I'm going. Others are asking wtf any of this has to do with systems or data platforms.
Here it is: the most fundamental piece of infrastructure you build isn't the part that executes the work. It's the wrapper around it. It's how well it logs decisions, the information it observed to reach those decisions, and the order of operations chained through the causal links. It's logging timestamps against each relevant event in a process. It's making the intentional choice about whether to open a database connection inside the inner function or the outer wrapper, because that single placement decides whether a failure is still recordable after it happens. The work itself isn't the part you're authoring anymore. The envelope is the real product.
This might sound like everyday plumbing. But these are the small decisions we all make left and right that can prevent problems months or years later when refactoring the foundational parts becomes brutally expensive.
Logging the failure that broke your logger
Recording when something succeeded is trivial for the most part. Recording when something failed takes real thought and it's harder than it looks for a funny reason. The most important failure to record is often the same one that also broke your ability to record it.
I spent a past life monitoring mission critical systems for regulatory compliance purposes and there were cases where we could flag that a problem happened but eventually realized that troubleshooting the why was flat out impossible. Those were always the toughest ones to swallow. These were seemingly simple design decisions made years earlier that came back to haunt us in ways nobody could have foreseen. The engineers who made them had usually moved on to greener pastures by then, so they weren't around to help answer the questions those decisions raised.
They were not at fault for this either though. There was no reason for them to plan for a world where the systems they built would later sit at the center of audits and regulatory scrutiny. But things are different now. The audit trails that used to be optional, arguably overengineering, back when data privacy and safety were still the wild west are now the same rails the automation relies on to track, learn, and evolve. If you're working with systems that lean heavily on agentic AI, this kind of foundational logging is the cornerstone of whether you can actually operate the thing at all. Without it, you're not running an autonomous system so much as praying at one.
Here's the concrete version. Say the way you write a "this failed" note is by writing a row to the same database via the same connection that the failed work was using. Now the failure happens. Maybe the transaction aborted or the connection itself died. Either way that connection is now poisoned - it rejects every command you send it until someone resets it. Your recording mechanism is sitting on a corpse. The one event you most needed to capture is the one you structurally can't because the tool you'd capture it with is part of what broke.
The fix isn't complicated once you see it: the failure note has to go out on a fresh connection, one that had nothing to do with the work that just died. You assume the working connection is poisoned and you don't touch it. You open a clean one, write the failure, and only then clean up.
But we shouldn't simply trust that fix because it sounds right. It should only be trusted when there's a test that deliberately poisons the connection first. It runs a query guaranteed to abort the transaction, confirms the connection is genuinely dead by checking that it now rejects everything, and then asserts that the failure event still landed on a different connection. The test doesn't check that the code tried to do the right thing. It breaks the thing on purpose and confirms whether or not the gate actually catches it.
That's the difference between a gate and a decoration. A decoration is code that would do the right thing if the bad case ever happened. A gate is code you've watched handle the bad case because you triggered the bad case and witnessed it working.
And this is the thing that generalizes. When the system runs without you, every safety property has to be enforced by construction rather than by hoping the good path holds. The good path always holds. That's why it's the good path. The engineering is entirely in the other paths. I.e. the ones you only see if you trigger them yourself.
The gate you're better off not needing
So far this has all been about building gates you can trust. But before you reach for a gate at all, theres often something more useful to consider: whether you can make the failure it guards against impossible in the first place.
In that regulatory compliance world, the default instinct was to enumerate every way a thing could fail and build a monitor for each. A single control might have a dozen distinct failure modes and the reflex was usually to build a dozen distinct monitors. One for each mode, each with its own alerting, thresholds, and maintenance burden. Stack them to check off the right boxes and you consider it fully covered.
But the biggest takeaway I walked away with was almost the opposite. In many cases, just a couple lines of changes in the upstream system itself could collapse half those failure modes entirely. Not monitor them better but make them cease to exist. A field that could arrive malformed a dozen ways gets constrained at the source and suddenly eight of your twelve monitors are watching for something that can no longer happen. That was orders of magnitude more robust than any amount of monitoring because the most reliable monitor is the one you eliminated the need for in the first place.
This is the move that doesn't show up in a dashboard and doesn't feel like progress in the moment. Adding a monitor feels like work. Removing the reason for a monitor feels like nothing. You ship two lines and delete eight alerts and there's no satisfying deliverable to point at. But fewer things that can break beats creating more monitoring for them any day. A gate you built and tested is good. A failure mode you made impossible is better because it no longer needs a gate at all.
The catch is that this only works on the failure modes you can see. You can't collapse an upstream cause you haven't recognized as a cause, and you can't gate a failure you never imagined. Everything in this post, the envelope, the fresh connection, the deleted failure modes, only covers the cases someone already thought of. Which means the failure that eventually bites is the one nobody saw coming. And the more of the work runs on its own, the more that's the only kind left to worry about.