I build a few different things, and I use AI assistants heavily on all of them. What I keep noticing is that the assistants are not uniformly good. On some projects they feel like a force multiplier. On others they produce confident, plausible, subtly-wrong work that costs me more to review than it would have to write.

For a long time I assumed the difference was difficulty — that the projects where AI struggled were just harder. That’s not it. Some of the projects it handles beautifully are, line for line, more complex than the ones where it flails.

The real dividing line turned out to be much simpler, and once I saw it I couldn’t unsee it: AI is good at the work you can check.

Two kinds of problem

Take a visual ETL tool — a canvas where you wire sources into transforms into sinks. AI assistants are excellent at this, and there are structural reasons why:

• The vocabulary is bounded. Sources, sinks, joins, filters, transforms. A small, closed set of concepts that recur everywhere.
• The pieces compose cleanly. Each node has explicit inputs and outputs. The boundaries are drawn for you.
• There’s a mountain of training data. Decades of Airflow, dbt, SSIS, Informatica, and every ETL pattern ever blogged.
• And, above all, the output is verifiable. Data goes in, a transform runs, data comes out. Does the SQL execute? Do the row counts reconcile? Did the schema survive? These are binary questions with mechanical answers.

Now take a different kind of project — say a tool that’s supposed to weigh arguments, or one that builds a knowledge graph out of messy documents. The assistant is markedly worse here, and it’s the same assistant. What changed is that the questions stopped having mechanical answers. Was that a good argument? Is this the right relationship between these two entities? Did it capture what the document actually meant? There is no query you can run that returns true or false. The output is plausible by construction — that’s what these models are built to produce — and plausibility is exactly the thing you can’t verify by machine.

The boundary is verification, not difficulty

That’s the whole insight, stated plainly:

The more deterministic the check on an output, the more of that work you can safely hand to an AI.

Not “how hard is the task.” Can a machine tell when the output is wrong? Where a cheap, automatic check exists — it compiles, the tests pass, the schema validates, the query runs, the numbers reconcile — an assistant can move fast and you’re safe, because when it’s wrong, something catches it immediately. Where no such check exists, the same assistant hands you confident output with no safety net, and you are the safety net, one careful read at a time.

The dividing line is verifiability, not difficulty: can a machine tell when the output is wrong?

I’ve written before that in a real system, AI should suggest while deterministic code decides. This is the same idea seen from the other end. It’s not just a rule for how to run AI in production — it’s a rule for predicting where AI will help you at all. The tasks AI is good at are precisely the tasks that come with a deterministic check to catch it when it isn’t.

What it looks like in practice

The AI app-builders are a nice illustration, because they’re genuinely impressive right up until they aren’t. Point one at a standard web app — the well-trodden shape, backed by the standard stack — and it scaffolds most of it startlingly fast. Then you steer somewhere off the beaten path (an in-browser analytical database, a stateful graph editor with its own connection rules) and it gets you maybe seventy percent of the way before you spend as long fighting the last thirty as you’d have spent building the whole thing.

That curve isn’t random. The fast part is the part with a dense training corpus and an obvious check — it looks like a million other apps, and you can see immediately whether it runs. The wall is where the pattern is novel and “is this correct?” stops being answerable by looking. The tool didn’t get dumber at the wall. It ran out of ground truth to stand on.

The move this suggests

So when I scope a build now, I look for the verification boundary before I decide how much to lean on AI.

Where the check is cheap and automatic, I let the assistant run — draft the transform, wire the nodes, generate the boilerplate — because a failing check will tell me the moment it drifts. Where there’s no check, I do one of three things: keep the human firmly in the loop, narrow the task until a check does exist, or spend the effort to build the check first and only then hand over the work behind it.

That reframing has been worth more to me than any prompt trick. The question was never “is this task too hard for the AI.” It was “if the AI gets this wrong, will anything notice?” When the answer is yes, go fast. When the answer is no, slow down — you’ve wandered off the edge of what plausibility can be trusted to do.