I use coding assistants across several kinds of projects, and their value varies widely. On some they save hours. On others they produce plausible mistakes that take longer to review than the original work would have taken.

I first blamed task difficulty, but some of the more complex projects were the ones where the assistant helped most. A better predictor was whether the output came with a cheap, deterministic check.

Two projects with different checks

Take a visual ETL tool — a canvas where you wire sources into transforms into sinks. AI assistants are excellent at this, and there are structural reasons why:

• The vocabulary is bounded. Sources, sinks, joins, filters, transforms. A small, closed set of concepts that recur everywhere.
• The pieces compose cleanly. Each node has explicit inputs and outputs. The boundaries are drawn for you.
• There’s a mountain of training data. Decades of Airflow, dbt, SSIS, Informatica, and every ETL pattern ever blogged.
• And, above all, the output is verifiable. Data goes in, a transform runs, data comes out. Does the SQL execute? Do the row counts reconcile? Did the schema survive? These are binary questions with mechanical answers.

Now take a different kind of project — say a tool that’s supposed to weigh arguments, or one that builds a knowledge graph out of messy documents. The assistant is markedly worse here, and it’s the same assistant. What changed is that the questions stopped having mechanical answers. Was that a good argument? Is this the right relationship between these two entities? Did it capture what the document actually meant? There is no query you can run that returns true or false. The output is plausible by construction — that’s what these models are built to produce — and plausibility is exactly the thing you can’t verify by machine.

Verifiability predicted the outcome better than difficulty

The more deterministic the check on an output, the more of that work you can safely hand to an AI.

When a cheap automatic check exists—the code compiles, tests pass, a schema validates, or numbers reconcile—the assistant can move quickly because errors surface immediately. Without such a check, each plausible output requires a careful human review.

The dividing line is verifiability, not difficulty: can a machine tell when the output is wrong?

I’ve written before that in a real system, AI should suggest while deterministic code decides. The same rule also predicts where an assistant is likely to save time during development: the surrounding system needs a reliable way to reject bad work.

What it looks like in practice

The AI app-builders are a nice illustration, because they’re genuinely impressive right up until they aren’t. Point one at a standard web app — the well-trodden shape, backed by the standard stack — and it scaffolds most of it startlingly fast. Then you steer somewhere off the beaten path (an in-browser analytical database, a stateful graph editor with its own connection rules) and it gets you maybe seventy percent of the way before you spend as long fighting the last thirty as you’d have spent building the whole thing.

The fast part combines familiar patterns with an obvious check: the application runs or it does not. Progress slows when the design becomes novel and correctness can no longer be established by executing the result.

How I use this distinction

When I scope a build now, I identify its available checks before deciding how much work to hand to an assistant.

Where the check is cheap and automatic, I let the assistant run — draft the transform, wire the nodes, generate the boilerplate — because a failing check will tell me the moment it drifts. Where there’s no check, I do one of three things: keep the human firmly in the loop, narrow the task until a check does exist, or spend the effort to build the check first and only then hand over the work behind it.

The practical question is: if the assistant gets this wrong, what will notice? When the answer is a test, compiler, schema check, or reconciliation, I can move quickly. When the answer is only my own later review, I keep the task smaller and the human decision explicit.

Next in “AI, and the work you can check” Four Results That Looked Better Than They Were →