The Cluster Was Idle the Whole Time

A Microsoft Fabric notebook ran for three hours to copy about fifty thousand files from ADLS Gen2 into a Fabric lakehouse. That part I could live with. The part that stuck with me was what the Spark cluster behind it was doing for those three hours: almost nothing. Dozens of cores, provisioned and paid for, sat idle while a single thread on the driver made the trips one file at a time.

I had brought a machine built to do a thousand things at once to a job I then insisted on doing one thing at a time.

The math is the whole story

The copy was a plain loop. Roughly:

for f in files:
    mssparkutils.fs.cp(f, dest)

Each iteration is a couple of network round trips — read the file, write the file, touch some metadata — call it 150 to 200 milliseconds once auth and storage latency are in. That feels instant. It isn’t, fifty thousand times in a row:

50,000 files × 0.2 s ≈ 10,000 s ≈ 2.8 hours

There’s no gremlin in there. No mysterious shuffle, no throttling, no bad join. The runtime was exactly the sum of fifty thousand small waits, performed in sequence, by one thread. The executors — the entire reason to be on a cluster at all — never received a single task. Spark can split one big computation across many cores beautifully, and it can do absolutely nothing with a loop you keep to yourself on the driver. I’d written single-threaded code and rented a hundred threads to watch it run.

Two problems wearing the same clothes

The real mistake was upstream of the loop. I’d reached for a distributed compute engine because the data was big — tens of thousands of files, a lift-and-shift between stores — without noticing that the work wasn’t a compute problem at all.

Moving fifty thousand files is orchestration. It’s enumerate-a-list, make-two-calls-each, handle-the-failures — a control-plane job, almost pure IO, with no arithmetic in sight. Spark is a data-plane engine: it earns its keep when there’s one large computation to fan out across cores. Hand it a pile of tiny independent IO operations one at a time and it has nothing to fan out. I’ve written before that in system after system the bottleneck is moving the data, not computing on it — and here was the punchline version of that, where the cost was entirely data movement and I had a cluster capable of moving it in parallel and used one thread anyway.

The fixes, in order of how much they admit the problem

Each step moves the work further off the engine that was never shaped for it.

Hand the list to the executors. Turn the file pairs into something Spark can distribute and let the cores you paid for actually do the copying in parallel. This alone takes the three hours down to minutes. It’s the smallest change and the least honest — you’re still using a compute engine as a file mover, just a parallel one now.
Stop iterating at all. A recursive or bulk copy lets the platform enumerate and parallelize the whole tree itself, instead of you scripting the trip file by file. Less code, and the work happens where it should.
Use the tool built for bulk movement. A Data Factory Copy activity — in Fabric, a pipeline Copy — is purpose-made for “move these bytes over there, in parallel, with retries” — no cluster, no notebook. Notebooks are for logic; they’re a bad place to shovel bytes. For a one-off migration this is usually both the fastest and the least code.

The lesson isn’t “parallelize your loops”

That’s the tactical fix. The actual lesson is the thing that put me in the loop in the first place: choose the engine for the shape of the work, not the size of the database or the prestige of the tool.

A distributed engine charges a coordination tax up front — planning, scheduling, shuffling, spinning up executors — and it only pays off past a break-even, when the working set of a single run is large enough that splitting the job saves more than splitting it costs. Below that line, the overhead is the bill. A nightly incremental that touches a few gigabytes, a reshape of one day’s data, fifty thousand small file copies — a single node finishes these first: a plain parallel loop, a copy utility, an in-process engine like DuckDB on one box. The trap is sizing the engine to the total store — “we have terabytes, so, Spark” — when any given run only ever touches a sliver of it. It’s the same reflex as bolting cluster machinery onto a laptop-scale problem: the tool’s ceiling is impressive and entirely beside the point.

The cluster wasn’t slow and the engine wasn’t broken. Both did exactly what I asked. I’d matched the tool to the size of the data instead of the shape of the job, and then written the job in the one shape the tool couldn’t help with. A bigger cluster would have changed nothing. The job simply wasn’t the kind a cluster is for, and no number of idle cores was ever going to turn it into one.