spark
Posts tagged "spark".
- The Cluster Was Idle the Whole Time
A Fabric notebook took three hours to copy fifty thousand files from ADLS Gen2 into a lakehouse. The Spark cluster behind it — dozens of cores — sat almost completely idle while one driver thread made the round trips one at a time. Distributed compute only speeds up the work you actually hand to it.
- The Missing Column That Didn't Crash
A column vanished from an upstream CSV and nothing broke — no error, no failed job, just wrong numbers for a week, with nothing to announce it. Spark had done exactly what I told it to: map the file to my schema by position, not by name. A loud failure would have been a gift.