performance
Posts tagged "performance".
- The Cluster Was Idle the Whole Time
A Fabric notebook took three hours to copy fifty thousand files from ADLS Gen2 into a lakehouse. The Spark cluster behind it — dozens of cores — sat almost completely idle while one driver thread made the round trips one at a time. Distributed compute only speeds up the work you actually hand to it.