Fabric Without Spark

“Fabric notebooks” usually means Spark. But Fabric also ships a regular Python notebook — a plain Python kernel, no Spark context, with Polars and DuckDB already installed. It starts in a couple of seconds instead of waiting on a cluster to spin up, and for data that fits on one machine that’s a real advantage. I wanted to know how far it goes on something past a toy, so I built a full NYC Yellow Taxi medallion pipeline on it — Bronze, Silver, Gold, delta-rs writes, a Data Factory loop over months — deliberately without Spark, on the smallest capacity Fabric sells: F2, two vCores, 16 GB.

The transformations were the easy part. The interesting bits were at the boundaries.

At this size, the engine barely matters

I benchmarked three engines — Polars, DuckDB, pandas — across three workloads on a single month of trips: 3,475,226 rows, 56 MiB of Parquet. Workload A is a filtered aggregation, B is a taxi-zone join, C is a scan of the partitioned Silver Delta table. Each number below is the median of five runs, with the engine order rotated every iteration so no engine gets to ride a warm cache the others paid to fill, and with every run’s row counts and totals checked against Polars before its time was allowed to count. A fast wrong answer isn’t a benchmark result.

A = filter + group-by, B = join + group-by, C = partitioned Delta scan + group-by. Median of five runs; memory is observed process RSS, not Fabric capacity usage.

The headline is boring in the good way: a representative month finishes in well under a second on the quick engines, and the single slowest number anywhere is pandas at 1.2 seconds. At this size, which engine you pick barely moves the wall clock.

Where they differ is the shape of the difference. DuckDB was fastest or tied on all three workloads and carried the least memory — the steady pick for these SQL-shaped aggregations. Polars was right there on speed but the most variable run to run; its Silver-scan times swung enough that I’d call them unstable rather than merely noisy. pandas was reliable and the most familiar, but about five times slower on the filter-heavy workload, and the first to lean on the memory ceiling: 2.1 GB peak on the join against roughly 1.9 for the others. On 16 GB that’s comfortable — but it’s also the number that would bite first if the month grew.

None of this is a Spark comparison. I never ran Spark, and the point isn’t “Python beats Spark.” It’s that for a month of this size the single-node engines are more than enough, and you skip the cluster entirely.

One honest limit: those memory figures are process RSS, sampled every 10 ms on one F2 session — not Fabric capacity-unit consumption, and not throttling. The Capacity Metrics app needs a Power BI license I didn’t have, so the benchmark deliberately makes no claim about platform-level cost. It answers a narrower question — does a representative month finish, correctly, quickly, and repeatably, on this box? — and the answer is yes.

The stack, and reading and writing

Everything is preinstalled — there’s no pip step. The split I settled on:

• Polars for the expression-heavy cleaning — lazy scans and columnar expressions.
• DuckDB for SQL-shaped aggregation; it queries Parquet and Arrow objects directly.
• PyArrow as the interchange format and for reading Parquet metadata.
• delta-rs (the deltalake package) to read and write Delta Lake without Spark.
• pandas for the familiar baseline.
• notebookutils for the runtime context and storage tokens.

Everything speaks Arrow, so handing a table from Polars to DuckDB to delta-rs is zero-copy and boring — which is the point.

Reading Parquet is a one-liner in any of them:

import polars as pl, duckdb, pyarrow.parquet as pq

lf   = pl.scan_parquet(path)                       # lazy Polars frame
rel  = duckdb.sql(f"FROM read_parquet('{path}')")  # DuckDB SQL
meta = pq.ParquetFile(path).metadata               # row/column counts, for validation

(One caveat lives below: against the mounted OneLake path, copy the file to local disk first.)

Reading Delta takes the table URI and a storage token, then it’s Arrow the rest of the way:

from deltalake import DeltaTable
arrow = DeltaTable(delta_uri, storage_options=opts).to_pyarrow_table()
# or straight into Polars: pl.read_delta(delta_uri, storage_options=opts)

That opts is the storage-options dict from the write example below — reading needs the same OneLake credentials.

Writing Delta is the least obvious part, so here it is whole. Pull the attached Lakehouse IDs and a storage token from the runtime, build the OneLake ABFS URI, and write:

ctx  = notebookutils.runtime.context
opts = {
    "bearer_token": notebookutils.credentials.getToken("storage"),
    "use_fabric_endpoint": "true",
}
target = (
    f"abfss://{ctx['defaultLakehouseWorkspaceId']}@onelake.dfs.fabric.microsoft.com/"
    f"{ctx['defaultLakehouseId']}/Tables/silver_trip"
)

from deltalake import write_deltalake
write_deltalake(
    target, arrow_table,
    mode="overwrite",
    partition_by=["service_year", "service_month"],
    predicate="service_year = 2025 AND service_month = 1",   # replace one month, not the table
    storage_options=opts,
)

That predicate is the useful bit — it overwrites only the matching partition, so re-running a month replaces just that month, the same keep-the-runner-dumb idempotency as the atomic Bronze download (stage a temp file, validate the Parquet footer, then swap it into place). If you’d rather skip Delta, plain Parquet is pq.write_table(...) or df.write_parquet(...) written into the Files/ area.

One honest boundary: delta-rs is not Fabric Spark. It’s solid for reads, appends, and partition replacement, but it doesn’t implement every Delta feature — check requirements before betting a production workload on it.

The tables show up on their own

The part that surprised me: I never wrote a CREATE TABLE. Writing the Delta files under the Lakehouse’s Tables/ folder is the whole registration — Tables/silver_trip becomes a table because Fabric treats a valid Delta folder there as a managed one. It appears in the Lakehouse explorer with a table icon (the Delta log and Parquet files beneath it show as Unidentified, which is expected — those are the physical files, not separate tables), and the SQL analytics endpoint auto-discovers it and exposes dbo.silver_trip to query, with no DDL anywhere. /Tables is the contract: write valid Delta there and you get a table; write under /Files and you get files. The one wrinkle is timing — the SQL endpoint runs its own metadata sync, so a table already visible in the Lakehouse can take a moment to appear on the SQL side. Refresh the endpoint metadata before deciding a write didn’t land.

The boundaries fought back

The medallion code was unremarkable: scan Parquet, clean with Polars, aggregate with DuckDB, write Delta. The scars were all at the seam between my code and the platform — and these two are the ones you’ll actually hit doing this, not quirks of my particular setup.

The mounted OneLake path isn’t a normal local path. Reading the OneLake-backed Parquet directly failed with Generic LocalFileSystem error: Upload aborted — mid-parse, on a file that was perfectly valid. The fix was to copy it to the notebook’s actual local disk first, validate the metadata there, and hand that to Polars. Different libraries exercise /lakehouse/default differently, and a local snapshot cleanly separates “the storage integration is being weird” from “the input is corrupt.”

Don’t mix GUID and friendly-name URIs. FriendlyNameSupportDisabled. OneLake accepts either workspace-name/lakehouse-name.Lakehouse/… or workspace-guid/lakehouse-guid/…, and I’d pasted the .Lakehouse suffix onto the GUID form. One stray suffix, and the error surfaced several layers down in the object store, nowhere near the line that built the path.

None of these are hard once you know them. All of them cost more time than the actual data work, and not one of them shows up in a demo.

The real question isn’t Python versus Spark

It’s the smallest execution model that reliably covers the workload’s size, growth, and compatibility. For a monthly batch that fits on one node, maps cleanly onto Polars or DuckDB, and needs no advanced Delta features, the Python notebook is a compact stack you can iterate on fast. Spark is still the answer the moment the data outgrows one machine, multiple writers need real concurrency, or you need Delta behavior and engine support that delta-rs doesn’t provide.

This is the same F2 medallion setup I used for the NHTSA evidence system, and the same lesson landed from a different direction: the transformations are the part everyone shows. The boundaries — filesystem quirks, URI conventions, metadata sync — are the part that decides whether it holds up.