Toggle dark/light theme
← Back to catalog
Architecture diagram
Expand

Spark Performance Engineering

Overview

Self-guided, hands-on tutorial for finding and fixing Apache Spark performance problems in Microsoft Fabric. Work through sub-optimal workloads on a synthetic Toy Brick lakehouse, following the code → data → execution tuning hierarchy with before-and-after benchmarks.

Class
Core
Type
Tutorial
Difficulty
Advanced
Deploy Time
~8 min
Complete Time
~90 min

Workloads

Data Engineering

Fabric Items Deployed

  • Lakehouse
  • Notebook
  • Environment
  • Spark Job Definition

Scenarios

Batch ProcessingStreaming

This tutorial deploys a self-guided, hands-on Apache Spark performance lab into your Microsoft Fabric workspace. Instead of abstract theory, you work through real, sub-optimal workloads on a synthetic Toy Bricks Manufacturing & Sales lakehouse — seeded for you during setup — and measure the impact of every fix with before-and-after benchmarks.

⏱️ Deploy time: ~8 minutes. After install you first run the source_to_bronze Spark Job Definition (~30 minutes of data generation), then work through three ~20-minute modules at your own pace — in your choice of the DataFrame API or Spark SQL track.

The Tuning Hierarchy

The lab is organized around the tuning hierarchy — code → data → execution — so you learn to identify which layer a problem lives in and reach for the right lever.

Module Theme Fix lever Examples
1 — Optimizing Code The query is written inefficiently Edit the transformation logic Predicate pushdown, column pruning, avoiding repeated scans, replacing Python UDFs, cartesian joins, driver OOM
2 — Optimizing Tables The table layout is suboptimal OPTIMIZE, clustering, deletion vectors, stats, schema Small-file compaction, optimize-write, Liquid Clustering, data-skipping stats
3 — Optimizing Execution Code and tables are fine, execution isn't Join hints, AQE, partition sizing, caching Broadcast joins, skew handling, shuffle-partition sizing, materialization, Native Execution Engine

How Modules 1 and 3 differ: Module 1 is when the query is written badly (fix = edit the code). Module 3 is when the code and tables are fine but Spark executes it sub-optimally (fix = a hint / config / .cache() / repartition, logic unchanged). Module 2 is when the table layout is the problem (fix = OPTIMIZE / clustering / schema / partitioning).

Choose Your Track — DataFrame API or Spark SQL

All three modules ship in two parallel tracks that teach the exact same concepts and follow the same benchmark loop — pick whichever matches how you write Spark. The exercise numbering is identical across tracks, so you can switch back and forth or work through both.

Track Folder How the queries are written
DataFrame API dataframe-lab/ Fluent PySpark: spark.table(...).filter(...).groupBy(...)
Spark SQL sql-lab/ spark.sql("SELECT ... FROM ...") from PySpark

The Spark SQL notebooks are suffixed _sql (e.g. 02_optimizing-tables_sql). Module 2's SQL track writes to a separate lab schema (bronze_fast_sql) so both tracks can run side by side without clobbering each other.

The Toy Bricks Data Model

Throughout the lab you use a synthetic Toy Bricks Manufacturing & Sales lakehouse. The schema has fact tables (manufacturing events, orders, inventory transactions) at the center and dimensions (parts, colors, themes, sets) on the periphery.

Toy Bricks Manufacturing & Sales ERD
Toy Bricks Manufacturing & Sales ERD

What Gets Deployed

Item Type Role
00_getting-started Notebook Entry point. Verifies prerequisites, orients you to the data model, and points you at the source_to_bronze job to seed the bronze layer.
01_optimizing-code Notebook Module 1 — rewrite the query. Reading the Spark UI, query plans, and Delta metadata, then fixing code-level anti-patterns (predicate pushdown, Python UDFs vs. native / NEE, driver collect()/toPandas() & OOM, cartesian / missing join keys). Ships in both tracks: dataframe-lab/ and sql-lab/ (as 01_optimizing-code_sql).
02_optimizing-tables Notebook Module 2 — change the data at rest. OPTIMIZE / compaction, Optimize Write, Liquid Clustering & data-skipping stats, deletion vectors, data types, partitioning strategy, storage-regression auditing (DESCRIBE HISTORY). Ships in both tracks: dataframe-lab/ and sql-lab/ (as 02_optimizing-tables_sql).
03_optimizing-execution Notebook Module 3 — tune how Spark runs it. Join strategies & broadcast, AQE & skew / salting, shuffle-partition sizing & spill, caching / materialization, streaming. Ships in both tracks: dataframe-lab/ and sql-lab/ (as 03_optimizing-execution_sql).
_benchmark_utils Notebook Shared helpers: df.benchmark(scenario, state) / benchmark_op(...) timers and get_table_metrics() / show_metrics(). Don't run it standalone — each module %runs it.
toy_bricks Lakehouse The synthetic Toy Brick lakehouse. source_to_bronze seeds its bronze Delta tables; the modules read from and re-benchmark against it.
toy_bricks_optimized Lakehouse Companion lakehouse that the source_to_bronze_optimized variant writes to for an optional deep dive comparing the execution and data layout quality produced by the two Spark Job Definitions.
datagen_env Environment Spark environment for the data-generation Spark Job Definitions. Bundles the arcflow and lakegen wheels used to generate and ingest synthetic landing data.
single_node_explore Environment Single-node Spark environment used for the interactive module notebooks.
source_to_bronze SparkJobDefinition Generates synthetic toy-brick landing data (mixed JSON + Parquet) into Files/landing/…, then ingests every table into bronze Delta tables under the bronze schema. Incremental — re-run to grow the dataset.
source_to_bronze_optimized SparkJobDefinition Optimized variant of the seeding job, used to contrast an efficient ELT implementation against the baseline.

How Each Exercise Works

Every exercise follows the same repeatable loop:

Step What you do
🐌 Benchmark Run the workload and capture a baseline time/metric.
🔍 Diagnose Inspect the plan, Spark UI, or Delta metadata to prove the root cause.
🔧 Fix Apply the change — a hands-on challenge, with an inline solution.
🚀 Re-benchmark Re-run and compare against the baseline.
💡 What Just Happened? Read a short explanation of why the fix worked.

Try It — The Workflow

  1. Install — jumpstart.install("spark-performance-engineering") and wait for the ~8-minute deploy.
  2. Seed the bronze layer — open 00_getting-started, then trigger the source_to_bronze Spark Job Definition. Click Run once; it runs for ~30 minutes to generate tables with hundreds of transactions that surface real-world performance challenges. Wait for Succeeded before starting the modules.
  3. Pick your track — decide whether to work in the DataFrame API (dataframe-lab/) or Spark SQL (sql-lab/) notebooks. The modules below are identical across tracks; the Spark SQL versions are suffixed _sql.
  4. Module 1 — Optimizing Code — open 01_optimizing-code in your chosen track. Learn the diagnostic toolkit (Spark UI, query plans, Delta metadata) and fix code-level anti-patterns. ~20 minutes.
  5. Module 2 — Optimizing Tables — open 02_optimizing-tables. Compact small files, apply Optimize Write and Liquid Clustering, tune schema and partitioning. ~20 minutes.
  6. Module 3 — Optimizing Execution — open 03_optimizing-execution. Tune join strategies, AQE and skew handling, shuffle-partition sizing, and caching. ~20 minutes.

Requirements

  • A Microsoft Fabric workspace with Spark enabled and a minimum of an F2 capacity assigned. Approximately 4 CUs are used for the duration of the lab.
  • Permissions to create Notebook, Lakehouse, Environment, and Spark Job Definition items.

Resources