Stateful Streaming Lakehouse

This jumpstart deploys a full Spark Structured Streaming lakehouse into your Microsoft Fabric workspace — a production-grade medallion pipeline (raw → bronze → silver) processing 9 entities from file and Eventstream sources, powered by ArcFlow, an open-source PySpark streaming ELT framework.

What You'll Learn

This jumpstart is designed as a hands-on workshop (Module 2 of the Stateful Streaming Lakehouse series). It covers two progression levels:

By the end you will have:

• Built a working raw → bronze → silver streaming pipeline
• Understood checkpoint state — how Spark remembers what it already processed
• Used the memory sink to develop safely without writing to Delta
• Seen how packaging transforms as a framework changes testability, observability, and scalability
• Understood the power of Sparks Structured Streaming API for managing state in both streaming and batch use cases

Entities

JSON files follow an envelope pattern: {"_meta": {...}, "data": [...]}. Parquet files are flat with OrganizationId and GeneratedAt columns added by the data generator.

How It Works

Part 1 — Core Concepts with Native Spark

The explore_streaming notebook walks through streaming primitives with zero abstractions:

1. Why Structured Streaming for batch pipelines? — The availableNow trigger gives you batch scheduling semantics with streaming state management. Re-run a job and it only processes new data.

2. Write streaming queries on top of files — Read a file source as a stream, write to Delta with a checkpoint. Re-run it and watch only new data process.

3. The memory sink development pattern — Stream data into driver memory to inspect and iterate on transforms without touching Delta tables. No checkpoints, no cleanup.

4. Eventstream (Kafka) source pattern — The same transforms work whether the source is files or a message broker. Parse the binary Kafka payload through cast → from_json → explode → expand.

5. Build raw → bronze → silver — Schema enforcement, snake_case column normalization, audit timestamps in bronze. Flatten and expand nested structs in silver.

Part 2 — Thinking Like a Software Engineer

The arcflow_elt_framework notebook takes the exact same transforms and packages them with ArcFlow:

The test_input / test_output methods let you validate the full transform chain end-to-end — raw through silver — without writing a single row to storage.

Spark Job Definition

The production pipeline runs as a headless Spark Job Definition (main.py) that:

1. Starts the LakeGen McMillan Industrial Group synthetic data generator — producing orders, shipments, scan events, and reference data
2. Initializes the ArcFlow Controller with pipeline configuration
3. Runs controller.run_full_pipeline(zones=['bronze', 'silver']) with await_termination=True

The same framework, same config, same transforms — just a different execution model. See Notebooks vs. Spark Jobs in Production for the tradeoffs, and Creating Your First Spark Job Definition for a step-by-step guide.

Key Patterns

Stateful Incremental Processing

Every streaming query uses checkpoint state to track what has been processed. Re-running a pipeline only processes new data — no duplicates, no full re-scans, no custom bookmarking logic.

query = (df.writeStream
    .format('delta')
    .option('checkpointLocation', 'Files/checkpoints/my_table')
    .trigger(availableNow=True)
    .toTable('bronze.my_table')
)

File Archival for Streaming from Object Storage

Target Schema

By the end of the workshop, you'll have a complete silver layer:

Resources

• Spark Structured Streaming Programming Guide
• Delta Lake Streaming Reads and Writes
• ArcFlow on GitHub — open-source PySpark streaming ELT framework
• Notebooks vs. Spark Jobs in Production
• Creating Your First Spark Job Definition
• Should You Infer Schema in Production?
• Fabric Dimensional Modeling