#structured-streaming

Articles tagged with #structured-streaming

Watermarking and Late Data Handling in Spark Structured Streaming
Late events are inevitable in distributed systems — watermarks tell Spark exactly when it is safe to finalize a window without waiting forever
Apr 19, 202627 min read138
Spark Structured Streaming: Micro-Batch vs Continuous Processing
Treats streams as unbounded tables — the execution model determines whether your pipeline hits milliseconds or minutes.
Apr 19, 202627 min read126
Stateful Aggregations in Spark Structured Streaming: mapGroupsWithState
mapGroupsWithState gives you a per-key mutable state machine in a streaming pipeline — it is the right tool when windows are not enough
Apr 19, 202628 min read137
Kafka and Spark Structured Streaming: Building a Production Pipeline
Kafka delivers events, Spark processes them — offset management, schema evolution, and exactly-once delivery determine production reliability
Apr 19, 202623 min read205