A Structured Streaming output option, called through writeStream.foreachBatch, that hands every micro-batch to your own batch code, which suits MERGE upserts and destinations lacking a streaming writer. Delivery is at least once, so the code should be safe to repeat.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains foreachBatch in context, with comparison tables and the common traps.
Terms in this definition
- Structured Streaming
Spark's engine for near real-time data: you write a batch-like query and it processes new data incrementally, recording progress in a checkpoint with exactly-once guarantees. Auto Loader and streaming tables run on it.
- Micro-batch
Rather than one record at a time, Azure Stream Analytics and Spark Structured Streaming process incoming streamed records in small groups as they arrive; each group is a micro-batch.
- MERGE
A SQL command that compares a source with a target table and, depending on whether rows match, inserts, updates or deletes them. Fabric Warehouse supports it as generally available, and Spark SQL offers MERGE INTO for Delta tables.
- Streaming
Sending model output piece by piece as server-sent events. Only the delivery changes; the answer's content, completeness and cost stay the same.
- LEAST
Pass up to 254 expressions to this T-SQL function and it hands back whichever is smallest. Both Fabric Warehouse and SQL analytics endpoints support it.