Micro-batch

In the Ultra Transcenders books

DP-900

Each book explains Micro-batch in context, with comparison tables and the common traps.

Terms in this definition

• Azure Stream Analytics

  Serverless service that processes streaming data with a SQL-like language, offering windowing functions and anomaly detection out of the box.

• Spark Structured Streaming

  Apache Spark's engine for near real-time streams, found in both Azure Databricks and Microsoft Fabric. Batch and streaming code share one set of APIs, incoming data is handled in increments, and it pairs naturally with Delta Lake.

Related terms

• Asynchronous checkpointing

  Lets a stateful stream begin its next micro-batch before the state checkpoint has completed, which lowers latency and preserves exactly-once guarantees. If an asynchronous checkpoint fails for any reason, the query fails too.

• Dead-letter table

  A pattern, not a built-in feature: bad rows from each micro-batch go into a separate Delta table, so the stream carries on despite them.

• foreachBatch

  A Structured Streaming output option, called through writeStream.foreachBatch, that hands every micro-batch to your own batch code, which suits MERGE upserts and destinations lacking a streaming writer. Delivery is at least once, so the code should be safe to repeat.

• Stream-static join

  Joins a stream with a Delta table, using the newest valid table version for every micro-batch. No state or watermark is involved, so facts already processed don't get corrected when dimensions change later.

• Streaming admission controls

  Limits on how much input each Structured Streaming micro-batch takes, mainly maxFilesPerTrigger and maxBytesPerTrigger, which is a soft limit. You can change them while keeping the checkpoint, but they only take effect once any batch already planned has finished.