Streaming

In the Ultra Transcenders books

AI-103

Each book explains Streaming in context, with comparison tables and the common traps.

Terms in this definition

• Server-Sent Events

  An older streaming method for talking to remote MCP servers over HTTP. VS Code drops back to it if a server can't do streamable HTTP; the abbreviation has nothing to do with server-side encryption.

Related terms

• Auto-TTL

  Rows older than a set number of days, measured from a timestamp column, are removed automatically (DELETE ROWS ... DAYS AFTER). It covers Unity Catalog managed Delta or Iceberg tables plus pipeline streaming tables, depends on predictive optimization, and runs at no guaranteed time.

• Checkpoint

  The storage location in which a streaming query records its offsets, state, commits and ID. Thanks to it, a restarted query picks up precisely where it left off. Never point two queries at the same one.

• Eventhouse

  Holds one or more KQL databases and is where Real-Time Intelligence in Microsoft Fabric keeps time-stamped streaming events for analysis, indexed by the moment they arrive.

• Flow

  The unit of work inside a Lakeflow pipeline that takes data from a source, transforms it and lands it in a destination like a streaming table. Streaming flows either append or update, and CREATE FLOW lets you declare a flow apart from the table it feeds.

• foreachBatch

  A Structured Streaming output option, called through writeStream.foreachBatch, that hands every micro-batch to your own batch code, which suits MERGE upserts and destinations lacking a streaming writer. Delivery is at least once, so the code should be safe to repeat.

• Lakeflow Connect

  Brings data into Azure Databricks with managed connectors for SaaS apps, files, streaming sources and databases, the last including CDC. The connectors run on serverless pipelines, and Unity Catalog governs them.

• ODX

  Offloaded Data Transfer. A capable SAN moves or copies data internally, exchanging tokens rather than streaming it through the server; NTFS supports it but ReFS doesn't.

• Real-time streaming semantic model

  Covers live-fed push, streaming and PubNub models, plus streaming dashboard tiles. Microsoft now steers people towards Real-Time Intelligence in Fabric, as these are being retired and new ones can only be made until 31 October 2027.