Streaming table

In the Ultra Transcenders books

DP-750

Each book explains Streaming table in context, with comparison tables and the common traps.

Terms in this definition

• Event

  Table in Log Analytics where entries from Windows event logs are kept.

• Deployment modes

  The two ways ARM can deploy: Incremental, the default, creates or updates what the template lists and ignores everything else; Complete also removes resource group contents absent from the template.

• CRUD

  Shorthand for create, read, update and delete, the four basic things you do with data. Data-plane roles in Azure Cosmos DB, for instance, authorise those operations on items.

• Databricks SQL

  Brings warehousing to an Azure Databricks lakehouse. ANSI SQL runs against Delta tables using SQL warehouses, compute tuned for heavy query concurrency and BI tools.

• READ FILES

  Granted on a storage credential or external location, this Unity Catalog permission lets tools like Auto Loader or COPY INTO pull files directly from cloud storage. WRITE FILES can't be used without it.

Related terms

• AUTO CDC

  The Lakeflow pipelines API for change data capture: it merges incoming changes into a streaming table, keeping either just the latest values (SCD 1) or full history (SCD 2), and uses an ordering column to cope with records that arrive late. A Python-only snapshot variant also exists.

• Flow

  The unit of work inside a Lakeflow pipeline that takes data from a source, transforms it and lands it in a destination like a streaming table. Streaming flows either append or update, and CREATE FLOW lets you declare a flow apart from the table it feeds.

• Full refresh

  A pipeline run that wipes the stored data and state of a streaming table or materialized view and then rebuilds it from every source record; anything already deleted at the source cannot come back.

• read_files

  A SQL table-valued function that loads files in formats such as CSV, JSON, XML, Parquet, Avro, ORC, text or binary and infers their schema to return rows. Used with the STREAM keyword inside a streaming table, it relies on Auto Loader.

• SCHEDULE clause

  Sets a timed refresh for a streaming table or materialized view. SCHEDULE EVERY takes an interval in hours, days or weeks, and SCHEDULE CRON takes a Quartz cron expression when exact or more frequent than hourly timing is needed.