Formerly Delta Live Tables, this framework has engineers state which datasets they want. Databricks then looks after orchestration, retries and much of the incremental processing.
Also called formerly Delta Live Tables, DLT.
Read more: Databricks docs
In the Ultra Transcenders books
Each book explains Lakeflow Pipelines in context, with comparison tables and the common traps.
Terms in this definition
- State
A parameter in OAuth that carries custom data through the authorisation flow and back, as with the "Support state parameter" option in APIM.
- Databricks
Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.
- Deployment modes
The two ways ARM can deploy: Incremental, the default, creates or updates what the template lists and ignores everything else; Complete also removes resource group contents absent from the template.
Related terms
- AUTO CDC
The Lakeflow pipelines API for change data capture: it merges incoming changes into a streaming table, keeping either just the latest values (SCD 1) or full history (SCD 2), and uses an ordering column to cope with records that arrive late. A snapshot variant also exists.
- AUTO CDC FROM SNAPSHOT
A Lakeflow pipelines API, available only in Python, for sources without a change data feed: it compares one snapshot with the next and maintains an SCD type 1 or type 2 target from the differences.
- CDC
Change data capture: rather than reloading whole snapshots, you process just the rows a source has added, changed or removed, with each change carrying its kind and an ordering value. Lakeflow pipelines handle such feeds through
AUTO CDC. - DAG
Directed acyclic graph: nodes (tasks or datasets) joined by dependency edges with no loops. Lakeflow Jobs and Lakeflow pipelines execute the nodes in order of their dependencies, running as many in parallel as they can.
- Lakeflow Pipelines Editor
Develop Lakeflow pipelines in the workspace with this IDE: it edits several files at once and adds an asset browser, a live graph, data previews and an issues panel, and you can run a single file or table on its own.
- Lakeflow Spark Declarative Pipelines
Also known as Lakeflow pipelines: in Azure Databricks you say which tables you want, and the framework deals with orchestration, incremental refresh and data quality checks. Databricks recommends it for production ingestion.
- Memory pipeline
In Beta, this resource takes a single session store and condenses what it holds into long-term memories in a single memory store for an agent. You can run it whenever you like or on a schedule, and Lakeflow pipelines are unrelated.
- Real-time Mode
A trigger using long-running batches that hand data to the next stage immediately, bringing end-to-end latency down to as little as five milliseconds. Lakeflow Pipelines layers it over continuous mode, and it can't read from or write to Delta tables.