Also known as Lakeflow pipelines: in Azure Databricks you say which tables you want, and the framework deals with orchestration, incremental refresh and data quality checks. Databricks recommends it for production ingestion.
Also called formerly Delta Live Tables, Delta Live Tables, DLT.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Lakeflow Spark Declarative Pipelines in context, with comparison tables and the common traps.
Terms in this definition
- Databricks
Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.
- Incremental refresh
Rather than reloading everything, a table is split into date-based partitions and only new or changed periods are reloaded. Semantic models rely on RangeStart and RangeEnd parameters here; Dataflow Gen2 uses buckets instead, and needs a fixed-schema destination.
Related terms
- SDP
Spark Declarative Pipelines: open-source Apache Spark tooling to declare batch or streaming data flows in Python or SQL. Databricks extends it as Lakeflow Spark Declarative Pipelines, adding
AUTO CDC, expectations and a queryable event log.
See Lakeflow Spark Declarative Pipelines in the full glossary