Change data capture: rather than reloading whole snapshots, you process just the rows a source has added, changed or removed, with each change carrying its kind and an ordering value. Lakeflow pipelines handle such feeds through AUTO CDC.
Also called change data capture.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains CDC in context, with comparison tables and the common traps.
Terms in this definition
- AUTO CDC
The Lakeflow pipelines API for change data capture: it merges incoming changes into a streaming table, keeping either just the latest values (SCD 1) or full history (SCD 2), and uses an ordering column to cope with records that arrive late. A Python-only snapshot variant also exists.
Related terms
- Capture instance
When
sys.sp_cdc_enable_tableturns on change data capture for a table, it builds this: a dedicated change table with its own query functions. A maximum of two can exist per source table. - Copy job
Moves data between a source and a destination in Fabric Data Factory with no pipeline required. Runs can copy everything or just changes (incremental mode, driven by CDC or a watermark column), and the item keeps track of where the previous run stopped.
- Extraction type
Describes the scope of data moved by each ingestion run, which can be everything at once, only new data in batches, a continuous stream, a change data capture (CDC) feed or a snapshot taken at one moment.
- Gating role
When change data capture is switched on for a table, an optional database role can be named in @role_name. Reading the change data then requires membership of that role plus SELECT permission on the captured columns.
- Incremental copy
In this Copy job mode the opening run brings over the whole data set, after which each run moves only rows that are new or have changed, spotted using CDC or a watermark column. State is tracked by the job, and a reset back to a full copy is possible either for everything or table by table.
- Incremental load
Rather than reloading everything each time, this approach does one full copy and afterwards moves only rows that are new or changed, found through CDC, change tracking or a high watermark. It scales better than full loads, provided change detection can be trusted.
- Lakeflow Connect
Brings data into Azure Databricks with managed connectors for SaaS apps, files, streaming sources and databases, the last including CDC. The connectors run on serverless pipelines, and Unity Catalog governs them.
- LSN
Log sequence number: a unique, ordered identifier given to every record in the transaction log. Change data capture sorts changes by their commit LSN (stored in __\$start_lsn) and uses a pair of LSNs to choose which changes to return.