A Lakeflow pipelines API, available only in Python, for sources without a change data feed: it compares one snapshot with the next and maintains an SCD type 1 or type 2 target from the differences.
Also called create_auto_cdc_from_snapshot_flow, APPLY CHANGES FROM SNAPSHOT.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains AUTO CDC FROM SNAPSHOT in context, with comparison tables and the common traps.
Terms in this definition
- Lakeflow Pipelines
Formerly Delta Live Tables, this framework has engineers state which datasets they want. Databricks then looks after orchestration, retries and much of the incremental processing.
- API
Short for application programming interface: a contract that client code calls programmatically, for example a web API secured with tokens or the Files, Images or Responses APIs.
- CDF
Change data feed. When switched on for a Delta table it records which rows changed from version to version, exposing extra columns that give the kind of change and the commit version. You can query it with
table_changes(), stream from it, or feed it toAUTO CDC. - snapshot
Captures a VM's disk state, power state, settings and, if requested, memory at a moment in time, using one delta disk per virtual disk. Snapshots should not be relied on as backups.
- NEXT
Restricted to visual calculations, this DAX function reads the value one step further along an axis of the visual matrix; writing OFFSET with 1 gives the same result.
- SCD
Slowly changing dimension. Type 1 keeps only current attribute values; type 2 retains every past version, bounded by
__START_ATand__END_AT. Pipelines get both viaAUTO CDC(STORED AS SCD TYPE 1orSTORED AS SCD TYPE 2), whileTRACK HISTORY ONnames which columns produce history.