History tracking

In the Ultra Transcenders books

DBX-DEP

Each book explains History tracking in context, with comparison tables and the common traps.

Terms in this definition

• Previous Versions

  Lists older copies of a file in Explorer, drawn from VSS shadow copies or, where an Azure file share is mounted, its share snapshots.

• Lakeflow Connect

  Brings data into Azure Databricks with managed connectors for SaaS apps, files, streaming sources and databases, the last including CDC. The connectors run on serverless pipelines, and Unity Catalog governs them.

• AUTO CDC

  The Lakeflow pipelines API for change data capture: it merges incoming changes into a streaming table, keeping either just the latest values (SCD 1) or full history (SCD 2), and uses an ordering column to cope with records that arrive late. A snapshot variant also exists.

• TRACK HISTORY

  Decides which columns count towards SCD Type 2 versioning in AUTO CDC. Changes to any other column just overwrite the current row.

• CRUD

  Shorthand for create, read, update and delete, the four basic things you do with data. Data-plane roles in Azure Cosmos DB, for instance, authorise those operations on items.

Related terms

• Bitemporal AUTO CDC

  A Beta form of AUTO CDC that adds a second time axis to SCD type 2. SEQUENCE BY supplies business time and SYSTEM SEQUENCE BY supplies system time, which makes it possible to see what the system held to be true at an earlier point.

• SCD

  Slowly changing dimension. Type 1 keeps only current attribute values; type 2 retains every past version, bounded by __START_AT and __END_AT. Pipelines get both via AUTO CDC (STORED AS SCD TYPE 1 or STORED AS SCD TYPE 2), while TRACK HISTORY ON names which columns produce history.

• Temporal table

  Lets you ask what the data looked like on a past date by keeping its change history. Azure Databricks options are SCD type 2, change data feed, or Delta history with time travel; retention and purpose vary.