Delta table

In the Ultra Transcenders books

DBX-DEPDBX-GENAI

Each book explains Delta table in context, with comparison tables and the common traps.

Terms in this definition

• Event

  Table in Log Analytics where entries from Windows event logs are kept.

• Databricks

  Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.

• Delta Lake

  Table format adding transactions to files in the data lake. In Synapse, Spark pools can write these tables, while serverless SQL pools can only query them.

• FORMAT

  A DAX function that turns a value into text according to a format string, for instance "MMMM" to show a month's name. Since the output is text, numeric operations can't use it, and dynamic format strings were introduced to get around that.

• CRUD

  Shorthand for create, read, update and delete, the four basic things you do with data. Data-plane roles in Azure Cosmos DB, for instance, authorise those operations on items.

Related terms

• Adaptive target file size

  A Spark setting in Fabric that chooses the target file size for each Delta table based on how big the table is, starting at 128 MB for small tables and rising towards 1 GB for very large ones, and reconsiders the choice every time OPTIMIZE runs. It is enabled by default in Runtime 2.0 and removes the need to tune the older delta.targetFileSize property by hand.

• ALTER TABLE ... SET MANAGED

  Moves a Delta table from external or foreign status to being managed by Unity Catalog while preserving history, views, permissions, settings and name. Use serverless compute, or Runtime 17.3 LTS and later.

• CDF

  Change data feed. When switched on for a Delta table it records which rows changed from version to version, exposing extra columns that give the kind of change and the commit version. You can query it with table_changes(), stream from it, or feed it to AUTO CDC.

• Column mapping

  A Delta table feature that separates the logical names of columns from their physical names, so columns can be renamed or dropped as a metadata-only change and names may contain characters like spaces.

• COPY INTO

  Loads files from a volume or cloud storage into a Delta table that already exists, skipping anything loaded before, so reruns are safe. Good for thousands of files; for millions, Auto Loader scales further.

• Dead-letter table

  A pattern, not a built-in feature: bad rows from each micro-batch go into a separate Delta table, so the stream carries on despite them.

• Delta generated columns

  Columns in a Delta table that are filled in automatically from other values in the same row each time data is written, and then saved. A typical use is deriving a DATE from a timestamp to help skip files; identity columns are one particular kind.

• DESCRIBE HISTORY

  A SQL statement showing every write to a Delta table, latest first, with the version, user and time of each. How far back it goes depends on delta.logRetentionDuration, which is 30 days unless changed.