Iceberg

In the Ultra Transcenders books

DBX-DEP

Each book explains Iceberg in context, with comparison tables and the common traps.

Terms in this definition

• FORMAT

  A DAX function that turns a value into text according to a format string, for instance "MMMM" to show a month's name. Since the output is text, numeric operations can't use it, and dynamic format strings were introduced to get around that.

• Unity Catalog

  Azure Databricks' governance solution covering both data and AI in one place, with centralised permissions, auditing, data discovery and lineage.

• CRUD

  Shorthand for create, read, update and delete, the four basic things you do with data. Data-plane roles in Azure Cosmos DB, for instance, authorise those operations on items.

• AUTO CDC

  The Lakeflow pipelines API for change data capture: it merges incoming changes into a streaming table, keeping either just the latest values (SCD 1) or full history (SCD 2), and uses an ordering column to cope with records that arrive late. A snapshot variant also exists.

• Streaming

  Sending model output piece by piece as server-sent events. Only the delivery changes; the answer's content, completeness and cost stay the same.

Related terms

• Apache Iceberg

  An open table format. Besides Delta, it is one of the formats Unity Catalog offers for managed tables, and other engines can work with those managed Iceberg tables over Iceberg's REST catalog interface.

• Auto-TTL

  Rows older than a set number of days, measured from a timestamp column, are removed automatically (DELETE ROWS ... DAYS AFTER). It covers Unity Catalog managed Delta or Iceberg tables plus pipeline streaming tables, depends on predictive optimization, and runs at no guaranteed time.

• CONVERT TO DELTA

  Turns Parquet into Delta where it stands: the current files stay put and a transaction log is added on top. Tables with partition evolution and Iceberg metastore tables can't use it.

• Deletion vectors

  Lets Delta Lake and Iceberg tables flag removed or changed rows through metadata rather than rewriting complete Parquet files, a soft delete that speeds up deletes, updates and merges. The physical rewrite happens later, when OPTIMIZE or a REORG TABLE purge runs.

• DESCRIBE DETAIL

  A SQL command reporting facts about a whole Delta or Iceberg table rather than its columns, such as where it is stored, how big it is, how many files it has, how it is partitioned and which protocol versions it uses.

• External data access

  A metastore option, disabled unless an admin switches it on, that opens Unity Catalog data to reads and writes from other engines via REST APIs (Unity's own or the Iceberg catalog). Each principal also needs the external use privilege on the schema.

• Iceberg reads

  A setting on a Delta table that writes Iceberg metadata as well, letting Iceberg clients query the data while all writes still go through Delta.

• Managed Iceberg table

  Stores data in Apache Iceberg format (USING iceberg) as a Unity Catalog managed table. Engines outside Databricks can query and update it via the Iceberg REST Catalog API, and Unity Catalog handles its upkeep.