Unity Catalog

In the Ultra Transcenders books

DP-900DP-750

Each book explains Unity Catalog in context, with comparison tables and the common traps.

Terms in this definition

• Databricks

  Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.

• Governance

  Keeping cloud deployments in line with an organisation's rules on technology, security and compliance, helped by Azure Policy, tags and resource locks.

• AGDLP

  Nesting pattern: users go into global groups, which go into domain local groups, which receive the permissions. AGUDLP adds universal groups for forests with several domains.

• Auditing

  Azure SQL capability that sends database audit logs to Log Analytics, Event Hubs or a storage account; that account is allowed to be in a different region.

Related terms

• ABAC policy

  Governed tags drive this kind of Unity Catalog rule: when a table or column has tags that satisfy its WHEN and MATCH COLUMNS conditions, a row filter or column mask UDF is put on it automatically. Policies are written with CREATE POLICY and can sit on a metastore, catalog, schema or table.

• Access Connector for Azure Databricks

  An Azure resource, provided by Microsoft, whose job is to carry a managed identity (system-assigned, user-assigned or both) that Unity Catalog can use. Storage credentials point at the connector's resource ID to gain access to ADLS.

• ADLS

  Paths beginning abfss:// point into this: Azure Storage with the hierarchical namespace switched on. In Unity Catalog it is typically the object store under both external locations and managed storage.

• AI-generated comments

  Drafted descriptions for Unity Catalog objects and their columns, which Catalog Explorer produces by reading metadata. Someone should check each one before accepting it, and it is not a dependable way of spotting PII.

• AI/BI dashboard

  The current dashboard type in Azure Databricks: AI helps you author it, its data stays governed by Unity Catalog, and publishing adds a companion Genie Agent. The legacy SQL dashboards it replaced can no longer be opened.

• ALL PRIVILEGES

  A shortcut grant in Unity Catalog covering every privilege relevant to an object. Four are deliberately left out: MANAGE, READ METADATA and the two external use privileges for schemas and locations.

• ANALYZE TABLE

  The ANALYZE TABLE ... COMPUTE STATISTICS statement gathers statistics on a table and its columns so the cost-based optimiser can plan queries better. On Unity Catalog managed tables, predictive optimization runs ANALYZE for you.

• Apache Iceberg

  An open table format. Besides Delta, it is one of the formats Unity Catalog offers for managed tables, and other engines can work with those managed Iceberg tables over Iceberg's REST catalog interface.