When to partition Delta tables, and when to use liquid clustering instead

Partitioning splits a table’s files into directories by the values of one or more columns. Learn’s current guidance is that most tables do not need it: Databricks recommends liquid clustering for all new tables, and unpartitioned Delta tables automatically get ingestion time clustering, which gives partitioning-like performance for date-based queries without tuning.

If you do partition, choose low or known cardinality columns (dates, regions), never high-cardinality ones such as timestamps or customer IDs. Partition columns must be top-level columns of a supported type (date, timestamp, string, numeric, boolean, binary and so on); you cannot partition by a struct field, map, array or VARIANT. Transactions in Delta are not defined by partition boundaries, so partitioning is not needed for atomicity. An ineffective scheme may need a full rewrite to fix, which is slow and expensive for large tables.

For managed Iceberg tables, Unity Catalog supports only liquid clustering and interprets PARTITIONED BY columns from external engines as clustering keys. To move an existing partitioned Delta table to liquid clustering in Databricks Runtime 18.1 and above, use ALTER TABLE ... REPLACE PARTITIONED BY WITH CLUSTER BY, which minimises downtime and works for managed and external tables (not for pipeline streaming tables and materialized views, where you change the definition to CLUSTER BY):

ALTER TABLE main.sales.events REPLACE PARTITIONED BY WITH CLUSTER BY (event_date, customer_id);
OPTIMIZE main.sales.events;

Common trap: Partitioning a fact table by its event timestamp or customer ID - partitioning works only for low or known cardinality columns and creates many tiny partitions otherwise. Use liquid clustering, which handles high-cardinality columns and avoids fixed partition boundaries.

Get the whole book

This note is one section of Ultra Transcenders DP-750: Implementing Data Engineering Solutions Using Azure Databricks, an independent study guide that explains every topic the exam covers by technology, with comparison tables, diagrams and the common traps, plus a glossary linked to Microsoft Learn.

About the book · DP-750 terms in the glossary · All DP-750 study notes