Existing tables force this comparison. Partitioning fixes physical boundaries, and Z-ordering arranges data within those boundaries during OPTIMIZE. Liquid clustering replaces both and is not compatible with either on the same table. Figure 13.1 shows the three layouts side by side.
| Factor | Partitioning | ZORDER | Liquid clustering |
|---|---|---|---|
| Cardinality | Low or known only | Any, including high | Any |
| Changing the columns | Full rewrite | New OPTIMIZE ZORDER BY |
ALTER TABLE, no rewrite required |
| When layout is applied | On write | Only during OPTIMIZE |
Partly on write, incrementally by OPTIMIZE |
| Predictive optimization | Compacts within partitions only | Ignores Z-ordered files | Runs clustering incrementally |
| Row-level concurrency | Not available | Conflicts with DML when used | Available |
| Rerun behaviour | Not applicable | Not idempotent; incremental | Incremental |
| Compatibility with others | Z-order inside partitions, not on partition columns | Used with partitions; not with liquid | Neither partitions nor ZORDER |
Liquid clustering is also the only way to skip data on a struct field without first extracting it into a top-level column, since partition columns must be top-level.
Decide on table size first. The documented guidance is firm:
| Table size | Guidance |
|---|---|
| Under 1 TB | Do not partition |
| 1 TB to 100 TB | Use liquid clustering; partitioning hurts more often than it helps |
| 100 TB or more | Partitioning might help, but try liquid clustering first and verify |
| Any partitioned table | Each partition should hold at least 1 GB |
Common trap: Date partitioning is a safe default for time-series tables - unpartitioned Delta tables already use ingestion time clustering, with performance similar to datetime partitioning and no tuning. Most tables under 100 TB need no partitioning, and an ineffective choice might need an expensive full rewrite to fix. If heavy
UPDATEorMERGEdisturbs that natural order, cluster on an event timestamp or creation date.
Habits from other data lakes do not carry over either: Delta provides ACID through the transaction log, not partition boundaries, and compute has no data locality tied to physical storage.
Z-ordering is still reasonable to understand for existing tables. Effectiveness drops with each extra column, and it produces files balanced by row count rather than size. It cannot be used with liquid clustering or on Iceberg tables. Bloom filter indexes are a separate relic: they are deprecated, and Databricks recommends dropping them in favour of predictive I/O and liquid clustering, then running VACUUM to clean the index files.
Moving an existing table across
Databricks recommends conversion in place, available in Databricks Runtime 18.1 and above for managed and external Delta tables.
-- New keys: run OPTIMIZE to benefit
ALTER TABLE t1
REPLACE PARTITIONED BY WITH CLUSTER BY (day, id);
OPTIMIZE t1;
-- Partition columns as initial keys, then adapt (managed only)
ALTER TABLE t2
REPLACE PARTITIONED BY WITH CLUSTER BY AUTO;| Current layout | Clustering keys to use |
|---|---|
| Hive-style partitioning | Partition columns |
| Z-order | The ZORDER BY columns |
| Partitioning and Z-order | Both sets of columns |
| Generated column reducing cardinality | The original column, without the generated one |
Converted partition columns become hierarchical keys automatically, and very different new keys cause a large recluster on the first OPTIMIZE. Batch readers see no downtime; batch writers see none on 15.4 and above. Streams need a restart, and a stream reading the table without schema tracking raises an exception and needs a new checkpoint and start version, though no commits are lost. Check the result with DESCRIBE EXTENDED and DESCRIBE HISTORY.
The command does not support pipeline streaming tables or materialized views, where the definition must change from PARTITIONED BY to CLUSTER BY, or tables shared with partition filtering. On managed Iceberg it raises an error, since partition definitions already act as clustering keys.