Delta Lake merges small files as soon as a write has succeeded, using the same cluster, and ignores files it has already compacted. History shows each run as an OPTIMIZE triggered automatically.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Auto compaction in context, with comparison tables and the common traps.
Terms in this definition
- Delta Lake
Table format adding transactions to files in the data lake. In Synapse, Spark pools can write these tables, while serverless SQL pools can only query them.
- OPTIMIZE
Bin-packs small files into larger ones, or reclusters liquid-clustered tables, when you run this SQL command; for Unity Catalog managed tables, predictive optimization takes care of it automatically.