Lets Delta Lake and Iceberg tables flag removed or changed rows through metadata rather than rewriting complete Parquet files, a soft delete that speeds up deletes, updates and merges. The physical rewrite happens later, when OPTIMIZE or a REORG TABLE purge runs.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Deletion vectors in context, with comparison tables and the common traps.
Terms in this definition
- Delta Lake
Table format adding transactions to files in the data lake. In Synapse, Spark pools can write these tables, while serverless SQL pools can only query them.
- Iceberg
An open format for tables. Within Unity Catalog, managed ones are available and foreign ones can only be read, while the docs cover
USING ICEBERGon materialized views but not on AUTO CDC streaming tables. - Metadata
Describes data rather than being the data itself, for instance how a file is laid out or what rows each chunk contains, so tools can read things efficiently.
- Deployment modes
The two ways ARM can deploy: Incremental, the default, creates or updates what the template lists and ignores everything else; Complete also removes resource group contents absent from the template.
- Parquet
Columnar file format, open source and aimed at analytics. Event Hubs Capture cannot produce it natively.
- Soft delete
Retains deleted backups or data so they can be recovered during a retention period.
- Physical
Describes the concrete, real-world parts that put a logical architecture into effect.
- OPTIMIZE
Bin-packs small files into larger ones, or reclusters liquid-clustered tables, when you run this SQL command; for Unity Catalog managed tables, predictive optimization takes care of it automatically.
Related terms
- Auto-Enable Deletion Vectors
Decides, for one workspace, if deletion vectors are switched on automatically when Delta tables are created on recent runtimes and SQL warehouses. Not every workspace starts with the same default.
- DELETE
Removes rows from a table: those that match a predicate, or every row if none is given. Where deletion vectors are on, rows may just be flagged in metadata and remain in the files until a purge and vacuum.
- Iceberg v3
An Apache Iceberg release that lets managed tables use deletion vectors and the
VARIANTtype, among other additions. - Predictive I/O
A set of Photon-only optimisations that make reads and updates faster. Updates rely on deletion vectors automatically, and no setting is needed for each query.
- REORG TABLE
With
REORG TABLE ... APPLY (PURGE), Delta rewrites files so that rows marked as deleted by deletion vectors are actually removed from storage, which is usually done before runningVACUUM. - Row-level concurrency
Sorts out clashes automatically when simultaneous writes touch different rows in one file. It requires deletion vectors and Databricks Runtime 14.3 LTS or later, and isn't available on partitioned tables.