Delta Lake

In the Ultra Transcenders books

AZ-305DP-900DP-750DP-600DP-700

Each book explains Delta Lake in context, with comparison tables and the common traps.

Terms in this definition

• Event

  Table in Log Analytics where entries from Windows event logs are kept.

• FORMAT

  A DAX function that turns a value into text according to a format string, for instance "MMMM" to show a month's name. Since the output is text, numeric operations can't use it, and dynamic format strings were introduced to get around that.

• Data lake

  Holds files of every kind, structured, semi-structured or unstructured, usually spread over a distributed file system, and lets engines like Spark apply a schema when reading. In Azure this is Data Lake Storage: Blob Storage with a hierarchical namespace enabled.

• Serverless

  Compute tier for single Azure SQL databases that scales automatically, pauses when idle and charges by the second. It is offered in General Purpose and Hyperscale, not Business Critical, and reserved capacity does not apply.

Related terms

• Apache Spark pool

  Spark compute in Azure Synapse used to build and update Delta Lake tables, run machine learning and stream changes from the Cosmos DB change feed.

• Auto compaction

  Delta Lake merges small files as soon as a write has succeeded, using the same cluster, and ignores files it has already compacted. History shows each run as an OPTIMIZE triggered automatically.

• Data lakehouse

  Combines a lake and a warehouse: raw data stays as files, yet Delta Lake lets you query it as SQL tables. Fabric Lakehouse and the lakehouse in Databricks work this way.

• Deletion vectors

  Lets Delta Lake and Iceberg tables flag removed or changed rows through metadata rather than rewriting complete Parquet files, a soft delete that speeds up deletes, updates and merges. The physical rewrite happens later, when OPTIMIZE or a REORG TABLE purge runs.

• Resource profile

  A ready-made bundle of Delta Lake and Spark settings tuned to a kind of workload, chosen through spark.fabric.resourceProfile. The options are writeHeavy (V-Order off, and what new workspaces get by default), readHeavyForSpark, readHeavyForPBI (V-Order on) and custom.

• Runtime 2.0

  The Fabric Spark runtime, now generally available and the recommended choice for production, built on Apache Spark 4.1 with Delta Lake 4.2, Python 3.13, Scala 2.13 and Java 21. Its predecessor, Runtime 1.3 on Spark 3.5, reached end of support on 30 September 2026 and moved into Long Term Support from 1 October 2026 until March 2027.

• Schema enforcement

  Delta Lake checks every write against the table's schema: columns being written must already exist and their types must be identical or safely convertible, or the write is rejected. Plain file formats like CSV or JSON get no such check.

• Spark Structured Streaming

  Apache Spark's engine for near real-time streams, found in both Azure Databricks and Microsoft Fabric. Batch and streaming code share one set of APIs, incoming data is handled in increments, and it pairs naturally with Delta Lake.