Runtime 2.0

In the Ultra Transcenders books

DP-700

Each book explains Runtime 2.0 in context, with comparison tables and the common traps.

Terms in this definition

• Apache Spark

  An open-source engine that spreads data processing over a cluster of machines so that big data sets are handled in parallel, in batches or as streams. In Azure you can use it through Microsoft Fabric or Azure Databricks.

• Delta Lake

  Table format adding transactions to files in the data lake. In Synapse, Spark pools can write these tables, while serverless SQL pools can only query them.

• LTS

  Applies to some Databricks Runtime versions, which Learn recommends for job compute in production. Feature work on such a version stops after about six months, and for three years afterwards it gets security and stability fixes backported.

Related terms

• Adaptive target file size

  A Spark setting in Fabric that chooses the target file size for each Delta table based on how big the table is, starting at 128 MB for small tables and rising towards 1 GB for very large ones, and reconsiders the choice every time OPTIMIZE runs. It is enabled by default in Runtime 2.0 and removes the need to tune the older delta.targetFileSize property by hand.

• ANSI mode

  A stricter Spark SQL behaviour, enabled by default since Apache Spark 4.0 (so also in Fabric Runtime 2.0), where a bad cast or an integer overflow fails the query rather than quietly producing null. Functions such as try_cast, try_add and try_divide keep the old null result, while switching ANSI mode off should be treated as a temporary workaround.

• EOSA

  Short for end-of-support announced: the lifecycle phase a Fabric Spark runtime enters once Microsoft names its retirement date, normally about half a year ahead. Runtime 1.3 sat here until 30 September 2026 before switching to a six-month Long Term Support period from 1 October 2026, and Runtime 2.0 is now the generally available runtime Microsoft points people to.

• File-level compaction targets

  A Fabric Spark option that leaves alone, during OPTIMIZE, any file that previously hit half or more of an older target size, so growing adaptive target sizes don't trigger endless rewriting. Runtime 2.0 onwards has it switched on.

• Real-time Mode

  A Structured Streaming mode in Spark, set with Trigger.RealTime on Fabric Runtime 2.0, where tasks stay running and handle each record on arrival rather than working in microbatches. Its output mode must be update, and its sources and sinks are limited to Kafka-compatible ones or a foreach sink, so files and Delta tables can't be used.

• Structured Streaming trigger

  Configured on the writer of a Spark streaming query, it governs when data gets processed and whether the query keeps going. Options: processing time, running microbatches one after another either straight away or every set interval; available now, which works through existing data then ends; and, from Runtime 2.0, low-latency Real-time Mode.