Classic compute's Apache Spark web console. Databricks advises checking the jobs timeline first, then the slowest stage, skew or spill, I/O and lastly the SQL DAG. Serverless and SQL warehouses offer a query profile in its place.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Spark UI in context, with comparison tables and the common traps.
Terms in this definition
- Classic compute
All-purpose, jobs and pipeline compute running in your own Azure subscription, which you set up and manage yourself. Its counterpart is serverless compute, which Azure Databricks runs for you.
- Apache Spark
An open-source engine that spreads data processing over a cluster of machines so that big data sets are handled in parallel, in batches or as streams. In Azure you can use it through Microsoft Fabric or Azure Databricks.
- Databricks
Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.
- FIRST
A DAX function available only inside visual calculations. It fetches the value at the start of one axis of the visual's matrix, which makes it handy for comparing each point with the first; its opposite is LAST.
- Skew
Unbalanced partitions leave a few tasks running long after the others finish. In Summary Metrics on the Spark UI, Max duration 50% or more higher than the 75th percentile points to it.
- Spill
Spark writing data to disk because shuffles, joins, sorts or aggregations have exhausted execution memory, which is costly. It is reported in the stage details of the Spark UI.
- SQL DAG
Graph showing the physical plan of a Spark query, reached through Associated SQL Query in the Spark UI. Operators report row counts, spill and peak memory; their timings add up task time.
- Serverless
Compute tier for single Azure SQL databases that scales automatically, pauses when idle and charges by the second. It is offered in General Purpose and Hyperscale, not Business Critical, and reserved capacity does not apply.
Related terms
- Query profile
A visual account of how a query ran, covering operators, time taken, rows and peak memory, used to track down bottlenecks. Viewing it requires owning the query or CAN MONITOR on the warehouse, and on serverless it stands in for the Spark UI.
- Shuffle
When Spark has to move rows between executors, as joins, sorts and aggregations require. Per stage, the Spark UI lists Shuffle Read and Shuffle Write; heavy shuffling frequently causes spill.