How many partitions Spark creates when shuffling, controlled by spark.sql.shuffle.partitions. For a stateful streaming query, the checkpoint's creation locks in this value, except where on-demand state repartitioning applies.
Also called spark.sql.shuffle.partitions.
Read more: Databricks docs
In the Ultra Transcenders books
Each book explains Shuffle partitions in context, with comparison tables and the common traps.
Terms in this definition
- Streaming
Sending model output piece by piece as server-sent events. Only the delivery changes; the answer's content, completeness and cost stay the same.
- Checkpoint
The storage location in which a streaming query records its offsets, state, commits and ID. Thanks to it, a restarted query picks up precisely where it left off. Never point two queries at the same one.
- EXCEPT
A set operator that gives the rows in one query's result that the second query doesn't return.
SELECT * EXCEPT (...)uses the same word to omit columns. - WHERE
Limits a SELECT, UPDATE or DELETE to just the rows meeting a condition. Omit it, and the statement hits every row.
- State
A parameter in OAuth that carries custom data through the authorisation flow and back, as with the "Support state parameter" option in APIM.
Related terms
- AQE
Adaptive query execution: Spark re-plans a query while it runs, for instance turning a sort-merge join into a broadcast join, merging shuffle partitions, dealing with skew and propagating empty relations. Databricks advises leaving it enabled.
- Autotune
A preview option in Fabric Spark that studies previous runs to pick better values for shuffle partitions, broadcast join threshold and max partition bytes query by query. It's disabled by default (
spark.ms.autotune.enabled), limited to Runtime 1.2, and incompatible with private endpoints and high concurrency mode.