Runs tasks on a worker node; Azure Databricks places exactly one on each worker. Autoscaling, losing a spot instance or out-of-memory errors can all take one away.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Executor in context, with comparison tables and the common traps.
Terms in this definition
- Agents (classic) API
First-generation Foundry Agent Service API, based on threads, messages and runs. It is deprecated, replaced by conversations and responses, and retires on 31 March 2027.
- Worker
A classic compute node hosting a Spark executor that carries out distributed work; with one executor on each worker the two words are used as synonyms. Worker node type sets its instance family, and Spark commands fail if there are no workers.
- Databricks
Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.
- Autoscaling
A classic compute option where you give a minimum and maximum worker count and Azure Databricks adjusts the number of workers within that range to match the load.
- ALL
A DAX function that ignores any filters and gives back every row of a table or every value of the named columns. Used within CALCULATE, it works as a modifier that clears filters, although REMOVEFILTERS states that intent more clearly where it is available.
Related terms
- Broadcast join
A join method that copies the smaller table to every executor so that the larger table needs no shuffle. You can force it with the
BROADCASThint (orBROADCASTJOIN/MAPJOIN), or AQE may pick it while the query runs. - Dynamic allocation
Lets a Spark application grow and shrink its executor count with demand, adding them when busy and giving them up when idle, up to a ceiling set by the workspace admin. In Spark pools the option is named Dynamically allocate executors and is enabled by default together with Autoscale.
- Single node
Compute made of a driver only, with no workers, running Spark locally using one executor thread per core except one reserved for the driver. Good for small jobs or libraries that don't distribute; it can't be turned into a multi-node cluster.
- Single-node pool
A custom Spark pool set to a minimum of one node, meaning the executor and driver run on the same machine. It is a good fit for development and light workloads.