A Spark pool built by a workspace Admin to their own specification, choosing the size and family of nodes, autoscale limits and whether executors scale dynamically. It needs a capacity that allows pool customisation and usually takes about three minutes to start, unless it's running as a live pool.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Custom Spark pool in context, with comparison tables and the common traps.
Terms in this definition
- Workspace
Teams in Power BI and Microsoft Fabric collaborate in this folder-style container, which groups items such as reports, semantic models and lakehouses, controls who can access them and is assigned a capacity.
- Autoscale
Azure Monitor capability that adds or removes instances on a schedule or once a metric rule has been true for its full duration. The maximum instance count merely sets a ceiling and never triggers scaling itself.
- Capacity
A reserved block of compute for a tenant whose size is fixed by its SKU; Fabric F SKUs express it in capacity units (CUs). Workspaces assigned to it unlock features like Copilot, and from F64 upwards people with free licences can view Power BI content.
Related terms
- Custom live pool
A time-window schedule, configured through an environment on a custom Spark pool, that keeps clusters ready so notebooks get a session in around five seconds. Trial capacities can't use it (a paid F SKU is needed), and Spark job definitions don't benefit.
- Single-node pool
A custom Spark pool set to a minimum of one node, meaning the executor and driver run on the same machine. It is a good fit for development and light workloads.