Fabric Spark pools: starter pools vs custom pools vs capacity pools

A pool describes the nodes a Spark session gets: node size, how many, and whether they scale. Fabric offers prewarmed starter pools for speed and custom pools for control, and both bill only while a session is active.

Starter pools

Every workspace gets a starter pool automatically. Its clusters are already running, so a session with no extra libraries or custom Spark properties typically starts in 5 to 10 seconds.

• Starter pools use Medium nodes only (8 vCores, 64 GB). Choosing another node size or customising compute switches to on-demand startup of 2 to 5 minutes.
• Autoscale and Dynamically allocate executors are both enabled by default. Workspace Admins can adjust the maximum nodes and executors with sliders, up to the SKU’s limit.
• Startup is best effort: if the region’s prewarmed capacity is exhausted, Fabric creates a new cluster (2 to 5 minutes). Workspaces with Private Link or managed virtual networks don’t get starter pools at all and always start on demand.
• Libraries add time: Quick mode environments install libraries at session start (30 seconds to 5 minutes); Full mode deploys a snapshot (1 to 3 minutes).
• Billing starts when the notebook or job starts running; idle time in the pool and session personalisation aren’t billed.

Custom Spark pools

A custom Spark pool lets the workspace Admin pick the node family (for example Memory optimized), node size, autoscale range and executor allocation. Prerequisites: the workspace Admin role, and the capacity admin must enable Customized workspace pools (on by default at capacity level).

• X-Large and XX-Large aren’t available on trial capacities.
• A custom pool supports at most 200 nodes. The node-to-executor ratio is always 1:1, with one node for the driver.
• Setting the minimum to one node creates a single-node pool, where the driver and executor share one node (each gets half its resources); it suits small workloads.
• Custom pools take about three minutes to start because nodes are acquired on demand. Creating a pool is free; billing covers only active sessions, not cluster acquisition or deallocation.
• Sessions expire after 20 minutes of inactivity by default, and an unused pool is deallocated 2 minutes after the session expires.
• An Admin can create several custom pools and select any of them as the workspace default pool. Changing the default from the starter pool to a custom pool lengthens session start to about three minutes.

Capacity pools and custom live pools

• Capacity admins can create capacity-level pools (2 to 3 minutes to start) that appear in the workspace pool list and in environments, and can then disable workspace-level customisation or starter pool usage entirely for central governance.
• Custom live pools keep clusters hydrated on a schedule so notebook sessions start in about 5 seconds during the active window. They need a paid Fabric capacity (not trial), support notebooks only (interactive, scheduled and pipeline-run, not Spark job definitions), must have a schedule, and work best with a Full mode environment.

Sizing against the capacity

One CU equals two Spark VCores and the capacity can burst to three times that. An F64 (128 VCores, 384 with burst) could run one job on 48 Medium nodes or several smaller ones.

• Job-level bursting is on by default, so one job can consume the full burst. A capacity admin can switch on Disable job-level bursting to cap each job at the base allocation (128 VCores on F64) and leave room for others. The switch isn’t available when Autoscale Billing for Spark is enabled.
• Optimistic job admission (the default for every workspace) admits a job if its minimum nodes fit, then grants autoscale requests only while cores are free. On an F32, jobs starting on one Medium node allow up to 24 concurrent jobs instead of three at full size.
• The Jobs tab option Reserve maximum cores for active Spark jobs turns optimistic admission off and reserves each job’s maximum cores.
• When no cores are free, interactive notebook and lakehouse jobs fail with HTTP 430 TooManyRequestsForCapacity; jobs from pipelines, the scheduler and Spark job definitions queue first in, first out and expire after 24 hours. A throttled capacity rejects new Spark jobs instead of queuing them.

Common trap: Selecting Large nodes for the starter pool to get faster sessions - starter pools are Medium-only; any other size (or customised compute) falls back to on-demand startup of minutes, not seconds.

Common trap: Expecting an interactive notebook to wait in the queue when the capacity is busy - queueing applies to pipeline, scheduled and Spark job definition runs; interactive notebooks are rejected with HTTP 430.

Get the whole book

This note is one section of Ultra Transcenders DP-700: Implementing Data Engineering Solutions Using Microsoft Fabric, an independent study guide that explains every topic the exam covers by technology, with comparison tables, diagrams and the common traps, plus a glossary linked to Microsoft Learn.

About the book · DP-700 terms in the glossary · All DP-700 study notes