Learn suggests steering clear of these where possible: shell scripts that execute on every classic compute node during boot, before Spark comes up. If you do need one, keep it cluster-scoped and store it in a volume; DBFS-hosted scripts are end-of-life.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Init scripts in context, with comparison tables and the common traps.
Terms in this definition
- WHERE
Limits a SELECT, UPDATE or DELETE to just the rows meeting a condition. Omit it, and the statement hits every row.
- EXECUTE
Allows calling a function or loading a registered model in Unity Catalog, and seeing its definition.
USE CATALOGandUSE SCHEMAare needed too. - Classic compute
All-purpose, jobs and pipeline compute running in your own Azure subscription, which you set up and manage yourself. Its counterpart is serverless compute, which Azure Databricks runs for you.
- Volume
Holds non-tabular files sitting in cloud storage under Unity Catalog governance, addressed as
/Volumes/<catalog>/<schema>/<volume>. A volume can be external or managed, andWRITE VOLUMEorREAD VOLUMEdecide who may use it.
Related terms
- Allowlist
Held at metastore level, it records which JAR files, Maven libraries and init scripts may run on compute using the standard access mode. It begins with no entries, and only holders of
MANAGE ALLOWLISTmay change it. - Compute policy
Limits, set by an admin, on what can be chosen when people create classic compute; they help contain cost and require particular custom tags, libraries or init scripts.