Keeps copies of remote Parquet files, Delta tables among them, on the local SSDs of worker nodes so the same data is read faster next time. Azure Databricks handles it automatically: no code is required and it invalidates itself when files change, unlike the Apache Spark cache.
Also called formerly Delta cache or DBIO cache, DBIO cache, Delta cache.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Disk cache in context, with comparison tables and the common traps.
Terms in this definition
- Parquet
Columnar file format, open source and aimed at analytics. Event Hubs Capture cannot produce it natively.
- LSDOU
The sequence in which Group Policy is processed: the local policy first, followed by site, domain and organisational unit policies. The nearest, last-processed setting takes effect unless Enforced or Block Inheritance alters that.
- Worker
A classic compute node hosting a Spark executor that carries out distributed work; with one executor on each worker the two words are used as synonyms. Worker node type sets its instance family, and Spark commands fail if there are no workers.
- CRUD
Shorthand for create, read, update and delete, the four basic things you do with data. Data-plane roles in Azure Cosmos DB, for instance, authorise those operations on items.
- NEXT
Restricted to visual calculations, this DAX function reads the value one step further along an axis of the visual matrix; writing OFFSET with 1 gives the same result.
- Databricks
Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.
- Apache Spark cache
A cache you request explicitly, in memory or on disk, for a table or DataFrame using
df.cache(),df.persist()orCACHE TABLE, and release withunpersist. It differs from the automatic disk cache and cannot be used on serverless compute.
Related terms
- LRU
Short for least recently used. When space runs out, both the Apache Spark cache and the disk cache evict the data that has gone longest without being used.