Auto Loader

In the Ultra Transcenders books

DP-750

Each book explains Auto Loader in context, with comparison tables and the common traps.

Terms in this definition

• Structured Streaming

  Spark's engine for near real-time data: you write a batch-like query and it processes new data incrementally, recording progress in a checkpoint with exactly-once guarantees. Auto Loader and streaming tables run on it.

• General-purpose v1

  The older storage account kind (Storage), which lacks access tiers, Archive and premium file shares and retires on 13 October 2026. Converting to ZRS requires first upgrading to GPv2, a one-way change.

• Checkpoint

  The storage location in which a streaming query records its offsets, state, commits and ID. Thanks to it, a restarted query picks up precisely where it left off. Never point two queries at the same one.

• Schema

  The middle part of a Unity Catalog name (catalog.schema.table), grouping tables, views, volumes, functions and models inside a catalog. A grant on it covers everything in it now and later, and nothing inside can be reached without USE SCHEMA.

Related terms

• COPY INTO

  Loads files from a volume or cloud storage into a Delta table that already exists, skipping anything loaded before, so reruns are safe. Good for thousands of files; for millions, Auto Loader scales further.

• EventGrid EventSubscription Contributor

  A built-in Azure role for managing Event Grid subscriptions. Granting it, alongside the relevant storage roles, allows Azure Databricks to create the resources for file events and for Auto Loader's notification mode without manual set-up.

• GCS

  Object storage offered by Google Cloud. Alongside Amazon S3, ADLS and Blob Storage, it is a source Auto Loader can ingest from.

• ORC

  Optimized Row Columnar (ORC), an Apache column-oriented format that stores indexes and statistics inside the file. You can use it with Auto Loader and external tables but not with managed tables.

• READ FILES

  Granted on a storage credential or external location, this Unity Catalog permission lets tools like Auto Loader or COPY INTO pull files directly from cloud storage. WRITE FILES can't be used without it.

• read_files

  A SQL table-valued function that loads files in formats such as CSV, JSON, XML, Parquet, Avro, ORC, text or binary and infers their schema to return rows. Used with the STREAM keyword inside a streaming table, it relies on Auto Loader.

• Rescued data column

  An extra column, named _rescued_data unless changed, in which Auto Loader and file readers keep values that don't fit the schema (for instance unexpected columns or mismatched types or letter case) as JSON, so nothing is lost.

• Schema drift

  When the structure of incoming data shifts, for example columns being added, renamed or changing type. Auto Loader deals with this using cloudFiles.schemaEvolutionMode together with the _rescued_data column, which keeps non-matching values rather than discarding them.