READ FILES

In the Ultra Transcenders books

DP-750

Each book explains READ FILES in context, with comparison tables and the common traps.

Terms in this definition

• Storage credential

  Wraps a long-lived credential, often an Access Connector's managed identity, that external locations use to reach ADLS. It is a Unity Catalog securable, and CREATE STORAGE CREDENTIAL is needed to make one.

• External location

  A securable object in Unity Catalog that combines a storage credential with a particular path in cloud object storage, so that who may read or write under that path is controlled centrally. Volumes, external tables and managed storage locations all depend on it.

• Unity Catalog

  Azure Databricks' governance solution covering both data and AI in one place, with centralised permissions, auditing, data discovery and lineage.

• LIKE

  Compares strings with a pattern that can contain the % and _ wildcards. Because it only understands character patterns, searching big volumes of text this way is much slower than using full-text search.

• Auto Loader

  The cloudFiles Structured Streaming source, which picks up new files from cloud storage or volumes as they arrive. Its checkpoint records progress so each file is processed exactly once, and it can infer and evolve the schema.

• COPY INTO

  Loads files from a volume or cloud storage into a Delta table that already exists, skipping anything loaded before, so reruns are safe. Good for thousands of files; for millions, Auto Loader scales further.

• General-purpose v1

  The older storage account kind (Storage), which lacks access tiers, Archive and premium file shares and retires on 13 October 2026. Converting to ZRS requires first upgrading to GPv2, a one-way change.

Related terms

• SharePoint Online

  Part of Microsoft 365, it stores, shares and manages content in cloud-based sites and document libraries. Reports from Power BI can be embedded in its pages, and Power BI can read files kept there.

• Streaming table

  Delta table built for incremental loading, declared with CREATE STREAMING TABLE (Databricks SQL or Lakeflow pipelines). Wrapping sources in STREAM, as in STREAM read_files(...), reads them as streams.