Databricks SQL

In the Ultra Transcenders books

DP-900

Each book explains Databricks SQL in context, with comparison tables and the common traps.

Terms in this definition

• Databricks

  Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.

• Lakehouse

  Fabric storage item in OneLake that holds structured and unstructured content side by side: managed Delta tables go under Tables, other files under Files. Spark is used for processing, and a read-only SQL analytics endpoint allows T-SQL queries.

• ANSI

  The American National Standards Institute, which made SQL a standard in 1986, a year before ISO did. Its standard types, for example INT, DECIMAL and VARCHAR, explain why SQL looks much the same from one vendor to another, even though each has its own dialect.

• Serverless

  Compute tier for single Azure SQL databases that scales automatically, pauses when idle and charges by the second. It is offered in General Purpose and Hyperscale, not Business Critical, and reserved capacity does not apply.

• Agents (classic) API

  First-generation Foundry Agent Service API, based on threads, messages and runs. It is deprecated, replaced by conversations and responses, and retires on 31 March 2027.

Related terms

• Query cache

  Repeated queries in Databricks SQL may be served from a local or remote result cache. Entries expire after 24 hours or as soon as a source table changes, and such cached answers come without a query profile.

• Query History

  A page in Databricks SQL where you can filter through past queries from SQL warehouses and serverless compute and open the profile of any of them.

• RTAS

  Short for REPLACE TABLE AS SELECT; Databricks SQL users typically write CREATE OR REPLACE TABLE ... AS SELECT. Schema and rows are swapped for whatever the query returns, and Delta logs this as another version.

• Standard connectors

  Part of Lakeflow Connect: Auto Loader for object storage, plus SFTP and message buses (Kafka, Pulsar, Pub/Sub). Choose them over managed connectors when you need more control, working in Databricks SQL, Structured Streaming or pipelines.

• Streaming table

  Delta table built for incremental loading, declared with CREATE STREAMING TABLE (Databricks SQL or Lakeflow pipelines). Wrapping sources in STREAM, as in STREAM read_files(...), reads them as streams.

• Temporary view

  A view that isn't saved to a catalog. It lives for the session in notebooks and jobs, for the query in Databricks SQL, and inside a Lakeflow pipeline (CREATE TEMPORARY VIEW or @dp.temporary_view) only that pipeline can see it.

• Verbose audit logs

  Optional workspace setting: every Databricks SQL query or notebook command run gets logged with its text (commandSubmit, commandFinish, runCommand).