Lakeflow Connect

In the Ultra Transcenders books

DP-750

Each book explains Lakeflow Connect in context, with comparison tables and the common traps.

Terms in this definition

• Databricks

  Analytics platform built on Apache Spark where data is processed in notebooks; Unity Catalog is the governance model it recommends.

• SaaS

  A cloud model in which you consume a finished application that the provider operates, Microsoft 365 for instance. Nearly everything is the provider's job; you look after your data, users and devices.

• Streaming

  Sending model output piece by piece as server-sent events. Only the delivery changes; the answer's content, completeness and cost stay the same.

• LAST

  Restricted to visual calculations, this DAX function reads a value from the final position along one of the visual's axes. Writing INDEX(-1) gives the same result.

• CDC

  Change data capture: rather than reloading whole snapshots, you process just the rows a source has added, changed or removed, with each change carrying its kind and an ordering value. Lakeflow pipelines handle such feeds through AUTO CDC.

• Serverless

  Compute tier for single Azure SQL databases that scales automatically, pauses when idle and charges by the second. It is offered in General Purpose and Hyperscale, not Business Critical, and reserved capacity does not apply.

• Unity Catalog

  Azure Databricks' governance solution covering both data and AI in one place, with centralised permissions, auditing, data discovery and lineage.

Related terms

• Integrated CDC

  A Lakeflow Connect design for SQL Server ingestion in which one pipeline points at a Unity Catalog connection, with no separate ingestion gateway needed.

• Standard connectors

  Part of Lakeflow Connect: Auto Loader for object storage, plus SFTP and message buses (Kafka, Pulsar, Pub/Sub). Choose them over managed connectors when you need more control, working in Databricks SQL, Structured Streaming or pipelines.