The edition setting for a Lakeflow pipeline: CORE covers streaming ingestion, PRO adds CDC and ADVANCED, the default, adds expectations. If the pipeline uses a feature its edition lacks, the update fails.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Pipeline product edition in context, with comparison tables and the common traps.
Terms in this definition
- Pipeline
Groups activities logically so that, together, they move and transform data, either one after another or side by side; found in both Azure Data Factory and Microsoft Fabric.
- Streaming ingestion
An eventhouse loading method in which data travels straight in the request body, giving close to real-time latency; it works well when lots of tables each get only small volumes. For most cases, queued (batched) ingestion remains the default because it handles more throughput.
- CDC
Change data capture: rather than reloading whole snapshots, you process just the rows a source has added, changed or removed, with each change carrying its kind and an ordering value. Lakeflow pipelines handle such feeds through
AUTO CDC. - Expectations
Optional data quality rules declared with
CONSTRAINT ... EXPECTon the tables and views a pipeline defines. Each incoming record is checked; on failure the pipeline can just record a warning (the default), discard the record or halt the update, and the results are logged. - CRUD
Shorthand for create, read, update and delete, the four basic things you do with data. Data-plane roles in Azure Cosmos DB, for instance, authorise those operations on items.