Typically a materialized view using expect_or_fail to confirm something across datasets, e.g. row counts agreeing. Because downstream tables aren't blocked by it, put checks in their own pipeline and chain pipelines with a job.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Validation table in context, with comparison tables and the common traps.
Terms in this definition
- Materialized view
A view that stores its query output ahead of time. A serverless pipeline maintains it, refreshing it on demand with
REFRESH MATERIALIZED VIEW, on a schedule or when upstream data changes, incrementally when it can. - EXPECT
Sets a data quality rule on a dataset in a SQL pipeline. To discard failing rows, add
ON VIOLATION DROP ROW; to halt everything,ON VIOLATION FAIL UPDATE; with neither, those rows are kept and recorded in the logs. - Pipeline
Groups activities logically so that, together, they move and transform data, either one after another or side by side; found in both Azure Data Factory and Microsoft Fabric.
- Job
A sequence of steps executed together on one agent or runner, or on the server for agentless work. While running, each occupies one of your parallel jobs.