Data profiling

In the Ultra Transcenders books

DBX-GENAIDP-750

Each book explains Data profiling in context, with comparison tables and the common traps.

Terms in this definition

• Data quality monitoring

  Combines two capabilities in Unity Catalog. Data profiling, once called Lakehouse Monitoring, generates metric tables, while anomaly detection checks how fresh and complete the tables in a schema are.

• Once flow

  A flow that executes a single time and repeats only if the pipeline is fully refreshed. It's a common choice for a first full load ahead of a change feed, and that refresh becomes dangerous once the original files no longer exist.

• Event

  Table in Log Analytics where entries from Windows event logs are kept.

• Statistics

  Information about how values are distributed in a column, which the warehouse optimiser uses to estimate plan costs. Fabric builds and updates it for you; manual CREATE, UPDATE and DROP STATISTICS are limited to single-column histograms.

• OVER

  Gives a T-SQL window function its window: PARTITION BY, ORDER BY and, if wanted, a ROWS or RANGE frame. Rankings and running totals can then be worked out while every row is kept.

• Dashboard

  Page in VCF Operations built from widgets that display object metrics, properties, alerts and relationships. Predefined and custom dashboards exist, and either kind can be shared.

• Anomaly detection

  Identifying data points or time-series values that look out of the ordinary, such as fraud or a spike from a sensor. Microsoft retired the dedicated Azure AI Anomaly Detector service on 1 October 2026.

• Schema

  The middle part of a Unity Catalog name (catalog.schema.table), grouping tables, views, volumes, functions and models inside a catalog. A grant on it covers everything in it now and later, and nothing inside can be reached without USE SCHEMA.