Sets a data quality rule on a dataset in a SQL pipeline. To discard failing rows, add ON VIOLATION DROP ROW; to halt everything, ON VIOLATION FAIL UPDATE; with neither, those rows are kept and recorded in the logs.
Read more: Databricks docs
In the Ultra Transcenders books
Each book explains EXPECT in context, with comparison tables and the common traps.
Terms in this definition
- Semantic model
Sometimes called an OLAP model, this Power BI and Fabric layer defines the measures, hierarchies, tables and relationships that reports and dashboards query; star schema design is the usual pattern.
- Serverless
Compute tier for single Azure SQL databases that scales automatically, pauses when idle and charges by the second. It is offered in General Purpose and Hyperscale, not Business Critical, and reserved capacity does not apply.
- Pipeline
Groups activities logically so that, together, they move and transform data, either one after another or side by side; found in both Azure Data Factory and Microsoft Fabric.
- Architecture Definition Document
A key deliverable bringing together the main architecture artifacts across the four domains for every relevant state: baseline, transition and target. It sets out, in qualitative terms, what the architect intends.
- ON VIOLATION
Decides the fate of invalid data under a pipeline expectation.
DROP ROWdiscards it andFAIL UPDATEstops the update, while omitting the clause lets the rows through to the target. - DROP
A DDL (Data Definition Language) statement that deletes a database object like a table. Once a table is dropped, its rows are gone unless a backup exists.
- CRUD
Shorthand for create, read, update and delete, the four basic things you do with data. Data-plane roles in Azure Cosmos DB, for instance, authorise those operations on items.
Related terms
- Agent's user account
Optional Microsoft Entra user object tied one-to-one to an agent identity, for services such as a mailbox, Teams or OneDrive that expect a user. No password or passkey exists for it, and privileged admin roles cannot be assigned to it.
- BGE
Databricks hosts this English embedding model as
databricks-bge-large-en; expect 1024-dimension output and a limit of 512 tokens on what it reads. - Classic time intelligence
The original DAX time functions, like TOTALYTD, which work from the date column of a table marked as a date table. They expect a Gregorian (or shifted Gregorian) calendar with no gaps in the dates and offer no week-level functions, which sets them apart from the calendar-based preview approach.
- Customer-managed keys (Azure AI Search)
Adds a further encryption layer to Azure AI Search, protecting indexes and synonym maps with an RSA key you hold in Key Vault or Managed HSM, which the service reaches using its managed identity. Expect queries up to 30-60% slower and no added capacity.
- Effective free space
The amount of free capacity vSAN reports once a chosen storage policy and the fault domain layout are taken into account. Expect it to be below the raw disk free space.
- Expectations
Optional data quality rules declared with
CONSTRAINT ... EXPECTon the tables and views a pipeline defines. Each incoming record is checked; on failure the pipeline can just record a warning (the default), discard the record or halt the update, and the results are logged. - finish_reason
Tells you why a model stopped producing a chat completion. A model can refuse and hand back text outside the schema, so look at this field first when you expect structured output.
- Hybrid search
Set
query_type="hybrid"in AI Search to run keyword and vector matching together, with Reciprocal Rank Fusion producing the final order. Expect a 200-result ceiling and roughly four times the resource use of ANN.