Lakeflow Pipelines expectations: warn, drop, fail and quarantine

Expectations are named SQL Boolean checks applied to each record as it passes through a query. There are three built-in actions; quarantine is a pattern built from them, not a fourth action.

CREATE OR REFRESH STREAMING TABLE customers(
  CONSTRAINT valid_age EXPECT (age BETWEEN 0 AND 120),
  CONSTRAINT valid_id EXPECT (id IS NOT NULL)
    ON VIOLATION DROP ROW
) AS SELECT * FROM STREAM(raw_customers);

The constraint must be valid SQL and cannot include custom Python functions, external service calls or subqueries that reference other tables. Names must be unique within a dataset, and they can be reused across datasets.

In Python, expect_all, expect_all_or_drop and expect_all_or_fail take a dictionary of name to constraint and apply one action to the group. SQL can hold several expectations on a dataset but has no way to group them or load them dynamically. Rules can live in a Delta table or Python module, read by a helper function.

Choosing the action is a per-constraint decision, and the production checklist gives the rule: fail for conditions that should stop everything, such as a broken primary key; drop for records that can be discarded, with a quarantine table capturing what was dropped; warn only where someone actually watches the trend. Figure 5.2 shows what each action does to an invalid row, and how quarantine is built.

The effect of a failed expectation

The scope of a failure depends on the pipeline mode.

The EXPECTATION_VIOLATION error reports the violated expectation and, for many queries, the input record. Metrics are not recorded for fail, so read the message and fix the pipeline code before rerunning.

Expectations enforce quality, not orchestration. A validation table, such as a materialized view that compares row counts under expect_or_fail, does not gate its readers: a downstream dataset does not wait for its result. To stop downstream work when validation fails, put the validation and the downstream work in separate pipelines and link them with task dependencies in a job, as covered in “Lakeflow Jobs”.

The quarantine pattern

Quarantine keeps bad rows for inspection without failing or losing them. A warn expectation records metrics, a computed column marks each row, and views split the two paths.

from pyspark import pipelines as dp
from pyspark.sql.functions import expr

rules = {
    "valid_pickup": "(pickup_zip IS NOT NULL)",
    "valid_dropoff": "(dropoff_zip IS NOT NULL)",
}
bad = "NOT({0})".format(" AND ".join(rules.values()))

@dp.table(temporary=True, partition_cols=["is_quarantined"])
@dp.expect_all(rules)
def trips_checked():
    return (
        spark.readStream.table("raw_trips")
        .withColumn("is_quarantined", expr(bad))
    )

@dp.view
def valid_trips():
    return spark.read.table("trips_checked").filter(
        "is_quarantined=false"
    )

@dp.view
def invalid_trips():
    return spark.read.table("trips_checked").filter(
        "is_quarantined=true"
    )

Where expectations do not reach

• Only streaming tables, materialized views and temporary views support them. Sinks do not.
• AUTO CDC FROM SNAPSHOT does not support them.
• A view is calculated only when queried, so its metrics may be missing, or may appear several times if several datasets read it.
• The pipeline edition must be ADVANCED to enforce expectations. ADVANCED is the default, PRO covers CDC, and CORE covers streaming ingest only.
• For standalone datasets, the pipeline UI’s Data quality tab is not available, so metrics come from the event log; see “Monitoring and alerting”.

Common trap: Using expect_or_drop when the bad rows may be needed later - dropped rows are gone and only a count is logged, so use the quarantine pattern to keep them.

Common trap: Relying on a validation table with expect_or_fail to block downstream tables in the same pipeline - in a triggered pipeline other flows continue to update, so the gate must be a job dependency between separate pipelines.

Get the whole book

This note is one section of Ultra Transcenders DBX-DEP: Databricks Data Engineer Professional, an independent study guide that explains every topic the exam covers by technology, with comparison tables, diagrams and the common traps, plus a glossary linked to the Databricks documentation.

About the book · DBX-DEP terms in the glossary · All DBX-DEP study notes