Full refresh

In the Ultra Transcenders books

DBX-DEPDP-750

Each book explains Full refresh in context, with comparison tables and the common traps.

Terms in this definition

• Pipeline

  Groups activities logically so that, together, they move and transform data, either one after another or side by side; found in both Azure Data Factory and Microsoft Fabric.

• State

  A parameter in OAuth that carries custom data through the authorisation flow and back, as with the "Support state parameter" option in APIM.

• Streaming table

  Delta table built for incremental loading, declared with CREATE STREAMING TABLE (Databricks SQL or Lakeflow pipelines). Wrapping sources in STREAM, as in STREAM read_files(...), reads them as streams.

• Materialized view

  A view that stores its query output ahead of time. A serverless pipeline maintains it, refreshing it on demand with REFRESH MATERIALIZED VIEW, on a schedule or when upstream data changes, incrementally when it can.

Related terms

• Auto full refresh

  An optional Lakeflow Connect policy, disabled unless turned on, that starts a full refresh by itself after something like a truncated table or an incompatible schema change. The gap between refreshes is at least min_interval_hours, which defaults to 24.

• DOUBLE

  An 8-byte floating-point type with double precision. A materialized view that sums or averages one may need a full refresh, which casting to DECIMAL avoids.

• Explicit flow

  A flow you declare on its own, apart from the target. A streaming table can then take data from multiple sources with no UNION, and adding another source needs no full refresh.

• Identity column

  Assigns a unique BIGINT to every row inserted into a Delta table, though not always in an unbroken sequence; it is declared as GENERATED ALWAYS AS IDENTITY or GENERATED BY DEFAULT AS IDENTITY. Concurrent writes are disabled and a rebuild can change the numbers, so it suits append-only tables that no full refresh will ever touch.