When the structure of incoming data shifts, for example columns being added, renamed or changing type. Auto Loader deals with this using cloudFiles.schemaEvolutionMode together with the _rescued_data column, which keeps non-matching values rather than discarding them.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Schema drift in context, with comparison tables and the common traps.
Terms in this definition
- Auto Loader
The
cloudFilesStructured Streaming source, which picks up new files from cloud storage or volumes as they arrive. Its checkpoint records progress so each file is processed exactly once, and it can infer and evolve the schema. - Rescued data column
An extra column, named
_rescued_dataunless changed, in which Auto Loader and file readers keep values that don't fit the schema (for instance unexpected columns or mismatched types or letter case) as JSON, so nothing is lost. - VALUES
Returns in DAX the distinct column values, or table rows, still visible after filters are applied, sometimes with an extra blank entry. CALCULATE often takes the result as a table filter.