For incremental loads, a saved record of the furthest point reached (for example the latest last-modified date or biggest increasing ID copied so far), which lets each run fetch only rows beyond it. Structured Streaming uses the word differently: withWatermark(column, delay) sets the lateness allowed for event-time data, after which Spark closes windows and discards state.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Watermark in context, with comparison tables and the common traps.
Terms in this definition
- Deployment modes
The two ways ARM can deploy: Incremental, the default, creates or updates what the template lists and ignores everything else; Complete also removes resource group contents absent from the template.
- Structured Streaming
Spark's engine for near real-time data: you write a batch-like query and it processes new data incrementally, recording progress in a checkpoint with exactly-once guarantees. Auto Loader and streaming tables run on it.
- State
A parameter in OAuth that carries custom data through the authorisation flow and back, as with the "Support state parameter" option in APIM.
Related terms
- Copy job
Moves data between a source and a destination in Fabric Data Factory with no pipeline required. Runs can copy everything or just changes (incremental mode, driven by CDC or a watermark column), and the item keeps track of where the previous run stopped.
- Incremental copy
In this Copy job mode the opening run brings over the whole data set, after which each run moves only rows that are new or have changed, spotted using CDC or a watermark column. State is tracked by the job, and a reset back to a full copy is possible either for everything or table by table.
- Incremental load
Rather than reloading everything each time, this approach does one full copy and afterwards moves only rows that are new or changed, found through CDC, change tracking or a high watermark. It scales better than full loads, provided change detection can be trusted.
- ROWVERSION
A binary column type whose value updates by itself every time its row changes. Copy job can use it, or datetime, date, integer and string-as-datetime columns, as the watermark for incremental copies.