The storage location in which a streaming query records its offsets, state, commits and ID. Thanks to it, a restarted query picks up precisely where it left off. Never point two queries at the same one.
Also called streaming checkpoint.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Checkpoint in context, with comparison tables and the common traps.
Terms in this definition
- General-purpose v1
The older storage account kind (
Storage), which lacks access tiers, Archive and premium file shares and retires on 13 October 2026. Converting to ZRS requires first upgrading to GPv2, a one-way change. - Streaming
Sending model output piece by piece as server-sent events. Only the delivery changes; the answer's content, completeness and cost stay the same.
- State
A parameter in OAuth that carries custom data through the authorisation flow and back, as with the "Support state parameter" option in APIM.
- WHERE
Limits a SELECT, UPDATE or DELETE to just the rows meeting a condition. Omit it, and the statement hits every row.
Related terms
- Auto Loader
The
cloudFilesStructured Streaming source, which picks up new files from cloud storage or volumes as they arrive. Its checkpoint records progress so each file is processed exactly once, and it can infer and evolve the schema. - Production checkpoint
Hyper-V's default kind, capturing data-consistent state through VSS, or a file system freeze for Linux guests, but no memory. Choosing ProductionOnly makes creation fail instead of quietly using a standard checkpoint.
- Stateful operation
Any Structured Streaming operation that must hold intermediate data between batches (stream-stream joins, deduplication or aggregations over windows, for instance). That data sits in a state store kept with the checkpoint, and TTL, watermarks and time-range conditions keep it from growing endlessly.
- Structured Streaming
Spark's engine for near real-time data: you write a batch-like query and it processes new data incrementally, recording progress in a checkpoint with exactly-once guarantees. Auto Loader and streaming tables run on it.