Stripping out repeated and almost identical documents as a pipeline stage. Spark ML can do this by pairing MinHash LSH with a similarity join.
Read more: Databricks docs
In the Ultra Transcenders books
Each book explains Deduplication in context, with comparison tables and the common traps.
Terms in this definition
- Pipeline stage
Groups jobs inside an Azure Pipelines run. Before each one begins, approvals and checks are evaluated, and stages follow one another unless dependsOn says otherwise.
- Pairing
Deployment pipelines link an item in one stage to its counterpart in the next. Deploying overwrites a linked item, but an unlinked item that just shares its name gets copied across as a duplicate.
- JOIN
Combines rows from two or more tables in one SELECT, most often by pairing a primary key with the foreign key that refers to it.
Related terms
- Data Reduction Ratio
Shows in Prism how much space erasure coding, compression and deduplication save between them. Thin provisioning and cloning are added only in the Overall Efficiency number.
- Fsas ETERNUS CS800
A deduplication appliance usable as a repository, with Veeam Data Movers talking to it.
- gateway server
A helper sitting in front of repositories that cannot run Veeam code, for example SMB shares or deduplication appliances, relaying data from the backup server or, for backup copy, the source repository.
- HPE StoreOnce
An HPE deduplication appliance reached via StoreOnce Catalyst and used as a repository. Since no Data Mover can run on it, a gateway server is required.
- Nutanix storage container
Carved logically from a storage pool, it contains vDisks and controls replication factor, erasure coding, deduplication and compression. Hyper-V mounts it as an SMB share; ESXi mounts it as an NFS datastore.
- Object space reservation
Deduplication and compression savings become ineffective once an object is thick provisioned through this vSAN storage policy rule, which reserves part of an object's logical size when it is created: none (thin, the default), 25, 50 or 75 per cent, or all of it (thick).
- Quantum DXi
A deduplication storage appliance from Quantum, supported as a backup repository in Veeam Backup & Replication.
- Stateful operation
Any Structured Streaming operation that must hold intermediate data between batches (stream-stream joins, deduplication or aggregations over windows, for instance). That data sits in a state store kept with the checkpoint, and TTL, watermarks and time-range conditions keep it from growing endlessly.