GDPR and CCPA require personal data to be permanently and completely deleted on a customer’s explicit request, within a specified period such as one calendar month. Sources, every medallion layer and retained files can all hold copies.
Deletion or obfuscation is the first decision, and complete erasure is the safest option. Where the design uses a non-natural key such as user_id rather than email, it may be possible to sever the link: delete the PII and leave non-PII activity in place.
| Step | Action | Why |
|---|---|---|
| 1 | Record the request in a control table | Drives a scheduled job |
| 2 | Delete from bronze, using MERGE for batches |
The recommended starting point |
| 3 | Propagate to silver and gold | Each layer holds its own copy |
| 4 | Delete from upstream sources | Rules also cover Kafka, files and databases |
| 5 | Run REORG TABLE ... APPLY (PURGE) |
Deletion vectors leave data in current files |
| 6 | Run VACUUM after the retention period |
Removes old files physically |
| 7 | Update the control table | Records completion |
A single user can go with a plain DELETE. For a batch, Databricks recommends a MERGE driven by a control table with a row for each requesting user.
DELETE FROM bronze.users WHERE user_id = 5;
MERGE INTO target
USING (SELECT user_id FROM gdpr_control_table) AS source
ON target.user_id = source.user_id
WHEN MATCHED THEN DELETE;Propagating erasure downstream
Materialized views and streaming tables handle source deletes in opposite ways.
| Dataset | On source delete | What to do |
|---|---|---|
| Materialized view | Handled automatically; may fully recompute to stay correct | Refresh, then run maintenance |
| Streaming table | A non-append change in a Delta source breaks the stream | Delete from the source, delete from the streaming table, set skipChangeCommits on the read |
| Streaming table, alternative | Full refresh clears state and reprocesses everything | Only if history is available and reprocessing is affordable |
| Stream needing the changes | Read the change data feed and handle deletes in code | See the chapter Delta table optimisation |
def users_bronze():
return (
spark.readStream
.option("skipChangeCommits", "true")
.table(f"{catalog}.{schema}.source_users")
)Common trap: Adding
skipChangeCommitsand considering the streaming table clean - the option makes the stream ignore the delete, so the row stays in the target. Delete it there with DML as well.
Common trap: Running a full refresh after an erasure when the source is a Kafka topic with short retention - anything beyond the source’s retention, such as a topic that ages out after 7 days, is not processed again and is lost. Reserve full refresh for cases where history is still available.
Making the deletion physical
Deletion vectors mean a DELETE may only mark rows in metadata. For tables with them enabled, including materialized views and streaming tables, also run REORG TABLE ... APPLY (PURGE), which rewrites files containing soft-deleted data. OPTIMIZE and auto-compaction also rewrite files, but compaction has no strict guarantee that every change gets applied.
REORG TABLE events APPLY (PURGE);
VACUUM events;These two statements cannot simply run back to back. REORG cleans the current files, but the old files stay for time travel, and VACUUM deletes them only once they have expired. Run VACUUM with a delay after REORG, or shorten the retention period and accept less history. A WHERE clause on REORG accepts only partition key filters, and dropping a column with column mapping enabled is a soft-delete too, so the same steps apply. Figure 16.1 shows where the data still sits after each step.
Maintenance may already be handled. Predictive optimization is the recommended route for managed tables, and without it Lakeflow Pipelines perform maintenance within 24 hours of a dataset being updated. Otherwise, schedule VACUUM yourself. For discovery, a documented notebook shows how Data Classification results can help find where a person’s data sits.