A parsing mode for CSV and JSON files that discards lines whose fields can't be parsed. Setting rescuedDataColumn means a type mismatch keeps the record instead.
Read more: Databricks docs
In the Ultra Transcenders books
Each book explains DROPMALFORMED in context, with comparison tables and the common traps.
Terms in this definition
- Cluster Shared Volumes
Shared failover cluster storage that all nodes write to simultaneously, typically for Scale-Out File Server or Hyper-V. On SAN-backed storage, ReFS-formatted volumes work in redirected mode.
- JSON
Text-based format for structured data. ARM templates and Cosmos DB documents are written in it, and most Azure REST APIs exchange request and response bodies as application/json.
Related terms
- FAILFAST
Stops a read as soon as a malformed record appears, whereas
DROPMALFORMEDandPERMISSIVEcontinue. SettingrescuedDataColumnchanges this: a wrong data type is saved rather than fatal, leaving errors for records that are broken or incomplete. - PERMISSIVE
The parser mode files get unless you choose otherwise. Fields it can't parse become null, with
_corrupt_recordavailable to capture what failed;DROPMALFORMEDandFAILFASTare the other options.