Functions that ship with Apache Spark and are tuned to run across a cluster. Compared with Python UDFs they handle growth more easily, making them a good fit for big datasets and for continuous or frequently run jobs.
Read more: Databricks docs
In the Ultra Transcenders books
Each book explains Built-in functions in context, with comparison tables and the common traps.
Terms in this definition
- Apache Spark
An open-source engine that spreads data processing over a cluster of machines so that big data sets are handled in parallel, in batches or as streams. In Azure you can use it through Microsoft Fabric or Azure Databricks.
Related terms
- AI Functions
Classic SQL warehouses aside, these built-in functions let SQL, notebooks and pipelines apply LLMs and research techniques to data.
ai_queryhandles general use, while task-specific ones likeai_classifywork without a prompt. - Manage fields
An Eventstream step for reshaping events: you can add, drop or rename fields and convert their types, and built-in functions handle strings, dates, times and maths.