A task in Lakeflow Jobs that loops over an array (supplied as JSON, a task value or a job parameter) and executes an inner task for every element, with a setting for how many iterations may run at the same time.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains For each task in context, with comparison tables and the common traps.
Terms in this definition
- Lakeflow Jobs
The orchestration product in Databricks, previously called Workflows. Each job contains at least one task, and the links between tasks make up a directed acyclic graph.
- OVER
Gives a T-SQL window function its window: PARTITION BY, ORDER BY and, if wanted, a ROWS or RANGE frame. Rankings and running totals can then be worked out while every row is kept.
- JSON
Text-based format for structured data. ARM templates and Cosmos DB documents are written in it, and most Azure REST APIs exchange request and response bodies as application/json.
- Task value
A value that code stores during a job run with
dbutils.jobs.taskValues.set()so later tasks can use it, ideally referenced as{{tasks.<task>.values.<key>}}. - Job parameter
A key-value setting declared for the whole job, which a run can override, and which is passed automatically to tasks that take key-value parameters; any other task can reference it as
{{job.parameters.<name>}}.
Related terms
- Metadata-driven job
A job design where a For each task iterates over results that a SQL task has queried from a control table. Extra work then means one more row, with no edits to the job itself.