A visual, low-code Microsoft Fabric item for preparing data, based on Power Query. It links to data sources, transforms the data and writes the output to targets like a lakehouse or a warehouse.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Dataflow Gen2 in context, with comparison tables and the common traps.
Terms in this definition
- Microsoft Fabric
Analytics platform from Microsoft delivered as SaaS on top of OneLake, offering capabilities including Direct Lake and shortcuts.
- LIKE
Compares strings with a pattern that can contain the % and _ wildcards. Because it only understands character patterns, searching big volumes of text this way is much slower than using full-text search.
- Lakehouse
Fabric storage item in OneLake that holds structured and unstructured content side by side: managed Delta tables go under Tables, other files under Files. Spark is used for processing, and a read-only SQL analytics endpoint allows T-SQL queries.
- Warehouse
Fabric item offering complete T-SQL support (DML, DDL, multi-table transactions) over Delta tables held in OneLake; lakehouse SQL analytics endpoints, by contrast, are read-only.
Related terms
- Data destination
Where a Dataflow Gen2 query writes its results, for example a lakehouse, a warehouse or an Azure SQL database. Each query may have a different one, and sending output to a warehouse needs staging enabled.
- Dataflow activity
Lets a pipeline run a dataflow, so Dataflow Gen2 refreshes can be orchestrated and given parameter values. Dataflow Gen1 can't be run this way.
- Dataflow Gen1
Power BI's first dataflow type, now legacy and frozen for new features. Dataflow Gen2 adds data destinations and a pipeline activity that Gen1 lacks, though Gen1 does support DirectQuery through the dataflow connector.
- Dataflow staging items
When staging is switched on, Dataflow Gen2 creates a DataflowsStagingLakehouse and a DataflowsStagingWarehouse in the workspace to keep intermediate data. Fabric looks after them, and you shouldn't work with them directly.
- Dynamic schema
If the shape of incoming data changes, this Dataflow Gen2 option throws away the target table and builds it afresh, which means any relationships or measures layered on top may disappear. It is offered only when Replace is the update method; choosing Fixed schema instead preserves the table and swaps just its rows.
- Fabric Data Factory
Presented as the next step on from Azure Data Factory, this Fabric workload moves and reshapes data using copy jobs, mirroring, Dataflow Gen2 and pipelines.
- Fast copy
A Dataflow Gen2 setting that hands large loads to the Copy activity engine used by pipelines, for instance CSV or Parquet files of at least 100 MB, or database sources of 5 million rows or more. With Require fast copy, the refresh fails rather than reverting to the normal engine.
- Fixed schema
With the Replace update method, this Dataflow Gen2 option keeps the existing destination table and swaps only its contents, so any relationships and measures survive. Warehouse destinations offer nothing else; the Dynamic schema alternative drops and rebuilds the table.