Groups activities logically so that, together, they move and transform data, either one after another or side by side; found in both Azure Data Factory and Microsoft Fabric.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Pipeline in context, with comparison tables and the common traps.
Terms in this definition
- ELT
Extract, load, transform: raw data lands in the target system first and is transformed there. See ETL for the opposite order.
- Azure Data Factory
Managed data integration service for ETL and ELT, built from pipelines, copy activities, triggers, mapping data flows and integration runtimes. It works in batches rather than routing individual transactions.
- Microsoft Fabric
Analytics platform from Microsoft delivered as SaaS on top of OneLake, offering capabilities including Direct Lake and shortcuts.
Related terms
- Activator
Rather than just displaying data as a dashboard does, this Real-Time Intelligence tool in Microsoft Fabric reacts to it. You define rules without writing code, and once a pattern or condition is spotted it can send an email or Teams message, or start a Power Automate flow or Fabric pipeline.
- Agent demands
Exists or equals conditions in a pipeline's pool that Pipelines checks against self-hosted agents' capabilities to choose one. With Microsoft-hosted agents, an image is selected instead.
- Auto-TTL
Rows older than a set number of days, measured from a timestamp column, are removed automatically (
DELETE ROWS ... DAYS AFTER). It covers Unity Catalog managed Delta or Iceberg tables plus pipeline streaming tables, depends on predictive optimization, and runs at no guaranteed time. - Azure Machine Learning component
Reusable building block for a pipeline, versioned and self-contained, much like a function: it declares a name, inputs and outputs, and bundles the command, code and environment it runs.
- Azure Pipelines for Slack
Brings pipeline run notifications into Slack channels and lets approvers respond to approval requests there.
- AzureAppConfigurationImport@10
Pushes key-values into an App Configuration store from a file (properties, YAML or JSON) during a pipeline run; tags, a label, a key prefix and strict mode can be added if wanted.
- Build validation
Requires a passing pipeline run on a pull request's pre-merge result before completion, valid until its build expiration lapses; an Azure Repos branch policy.
- Business hours
A pipeline check restricting stage starts to a configured time window.