Copy job

In the Ultra Transcenders books

DP-600DP-700

Each book explains Copy job in context, with comparison tables and the common traps.

Terms in this definition

• Fabric Data Factory

  Presented as the next step on from Azure Data Factory, this Fabric workload moves and reshapes data using copy jobs, mirroring, Dataflow Gen2 and pipelines.

• Pipeline

  Groups activities logically so that, together, they move and transform data, either one after another or side by side; found in both Azure Data Factory and Microsoft Fabric.

• Agents (classic) API

  First-generation Foundry Agent Service API, based on threads, messages and runs. It is deprecated, replaced by conversations and responses, and retires on 31 March 2027.

• Deployment modes

  The two ways ARM can deploy: Incremental, the default, creates or updates what the template lists and ignores everything else; Complete also removes resource group contents absent from the template.

• CDC

  Change data capture: rather than reloading whole snapshots, you process just the rows a source has added, changed or removed, with each change carrying its kind and an ordering value. Lakeflow pipelines handle such feeds through AUTO CDC.

• Watermark

  For incremental loads, a saved record of the furthest point reached (for example the latest last-modified date or biggest increasing ID copied so far), which lets each run fetch only rows beyond it. Structured Streaming uses the word differently: withWatermark(column, delay) sets the lateness allowed for event-time data, after which Spark closes windows and discards state.

• WHERE

  Limits a SELECT, UPDATE or DELETE to just the rows meeting a condition. Omit it, and the statement hits every row.

• PREVIOUS

  Fetches the value of the preceding element along an axis of the visual matrix. This DAX function works only in visual calculations and is equivalent to OFFSET with -1.

Related terms

• Copy job activity

  An activity that lets a pipeline run a Copy job, either one that already exists or one created on the spot. This means the copy can sit alongside other steps and be started by the pipeline's schedules or event triggers.

• Full copy

  In a Copy job, the option to move the entire source dataset to the destination each time it runs; incremental copy is the alternative.

• Incremental copy

  In this Copy job mode the opening run brings over the whole data set, after which each run moves only rows that are new or have changed, spotted using CDC or a watermark column. State is tracked by the job, and a reset back to a full copy is possible either for everything or table by table.

• ROWVERSION

  A binary column type whose value updates by itself every time its row changes. Copy job can use it, or datetime, date, integer and string-as-datetime columns, as the watermark for incremental copies.

• Upsert

  Updates rows whose key columns match while inserting all others. Copy activity provides it when writing to lakehouse tables; Copy job names it Merge.