Using nested pipelines in Azure Data Factory

Nested pipelines in Azure Data Factory (ADF) are a way to organize and manage complex workflows. They allow you to create a hierarchy of pipelines, where one pipeline can call another as a part of its execution. This is particularly useful for managing large data processing jobs that need to be broken down into smaller, more manageable pieces.

Here’s an overview of how nested pipelines work in ADF:

• Control Flow Activities: ADF provides several control flow activities that can contain other activities. These include ForEach, If Condition, Until, and Switch activities¹.
• Containers: Think of nested activities as containers that hold one or more other activities. These containers can execute depending on the top-level control flow activity¹.
• Navigating Nested Activities: You can navigate to the contained activities in a nested activity by selecting the activity tab in the ADF UI and then clicking the pencil icon to drill down into the inner activities panel¹.
• Limitations: While you can use nested activities within control flow activities, you cannot nest ForEach loops directly. However, you can execute another pipeline within a ForEach loop, which can then contain another loop³⁴.

It’s important to note that while nesting can add flexibility to your data workflows, it also introduces complexity. Therefore, it’s essential to plan and design your nested pipelines carefully to ensure they are maintainable and efficient.

For more detailed information and best practices on using nested pipelines in Azure Data Factory, you can refer to the official Microsoft documentation¹.

(1) Nested activities – Azure Data Factory & Azure Synapse. https://learn.microsoft.com/en-us/azure/data-factory/concepts-nested-activities.
(2) Nested ForEach loops in Azure Data Factory – Syntera. https://www.syntera.ch/blog/2022/12/15/nested-foreach-loops-in-azure-data-factory/.
(3) Nested ForEach in ADF (Azure Data Factory) – Stack Overflow. https://stackoverflow.com/questions/72684969/nested-foreach-in-adf-azure-data-factory.
(4) Linking nested pipeline runs to parent pipeline – Microsoft Q&A. https://learn.microsoft.com/en-us/answers/questions/576740/linking-nested-pipeline-runs-to-parent-pipeline.
(5) undefined. https://learn.microsoft.com/en-us/rest/api/datafactory/activity-runs/query-by-pipeline-run.
(6) undefined. https://learn.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/query-by-factory.