Mapping data flow is an inbuilt data transformation tool in Azure data factory. It provides data transformation and visualization interface where no or very limited coding is needed. There are various transformation categories such as schema modifiers, row modifier, multiple inputs and outputs, formatters, flowlets and destination or sink. These categories are having number of... Continue Reading →
Azure Data Factory – Aggregate transformation in mapping data flow
Mapping data flow comes with many transformation options. While working with data flows, you need to incorporate appropriate transformations to get the desired result. The Aggregate transformation helps to perform aggregations of data using Count, Min, Max, and Sum with expression builder in ADF. So let's begin with the implementation:- Implementation We are going to... Continue Reading →
Azure Storage – Encrypt and Decrypt blobs using Azure Key Vault
The blob storage accessibility can be control using Role-Based Access Control (RBAC). Team members have read permission then they can view data inside blob storage which is quite useful for non-prod environment scenarios. But, if this is not a good use case for production environment. Production environment data security can be improved using Azure Data... Continue Reading →
Azure Data Factory – Remove Duplicate Data using Data Flow
Generally, Azure Data Factory aggregate transform has been used to perform COUNT, SUM, MIN, and MAX. The aggregate transform uses Azure Data Factory (ADF) expression to perform these computations. However, the aggregate transform can be used with a select transform to remove duplicate data. Let's see how we can achieve it. Data Source Just to... Continue Reading →
Azure Data Factory – Select Transformation in Data Flow
The post, Data Flow joins in Azure Data Factory uses select transformation and all the columns from input sources are sent to a blob storage which is a sink source in this case. But, this cannot be a real time requirement specially when there are many input data sources. In real time scenario, we only... Continue Reading →
Azure Data Factory – Using Data Flow Joins
The data flow is a visual data transformation feature in Azure Data Factory where no coding is required. Well, no coding does not mean that no effort is required to create a data flow, but we have visual representation of transformation components that can be put together to create data flow. Some examples - Join,... Continue Reading →
How to Create Event Trigger in Azure Data Factory
The event trigger works with storage account blob container. It gets triggered if any blob file is created or deleted so event trigger is scoped to trigger for such file. It supports only for Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts. Steps to Perform You need to create below services to... Continue Reading →
Azure Data Factory – How to Create Schedule Trigger
The schedule trigger runs automatically and invoke pipelines in an ADF. We can create multiple triggers to run a pipeline and create single trigger to run multiple pipeline. Let's suppose you have already implemented an ADF pipeline, if not you can find an example here to create an ADF pipeline so you need to create... Continue Reading →
Azure Data Factory – How to Parameterize Linked Service
The ADF linked Services are the connectors, between source and sink data stores, which are used to move data using pipeline activities. In real time scenario, we need to deal with different databases, blob storage containers, KeyVault secrets in various environments like development, QA, UAT etc. This article describes a general approach to overcome this... Continue Reading →
Azure Data Factory – How to Get Database Password from Key Vault
Introduction: Azure Data Factory (ADF) supports Azure Key Vault linked service. This linked service connects your ADF pipeline to a Key Vault to fetch secrets. Overview of Data factory and Key Vault are covered in previous articles. Azure Key Vault using CLI Create Azure Data Factory Pipeline and perform copy activity This approach can be... Continue Reading →