The event trigger works with storage account blob container. It gets triggered if any blob file is created or deleted so event trigger is scoped to trigger for such file. It supports only for Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts.
Steps to Perform
You need to create below services to implement a pipeline to perform copy activity.
- Create a resource group
- Create Storage Account with two Blob container named – source and destination
- Create Data Factory named – “adf-blobevent-trigger-demo”
*Steps to create these resources are not covered in this article.
Create Arm Template
- Save JSON to a arm_template.json file
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"factoryName": {
"type": "string",
"metadata": "Data Factory name",
"defaultValue": "adf-blobevent-trigger-demo"
},
"AzureBlobStorage_connectionString": {
"type": "secureString",
"metadata": "Secure string for 'connectionString' of 'AzureBlobStorage1'"
},
"EventTrigger_scope": {
"type": "string"
}
},
"variables": {
"factoryId": "[concat('Microsoft.DataFactory/factories/', parameters('factoryName'))]"
},
"resources": [
{
"name": "[concat(parameters('factoryName'), '/EventPipeline')]",
"type": "Microsoft.DataFactory/factories/pipelines",
"apiVersion": "2018-06-01",
"properties": {
"activities": [
{
"name": "EventPipeline",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "BinarySource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": true
},
"formatSettings": {
"type": "BinaryReadSettings"
}
},
"sink": {
"type": "BinarySink",
"storeSettings": {
"type": "AzureBlobStorageWriteSettings"
}
},
"enableStaging": false
},
"inputs": [
{
"referenceName": "SourceDataset",
"type": "DatasetReference",
"parameters": {}
}
],
"outputs": [
{
"referenceName": "DestinationDataset",
"type": "DatasetReference",
"parameters": {}
}
]
}
],
"annotations": []
},
"dependsOn": [
"[concat(variables('factoryId'), '/datasets/SourceDataset')]",
"[concat(variables('factoryId'), '/datasets/DestinationDataset')]"
]
},
{
"name": "[concat(parameters('factoryName'), '/AzureBlobStorage1')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"annotations": [],
"type": "AzureBlobStorage",
"typeProperties": {
"connectionString": "[parameters('AzureBlobStorage_connectionString')]"
}
},
"dependsOn": []
},
{
"name": "[concat(parameters('factoryName'), '/DestinationDataset')]",
"type": "Microsoft.DataFactory/factories/datasets",
"apiVersion": "2018-06-01",
"properties": {
"linkedServiceName": {
"referenceName": "AzureBlobStorage1",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "Binary",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"container": "destination"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureBlobStorage1')]"
]
},
{
"name": "[concat(parameters('factoryName'), '/SourceDataset')]",
"type": "Microsoft.DataFactory/factories/datasets",
"apiVersion": "2018-06-01",
"properties": {
"linkedServiceName": {
"referenceName": "AzureBlobStorage1",
"type": "LinkedServiceReference"
},
"annotations": [],
"type": "Binary",
"typeProperties": {
"location": {
"type": "AzureBlobStorageLocation",
"fileName": "image.PNG",
"container": "source"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureBlobStorage1')]"
]
},
{
"name": "[concat(parameters('factoryName'), '/EventTrigger')]",
"type": "Microsoft.DataFactory/factories/triggers",
"apiVersion": "2018-06-01",
"properties": {
"annotations": [],
"runtimeState": "Started",
"pipelines": [
{
"pipelineReference": {
"referenceName": "EventPipeline",
"type": "PipelineReference"
},
"parameters": {}
}
],
"type": "BlobEventsTrigger",
"typeProperties": {
"blobPathBeginsWith": "/source/blobs/image",
"ignoreEmptyBlobs": true,
"scope": "[parameters('EventTrigger_scope')]",
"events": [
"Microsoft.Storage.BlobCreated"
]
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/pipelines/EventPipeline')]"
]
}
]
}
Create Data Factory using Arm Template
- Go to Resource Group > Azure Data Factory > Author & Monitor and wait for Azure data factory to open.
- Go to Arm Template > Import Template from the top menus.
- Click Build your own template in editor.
- Load the arm_template.json file.
- Save the file.
- Enter the blob storage connect string and trigger scope.
- Check the Terms & Conditions and click purchase.
- Wait for the resource to get deployed in ADF.
Point 6 : Scope should be formatted as “/subscriptions/<Your Subscription ID>/resourceGroups/<Data Factory Name>/providers/Microsoft.Storage/storageAccounts/<Storage Account Name>”
One ARM template will get deployed successfully then go to data factory and verify pipeline, datasets and trigger.
Go to Storage Account > Containers > Source container and upload any image that start with name “Image”. For Example – image.png, image_1.jpg. This event will trigger data factory pipeline which will perform copy activity between source and destination container.
Important : You need to enable to trigger in ADF and publish changes if trigger is not enabled by default.
Hope, this gives an understanding about how event triggers work and create an ADF using an ARM template.
If you have any suggestions/feedback, please put them in the comment box.
Happy Learning 🙂
Leave a Reply