Blob Storage – Object Replication using Azure PowerShell

Azure Blob Storage got a new feature named Object Replication. Currently, this feature is only supported for the block blob. Object replication asynchronously copies blobs between a source storage account and a destination account.

Object replication helps to minimize latency and data distribution, and the implementation requires replication policies and rules to be set on source and destination accounts.

This feature would be helpful if you want to distribute blobs from the production environment container to the other region containers to use for any other purpose, for example – data analysis or management – related activities based on certain rules and policies.

Some Important Points

There are some important points which must be taken care of for storage accounts to implement object replication –

• Must be a general purpose V2 storage account and blob versioning enabled.
• The source account must have change feed enabled.
• Can be in the same or different Azure regions.
• Can be in same subscription or different subscription.
• Can be in different tenants or Azure Active Directory.
• The source account can be associated with up to two destination storage account to configure rules and policies.
• The storage accounts require Contributor roles to configure replication rules.

Implementation

We need to perform the below steps to implement Object replication between two storage accounts. The demo is prepared using Azure PowerShell to support automation.

Step – 1 : Create a resource group

$rgName = "rgobjectrelication"
$rglocation = 'eastus'
New-AzResourceGroup `
        -Name $rgName `
        -Location $rglocation -Force

Step – 2 : Create a source storage account and container in “eastus” location

$sourceaccount = "sasourceaccount"
$sourcecontainername = "sourceblobcontainer"
$sourcelocation = 'eastus'

New-AzStorageAccount `
    -ResourceGroupName $rgname `
    -Name $sourceaccount `
    -Location $sourcelocation `
    -SkuName Standard_RAGRS `
    -Kind StorageV2

Get-AzStorageAccount `
    -ResourceGroupName $rgname `
    -StorageAccountName $sourceaccount | `
    New-AzStorageContainer $sourcecontainername

Step – 3 : Create a destination storage account and container in northeurope

$destinationaccount = "sadestinationaccount"
$destinationcontainername = "destinationblobcontainer"
$destinationlocation = 'northeurope'

New-AzStorageAccount `
   -ResourceGroupName $rgname `
   -Name $destinationaccount `
   -Location $destinationlocation `
   -SkuName Standard_RAGRS `
   -Kind StorageV2

Get-AzStorageAccount `
    -ResourceGroupName $rgname `
    -StorageAccountName $destinationaccount | `
    New-AzStorageContainer $destinationcontainername

Step – 4 : Enable blob versioning and change feed on source account.

Update-AzStorageBlobServiceProperty `
    -ResourceGroupName $rgname `
    -StorageAccountName $sourcestorageaccount `
    -EnableChangeFeed $true `
    -IsVersioningEnabled $true

Step – 5 : Enable blob versioning on destination account.

Update-AzStorageBlobServiceProperty `
   -ResourceGroupName $rgname `
   -StorageAccountName $destinationstorageaccount `
   -EnableChangeFeed $true `

Step – 6 : Create replication rules between source and destination account containers

$prefixrule = New-AzStorageObjectReplicationPolicyRule `
         -SourceContainer $sourcecontainername `
         -DestinationContainer $destinationcontainername `
         -PrefixMatch Sales,Marketing

$policies= Set-AzStorageObjectReplicationPolicy `
                -ResourceGroupName $rgname `
                -StorageAccountName $destinationaccount `
                -PolicyId default `
                -SourceAccount $sourceaccount `
                -Rule $prefixrule

Set-AzStorageObjectReplicationPolicy -ResourceGroupName $rgname `
    -StorageAccountName $sourceaccount `
    -InputObject $policies

Script Output

To test the behavior of the script, we could combine all the script blocks into one PowerShell (.ps1) file and run it using the Windows Command Prompt.

The expected behavior is when files with the prefix Sales and Marking (Step 6) uploaded into the source container then object replication will happen in the destination account asynchronously.

1. Use Connect-AzAccount command to connect Azure subscription to run the script

2. Upload some files into source container <sasourceaccount> with the names starting with Sales and Marketing.

3. Object replication policy has copied the files into destination blob container.

So, this is how we can leverage object replication to copy files between source and destination account containers. Though, object replication costs to perform write in the destination container, but it is quite useful.