Blob Storage – Bulk Files Upload Using Azure PowerShell

Azure Storage is one of the broadly used service. We are going to use a blob container of a Storage Account to upload files in bulk. Let’s consider a scenario where 100 or 1000 of various files should be upload in blob storage before starting some actual work or testing.

It becomes more tedious if it has to perform in multiple environments, for example – Development, QA, Staging, load, etc. Azure PowerShell template is available in VSTS DevOps, and the script will get integrated with various release pipelines.

In the world of automation, who wants to spend time on such activities again and again? Let’s suppose, we would need to upload various types of file .txt, .docx, .xlsx, image, and pptx.

So, let’s automate it.

Implementation Steps

So, we would need perform below steps in the Azure PowerShell script.

  • Connect Azure
  • Create a Resource Group
  • Create a Storage Account and a Blob Container
  • Upload files to the Blob Container

Azure PowerShell Script

The script has a variable $numberOfBatch which is used in a for loop to create that many set of files.

The script will upload files by renaming it to a GUID value to have a unique name in the blob container.

Define some variables and create a resource group.

#Define Variables 
$rgname = 'Bulk-Upload'
$rglocation = 'EastUS'
$storageaccountname = 'bulkuploadstorage'
$containerName = 'bulkblobcontainer'
$fileslocation = "C:\Temp\Files\"
$numberOfBatch = 10

#Create a resource group
New-AzResourceGroup -Name $rgName -Location $rglocation -Force

Create a Storage Account and a Blob Container

#Create a storage account 
#and if already exists then skip the account creation  

$storageAccount = Get-AzStorageAccount `
                            -ResourceGroupName $rgname `
                            -Name $storageaccountname `
                            -ErrorAction SilentlyContinue
        $context = $storageAccount.Context

if(!$storageAccount)  
{  
     $storageAccount = New-AzStorageAccount `
                         -ResourceGroupName $rgname `
                         -Name $storageaccountname `
                         -Location $rglocation `
                         -SkuName Standard_RAGRS `
                         -Kind StorageV2
      $context = $storageAccount.Context    
} 

$container = Get-AzStorageContainer `
                    -Name $containerName `
                   -Context $context `
                  -ErrorAction SilentlyContinue
if(!$container)  
{  
    $container = New-AzStorageContainer `
                      -Name $containerName `
                      -Context $context `
                      -Permission blob    
 } 

Looping for the variable $numberOfBatch times and upload files which are available at the location variable $fileslocation.

#Upload files and get the files count
#uploaded in the blob container.
if($container)
{
  $files = Get-ChildItem -Path $fileslocation
  for ($i = 1 ; $i -le $numberOfBatch; $i++)
  {
    write-host -ForegroundColor Yellow "Running Batch: " $i
    foreach ($file in $files)
    {
         $ext = (Split-Path -Path $file -Leaf).Split(".")[1];
         Set-AzStorageBlobContent `
         -File $file `
         -Container $containerName `
         -Blob $([guid]::NewGuid().ToString() + "." + $ext)  `
         -Context $context
     }
   }
$blob = Get-AzStorageBlob `
           -Container $containerName `
           -Context $context ` 
$totalfiles = $blob.Count                                                          
write-host -ForegroundColor Green "$totalfiles `
             "files are uploaded successfully"
}

Script Output

To test the behavior of the script, we could combine all the script blocks into one PowerShell (.ps1) file and run it using the Windows Command Prompt.

Use Connect-AzAccount command to connect Azure subscription

Script Execution
Execution In-Progress
Blob Container Output

If you want to upload only a set of files from a physical location, change the variable $numberOfBatch value to zero (0).

So, we have seen how we can upload files in bulk in a blob container using Azure PowerShell.

Happy Learning 🙂

2 thoughts on “Blob Storage – Bulk Files Upload Using Azure PowerShell

Add yours

  1. I am facing an error while running this script on on prem server in powershell. There are couple of other errors as well but starts what i got first.

    Get-AzStorageContainer : Could not get the storage context. Please pass in a storage context or set the current
    storage context.
    At line:1 char:14
    + $container = Get-AzStorageContainer
    + ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : CloseError: (:) [Get-AzStorageContainer], InvalidOperationException
    + FullyQualifiedErrorId : InvalidOperationException,Microsoft.WindowsAzure.Commands.Storage.Blob.Cmdlet.GetAzureSt
    orageContainerCommand

    1. Hope you have installed Azure Powershell module and before running the script you much have connected to Azure Subscription from commpt using Connect-AzAccount

Leave a Reply

Up ↑

%d bloggers like this: