PowerShell: Automatically Deploy an Azure Test or Development Environment as Code (IaC)

We often need to build testing, packaging or development environments in Azure. Often these are built in an ad-hoc way, manually. This is fine, if a little time consuming, but Azure makes it easy to create scripts that will automate the provisioning of these environments. In fact, automation should really be the preferred method of deployment as it ensures that environments can be deployed or re-deployed rapidly whilst maintaining build quality.

The script that I am going to describe here uses PowerShell Azure cmdlets to provide a very flexible method of deploying the Azure environment, and deploying as many fully configured VMs (desktops or servers) as you may need.

This script relies on the AZ PowerShell module, and this should be installed on any workstation prior to running the script by running the following command. The script uses functions, for code minimisation and to simplify code reuse.

Install-Module -Name Az

Azure Environment and Networking

All the variables required for the script are set at the beginning, and I have added comments in the script describing all of these. All output is sent to the console.

The ConfigureNetwork function creates a Virtual Network, a subnet and a Network Security Group (NSG) name. The NSG also has a rule enabled for RDP. You can add other rules if you need them. If you don’t need the networking configured simply comment out the ConfigureNetwork call later in the script.

function ConfigureNetwork {
    $virtualNetwork = New-AzVirtualNetwork -ResourceGroupName $RGName -Location $Location -Name $VNet -AddressPrefix 10.0.0.0/16
    $subnetConfig = Add-AzVirtualNetworkSubnetConfig -Name default -AddressPrefix 10.0.0.0/24 -VirtualNetwork $virtualNetwork
        
    $rule1 = New-AzNetworkSecurityRuleConfig -Name rdp-rule -Description "Allow RDP" -Access Allow -Protocol Tcp -Direction Inbound -Priority 100 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * -DestinationPortRange 3389
    $nsg = New-AzNetworkSecurityGroup -ResourceGroupName $RGName -Location $location -Name $NsgName -SecurityRules $rule1
        # $rule1, $rule2 etc
        If ($nsg.ProvisioningState -eq "Succeeded") {Write-Host "Network Security Group created successfully"}Else{Write-Host "*** Unable to create or configure Network Security Group! ***"}
    $Vnsc = Set-AzVirtualNetworkSubnetConfig -Name default -VirtualNetwork $virtualNetwork -AddressPrefix "10.0.1.0/24" -NetworkSecurityGroup $nsg
    $virtualNetwork | Set-AzVirtualNetwork >> null
        If ($virtualNetwork.ProvisioningState -eq "Succeeded") {Write-Host "Virtual Network created and associated with the Network Security Group successfully"}Else{Write-Host "*** Unable to create the Virtual Network, or associate it to the Network Security Group! ***"}
}

I wanted the script to support fully configured VMs, which can be configured using scripts and Azure’s Custom Script Extension functionality. This enables scripts to be run during, or after VM deployment. These scripts need storing somewhere, so I have included a CreateStorageAccount function.

function CreateStorageAccount {
    If ($StorAccRequired -eq $True)
        {
        $storageAccount = New-AzStorageAccount -ResourceGroupName $RGName -AccountName $StorAcc -Location uksouth -SkuName Standard_LRS
        $ctx = $storageAccount.Context
        $Container = New-AzStorageContainer -Name $ContainerName -Context $ctx -Permission Container
        If ($storageAccount.StorageAccountName -eq $StorAcc -and $Container.Name -eq $ContainerName) {Write-Host "Storage Account and container created successfully"}Else{Write-Host "*** Unable to create the Storage Account or container! ***"}    
        #$BlobUpload = Set-AzStorageBlobContent -File $BlobFilePath -Container $ContainerName -Blob $Blob -Context $ctx 
        Get-ChildItem -Path $ContainerScripts -File -Recurse | Set-AzStorageBlobContent -Container "data" -Context $ctx
        }
        Else
        {
            Write-Host "Creation of Storage Account and Storage Container not required"
        }   
}

This function creates a Storage Account and a Container (for holding the scripts). It then copies all files found in the $ContainerScripts variable up to the Container. Again, if this functionality is not required (e.g. you are storing your scripts in GitHub) you can comment out the CreateStorageAccount call later in the script.

Creating the VM/s

To provide the correct details for the image, the available SKUs need to be listed. This is done with the following command (don’t forget to enter the location that you want to use when creating the resource, as availability varies between regions).

Get-AzVMImageSku -Location "uksouth" -PublisherName "MicrosoftWindowsServer" -Offer "WindowsServer"

To identify available Windows 10 SKUs, use this command.

Get-AzVMImageSku -Location "uksouth" -PublisherName "MicrosoftWindowsDesktop" -Offer "Windows-10"

From this data we can put together the image name to be used for the VM. For example, We want to deploy the latest 20h2 Windows 10 image, so we would use the following string for the ImageName parameter.

MicrosoftWindowsDesktop:Windows-10:20h2-ent:latest

The CreateVMp function creates the VM, sets the local Admin User and Password, and creates a Public IP address.

function CreateVMp($VMName) {
    $PublicIpAddressName = $VMName + "-ip"

    $Params = @{
    ResourceGroupName = $RGName
    Name = $VMName
    Size = $VmSize
    Location = $Location
    VirtualNetworkName = $VNet
    SubnetName = "default"
    SecurityGroupName = $NsgName
    PublicIpAddressName = $PublicIpAddressName
    ImageName = $VmImage
    Credential = $VMCred
    }

    $VMCreate = New-AzVm @Params
    If ($VMCreate.ProvisioningState -eq "Succeeded") {Write-Host "Virtual Machine $VMName created successfully"}Else{Write-Host "*** Unable to create Virtual Machine $VMName! ***"}
}

Configuring the VM/s

Configuring VMs is done by means of a script, run on the newly provisioned VM. Obviously, a script allows you to do anything that you want. In my example the script is run from the Storage Account but copies media from a GitHub location and then installs Orca. You can use the same technique to do pretty much anything that you need to.

The script is run using the Custom Script Extension functionality in Azure. The function, RunVMConfig, can be run against any Azure Windows VM, it does not have to have been created using this script.

function RunVMConfig($VMName, $BlobFilePath, $Blob) {

    $Params = @{
    ResourceGroupName = $RGName
    VMName = $VMName
    Location = $Location
    FileUri = $BlobFilePath
    Run = $Blob
    Name = "ConfigureVM"
    }

    $VMConfigure = Set-AzVMCustomScriptExtension @Params
    If ($VMConfigure.IsSuccessStatusCode -eq $True) {Write-Host "Virtual Machine $VMName configured successfully"}Else{Write-Host "*** Unable to configure Virtual Machine $VMName! ***"}
}

The script that will be run is taken from the Storage Account Container, but it will also accept the RAW path to a GitHub repository. You would then not need to create the Storage Account.

Script Main Body

The main body of the script is actually quite short, due to the use of functions. The Resource Group is created, and then the ConfigureNetwork and CreateStorageContainer functions are called (if required, they can be commented out).

# Main Script

# Create Resource Group
$RG = New-AzResourceGroup -Name $RGName -Location $Location
If ($RG.ResourceGroupName -eq $RGName) {Write-Host "Resource Group created successfully"}Else{Write-Host "*** Unable to create Resource Group! ***"}

# Create VNet, NSG and rules (Comment out if not required)
ConfigureNetwork

# Create Storage Account and copy media (Comment out if not required)
CreateStorageAccount

# Build VM/s
$Count = 1
While ($Count -le $NumberOfVMs)
    {
    Write-Host "Creating and configuring $Count of $NumberofVMs VMs"
    $VM = $VmNamePrefix + $VmNumberStart
    CreateVMp "$VM"
    RunVMConfig "$VM" "https://packagingstoracc.blob.core.windows.net/data/VMConfig.ps1" "VMConfig.ps1"
    # Shutdown VM if $VmShutdown is true
    If ($VmShutdown)
        {
        $Stopvm = Stop-AzVM -ResourceGroupName $RGName -Name $VM -Force
        If ($RG.ResourceGroupName -eq $RGName) {Write-Host "VM $VM shutdown successfully"}Else{Write-Host "*** Unable to shutdown VM $VM! ***"}
        }
    $Count++
    $VmNumberStart++
    }

In order to provision multiple VMs, a While loop is used, to rerun code until the required number of VMs are provisioned. The VM name is created by concatenating variables, and the name is passed, first to the CreateVMp function, and then to the RunVMConfig function. A variable at the beginning of the script is used to check if the newly provisioned VM should be shutdown or not. This is very useful as an aid to manage costs.

The full script is located here.

https://raw.githubusercontent.com/HigginsonConsultancy/Scripts/master/CreateConfigureVMs.ps1

The VM config script I used is written to output into the Windows Event Log for ease of fault finding. It is located here.

https://raw.githubusercontent.com/HigginsonConsultancy/Scripts/master/VMConfig.ps1

The script outputs to the console and successful completion shouls look something like this.

Resource Group created successfully
Network Security Group created successfully
Virtual Network created and associated with the Network Security Group successfully
Storage Account and container created successfully
Account              SubscriptionName          TenantId                             Environment
-------              ----------------          --------                             -----------
graham@*********.*** Microsoft Partner Network ********-****-****-****-********73ab AzureCloud 

ICloudBlob                         : Microsoft.Azure.Storage.Blob.CloudBlockBlob
BlobType                           : BlockBlob
Length                             : 2034
IsDeleted                          : False
BlobClient                         : Azure.Storage.Blobs.BlobClient
BlobProperties                     : Azure.Storage.Blobs.Models.BlobProperties
RemainingDaysBeforePermanentDelete : 
ContentType                        : application/octet-stream
LastModified                       : 11/30/2020 10:03:49 AM +00:00
SnapshotTime                       : 
ContinuationToken                  : 
Context                            : Microsoft.WindowsAzure.Commands.Storage.AzureStorageContext
Name                               : VMConfig.ps1

Creating and configuring 1 of 1 VMs
Virtual Machine PVMMSI500 created successfully
Virtual Machine PVMMSI500 configured successfully
VM PVMMSI500 shutdown successfully

Troubleshooting a Custom Script Extension as part of an ARM Template Deployment

I recently decided to test the use of a Custom Script Extension, as I wanted to deploy a VM with client software installed. I created a Powershell script that downloaded, extracted and installed the MSI tool Orca. If I could get this to deploy and run successfully it would give me a basis to install any software I needed, plus perform any other configuration I might need.

I added lots of error logging to the script so that I would have plenty of information when it came to troubleshooting. The script I used is located here.

Create ARM Template

I took an existing ARM template that I used to create a Windows 10 VM, renamed it and added the Custom Script Extension code. The template is here and the parameter file I used is here.

The Custom Script Extension code itself is shown here.

To deploy the template I used the following Powershell commands (after installing the Az module).

Connect-AzAccount
New-AzResourceGroupDeployment -Name TestDeployment -ResourceGroupName HiggCon -TemplateFile C:\Dev\VM\VMcse.template.json -TemplateParameterFile C:\Dev\VM\VM.parameters.json

And that is where the problems started…

There is a part of me that always hopes that something like this will work first time, but I have to be honest, it hardly ever does! So the error I got back is below.

The deployment failed

I looked back through my Custom Script Extension code (which I had got, and edited, from the Microsoft documentation), and found the following line that I hadn’t noticed.

"name": "virtualMachineName/config-app",

Hmm, that didn’t look right, so I changed it to this.

"name": "[Concat(parameters('virtualMachineName'),'/config-app')]",

Hurrah, the deployment was then successful…but.

Looking in Azure, I could see that the VM appeared to have been created successfully. Unfortunately, after logging into the VM, Orca wasn’t installed.

I looked in the Eventlog. There were no entries from my script so it looked like the script hadn’t run, but why?

First I looked in the C:\WindowsAzure\Logs\WaAppAgent.Log file. It looked like some Custom ScriptExtensions had run successfully, but they looked Azure specific, with no hint that my script had run. After some digging I managed to find the VMConfig.ps1 file, that was my script, in C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.8\Downloads\0. So that showed that the script had been copied down correctly. I tried running the script manually…and got the “Running Scripts is disabled” message. Doh!

I changed my command to execute to the following, deleted the VM and reran the deployment.

"commandToExecute": "powershell -ExecutionPolicy Unrestricted -File VMConfig.ps1"

Success! My Deployment worked AND my script worked. Orca was installed successfully. My script had also written successfully to the Event Viewer.

I looked in C:\WindowsAzure\Logs\WaAppAgent.Log and found these entries near the bottom.

Its not exactly clear, but I think it is related to my script as it wasn’t there previously. The middle line includes this “ExtensionName: ”’ seqNo ‘0’ reached terminal state”. My script was downloaded to a directory named 0, so my thinking is that they are related. The end of the last line stating “completed with status: RunToCompletion” sounds right too. Unfortunately it is difficult to know definitely.

Summary

So what have we learnt? Well first of all, I think that when writing a script to deploy using Custom Script Extension, it makes a lot of sense to include as much logging as possible, and to test if fully before trying to deploy, as it will be difficult to determine what is wrong otherwise.

The deployment can really be split into two sections, Deployment and Execution.

Deployment

It is easy to misconfigure the ARM template, as I found to my cost. I used Visual Studio Code to edit the template as it supports ARM templates and identifies error syntax. Make sure all errors are resolved before trying to deploy. The “New-AzResourceGroupDeployment” Powershell cmdlet will tell you if the deployment was successful or not. The error messages aren’t always easy to interpret though.

Execution

Once successfully deployed, if you don’t get the result you are expecting check the

C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.8\Downloads

directory for a subdirectory holding your script. That way you know that it is actually there. Then trying running the script and see what error messages you get.

I also looked in

C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.8\Status

and found a “0.status” file that gave me a little more information on my script, and a success status code.

In

C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.8\RuntimeSettings

I found a file named “0”, which contained, amongst other things, the URL the script was download from and the command to execute. So be aware if you are working in a secure environment, or are using a public repository.