Key Vault Extension

Rotating server certificates in a large estate has always been a administration hassle, so let this key vault extension take the heavy lifting for both Azure and Azure Arc-enabled VMs.

Introduction

Another nice extension is the Key Vault Extension. This will watch a certificate uploaded into a key vault and then pull it down to the server. Therefore you can rotate your certificates and allow the extensions to do all of the hard work.

On Windows server it will be installed into the selected certificate store.

For linux it is downloaded to a directory and you then install. (This last stage should be scripted.). As this is the slightly more complex version then we will work on the linux VMs for this one.

This is a guided lab rather than a challenge.

Create a key vault

Get a uniq value

This will help ensure the FQDNs are unique.

uniq=$(az account show --query id --output tsv | cut -f1 -d-)

Create the key vault

kv=arc-pilot-$uniq
az keyvault create --name $kv --retention-days 7 --resource-group arc_pilot --location uksouth

Create a self signed certificate

The certificate will be used in the Key Vault Extension lab.

az keyvault certificate create --name self-signed-cert --vault-name $kv --policy "$(az keyvault certificate get-default-policy)"

Access policy

For the Key Vault Extension to work, you need to ensure that the managed identity has access to get the secret. (Remember that certificates are uploaded into the certificate, key and secret areas of a key vault. The certificate part is just the public part, whereas the secret includes everything.)

List the managed identities

Display a table of the Azure Arc-enabled servers and their managed identities.

az connectedmachine list --resource-group arc_pilot --query "[].{name:name, identity:identity.principalId}" --output table

Create a variable with the objectIds for the managed identities

managedIdentityIds=$(az connectedmachine list --resource-group arc_pilot --query "[].identity.principalId" --output tsv)

Get the key vault name

kv=$(az keyvault list --resource-group arc_pilot --query [0].name --output tsv)

Create the access policies
```
for id in $managedIdentityIds; do az keyvault set-policy --name $kv --secret-permissions list get --resource-group arc_pilot --object-id $id; done
```
Note that you could add the managed identities into a security group and then use the objectId for the group as a single access policy or RBAC role assignment. This is a more elegant approach when working at scale.

Key Vault Extension

Set the variables

kv=$(az keyvault list --resource-group arc_pilot --query [0].name --output tsv)
cert="self-signed-cert"

Create the Key Vault Extension

az connectedmachine extension create --name KeyVaultForLinux --publisher Microsoft.Azure.KeyVault --type KeyVaultForLinux --machine-name ubuntu-03 --resource-group arc_pilot --settings "{\"secretsManagementSettings\":{\"pollingIntervalInS\":\"60\",\"observedCertificates\":[\"https://$kv.vault.azure.net/secrets/$cert\"]},\"authenticationSettings\":{\"msiEndpoint\":\"http://localhost:40342/metadata/identity\"}}"

The polling interval has been set to 60 seconds rather than the default 3600, or hourly.

Note the quoted JSON string. These can be a syntactical nightmare, so the Creating quoted strings section at the end of this page has some useful hints.

Repeat

Repeat the last step for the other linux VMs.

Stretch targets (optional)

Create the Key Vault Extension for the same certificate on the Windows servers

Use the CA store.
Update the certificate in the key vault

Certificates

The extension will poll the key vault secret every minute and will download updates to /var/lib/waagent/Microsoft.Azure.KeyVault.Store/ in PEM format. This directory is only accessible to root.

SSH to the server

Set the variables

kv="<myKeyVaultName>"
cert="self-signed-cert"

Once the extension creation has succeeded then you can check the folder.
```
sudo ls /var/lib/waagent/Microsoft.Azure.KeyVault.Store/
sudo cat /var/lib/waagent/Microsoft.Azure.KeyVault.Store/$kv.$cert
```
Note the secret’s version is included in the full filename. The short form is a symbolic link to the latest of these files.

On Windows it will add to the specified certificateStoreName.

Certificate Installation

You would usually have a cronjob to check for new certificates in that location and then run these install steps. We’ll do it manually.

Set pem variable

You should have the kv and cert variables from the previous step.
```
pem=/var/lib/waagent/Microsoft.Azure.KeyVault.Store/$kv.$cert
```

Convert from PEM to DER

sudo openssl x509 -outform der -in $pem -out /usr/local/share/ca-certificates/$cert.crt

Update CA certificates
```
sudo update-ca-certificates
```

Success criteria

Screen share with your proctor to prove:

one of your linux VMs has the certificate

 openssl x509 -in  /usr/local/share/ca-certificates/$cert.crt -inform der -noout -text

one of your Windows VMs has the certificate in the store (stretch)

Troubleshooting

The extension manager writes out to /var/lib/GuestConfig/ext_mgr_logs.

The extension agents write out to the /var/lib/waagent/ area on linux.

We’ll see the Windows equivalents for the Custom Script Extension in the next lab.

Creating JSON strings

This section is purely for info.

If you are having difficulty creating JSON strings then first create a JSON file in vscode to check it is syntactically valid.

settings.json

{
  "secretsManagementSettings": {
    "pollingIntervalInS": "3600",
    "certificateStoreLocation": "/var/lib/waagent/Microsoft.Azure.KeyVault.Store/",
    "certificateStoreName": "ignored",
    "observedCertificates": [
      "https://$kv.vault.azure.net/secrets/$cert"
    ]
  },
  "authenticationSettings": {
    "msiEndpoint": "http://localhost:40342/metadata/identity"
  }
}

The default certificateStoreLocation has been included for completeness. For Windows just specify the machine name.

Quoted string

If your JSON includes bash variables then you can convert it to a quoted JSON string using jq.

jq @json settings.json

Output:

"{\"secretsManagementSettings\":{\"pollingIntervalInS\":\"3600\",\"certificateStoreName\":\"ignored\",\"certificateStoreLocation\":\"/var/lib/waagent/Microsoft.Azure.KeyVault.Store/\",\"observedCertificates\":[\"https://$kv.vault.azure.net/secrets/$cert\"]},\"authenticationSettings\":{\"msiEndpoint\":\"http://localhost:40342/metadata/identity\"}}"

Single quoted string

If all of the values are hardcoded then you can just minify it and then enclose in single quotes. This will be a little more readable.

jq -Mc . settings.json

Output:

{"secretsManagementSettings":{"pollingIntervalInS":"3600","observedCertificates":["https://$kv.vault.azure.net/secrets/$cert"]},"authenticationSettings":{"msiEndpoint":"http://localhost:40342/metadata/identity"}}

Surround with single quotes.

'{"secretsManagementSettings":{"pollingIntervalInS":"3600","observedCertificates":["https://$kv.vault.azure.net/secrets/$cert"]},"authenticationSettings":{"msiEndpoint":"http://localhost:40342/metadata/identity"}}'

JSON file

If the file will be present then keep it simple and just specify the filename.

az connectedmachine extension create --name KeyVaultForLinux --publisher Microsoft.Azure.KeyVault --type KeyVaultForLinux --machine-name ubuntu-02 --resource-group arc_pilot --settings settings.json

Resources

Next we will use the Custom Script Extension to run PowerShell and bash scripts on the connected servers.

Table of Contents

Key Vault Extension

Introduction

Create a key vault

Access policy

Key Vault Extension

Stretch targets (optional)

Certificates

Certificate Installation

Success criteria

Troubleshooting

Creating JSON strings

Quoted string

Single quoted string

JSON file

Resources

Next