Using azcopy in GitHub actions with federated credentials

Learn how to seamlessly copy files from GitHub Actions to Azure Blob Storage without providing secrets using azcopy.

Using azcopy in GitHub actions with federated credentials

I recently needed to copy files that were created in a GitHub action to Azure Blob Storage

The script should run seamlessly locally and on GitHub actions, without having to provide any secrets, I ran into a few issues so hopefully this will be useful to someone.

To do this you first need to create:

  • a resource group
  • a managed identity (or service principal but we'll be using managed identity in this blog)
  • a federated credential
  • a storage account
  • permissions for the managed identity to access the storage account

Terraform example:

locals {
  org        = "your-github-org" # replace-me
  repository = "your-repository-name" # replace-me
}

resource "azurerm_resource_group" "example" {
  name     = "example"
  location = "West Europe"
}

resource "azurerm_user_assigned_identity" "example" {
  location            = azurerm_resource_group.example.location
  name                = "example"
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_federated_identity_credential" "example" {
  name                = "${local.repository}-main"
  resource_group_name = azurerm_resource_group.example.name
  audience            = ["api://AzureADTokenExchange"]
  
  # this is the default issuer, enterprise customers may customise it
  # see https://docs.github.com/en/enterprise-cloud@latest/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect#customizing-the-issuer-value-for-an-enterprise
  issuer              = "https://token.actions.githubusercontent.com"
  
  parent_id           = azurerm_user_assigned_identity.example.id
  subject             = "repo:${local.org}/${local.repository}:ref:refs/heads/main"
}

resource "azurerm_storage_account" "example" {
  name                     = "storageaccountname"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_storage_container" "example" {
  name                  = "example"
  storage_account_name  = azurerm_storage_account.example.name
  container_access_type = "private"
}

resource "azurerm_role_assignment" "example" {
  principal_id = azurerm_user_assigned_identity.example.principal_id
  scope        = azurerm_storage_account.example.id

  role_definition_name = "Storage Blob Data Contributor"
}

main.tf

Azure CLI example:

az group create --name example --location westus

PRINCIPAL_ID=$(az identity create \
  --name example \
  --resource-group example \
  --query principalId \
  -o tsv)

ORG="your-github-org" # replace-me
REPO="your-github-repo" # replace-me
export AZURE_STORAGE_ACCOUNT="replace-me"

az identity federated-credential create \
  --name "${REPO}-main" \
  --identity-name example \
  --resource-group example \
  --issuer "https://token.actions.githubusercontent.com" \
  --subject "repo:$ORG/$REPO:ref:refs/heads/main" \
  --audiences myAudiences

ID=$(az storage account create \
  --resource-group example \
  --name ${AZURE_STORAGE_ACCOUNT} \
  --sku Standard_LRS \
  --kind StorageV2 \
  --query id \
  -o tsv)

az storage container create -n example

az role assignment create --assignee "${PRINCIPAL_ID}" \
--role "Storage Blob Data Contributor" \
--scope "${ID}"

create-federated-credentials.sh

Then create three secrets in the GitHub repository settings, see GitHub's instructions.

The secrets you need are:

  • AZURE_CLIENT_ID
  • AZURE_TENANT_ID
  • AZURE_SUBSCRIPTION_ID

Now you're ready to create the GitHub action:

name: Copy files
on:
  push:
    branches: ["main"]

permissions:
  id-token: write # Require write permission to Fetch an OIDC token.

jobs:
  copy:
    runs-on: ubuntu-latest
    steps:
      # Login to Azure using federated credential
      # No secret is required due to OIDC authentication
      - name: Azure CLI Login
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Install azcopy
        run: |
          sudo apt-get update
          sudo apt-get install -y wget

          # when ubuntu-latest changes this needs updating from 22.04
          wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb
          sudo dpkg -i packages-microsoft-prod.deb
          sudo apt-get update
          sudo apt-get install -y azcopy
          rm -f packages-microsoft-prod.deb
      - name: Create file
        run: |
          echo "Hello, world!" > hello.txt
      - name: Copy file to Azure Blob Storage
        env:
          AZURE_STORAGE_ACCOUNT: your-storage-account-name
          AZURE_CONTAINER_NAME: example
          AZCOPY_AUTO_LOGIN_TYPE: AZCLI
          AZCOPY_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
        run: |
          azcopy sync --compare-hash=md5 \
            --delete-destination=true \
            --include-pattern="*.txt"  \
            "." "https://$AZURE_STORAGE_ACCOUNT.blob.core.windows.net/$AZURE_CONTAINER_NAME/"

.github/workflows/copy.yml

There are two critical environment variables in the Copy file to Azure Blob Storage step (which are currently not well documented):

  • AZCOPY_AUTO_LOGIN_TYPE=AZCLI
    • This instructs azcopy to use your existing Azure CLI login without having to run azcopy login
  • AZCOPY_TENANT_ID=${{ secrets.AZURE_TENANT_ID }}
    • There is a bug in azcopy's federated identity functionality where it tries to authenticate with the Microsoft tenant if no tenant is provided, even though the information should be provided by the Azure CLI login, this is the recommended workaround.

Push the GitHub action to the main branch and run it, the workflow should pass and the output should look roughly like:

Run azcopy sync --compare-hash=md5 \
INFO: Login with AzCliCreds succeeded
INFO: Authenticating to destination using Azure AD
INFO: XAttr hash storage mode is selected. This assumes all files indexed on the source are on filesystem(s) that support user_xattr.
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job 6ece7230-2b26-9249-5738-c15551e9c5e3 has started
Log file is located at: /home/runner/.azcopy/6ece7230-2b26-9249-5738-c15551e9c5e3.log

INFO: One or more hash storage operations (read/write) have failed. Check the scanning log for details.

100.0 %, 1 Done, 0 Failed, 0 Pending, 1 Total, 2-sec Throughput (Mb/s): 0.0001

Job 6ece7230-2b26-9249-5738-c15551e9c5e3 Summary
Files Scanned at Source: 1
Files Scanned at Destination: 0
Elapsed Time (Minutes): 0.0334
Number of Copy Transfers for Files: 1
Number of Copy Transfers for Folder Properties: 0 
Total Number of Copy Transfers: 1
Number of Copy Transfers Completed: 1
Number of Copy Transfers Failed: 0
Number of Deletions at Destination: 0
Total Number of Bytes Transferred: 14
Total Number of Bytes Enumerated: 14
Final Job Status: Completed

If you check the storage account you'll find the file has been uploaded:

Image showing result of uploading the file

This blog post has covered quite a bit, at the end of this you should have:

  • Created a managed identity with a federated credential, that has a trust relationship with a GitHub repository on its main branch allowing that branch to login with that managed identity without needing a password or certificate.
  • That identity has RBAC permissions on a storage account and GitHub actions using that identity is able to upload a file with azcopy