Domain of Thrones: Part II

Written by Nico Shyne & Josh Prager

Introduction Part II

In the first installment of “Domain of Thrones,” we meticulously explored an array of six distinctive domain persistence techniques:

• Credential Theft on the Domain Controller (DC)
• NTDS Access
• DCSync
• Golden Ticket
• Diamond Ticket
• Active Directory Certificate Services (AD CS)

These adversarial methods facilitate an elevated level of access to the targeted domains, thereby challenging the defenders with a considerably strenuous eviction and remediation process.

Our initial post ended by offering high-level recommendations on determining the scope of a compromise when evidence confirms the establishment of domain persistence. Additionally, we covered initial recommendations for promoting Windows servers to domain controllers to replace the compromised domain controllers.

Looking ahead to “Domain of Thrones: Part II,” we will provide post-compromise guidance for the rotation of domain secrets. This sequel will also dive into Windows Security event auditing that provides detection capabilities surrounding domain persistence behavior. Such auditing is crucial to reduce adversaries’ time in the domain and strengthen defenses against further compromising activities.

Changing the Guard Post-Siege

Following the reconstruction strategies delineated in “Domain of Thrones: Part I” for compromised domain controllers, defenders must also focus on the rotating of the domain’s secret material. Defenders must rotate the secrets of any domain controller identified within the scope of the compromise. Additionally, consider children-domain controllers within scope if the in-scope domain controller is a forest root, and one or more child domains exist.

As per Microsoft’s remediation guidance, ensure the below steps are performed from a “trusted device, built from a clean source.” Microsoft emphasizes the first step to avoid accidentally providing an embedded adversary with a privileged workstation and account during the remediation efforts. Additionally, defenders should create and utilize brand-new dedicated accounts for remediation activity.

Defenders can use the Active Directory PowerShell module to identify accounts modified (whenChanged property)before the last date of malicious activity. For instance, if an incident report shows the latest activity date, accounts modified before this date should have their credentials rotated. This broad rotation ensures that all accounts linked to domain persistence are updated, even without direct evidence of compromise by adversaries. The actual credential rotation method depends on the organization’s identity manager. However, for those using on-prem Active Directory or Azure Hybrid, specific commands can help identify accounts requiring rotation (Figure 1).

$date = "09/18/2023"
$users_to_be_rotated = Get-ADObject | Where-Object {$_.ObjectClass -eq 'user' -and $_.whenchanged -lt $date}

The accounts and objects that will need rotation are to include the following:

• Machine accounts
• User accounts
• Service accounts
  – Per domain KRBTGT account (the authors will discuss specific rotation steps in more detail below)
• Trust keys and objects related to trust of all other domains
• Group-managed service accounts
• Key distribution service root keys

Rotating Machine Accounts

WARNING: Resetting the client’s machine account password locally will disconnect the client from the domain until the domain trust is re-established.

By default, machine account passwords do not expire in Active Directory, and they are typically exempt from the active domain password policy. We recommend setting the maximum machine account password age to 30 days, with the client initiating the password change. The client’s netlogon service handles machine password updates via the Workstation Scavenger Thread. Create a script to rotate the machine account passwords and execute it remotely on each client identified within the scope of the compromise, using the Reset-ComputerMachinePassword cmdlet to reset the client machine account password. Then, rejoin the client to the domain, where the client provides the new password to Active Directory. The example script below can remotely reset the machine account password and re-establish trust with Active Directory (see Figure 2).

# ResetAndTestComputer.ps1 
#PowerShell Script to Reset Computer Machine Password and Test Computer Secure Channel
# Parameters
param(
    [string]$DCname,
    [string]$DomainUser
)
# Function to reset computer machine password
function Reset-ComputerPassword {
    param(
        [string]$server,
        [string]$credential
    )
    # Reset the computer machine password
    Reset-ComputerMachinePassword -Server $server -Credential $credential
}
# Function to test and repair computer secure channel
function Test-RepairComputerSecureChannel {
    param(
        [string]$credential
    )   
    # Test and repair the computer's secure channel
    Test-ComputerSecureChannel -Repair -Credential $credential
}
# Execute the functions with provided parameters
Reset-ComputerPassword -server $DCname -credential $DomainUser
Test-RepairComputerSecureChannel -credential $DomainUser
Write-Output "Operations completed successfully!"

The below command can execute the above example script on clients with PSRemoting enabled.

Invoke-Command -ComputerName RemoteComputerName 
-FilePath C:PathToResetAndTestComputer.ps1 
-ArgumentList "DCname", "DOMAINUser" -Credential DOMAINUser

Rotating User Accounts

Adversaries can establish domain persistence through malicious code execution and privileged account compromise. In recent intrusions, adversaries have used the administrative permissions of compromised user accounts to establish persistent access to on-premises and cloud environments.

You can complete user account rotation using the Active Directory PowerShell module and the Set-ADUser cmdlet. Defenders must identify all account types that allow interactive logon and set the “User must change password at next login” flag by adding the ChangePasswordAtLogon $true argument for each user. Defenders should notify the organization stating that a password change is required. After several days, defenders must lock the accounts that have not had passwords rotated.

Some accounts will likely have the passwordNeverExpires or cannotChangePassword property set to true, in which case, the Set-ADUser cmdlet for setting changePasswordAtLogon will not work, and their passwords will need to be reset manually via Active Directory Users and Computers or by setting passwordNeverExpires or cannotChangePassword to false.

Microsoft provides additional user rotation guidance focused on MFA-enabled user accounts for Azure hybrid environments. After the user accounts have been rotated, the Revoke-AzureADUserAllRefreshToken AzureAD PowerShell cmdlet should be executed to revoke the refresh tokens of potentially compromised user accounts. This additional measure revokes all access for user accounts, including the compromised user accounts, to Azure Active Directory resources.

Rotating Service Accounts

Rotating the passwords of service accounts can pose a significant challenge, especially for organizations heavily reliant on legacy applications. In large organizations, legacy services often operate under the expectation of static service account passwords, creating a persistent vulnerability that is challenging to mitigate.

Defenders (or internal offensive operators) can proactively identify vulnerable service accounts susceptible to cracking by using open-source tools to retrieve the NTLM hash of service accounts with an associated Service Principal Name (SPN). Defenders can then attempt to crack the service account NTLM passwords with hashcat. When defenders know the crackable accounts, they can set additional auditing on these vulnerable accounts. Defenders can promptly identify and respond to malicious activities by closely monitoring ticket requests and activities associated with these service accounts.

Adopt a strategic approach for services that would allow for the rotation of the service account password. During incident response, separate the service accounts suspected of being compromised based on telemetry data. Involving security engineers and system administrators in the review process is essential. Their combined expertise will help make informed decisions about which compromised service accounts can be immediately rotated and which ones need further testing. Security engineers should also ensure that service account passwords meet robust security standards, recommending a minimum length of 30 characters to enhance security resilience.

Rotating the KRBTGT Account

The domain Kerberos service account, KRBTGT, is part of the domain’s Kerberos distribution center, which handles all Kerberos ticket requests. As mentioned above, adversaries who have compromised the KRBTGT service account can generate Golden Tickets, which enables an adversary to generate their own TGT using the KRBTGT account password hash. This generated TGT can be modified to have an extremely long lifetime of up to 10 years. Due to the trust that each client has in the KRBTGT password hash, preventing the abuse of a Golden Ticket is extremely difficult. Rotating this particular service account is critical to remediation and recovery operations. The recommendations for rotation will often state to rotate the password of this account twice. However, the complexity of rotating the KRBTGT account password is more nuanced than simply doing it twice (Figure 3).

Many recommendations suggest changing the KRBTGT account password twice to address password history concerns. An account’s password history stores both the current password and the last one (n-1). When rotating the KRBTGT account password, the first password change must be replicated to all secondary domain controllers. To validate that the replication has been completed, security engineers can run the replication admin “show replication” command from a Windows terminal:
repadmin.exe /showrepl dc2 (Figure 4).

Once replication is complete, the second password rotation can occur to ensure that the previously compromised KRBTGT account password hash is no longer lingering in the environment as a potential attack vector. Additionally, this process will need to be conducted on every domain within the scope of the compromise. Microsoft provides a script, New-KrbtgtKeys.ps1, that will both change the KRBTGT account password and validate that replication was successful before the second change with repadmin.exe.

Rotating Trust Realm Objects

In a forest with multiple Active Directory domains connected by trust relationships, compromising a Tier 0 account, like a domain admin or the KRBTGT service account, affects all domain controllers within the domain due to their trust relationship with the compromised Tier 0 principal. However, the adversary will not be able to request resources within another domain within the same forest because creating service tickets requires the target service’s password data, and the compromised domain only has the credential information for its own principals. This logical separation is called a Kerberos Realm. To provide a method by which users within a child domain can request resources in the ROOT domain, Microsoft designed a secure password to be shared between the multiple domains in the same forest. This trust password is then converted into a cryptographic key called the Inter-Realm Trust Key. This collection of Kerberos trust between multiple domains is called a Kerberos Trust Realm. When a user requests a resource in another domain, the remote domain’s Key Distribution Center (KDC) returns a signed TGT. However, unlike a typical TGT signed by the KRBTGT account of the domain, the Trust Realm TGT is signed by the inter-realm key.

Adversaries can abuse Kerberos Trust Realms to regain privileged access in an environment that has begun the eviction process and rotated the KRBTGT account. Once the KRBTGT account is rotated, the adversary can no longer generate Golden Tickets because the password hash has changed. However, if the adversary retrieves credentials from the domain controller, the adversary can generate forged inter-realm trust keys, enabling the adversary to regain access utilizing enterprise admin accounts. Due to the inherent trust between domains in the forest and the trust password, an enterprise admin in one domain is part of the Administrators group in every other domain in the forest.

From a remediation perspective, the inter-realm trust password automatically changes every 30 days, according to section 6.1.6.9.6.1 of the Active Directory Technical Specification. Still, the last trust password will be retained in the SAM database until the following change. Defenders must manually initiate the trust password update and wait for it to replicate to all domains before rotating it a second time. Microsoft provides guidance on rotating trust keys with netdom.exe. Incident responders must ensure they rotate the password on the incoming trust side and use the same password on the outgoing trust side. This procedure ensures that the domain controllers do not replicate the compromised domain controller’s password, which may be compromised.

Defenders should rotate the inter-realm trust password on the recovered domain controller from the compromise first. The command using netdom.exe will differ slightly depending on whether the affected domain was the parent or child domain in the trust realm. Netdom trust /resetOneSide will write a new trust password on the targeted domain (child or parent) that the command was executed on. The /PasswordO:* command will prompt the CLI to have the administrator input the password. The /PasswordT command should use the same password on either side of the trust relationship.

In the below screenshots, eagle.banking will represent the parent domain and island.banking will mean the child domain.

To reset the inter-realm trust password with netdom.exe on a parent domain (Figure 5), defenders can execute the following syntax:

netdom trust <PARENT_DOMAIN_NAME> /domain:<CHILD_DOMAIN_NAME> 
/resetOneSide /passwordT:<PASSWORD> /userO:<ADMINISTRATOR_ACCOUNT> 
/passwordO:*

To reset the inter-realm trust password with netdom.exe on a child domain (Figure 6), defenders can execute the following syntax:

netdom trust <CHILD_DOMAIN_NAME> /domain:<PARENT_DOMAIN_NAME> 
/resetOneSide /passwordT:<PASSWORD> /userO:<ADMINISTRATOR_ACCOUNT> 
/passwordO:*

The above commands must only be executed once per domain because netdom.exe will automatically reset the password twice.

Rotating Group Managed Service Accounts

The Microsoft Key Distribution Service (KDS) root key on the domain controller is used to generate group-managed service account (gMSA) passwords. Domain controllers require the root key to create these gMSA passwords. For gMSA specifically, defenders can conduct the following steps to rotate the gMSA passwords:

1. Delete the old root key
2. Create a new root key
3. Restart the KDS on all domain controllers once the key is recreated.
   – The restart will ensure the previous key is not used after deletion due to caching of the key.

Microsoft introduced Managed Service Accounts (MSA) in Windows Server 2008, which precedes gMSAs. To rotate Standalone MSAs, a different process is required than the one used for gMSA. To rotate passwords for standalone MSAs, use the Active Directory PowerShell module and the Reset-ADServiceAccountPassword cmdlet on local computers. Execute the cmdlet on the standalone client where you have installed the MSA.

Remediation of Certificate Authorities

The industry standard for Public-Key Infrastructure (PKI) implementation is to establish the first Certificate Authority (CA) as the root, accompanied by the generation of the root certificate and subsequent authorization of intermediate CAs. This practice allows organizations to efficiently distribute and revoke certificates without the direct involvement of the root CA. Industry best practices recommend keeping the root CA secured and offline to mitigate potential attack vectors. This strategic isolation acts as a safeguard, eliminating avenues of compromise.

When the root CA is compromised, a rigorous process of revocation and reauthorization of all certificates issued by intermediate CAs becomes imperative. This revocation process starts with logging into the root CA as a Certification Authority Administrator or Certificate Manager and revoking all subordinate CA certificates. The certificates are then marked as revoked and moved to the Revoked Certificates folder. The revoked subordinate CA certificates will appear on the certificate revocation list (CRL) the next time it is published. An adversary with unidentified persistent access to the root CA could un-revoke any certificates listed as “Certificate Hold” in the revocation reason code. Defenders should execute the following certutil.exe command for the certificates that contain a revocation reason code of “Certificate Hold” to label the revoked certificate as having a new reason code of “CA Compromise.”

certutil -revokeCertificateSerialNumber2

After revoking the subordinate CA certificates, defenders should manually publish the CRL on the root CA with the certutil -crl command. This step sets the new CRL as the official list of revoked subordinate CA certificates until the instantiation of a new root CA. Additionally, clients will cache CRLs locally to speed up the certificate verification process. This means that clients will not normally download the newly published CRL until the locally cached CRL has expired. To force clients to download the newly published CRL, defenders will need to invalidate the current cache of CRL on each client. The following command will immediately invalidate all locally cached CRL entries on the client:

certutil -setreg chainChainCacheResyncFiletime @now

Once the new CRL is manually published and each client has purged its cache of previously trusted certificates, the original root CA must be decommissioned. Microsoft recommends saving any data, logs, or databases required for retention and sequentially reformatting the root CA’s hard drives. Once the root CA is retired, generate a new key pair on a newly implemented root CA and reissue subordinate CA certificates. Microsoft notes that the newly generated CA keys should use a larger key size and a currently uncompromised algorithm.

Recovering from a compromise of the root CA is generally a meticulous and exhaustive process that significantly impedes operational workflows and production. Organizations should proactively strategize and implement measures to isolate the root CA if they haven’t already, strengthen security protocols, and reduce potential threats.

Recommended Logging for Domain Compromises

We’ve reached the end of the series. The dragons burned down the city, the invaders breached the wall, but we’ve started rebuilding. Once an organization recovers from a domain compromise, information security management often asks the question: “How can we be better prepared for a future attempt?” The answer relies upon implementing scalable auditing and reducing adversary dwell time, preventing them from reaching a level that necessitates a complete environment rotation.

Microsoft Defender for Identity (MDI) provides security event IDs related to many attack techniques discussed in our previous blog, “Domain of Thrones: Part 1.” The following event IDs focus on persistence and privilege escalation related to Kerberos abuse:

• Event ID: 2009 (Encryption Downgrade)
  – This event filters for TGS_REQ (service requests) where the encryption type is a less secure cipher than previously requested service requests over five days.
• Event ID: 2027 (Nonexistent Account)
  – This event filters for TGTs (ticket-granting tickets) provided to resources where the account related to the TGT is nonexistent.
• Event ID: 2032 (Ticket Anomaly)
  – This event analyzes TGTs for “unique characteristics” corresponding to forged tickets. While we don’t know precisely the characteristics of the forged tickets that cause this alert to fire, open-source research indicates that the ticket anomalies are likely related to ticket lifetimes, such as the MaxTicketAge parameter of a TGT. The expected signal to fire for MaxTicketAge representing an anomalous lifetime would be Event ID: 2022 (Ticket Time Anomaly). However, Synack’s tests prove that event ID 2032 Ticket Anomaly will fire instead. Additionally, this alert likely considers the published anomalies in TGTs that offensive tools generate when forging tickets.
• Event ID: 2040 (Role-Based Constrained Delegation (RBCD) Ticket Anomaly)
  – This event filters for TGTs created by setting the RBCD permissions via the KRBTGT account for an account with an SPN.

MDI retrieves logs from the MDI sensor located on the domain controllers or the AD FS servers and parses the network traffic, windows security event traces, and Active Directory data. MDI then analyzes these logs and builds baseline profiles. When deviations from the baseline profiles occur, and these deviations match specific filtering, defenders will retrieve an alert from Microsoft 365 Defender. Enabling these security alerts is a good start, but heavy baselining will be required to identify attacks of the gemstone variety. Additionally, many of these alerts have trivial bypasses for an advanced adversary. Defenders should instead try adding context to existing technique-focused alerting rather than just alerting ticket anomalies alone.

Many organizations attempt to aggregate only the pertinent telemetry to decrease adversary dwell time and ingestion rates. To facilitate a more straightforward selection process, Microsoft has thoughtfully curated a comprehensive chart. This chart outlines the necessary audit policies and their matching event IDs (see Figure 7). It’s a valuable tool for organizations to strengthen their security measures.

Furthermore, we advise employing nested security groups for administrative permissions instead of assigning them to individual users. Access should be exclusively reserved for essential personnel based on roles. Infrequent occurrences of Windows Security event IDs 4732 or 4733, which signify alterations in group membership, warrant immediate scrutiny.

Audit procedures should include checking Windows Security event ID 4776, which verifies account credentials. A domain controller logs this event whenever it tries to validate an account’s credentials using NTLM over Kerberos or when NTLM logon attempts occur to the local SAM account. Event ID 4776 helps identify the forced use of NTLM, which enables defenders to detect attacks that primarily focus on using an NTLM hash of a password. Microsoft explains that this event exclusively originates from the authoritative source of the credentials. In Active Directory, the domain controller acts as this authoritative source, and the Source Workstation field will display the event ID 4776 alongside the identification of the requesting client. This event, particularly noticeable during tool testing involving forged tickets, can operate with nested security groups to identify unusual NTLM authentication requests.

Like Windows Security event ID 4776, logon telemetry originates from various sources based on the authentication method and credential source. Interactive logons generate event ID 4624 at a local capacity (Logon Type 2 — “Interactive”), with the client serving as the authoritative source of the credentials. In instances where remote logins employ Kerberos to validate the presented credentials (Logon Type 3 — “Remote”), the authoritative source of the credential is the domain controller. However, when adversaries send NTLM password hashes directly to a remote client, the 4624 Logon Type 3 is generated on the remote host (and not the domain controller), as the remote host is the authoritative source of the credential. A similar situation occurs when adversaries provide stolen credentials to a thread within an attacker-controlled process with methods such as Cobalt Strike’s make_token module. This method will generate a 4624 Logon Type 9 — NewCredentials with the client hosting the compromised process as the authoritative source of the event.

It’s vital to recognize this distinction, as many organizations mistakenly think they’re capturing all logon data when they’re only collecting remote logon events from the domain controller. By only ingesting logon telemetry from domain controllers, defenders are inherently blind to many techniques that adversaries must execute to propagate successfully.

Organizations should not only utilize nested security groups correlated with Windows event IDs but also consider integrating custom tools like bots or automated scripts. These scripts, utilizing utilities such as klist, can examine cached tickets to ensure they correspond with the expected user. Due to the legitimate appearance of these tickets, defenders often must collate information from various sources to identify signs of adversary manipulations or interactions with these valid tickets.

The 8th Season

Much like the final season of Game of Thrones, trying to evict a persistent adversary out of your domain can be trying. Since Kerberos is one of Microsoft’s authentication technologies, the “gemstone” variety of techniques can be challenging for defenders to identify. The detection in-depth approach is more challenging than just implementing a vendor solution. We believe constant testing and research is the only way to incorporate a defensive praxis in your organization and environment because the only thing worse than a compromise is a reoccurring compromise.

Organizations often pursue cost-effective remediation steps — short-cutting the more resource-intensive remediation requirements, leading to a false sense of security. While telemetry might show past adversary access to privileged accounts, it doesn’t confirm current access. Due to telemetry and resource constraints, no organization can ensure complete adversary eviction, emphasizing the need for comprehensive remediation.

Detection engineers should validate these techniques in labs and their environments. The provided detection methods are foundational and require context for accuracy. A dedicated sprint should focus on detecting these techniques, indicative of total domain compromise.

References

• Forest Recovery Guide, Microsoft, https://github.com/MicrosoftDocs/windowsserverdocs/tree/main/WindowsServerDocs/identity/ad-ds/manage/forest-recovery-guide
• Active Directory Forest Recovery — Devise an AD forest recovery plan, Microsoft, https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/forest-recovery-guide/ad-forest-recovery-devise-a-plan
• Active Directory Forest Recovery — Recover a single domain in a multidomain forest, Microsoft, https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/forest-recovery-guide/ad-forest-recovery-recover-single-domain-multidomain-forest
• AD DS Audit Policy Recommendations, Microsoft, https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/plan/security-best-practices/audit-policy-recommendations

---

Domain of Thrones: Part II was originally published in Posts By SpecterOps Team Members on Medium, where people are continuing the conversation by highlighting and responding to this story.