Summary
In late May 2021, Secureworks® Counter Threat Unit™ (CTU) researchers investigated the protocol that the Azure Active Directory (AD) Connect Health agent for AD Federation Services (AD FS) uses to send AD FS sign-in events to Azure AD. This research revealed a flaw in the protocol that could be exploited by a threat actor who has local administrator access to the AD FS server. If the threat actor can extract the credentials that the agent uses to authenticate to Azure AD, they could tamper with Azure AD sign-ins log events or pollute the sign-in log with fake sign-in events to hide unauthorized authentication events.
CTU™ researchers reported the flaw to Microsoft on May 31. Microsoft confirmed the behavior on June 16 and released a "fix" on July 7. CTU researchers verified that the change addressed the issue.
Azure AD Connect Health
The Azure AD Connect Health agent allows configuration and health information from on-premises AD FS servers to be monitored centrally in Azure AD. Since March 2021, the Azure AD Connect Health agent also sends AD FS sign-in and sign-out events to Azure AD.
The agent collects the sign-in events from the Windows Security log on the AD FS server, sends them to Azure Blob storage, and sends a notification using Azure Service Bus. Within Azure AD, the sign-in events are stored in the ADFSSignInLogs log. This log cannot be viewed directly from the Azure portal and requires an Azure subscription to be viewed in Log Analytics. However, the AD FS sign-in events are included in the standard Azure AD sign-ins log alongside sign-in events not sourced from AD FS.
After compromising an AD FS server, attackers could spoof the sign-ins log with fake sign-in events. For example, a threat actor could hide unauthorized activity within a flood of spoofed log events. This behavior represents abuse of legitimate functionality, not a vulnerability in the platform.
Structure
In Azure AD, there are distinct logs for different types of sign-ins. Azure AD sign-in events are stored in SignInLogs, and AD FS sign-ins are stored in ADFSSignInLogs (see Figure 1).
Figure 1. Sign-in events in Azure AD (left) and AD FS (right). (Source: Secureworks)
Organizations with an Azure subscription can export ADFSSignInLogs to a Log Analytics workspace for viewing and analysis (see Figure 2).
Figure 2. Log Analytics workspace view of sign-in events. (Source: Secureworks)
Administrators can view sign-in logs in the Azure Admin Portal. However, there is no dedicated tab for ADFSSignInLogs. Instead, AD FS sign-in events are shown in the ‘User sign-ins (interactive)' tab alongside Azure AD sign-in events. Figure 3 shows how the AD FS log-in events from Figure 2 appear in the Azure AD sign-ins log.
Figure 3. AD FS sign-in events displayed alongside Azure AD sign-ins. (Source: Secureworks)
The Azure AD Connect Health agent for AD FS consists of three services. The one that is responsible for sending the events to Azure AD is the Azure AD Connect Health AD FS Insights Service (see Figure 4).
Figure 4. Azure AD Connect Health AD FS Insights Service. (Source: Secureworks)
Process and protocol
Figure 5 illustrates how AD FS sign-in events are gathered and sent to Azure AD.
Figure 5. AD FS sign-in events sent to Azure AD. (Source: Secureworks)
The following procedure is used to gather and send AD FS sign-in events to Azure AD:
A user signs in to AD FS using any method configured and available for the user.
During and after a successful or failed sign-in, AD FS server writes multiple auditing events to its Security log. Auditing is turned on during the installation of the agent and is a requirement for gathering events.
These events use Event ID 1200 and contain details about the sign-in event (see Figures 6 and 7). The Azure AD Connect Health agent gathers most of the information it needs from Event ID 1200 events, although it also gathers certain user identification information from other Event IDs.
Figure 6. Auditing Event ID 1200 created during a successful or failed sign-in. (Source: Secureworks)
Figure 7. XML content of an Event ID 1200 event. (Source: Secureworks)The Azure AD Connect Health agent periodically reads auditing events from the AD FS server Security log. This auditing event information is then uploaded to Azure AD.
The agent gets a Service Access Token from Azure AD. The token is fetched by making an HTTP POST request to the following URL:
https: //s1 . adhybridhealth . azure . com/oauth2/token
The body of the request includes the following string:
grant_type=client_credentials&client_secret=<client_secret>&client_id=<tenant_id>_<machine_id>
The <client_secret> variable is known as an AgentKey and is stored in the AD FS server registry. The AgentKey is protected using Microsoft Data Protection API (DPAPI) with a hard-coded entropy "ra4k1Q0qHdYSZfqGxgnFB3c6Z025w4IU". The client_id is a combination of the <tenant_id> and <machine_id>. Both of these values are stored in the registry (see Table 1).
Parameter Registry location client_secret HKLM\SOFTWARE\Microsoft\ADHealthAgent\AgentKey tenant_id HKLM\SOFTWARE\Microsoft\ADHealthAgent\TenantId machine_id HKLM\SOFTWARE\Microsoft\Microsoft Online\Reporting\MonitoringAgent\MachineIdentity Table 1. Registry location for Service Access Token request parameters.
When the request is successful, the response is a JSON file containing the service access token (see Figure 8).
Figure 8. JSON file containing service access token. (Source: Secureworks)The agent gets a Blob Upload Key that is required to send the auditing event information to Azure AD. The key is fetched by making a HTTP GET request to the following URL:
https: //s1 . adhybridhealth . azure . com/providers/Microsoft.ADHybridHealthService/monitoringpolicies/<service_id>/keys/BlobUploadKey
The <service_id> refers to the ID of the AD FS service registered to Azure AD during the first agent installation. This ID is not shown in the Azure Portal, but it is stored in the HKLM\SOFTWARE\Microsoft\ADHealthAgent\ADFS\ServiceId registry.
The Service Access Token from the previous step is included in the Authorization header:
Authorization: Bearer <service access token>
The response to a successful request contains a URL for the Blob storage with a valid shared access signature (SAS) token. The <service_id> is the same service ID sent in the request.
https: //adhsprodweuaadsynciadata . blob . core . windows . net/adfederationservice-<service_id>?sv=2018-03-28&sr=c&sig=RCrQOWOLr%2FjHIX6%2FxCti1bPmbHgkp4T9eLS07uP%2FyKM%3D&se=2021-07-10T08%3A01%3A46Z&sp=w
The agent gets an Event Publisher Key that is required to send the signature of the events blob to Azure AD. Azure AD uses this signature to verify that the event information uploaded by the Azure AD Connect Health agent is authentic. Azure AD starts processing the event information uploaded to Blob storage only after it receives a notification via Azure Service Bus that contains this signature.
The Event Publisher Key is fetched by making a HTTP GET request to the following location. The <service_id> is the same as in step 5. The service access token obtained in that step is also used for authentication.
https: //s1 . adhybridhealth . azure . com/providers/Microsoft.ADHybridHealthService/monitoringpolicies/<service_id>/keys/EventHubPublisherKey
The response contains a JSON file that consists of a single string containing an Azure Service Bus endpoint and other related information that includes another SAS token:
"Endpoint=sb://adhsprodweuehadfsia . servicebus . windows . net/;SharedAccessSignature=SharedAccessSignature sr=sb%3a%2f%2fadhsprodweuehadfsia . servicebus . windows . net%2fadhsprodweuehadfsia%2fPublishers%2f658fe106-a59d-404e-985b-
0c1bf3b4f72d&sig=4%2bZ%2bNurnA4%2b4t6dvTG8kqraJMlNzxKF0KFjiBIaZUw4%3d&se=1625904056
&skn=RootManageSharedAccessKey;EntityPath=adhsprodweuehadfsia;Publisher=658fe106-a59d-404e-
985b-0c1bf3b4f72d"
The auditing event information is sent to Blob storage as a JSON file that consists of an array of event objects (see Figure 9). Blob storage stores large amounts of unstructured data and is ideal for handling the uploaded auditing event information, which could be large in size.
Figure 9. Example Event ID 1200 event uploaded to Blob storage. (Source: Secureworks)Azure AD only uses the User Principal Name (UPN), although Event ID 1200 contains other identity information (e.g., rows 14, 16, 36, and 37 in Figure 9). All sign-in events are sent to Azure AD. However, only events with a UPN corresponding to a user account that exists in Azure AD are added to ADFSSignInLog.
The JSON file contents are compressed using Gzip. The Azure AD Connect Health agent sends the compressed JSON file to the Blob storage by issuing an HTTP POST to the URL received in step 5. The URL is modified by adding a filename and api-version. The <service_id> is the same as in the previous steps, and <id> is a random GUID generated to uniquely identify the sent events.
https: //adhsprodweuaadsynciadata . blob . core . windows . net/adfederationservice-<service_id>/<id>.json?sv=2018-03-28&sr=c&sig=RCrQOWOLr%2FjHIX6%2FxCti1bPmbHgkp4T9eLS07uP%2FyKM%3D&se=2021-07-10T08%3A01%3A46Z&sp=w&api-version=2017-04-17
The following HTTP headers are used while making the request. The <MD5Hash> value is an MD5 hash of the Gzip-compressed JSON file. The <id> value is the same ID used in the request URL.
User-Agent: Azure-Storage/8.2.0 (.NET CLR 4.0.30319.42000; Win32NT 10.0.17763.0)
x-ms-version: 2017-04-17
Content-MD5: <MD5Hash>
x-ms-blob-type: BlockBlob
x-ms-client-request-id: <id>
The agent notifies Azure AD about the sent auditing event information via Azure Service Bus. It sends the notification and a signature that results from signing the notification with a signing key. Figure 10 lists example PowerShell code that could be used to derive the signing key.
Figure 10. Commented PowerShell showing the generation of the signing key. (Source: Secureworks)The following notification string is signed with this key. The <tenant_id>,<service_id>, and <machine_id> variables use the values obtained in steps 4 and 5. The <blob_url> is the URL used in step 7 without query parameters. The <date_string> is the signing time (UTC) in sortable format (e.g., 2021-07-09T10:43:35).
<tenant_id>,<service_id>,<machine_id>,Adfs-UsageMetrics,<blob_url>,<date_string>
The signature is calculated by converting the notification string to a byte array of Unicode values. A HMACSHA512 hash of this converted string is then generated using the previously calculated signing key. Finally, the signature is Base64-encoded.
The Azure AD Connect Health agent connects to Azure Service Bus using the following URL, where <endpoint_host> is the host name of the endpoint retrieved in step 6:
https: //<endpoint_host>/$servicebus/websocket
Figure 11 shows the message containing the notification string to be signed and the actual signature.
Figure 11. Notification string and signature. (Source: Secureworks)
Flaw in the protocol
CTU researchers noticed that the Request ID column in the Azure AD sign-ins log matched the UniqueID property of AD FS sign-in events (see Figure 12). By using an existing Request ID as the UniqueID for a fake sign-in event, CTU researchers overwrote existing Azure AD sign-ins log events.
Figure 12. Matching UniqueID and Request ID. (Source: Secureworks)
An attacker could modify details of existing sign-in events (e.g., date, user, IP address) to hide malicious activities. A threat actor could also abuse the protocol to hide unauthorized access by creating large numbers of faked events.
Conclusion
Microsoft addressed the tampering flaw by ensuring that all AD FS sign-in events are assigned a randomly generated Request ID. This change mitigates the exploitation technique identified by CTU researchers. However, the sign-ins log spoofing technique cannot be detected with the information available to Microsoft 365 and Azure customers.
Organizations should follow Microsoft's recommendations for securing AD FS servers and protecting cloud environments from on-premises attacks. Endpoint monitoring on AD FS servers can help to detect credential theft that could let attackers exploit this flaw, as well as other techniques that threat actors use to pivot from on-premises environments into cloud environments. Organizations should also follow Microsoft's guidance for securing global administrator accounts. These highly privileged accounts can provide an attacker with wide-ranging access to accounts and data.