XZ Backdoor / RCE (CVE-2024-3094) is the Biggest Supply Chain Attack Since Log4j

Listen to post: Getting your Trinity Audio player ready...

A severe backdoor has been discovered in XZ Utils versions 5.6.0 and 5.6.1, potentially allowing threat actors to remotely access systems using these versions within SSH implementations.

Many major Linux distributions were inadvertently distributing compromised versions. Consult your distribution’s security advisory for specific impact information. While the attacker’s identity and motivation remain unknown, the sophisticated and well-hidden nature of the code raises concerns about a state-sponsored attacker.

Cato does not use a vulnerable version of “XZ / liblzma” and Cato’s code and infrastructure are not vulnerable to this backdoor / RCE.

Cato recommends that enterprises patch immediately. They should update XZ Utils from their Linux distribution’s repositories as soon as possible. In addition, they should review all SSH configurations for potentially impacted systems, implement strict security measures (e.g., strong authentication and access controls) and actively monitor network traffic and system logs for anomalies, especially related to SSH activity on vulnerable systems. This situation is still developing. Monitor sources like your distribution’s security advisories and trusted security news outlets for updates and enhanced detection methods.

What is XZ?

XZ Utils is a collection of free software tools used for highly efficient lossless data compression. It works with the .xz file format, known for its superior compression ratios compared to older formats like .gz or .bz2. The primary tools within XZ Utils (xz, unxz, xzcat, etc.) are used through your system’s terminal or command prompt.

• xz: Main command-line tool for compression and decompression.
• liblzma: A library with programming interfaces (APIs) for use in development.

Many major Linux distributions (Debian, Ubuntu, Fedora, etc.) employ XZ to compress software packages within their repositories. This significantly reduces storage costs and speeds up users’ downloads.
The main Linux kernel source is distributed as an XZ-compressed tar archive. Mac OS also comes preinstalled with XZ.

It’s important to note that XZ is open source.

How was the Backdoor Discovered?

Andres Freund, a PostgreSQL developer and software engineer at Microsoft, discovered the backdoor on March 29, 2024. He observed some unusual behavior on Debian testing systems. Logins via SSH were consuming abnormally high CPU resources, hence, slower SSH performance, and he also encountered valgrind errors (a memory debugging tool) related to liblzma (a core component of XZ Utils).

He posted his discovery on “Openwall”. Openwall is a project aimed at enhancing computer security by providing a collection of open-source software, resources and information to improve system and network security.

This is a screen from the discussion that Andres Freund started in Openwall.

Delving into the source code, he discovered a very odd and out-of-place M4 macro. This macro appeared to be intentionally designed to introduce malicious code during the build process. The backdoor logic was heavily obfuscated to avoid easy detection.

What is Known About the Backdoor So Far?

The backdoor was committed on February 23, 2024 by “JiaT75”.

Even if you have the “XZ” (liblzma) vulnerable version, it does not mean that you are affected.

In the build code itself, multiple conditions trigger the payload.

Here are a few examples:

This condition checks that the target build is for x86_64 and Linux systems. Otherwise, it terminates.

This condition checks that the build process should be done with “gcc”. Otherwise, it terminates.

From what we know so far here, are the steps in the malicious build process:

1. Checking various configuration settings and environment variables to ensure the build environment meets certain criteria (e.g., using GCC compiler, GNU linker, x86-64 architecture, etc.).
2. If the build environment is suitable, the script modifies the Makefiles and build configuration to enable the injection of the malicious code.
3. The script checks for specific source files related to CRC (cyclic redundancy check) algorithms used in XZ.
4. The script then attempts to inject a modified version of the CRC code into the XZ utility. It does this by:
   1. Extracting and decrypting a payload file (good-large_compressed.lzma) using a decryption algorithm.
   1. Saving the decrypted payload as liblzma_la-crc64-fast.o.
   1. Replacing the original CRC code with the modified version, including the decrypted payload.
5. The script compiles the modified CRC code using the GCC compiler with specific flags and options.
6. If the compilation is successful, the script replaces the original CRC object files (.libs/liblzma_la-crc64_fast.o and .libs/liblzma_la-crc32_fast.o) with the modified versions.
7. The script links the modified object files into the XZ library (liblzma.so).
8. After the build and successful installation, the backdoor intercepts execution by substituting ifunc resolvers for crc32_resolve() and crc64_resolve() , changing the code to call _get_cpuid()
   1. “ifunc” is a glibc mechanism that allows you to implement a function in different ways and choose between implementations while the program is running.
9. Afterwards, the backdoor monitors the dynamic connection of libraries to the process through an immediately installed audit hook, waiting for the connection of the RSA_public_decrypt@got.plt library.
10. Having seen the RSA_public_decrypt@got.plt connection, the backdoor replaces the library address with the address of the controlled code.
11. Now, when connecting via SSH, in the context before key authentication, the process will execute code controlled by the attacker.

As you can see, it’s a sophisticated and stealthy attack that can only be carried out by a nation-state-sponsored adversary.

The “xz” Github and the official site were taken down.

Supply chain attacks & Critical infrastructure: CISA’s approach to resiliency | Watch Master Class

Who is Behind the XZ Backdoor?

The backdoor commit was made by an individual using the name “Jia Tan” and the username “JiaT75”. This GitHub account was created in 2021 and has been active since then. They “contributed” to a few projects, including “OSS-Fuzz” by Google. But they were mainly active in the “xz” project.

How was “JiaT75”’s commit to “xz” approved? You can read the full chain of events on Evan Boehs’s blog. In short, the path to implementing the backdoor began approximately two years ago. The project’s main developer, Lasse Collin,was accused of slow progress. User Jigar Kumar insisted that xz needed a new maintainer for development. They demanded that patches be merged by Jia Tan, who contributed to the project voluntarily.

In 2022, Lasse Collin admitted to a stranger that he was in a difficult position: mental health issues, lack of resources and physical limitations were hindering his progress and the project’s pace. However, with Jia Tan’s contributions, he said he might be able to take on a more significant role in the project.

In 2023, Jia Tan replaced Lasse Collin as the main contact for oss-fuzz, a fuzzer for open-source projects from Google. In 2024, he commits the infrastructure that will be used in the exploit. The commit is attributed to Hans Jansen, a user who seems to have been created solely for this purpose. Jia Tan submits a pull request to oss-fuzz urging the disabling of some checks, citing the need for ifunc support in xz.

In 2024, Jia Tan changed the project link in oss-fuzz from tukaani.org/xz to xz.tukaani.org/xz-utils and completed the backdoor’s finishing touches.

Jia Tan, whoever he may be, started building this attack in 2021, gaining the trust of the primary maintainer of the XZ project. The amount of time and dedication from Jia Tan can only be attributed to a persistent adversary.

What Does This Mean for Other Open-source Projects?

1. Creating a validation process for entities that commit the code is important, especially for repositories that can affect other software.
   1. As demonstrated by @hasherezade, it is very easy to spoof the account that commits to Github.
2. Conduct a proper code review until you understand what is being committed.
   1. In the “XZ” backdoor commit, that backdoor is in the “XZ” file. You could spot the malicious code only if you had run and analyzed it.
3. Maintaining open-source projects requires a lot of time and dedication. You need to vet the person you want to hand the project over to.

What Can Cato’s Customers Do?

Check Which Version of “XZ” is Installed On Your Systems

Check the version of “XZ” on your system. Versions 5.6.0 or 5.6.1 are affected.

Run the following command in your terminal:

• xz –version
• apt info xz-utils

You can also check https://repology.org/project/xz/versions for affected systems.

Downgrade to an older version if possible.

XZ malicious package detection

We’ve verified that there have been no indications of downloading the known malicious files based on hashes or file names for the past six weeks —this is at least for customers who have TLSi enabled. (Note, however, that it is possible that malicious files could reach users in other forms of distribution, i.e., part of a package.)

The BitDefender Anti-Malware engine classifies the XZ package files as malicious files and blocks them if Anti-Malware is enabled.

SSH Traffic

Until the verification and downgrade process are completed, apply strict access policies on Inbound SSH traffic – limiting access to trusted sources and only in case of actual necessity.

Cato’s Multi-layer Detection and Mitigation Approach

Cyber-attacks are usually not an isolated event. They have multiple steps. Cato has multiple detections and mitigations across the entire kill chain, including initial access, lateral movement, data exfiltration, and more.

Cato’s Infrastructure

After checking Cato’s infrastructure, we can confirm that Cato is not using the vulnerable version of XZ / liblzma.

Final Thoughts

We still do not know the full extent of this backdoor’s impact. There is always fallout in such cases as the security community delves deep and uncovers more information about possible attacks.

The initial commit was on February 23, 2024 and it was discovered on March 29, 2024. This is a significant window for malicious activity to occur.

In security incidents, multiple layers of detection and mitigation capabilities are crucial to halt the attack through various means.

We are continuing to research and monitor for further developments.