A Hacker Spent Two Years Earning Trust to Backdoor the Internet

The compression library that almost broke the internet

xz Utils is a compression library and command-line tool present on virtually every Linux system. It compresses man pages, kernel modules, firmware packages, and package archives. It is a dependency of systemd, which means it is a dependency of OpenSSH on most Linux distributions. It is the kind of low-level infrastructure that nobody thinks about until something goes terribly wrong [1].

On March 29, 2024, Andres Freund, a PostgreSQL developer at Microsoft, posted to the oss-security mailing list that he had discovered a backdoor in xz Utils versions 5.6.0 and 5.6.1. The backdoor, assigned CVE-2024-3094 with a CVSS score of 10.0 (the maximum), was designed to give the attacker pre-authentication remote code execution on any system running OpenSSH linked against the compromised liblzma library [2].

The backdoor was weeks away from shipping in the stable releases of Debian and Fedora. If it had landed, the attacker would have had a skeleton key to a significant fraction of the servers on the internet.

The two-year social engineering campaign

The most remarkable aspect of CVE-2024-3094 is not the technical sophistication of the backdoor. It is the social engineering campaign that placed it there.

The xz Utils project was maintained by a single developer, Lasse Collin, who had maintained the project since 2009. By 2021, Collin was burned out and struggling to keep up with the project. He was open about this on mailing lists, noting delays in reviewing patches and releasing updates [3].

In late 2021, an account using the name "Jia Tan" (GitHub username JiaT75) began submitting patches to xz Utils. The patches were legitimate, helpful, and well-crafted. They fixed bugs, improved performance, and added test infrastructure. Over the following months, Jia Tan became a regular contributor.

Simultaneously, sockpuppet accounts began pressuring Collin on the mailing list. Users with names like "Jigar Kumar" and "Dennis Ens" complained about slow release cycles, demanded faster patch reviews, and suggested that the project needed a co-maintainer. The pressure was coordinated and designed to make Collin feel guilty about his pace of work [4].

The timeline:

• Late 2021: Jia Tan begins contributing patches
• Early 2022: Sockpuppet accounts pressure Collin to add a co-maintainer
• Mid 2022: Collin grants Jia Tan commit access
• 2022 to 2023: Jia Tan makes hundreds of legitimate contributions, building trust and establishing a track record
• Late 2023: Jia Tan begins introducing the backdoor components, disguised as test infrastructure improvements
• February 2024: xz Utils 5.6.0 is released with the backdoor. Jia Tan is listed as a co-maintainer
• March 2024: xz Utils 5.6.1 is released with additional obfuscation of the backdoor
• March 29, 2024: Andres Freund discovers the backdoor and reports it

Two years of patient work. Hundreds of legitimate commits. Social engineering of a burned-out volunteer. This was not a smash-and-grab. This was a state-level intelligence operation.

The technical backdoor

The backdoor was elegantly constructed to avoid detection through normal code review [5]:

The delivery mechanism. The malicious code was not in the xz Utils source code repository itself. It was hidden in binary test files (compressed .xz files used for testing) that were checked into the repository. The build system was modified so that during the compilation of liblzma, a script extracted the malicious code from these test files and injected it into the compiled library.

The trigger. The backdoor only activated when the following conditions were met:

1. The system was running on x86-64 architecture (amd64)
2. The library was linked against OpenSSH's sshd process
3. The build was part of a Debian or RPM package build (not a developer build from source)
4. Certain environment variables were not set (to avoid triggering in sandboxed or testing environments)

The payload. When activated, the backdoor intercepted the RSA signature verification function used during SSH authentication. It checked incoming connections for a specific cryptographic key embedded in the backdoor. If the correct key was present in the connection, the backdoor executed an attacker-supplied command before authentication completed. This is pre-authentication remote code execution: the attacker did not need a valid username or password [6].

The obfuscation. The malicious code was:

• Split across multiple binary test files
• Compressed and encoded to avoid pattern matching
• Extracted and assembled only during the build process
• Conditional on specific build environments (Debian/RPM packaging)
• Designed to produce identical behavior to the unmodified library in all normal usage

A standard code review of the xz Utils source code would not reveal the backdoor because the malicious code did not exist in the source files. It existed in binary test data and was injected during compilation.

How Andres Freund found it

The discovery was accidental. Freund was benchmarking PostgreSQL performance on a Debian Sid (unstable) system and noticed that SSH logins were taking approximately 500 milliseconds longer than expected, consuming noticeably more CPU than they should [7].

Most developers would have shrugged this off as a system configuration issue. Freund did not. He profiled the sshd process and traced the CPU usage to the liblzma library, which should not have been doing significant computation during SSH authentication.

He examined the xz Utils source code and found nothing unusual. Then he looked at the binary test files and the build scripts, and he found the injection mechanism.

His email to oss-security at 10:50 AM UTC on March 29, 2024, began: "After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer: The upstream xz repository and the xz tarballs have been backdoored."

Within hours, every major Linux distribution had issued advisories and rolled back to xz Utils 5.4.x. Debian, Fedora, openSUSE, and Arch Linux all confirmed they had been shipping or were about to ship the compromised versions. Debian Unstable and Fedora Rawhide had already distributed it. The stable releases of both distributions had not yet received it.

The margin was approximately one to two weeks. If Freund had not noticed the CPU anomaly, the backdoor would have shipped in Debian 13 (Trixie) and Fedora 40 stable, reaching millions of production servers.

State-sponsored attribution

The sophistication, patience, and operational security of the attack strongly suggest state sponsorship. Multiple intelligence community sources and security researchers have pointed to characteristics consistent with Chinese or Russian state-backed operations [8]:

• Two-year timeline. No financially motivated criminal group invests two years of continuous effort in a single project with no intermediate payoff
• Sockpuppet coordination. The pressure campaign against Collin used multiple carefully managed fake identities that never appeared in any other context
• Technical sophistication. The backdoor design, including its conditional activation, environment-aware triggering, and build-system integration, required expertise in compiler toolchains, SSH internals, and Linux packaging systems
• Target selection. A backdoor in liblzma linked to OpenSSH provides access to the largest possible set of Linux servers on the internet. This is a strategic intelligence target, not a criminal one
• Operational patience. State intelligence operations routinely invest years in developing access. Criminal operations optimize for speed

No definitive attribution has been made publicly. The Jia Tan account was carefully constructed to avoid revealing its origin. The email address used is not linked to any known individual or organization.

The open-source maintenance crisis

CVE-2024-3094 exposed a critical structural vulnerability in the open-source software ecosystem: the entire internet depends on code maintained by individuals who are unpaid, burned out, and overwhelmed [9].

Key facts:

• xz Utils was maintained by one person for 15 years
• OpenSSL was maintained by two people when Heartbleed was discovered in 2014
• core-js, a JavaScript polyfill used by 75% of the top 1,000 websites, is maintained by one person who was imprisoned in Russia for unrelated charges
• curl, used by virtually every internet-connected device, is maintained primarily by one person (Daniel Stenberg)
• Log4j, whose vulnerability caused global chaos in December 2021, was maintained by a handful of volunteers

The xkcd comic showing all of modern infrastructure resting on a tiny block labeled "a project some random person in Nebraska has been mass maintaining since 2003" is not satire. It is a system architecture diagram.

Supply chain attacks in 2025: the epidemic continues

The xz Utils backdoor was the most dramatic supply chain attack of 2024, but it was far from the only one. The problem has accelerated [10]:

npm and PyPI poisoning. In 2025, security researchers identified over 1,700 malicious packages on npm and PyPI attributed to North Korean threat actors. These packages used typosquatting (names similar to popular packages), dependency confusion (publishing internal package names to public registries), and trojanized forks of legitimate projects.

GitHub Actions compromise. Attackers compromised popular GitHub Actions used in CI/CD pipelines, injecting malicious code that executed during builds. Because GitHub Actions run with access to repository secrets, a single compromised action can leak API keys, deployment credentials, and signing certificates across thousands of projects.

Compromised maintainer accounts. Multiple npm and PyPI maintainer accounts were compromised through credential stuffing (using passwords leaked in other breaches) and phishing. The attackers published malicious versions of the maintainer's legitimate packages.

Build system attacks. Beyond xz Utils, researchers found evidence of build-system manipulation in other projects, where malicious code existed only in release tarballs but not in the source repository, making it invisible to anyone reviewing the repository code.

What this means for security

The xz Utils backdoor teaches several lessons:

Trust is the vulnerability. The open-source ecosystem is built on trust. Trust that maintainers are who they say they are. Trust that contributors have good intentions. Trust that the code you download is the code that was reviewed. The xz attack exploited every layer of this trust model.

Binary artifacts in repositories are dangerous. Any project that includes binary files (compiled code, compressed test data, images with embedded data) has a potential injection vector that bypasses source code review.

Reproducible builds matter. If the binary package produced by a build can be independently reproduced from the source code, hidden injections are detectable. Projects like Reproducible Builds (reproducible-builds.org) are working on this, but adoption remains limited.

Maintainer sustainability is a security issue. A single burned-out maintainer is a single point of failure. Funding open-source maintenance through initiatives like the OpenSSF, GitHub Sponsors, and Tidelift is not charity. It is infrastructure investment.

Dependency auditing must go deeper. Scanning for known CVEs in your dependency tree is not enough. You need to understand who maintains your dependencies, how the build process works, and whether the released artifacts match the source code.

The internet's most critical software runs on the donated labor of volunteers. Adversaries with nation-state resources have noticed, and they are patient enough to spend two years proving it.

Sources

1. xz Utils Project, "XZ Utils Documentation," tukaani.org/xz
2. NIST National Vulnerability Database, "CVE-2024-3094," March 2024
3. Lasse Collin, xz-devel mailing list archives, 2021 to 2022
4. Evan Boehs, "Everything I Know About the XZ Backdoor," boehs.org, March 2024
5. Filippo Valsorda, "The xz Attack Shell Script," filippo.io, April 2024
6. Andres Freund, "Backdoor in Upstream xz/liblzma Leading to SSH Server Compromise," oss-security mailing list, March 29, 2024
7. Andres Freund, personal blog and interviews, March to April 2024
8. Wired, "The Mystery of Jia Tan, the XZ Backdoor Mastermind," April 2024
9. Nadia Eghbal, "Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure," Ford Foundation, 2016
10. Phylum, "2025 Software Supply Chain Security Report," 2025