AUR Chaos malware: an analysis

Recently, an incident involving malware in the AUR made the headlines. I read a lot of things around this topic, both right and wrong, and sometimes misleading. I was involved in the incident handling I chose to write this blog post, not only for transparency but also for laying down what I learned both during and after the incident.

Incident handling timeline

First, I will try to present factually what happened and how did we react within the Arch Linux team. The incident was mainly handled by bertptrs, gromit and I.

Something to keep in mind while reading this part is that members of the Arch Linux staff are all volunteers; we happily give some of our free time to a distro we love, but we are not on call 24/7 even if we obviously always do our best to handle such incidents as quickly as possible. From this perspective, the incident happened “at a nice time” because it was Friday afternoon / evening in Europe, where a large part of the Arch Linux team is living.

All times given in this post are in UTC+2 (Paris summer hour, i.e. my time). This part will only include the timeline of our incident response and detailing actions we took in order to stop the crisis. I will deliberately be evasive on the content of the malware, as I address it in depth in the following section.

First wave

At around 5pm on Friday, July 18th, we (the Arch team) became aware of three malicious packages having been pushed into the Arch User Repository (AUR). These packages, librewolf-fix-bin, firefox-patch-bin, and zen-browser-patched-bin, were copycats of well-known web browsers that claimed to “fix” bugs or performance issues. What they effectively did was installing a Remote Access Trojan (RAT). The submitter of packages had already deleted their AUR account by the time we started the investigation. This slowed us a bit because we don’t keep information of deleted accounts. We also quite quickly saw that the GitHub repo hosting the malicious patch has been deleted, preventing further installation of the malware by users.

After creating a shared document to coordinate our efforts, follow each other’s actions and collect the evidence, the first thing we rushed to do was to delete the offending packages. This is a quite interesting part that I will explain in depth below. We took the precaution of cloning the packages first and saving them for later investigation. The packages were fully deleted a little after 6pm.

We then focused on disabling the AUR user that uploaded the malicious packages, but because they had already deleted their account not much was left to do. We also spent a little time trying to quantify the amount of users that potentially downloaded the infected packages (I’ll go back on that later). We finally drafted an announcement mail that was sent a little before 7pm. All in all, we tackled the situation in less than two hours, deleting the infected packages, ensuring the removal of the corresponding AUR account, and sent out the announcement to users, which I think is not bad.

Second wave

Once the announcement mail sent, and after we thought the incident was over, I started my Friday night by enjoying a movie. When I checked my phone in the middle of the film, at around 11pm, I saw that a second wave of infected packages had been uploaded, with the Arch team becoming aware at 10pm. These packages were minecraft-cracked, ttf-ms-fonts-all, vesktop-bin-patched and ttf-all-ms-fonts. These again tried to impersonate well-known packages by pretending fixes, or by naming packages in a way that may make them attractive to unsuspecting users. This time, the “patch” that allowed the malware installation was hosted on Codeberg, but the repo was deleted before we became aware of these packages. They were uploaded between 6pm and 7pm the very same day.

I arrived one hour late to the party and most of the situation had already been handled by my fellow Arch Team members. I stilled joined a call where we were trying to identify the attacker (we quickly identified that the two waves were coming from the same person, as the “patch” script hosted on GitHub and Codeberg were identical, along with other factors). This time it wasn’t a single AUR account involved but several sockpuppets accounts being used to upload packages and leaving comments on other infected packages, praising their “quality”.

My fellow Arch Team members did a very good job at hunting down sockpuppets accounts before I joined the call, and were also working on identifying potential other AUR packages being infected by the same attacker by using identical patterns across packages. The infected packages were all using an identical line, chmod +x "$pkgdir"/opt/$pkgname/patches/main.py, that was used to make the main.py script (the “patch” retrieved from a remote git repo) executable. A little after midnight, after exploring the whole content of the AUR, we confirmed that no infected package was left (or more precisely, no infected package using the same infection method was left).

I was myself focused on analyzing the malware in order to try to get back to the source. I was quickly able to identify the Command and Control (C2) server to which the malware was phoning to. While investigating where this C2 server was hosted in order to bring it down, the server was disconnected from the Internet between 11:30pm and midnight. I am not sure when exactly the C2 was disconnected, because I first thought that I got banned from communicating with the C2 due to my interactions with it. A little after, I was able to confirm that the C2 was indeed down not only for me but for everyone. This concluded our incident handling, and we ended the call a little after midnight. We decided against sending another warning mail as the packages were only available on the AUR a few hours and were non-functional most of the time.

Focus on the deletion of packages

Our deletion method

Deleting properly an AUR package is less simple than it seems, so I figured that the topic deserved its own section, especially as it made me learn new things about git. First of all, aurweb (the website behind aur.archlinux.org) does not really delete a package from cgit, the web interface used for browsing git repositories of AUR packages. Instead, the package is stored as is, and it is possible to retrieve the content of the package by cloning the name. Example with the podman-desktop-bin package I used to maintain before promoting it to an official Arch Linux package in extra (you can check that the package does not currently exist on the AUR):

 $ git clone https://aur.archlinux.org/podman-desktop-bin
Cloning into 'podman-desktop-bin'...
remote: Enumerating objects: 146, done.
remote: Counting objects: 100% (146/146), done.
remote: Compressing objects: 100% (74/74), done.
remote: Total 146 (delta 69), reused 146 (delta 69), pack-reused 0 (from 0)
Receiving objects: 100% (146/146), 28.73 KiB | 14.36 MiB/s, done.
Resolving deltas: 100% (69/69), done.

We cloned something, then doing git log inside this folder:

 $ git --no-pager log
[SNIP]
commit 16ccf24b74afc5a0f6a0c3ea2d33cfd19f68f20e
Author: Quentin MICHAUD <[REDACTED]>
Date:   Tue Mar 4 23:59:49 2025 +0100

    Bump to 1.17.1
[SNIP]

We can indeed retrieve the history of all the commits done to this package.

Thus, if we want to remove completely remove the malware from the AUR, we need to delete the history by force-pushing something else. We first tried to delete the branch with AUR_OVERWRITE=1 git push origin --delete master (check about AUR_OVERWRITE here), but it was refused by aurweb because the way aurweb is processing pushes expect the newly pushed git object to exist (relevant code line here). In the case of a branch deletion, the new object is a null sha1 (40 zeros), meaning the call to repo.walk() fails. This could (should?) be fixed by adding a condition checking if the new object isn’t a null sha1 and acting accordingly. This would allow us to indeed delete branches, which would have come handy here and may come handy again in the future.

The next best solution was to force-push some content to overwrite the malware. But simply force-pushing a commit with random content won’t do: we have to satisfy the requirements of an AUR package. Indeed, aurweb verify that what we are pushing are not random things but indeed and AUR package. A minimal package consists of an empty PKGBUILD and a minimal .SRCINFO containing the following:

pkgbase = firefox-patch-bin
        arch = any
        pkgrel = 1
        pkgver = 0

pkgname = firefox-patch-bin

Normally, the .SRCINFO file is generated from the PKGBUILD using makepkg --printsrcinfo. .SRCINFO is used by aurweb and pacman to easily get metadata from the packages, because the PKGBUILD can quickly become complex and hard to parse. This also means that we cannot push the same empty repo to each affected package, because of the .SRCINFO containing the package name, we needed to create one empty repo per package. One of my fellow Arch Team member quickly wrote a script handling that and that’s what allowed us to delete the malware from the AUR.

A complete deletion?

But did we REALLY delete them? If you already struggled with git in your life (I surely did), you know that when you mess up your git repository and overwrite or in any way lose some commits, you may be able to retrieve them using less-known git commands such as git reflog and git fsck. So, knowing that we can access deleted packages repo history as I mentioned before, could we retrieve the commits containing the malware?

Investigating the matter, it seems that locally, there isn’t any git command that could retrieve all objects stored in a remote repository. Only objects that are referenced (e.g., by a branch, a tag, or another commit) can be retrieved using the git command. I learned along the way about commands like git rev-list, git clone --mirror or git ls-remote, but none of them are able to list unreferenced (orphan) objects in a remote repository.

But on the remote repository (the machine hosting the AUR), could we retrieve these objects? As I’m not part of the Arch Linux DevOps team, I have no way to investigate this myself. However, if these orphan objects are still available in the repositories, I may be able to access them from the cgit interface.

Interestingly, less than a month ago, a security researcher was able to scan GitHub orphan commits and retrieve secrets inside them. Something similar could be achieved here by searching the orphan commits in cgit’s interface. However, there are several things that stop us for doing that. First, the tool used by the researchers is using a GitHub archive indexing all the events happening on GitHub. As far as I know, something similar doesn’t exist for the AUR. I know of the Software Heritage archive that backup AUR contents, but it is archiving at fixed times, and it seems that it didn’t take a snapshot when our malicious packages were online.

Second, the blog post suggests another method of finding orphaned commits, which is leveraging the ability of git to address objects by the first four hex digits of their sha1 identifier if this one is unique. Assuming the number of objects in one repository isn’t huge, you would have a decent chance of finding said commit by bruteforcing the 65536 possibilities of commit identifiers. However, cgit doesn’t have an API allowing to search for commits and even the web interface doesn’t seem to have a possibility to search using a shortened commit identifier. Retrieving a commit using its full hash is possible but of course, bruteforcing a sha1 identifier is near impossible.

As we saved the infected malware before deleting them, I have still access to the original git repositories of the infected packages, which allows me to retrieve the full commit id. And sure enough, when accessing the cgit interface of the firefox-patch-bin package and providing the correct commit id:

The commit is still available on the remote! Because, as we thought, the orphan commit is still available as an object in the git repository, and knowing the commit id allow us to actually see the commit in cgit’s interface. Using this commit id, I can also run git fetch <commit-id> and git checkout <commit-id> from the firefox-patch-bin repo to retrieve this commit on my machine.

So, now that we know the commits are still there, how could we REALLY delete them? Well, git provides a way to do that. First, we mark all entries as expired using git reflog expire --expire=now --all. Then, we force the garbage collector to run (yes, git has a garbage collector) with git gc --prune=now --aggressive. I tried it on a local repo with an orphan commit, and it works, the commit was completely deleted. So in theory running these commands on the machine hosting the AUR in the correct repos would work.

However, doing that on the AUR machine would potentially have an impact on all AUR packages. Indeed, the AUR is one big git namespace, meaning even if we, as users, see each AUR package as an independent project, the git objects such as commits are stored in one big object store. This means that the commands leveraging the reflog or the garbage collector applies to ALL AUR packages. Not sure if that is something we as the Arch Team is something that we would like to experiment, just for deleting commits that are impossible for users to find anyway…

Investigation

I’ll now focus on exploring what the malware did and how it was propagated.

Malware propagation

Interestingly, the attempts of propagating the malware are what made it detected. We can find a user advocating for one of the malicious packages on Reddit… And plenty of comments calling out the user and warning about the malware. Thankfully for us, at least a part of AUR users actually read PKGBUILDs before installing them!

We can check the user profile of the OP to see that the main thing this account ever did was trying to advocate for malware. This makes it very probable that they are our attacker or at least linked to them.

Sadly, all comments are deleted now, but thanks to a strange behavior of Reddit, we can explore part of the posts and comments of the user that advocated for the malicious packages. Using Reveddit, we can see some deleted comments of the user (sadly not the posts because they have been deleted by administrators) to understand the pattern a little.

Infected packages

Let’s focus on the packages themselves. First of all, it is important to note that these were not packages that were already present and well-used on the AUR and then for infected. You couldn’t have been infected by this malware by simply updating your computer. It was new packages that tried to impersonate well-known packages, and you could only get infected if you installed these specific packages during the small time frame where they were functional. All in all it was a quite small risk.

Where the malware was smart, however, was not only in the name they used, but in the way they use the provides field of the PKGBUILD. They claimed to provide the same package that they were impersonating, listing the package as a possibility for a user trying to install a package. An example with the output of paru and librewolf when the malware was still online:

 $ paru -S librewolf
:: Resolving dependencies...
:: There are 5 providers available for librewolf:
:: Repository AUR:
    1) librewolf  2) librewolf-allow-dark  3) librewolf-bin  4) librewolf-fix-bin  5) librewolf-kde-appmenu
Enter a number (default=1):

If for some reason an unsuspecting user decided that librewolf-fix-bin feel a better option than librewolf-bin (without reading the PKGBUILD), they could have chosen to install it quite easily. Due to this mechanism the package was quite discoverable even if available for only a few days.

Finally, another smart thing the malware did was to use a post install script instead of trying to execute the malware directly in the installation phase. Indeed, the good practice of building an Arch package dictates that packages must be built in a chroot. Now, most AUR helpers don’t do that by default (you can enable it in paru using --chroot), but that’s something important to keep in mind when troubleshooting an AUR package that doesn’t build on one’s system. Building in a clean chroot ensures the packages and configuration of your system doesn’t interfere with the build and vice versa. The commands ran in a chroot are isolated from the rest of your system, and commands run there won’t permanently impact it (such as try to start a systemd service).

The install functions family allows running commands directly on your system before or after installation. These commands are not run isolated from your system and thus can have long-lasting effects. Moreover, even when building outside a chroot the build process will most of the time be executed as an unprivileged user, while the install scripts are run as root. This is where the attacker decided to execute the first stage of their malware, allowing to avoid the potential isolation of a user building their packages in a chroot and being able to get more privileges from the start.

The relevant line executed in the post_install script of malicious packages is the following:

python /opt/firefox-patch-bin/patches/main.py

The main.py script comes from the git repositories claiming to provide “fixes” or “patches” and is the first step towards installation of the malware.

Stage one: Remote git repository

The remote git repositories contained several files but the main.py script is the most interesting bit. I won’t go in every detail (the file is available here for those interested) but describe the main steps. Thanks to Arch Linux user fk29g for saving this repo when it was still online and accepting to share it with me for the purposes of my investigation and of this blog post.

The script first checks if the platform is Linux or Windows, then run the corresponding steps. The script then downloads a binary named systemd-initd and stores it in /usr/local/share if it has root privileges, and in ~/.local/share otherwise. It then setups a systemd service that executes the newly downloaded binary and makes it wanted by default.target, which will trigger the script at boot on most machines. The service is then enabled and started, either as root or as an unprivileged user depending on the permission of the script. All in all, this Python script installs a malicious binary and attempts to conceal it as a legitimate systemd init binary.

The malware files (including the Windows ones) are still hosted to this day on a public filesharing website.

Stage 2: Remote Access Trojan

The file command quickly tells us that the final binary to investigate is a classic ELF binary, coming from a Go program. The VirusTotal analysis clearly highlight it as the Chaos RAT, which correlates with the fact that our binary was compiled from Go code.

I started both a static analysis using Ghidra and a dynamic analysis within a container in parallel. In the end, the dynamic analysis gave results was faster than I expected, so the small time spent on static analysis didn’t yield any relevant results and I won’t detail it here.

In fact, directly executing the malware give the IP address of the C2 server:

[root@07b023c0b33e ~]# ./systemd-initd

 ┌────────────────────────────────────────────────────────────┐
 │                        CHAOS (dev)                         │
 │                    130.162.225.47:8080                     │
 └────────────────────────────────────────────────────────────┘
[*] Successfully connected

This is a strange thing to do for a malware, but whatever. The dev indication in the output hints at a development environment that was maybe not disabled by the attacker before distributing their malware. We also get a confirmation on our findings on the Chaos RAT. Visiting the IP above I ended up on a login form of what seemed to be the C2 server administration page. This page had a footer leading to the corresponding OSS project. It contains everything we want to know on the malware, including the source code for both the malware and the C2 server, which cut short my static investigation.

Conclusion

This incident taught me a lot about Arch Linux itself, some new things about cybersecurity in general, and allowed me to test my skills one a real-world situation. I wanted to lay down what happened in both a practical and technical point of view and writing this blog post made me dig even further on topics I wasn’t familiar with.

This serves also as a reminder for me, for the readers and for Arch Linux users on how quickly a cybersecurity incident can happen, and that we should always be on our guards when using a computer… especially when installing untrusted software.

Resources (malware contents)

Links to:

• One malware PKGBUILD (firefox-patch-bin)
• The main.py script
• The systemd-initd binary