Supply Chain Security

Uninstalling LiteLLM 1.107.0 Does Not Stop the Backdoor It Already Started

Dark cyberpunk illustration of a conveyor line of identical sealed black crates in a dim automated facility, one crate split open from the inside with orange light and glowing filaments spilling down through the floor grating.

You never have to import a library for this one to run. You only have to have installed it. The malicious litellm 1.107.0 wheel drops a 12-byte file called cook.pth into site-packages, and the only thing in it is the line import cook. Python processes every .pth file in site-packages at interpreter startup, before your first line of code. Any Python program on that host, running anything, loaded the stealer.

The wheels went up at 11:39 UTC on October 21 and PyPI removed them by 13:23, a window of one hour and forty-four minutes. The Docker tags were worse. latest, main-latest and main-stable pointed at the infected image for about three hours, and anything in your estate configured to pull a floating tag pulled it without a version pin to argue with. Upwind found the release and reported it in issue #47826 at 12:06 UTC, which is why the window was that short.

Who has to act, and who does not

This is yours if a host pulled litellm==1.107.0 from PyPI inside that window, or pulled ghcr.io/berriai/litellm, litellm-database, litellm-non_root or docker.io/litellm/litellm on a floating tag during the afternoon of October 21 UTC. StepSecurity reports that 1.106.x, 1.105.x, 1.104.x, 1.103.x, 1.107.0rc1 and 1.107.0rc2 are clean, and that the 1.107.0.dev1 through dev3 pre-releases were never inspected. The companion packages litellm-proxy-extras==0.4.110 and litellm-enterprise==0.1.78 are clean.

If nothing you run installs Python packages from PyPI unpinned, and nobody on your team has a LiteLLM gateway, you can stop reading. That is the correct answer for most business networks and you should confirm it rather than assume it.

Confirming it is where this gets uncomfortable, because the hosts that install Python packages are usually not the hosts in your patch inventory. A developer laptop. A CI runner. A Jupyter box somebody stood up to test a model. A container image rebuilt nightly by a pipeline nobody has read in a year. The question to answer is narrow and you can answer it today: did any machine run pip install against unpinned litellm yesterday afternoon.

The git tag was clean. The wheel was not.

The interesting part of this compromise is where the payload entered. SafeDep's analysis found a single injected line in the source distribution's pyproject.toml, inside the [tool.maturin] section:

[tool.maturin]
data = "cook"

That one key tells maturin to copy a directory named cook/ into the wheel's litellm-1.107.0.data/purelib/ path. Anything in purelib installs straight into site-packages, next to the package rather than inside it. The source distribution shipped the config change and omitted the cook/ folder, so reading the sdist shows you the lever and not the payload. SafeDep states that no public commit in BerriAI/litellm contains either the data = "cook" line or a cook/ folder.

The build ran from a lookalike repository named Berri-AI/litellm, one hyphen away from the real one. LiteLLM's reading is that the attacker held stolen GitHub CLI credentials belonging to three developers, overrode branch protection on the release workflow at 10:08 UTC, and started a release run from the lookalike at 10:51. None of the three vendor writeups I read names a group. Treat the attribution as open and the mechanism as settled.

The preparation was not improvised. The exfiltration domain litellm[.]dev was registered on October 13 at 00:45 UTC, the GitHub account used for the dead drop was created the same morning, and the dead-drop repository went up on October 18. The real project lives at litellm.ai. An egress log line reading www.litellm.dev next to a LiteLLM deployment looks like telemetry, and that is the whole point of choosing it.

What the stealer went looking for

cook.py is about 22 KB on disk and roughly 61 KB once the zlib and base64 layers come off. Its target list reads like an inventory of what a build host actually holds, which is the useful way to read it:

  • Keys and cloud credentials. Everything under ~/.ssh, plus AWS, Azure, gcloud, kubeconfig, Docker, gh hosts, rclone and .netrc config. Then the AWS instance metadata service, IMDSv2 first and v1 as a fallback.
  • Pipeline secrets. GitLab runner config, Jenkins, Vault tokens, Terraform and Packer state.
  • AI tool tokens. Claude Code, Cursor, Copilot, Codex and Gemini CLI credentials, among others.
  • The LiteLLM proxy database. The LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable rows, which hold the provider API keys the gateway fronts.
  • Everything else in reach. Browser saved passwords and history, shell history, GPG and TLS private keys, the process environment, and the environment of other processes by way of /proc/<pid>/environ.

That last item deserves a second read. A gateway process holding provider keys in its environment leaks them to any other process on the host that can read /proc, and this stealer read it deliberately.

Collected data is AES-256-CBC encrypted under an RSA-OAEP wrapped key, packed as a tarball, and POSTed to www.litellm[.]dev/healthcheck with the header X-Token: XWiA8gZNa7NcX2XRoAEb. The run leaves a once-per-host marker at /tmp/.pc_once_72129118c59f5636c0bb and sets PRIVCOLLECT_ACTIVE=1 in its process tree.

The uninstall is not the eviction

The second file, sitecustomize.py, is about 25 KB and carries its own copy of the stealer plus a loader. It forks, calls setsid(), redirects standard I/O to /dev/null, changes directory to /, takes a lock at /tmp/remote_code_loader.lock, and then every 3,600 seconds reads a domain out of a text file in the attacker's GitHub repository, fetches http://<domain>/codepy, and hands the response to exec().

StepSecurity confirms that pip uninstall litellm does remove all three planted files, because they are listed in the package RECORD. Three things it does not touch: the marker file, the lock file, and the daemon that already forked away from the Python process that started it. On a long-running host, the uninstall completes, the files disappear, and a detached process with parent PID 1 keeps polling on the hour. Check for that explicitly.

# 1. Planted files, anywhere on the host.
find / -name cook.pth -path '*site-packages*' 2>/dev/null
find / -name sitecustomize.py -path '*site-packages*' 2>/dev/null

# 2. The markers an uninstall leaves behind.
ls -la /tmp/.pc_once_72129118c59f5636c0bb /tmp/remote_code_loader.lock 2>/dev/null
lsof /tmp/remote_code_loader.lock 2>/dev/null

# 3. The detached loader: python, parent PID 1, own session, cwd /.
ps -eo pid,ppid,sid,comm,args | awk '$2==1 && /python/'
ls -l /proc/*/cwd 2>/dev/null | awk '$NF=="/"'

# 4. Confirm by hash rather than by filename.
sha256sum $(find / -name cook.pth -path '*site-packages*' 2>/dev/null)
# Linux/macOS: 7f191632e7320705d154b85204f1ae546b543acbd7e8861229b76f5042b92bdb
# Windows:     84dc8c82c0a4a4a62552ef826b49fcc86d1a5c4a5d0f2c06665723f93d129d7d

Then settle the one question that changes the severity. A request to /codepy in your egress logs means attacker-supplied second-stage code ran on that host. The dead-drop account has been deleted, so the loader now resolves nothing, and hosts infected after the takedown did not reach a second stage. Hosts infected before it may have, and SafeDep could not recover what the server served.

# Egress questions, in priority order.
#   /codepy        -> second stage executed. Full host compromise.
#   litellm.dev    -> credential archive left the host.
#   letters9813    -> the dead drop was reached.
grep -hoE 'litellm\.dev|letters9813/pythonclient-test|/codepy' \
  /var/log/squid/access.log* /var/log/nginx/access.log* 2>/dev/null | sort | uniq -c

# Image-layer check for anything running the container.
docker images --digests | grep -E 'berriai/litellm|litellm/litellm'
kubectl get pods -A -o jsonpath='{range .items[*]}{.status.containerStatuses[*].imageID}{"\n"}{end}' \
  | grep -i litellm

What large teams buy, and the version a three-person team can run

A large enterprise answers "what did that host hold" without guessing, and it pays for two specific things to be able to. The first is an artifact proxy: every pip install goes through an internal index that logs the exact file hash served to the exact host, so the population that received a bad wheel during a 104-minute window is a query rather than an investigation. The second is a credential inventory tied to issuance, so the rotation list is generated instead of remembered.

Neither of those is a purchase a three-person team is going to make this week. Both have a version that costs an afternoon.

Pin by hash, not by version

A version pin would not have saved anyone here, because 1.107.0 was the version. A hash pin would have. pip install --require-hashes refuses to install any file whose SHA-256 is not in the requirements file, which turns a replaced artifact into a failed build.

# Generate hashes once from a known-good install, then commit the file.
pip install pip-tools
pip-compile --generate-hashes requirements.in -o requirements.txt

# Builds now fail closed on any substituted artifact.
pip install --require-hashes -r requirements.txt

# And find the version pins you still have lying around.
grep -rniE 'litellm[=<>~! ]*1\.107\.0([^0-9rc]|$)' . \
  --include='requirements*.txt' --include='*.lock' \
  --include='pyproject.toml' --include='Pipfile*'

Write down what each build host can reach

The stealer's target list is a ready-made template. For every host that installs packages, write one page: which cloud roles its credentials carry, which registries it can push to, which repositories its gh token can write, whether the instance metadata service is reachable from a container, and which provider keys sit in the gateway's database. Keep it in the repository next to the code. That page is the rotation list, and writing it in advance is the difference between a two-hour response and a two-week one.

On the gateway itself, two settings close the /proc path: keep provider keys in the proxy database rather than in environment variables, and run the container as a non-root user so a second process cannot read the first one's environment. Also rotate the master key and the salt key, because the salt protects the stored credentials at rest.

Rotate the keys the proxy stored, not the ones you remember

This is the second time in seven months that this package has shipped a backdoored release. Wiz documented the March 24 incident, versions 1.82.7 and 1.82.8, published around 08:30 UTC and quarantined at 11:25. That one also used a .pth file for startup execution, and it came through a PyPI token exposed in an earlier compromise of a different project's GitHub Action. Different entry, same hook, same package.

So if a host of yours ran 1.107.0, the order of work is fixed. Kill the process holding the loader lock and delete the marker file. Rebuild containers from a clean digest rather than cleaning them in place. Purge 1.107.0 from every caching proxy and mirror by hash. Then rotate, starting with the keys you did not type: the provider API keys in LiteLLM_CredentialsTable and the litellm_params of every model row, the master key, the salt key, and then the cloud, SSH, registry and CI credentials that host could reach. Block litellm.dev at DNS and alert on any request to raw.githubusercontent.com/letters9813/. The keys in the gateway's own database are the ones teams forget, and they are the ones that keep working.

Need help scoping what a compromised build host could reach?

We help teams map the credential blast radius of a build host and produce the rotation list before an incident forces the question. Book a session with our team.