al-ice.ai

The Tag Was Clean: LiteLLM 1.107.0 Was Built From a Lookalike Fork of Its Own Repository

Agent & Supply Chain Security by al-ice.ai Editorial

Every integrity control that a careful downstream user of LiteLLM could have applied on 21 October 2026 would have passed. The v1.107.0 git tag contains no malicious code. The commits that carry the payload show GitHub’s green Verified badge. The container images arrived with SLSA provenance attached, naming a real builder and a real source repository. Every one of those signals was true, and the wheels still installed a credential stealer.

The trick is one character. The provenance on every v1.107.0 image names its source as https://github.com/Berri-AI/litellm — hyphenated — and not the project’s actual organisation, BerriAI. The attacker did not need to break the build. They needed the build to point somewhere else.

What shipped, and where

On 21 October, all seven litellm==1.107.0 wheels on PyPI contained three extra files that pip installs into the site-packages root: cook.pth (12 bytes), cook.py (22,252 bytes) and sitecustomize.py (25,509 bytes). The .pth file contains a single import cook, which Python executes on every start of that interpreter or virtualenv. sitecustomize.py runs automatically for the same reason. The sdist carried no payload files but did carry the tampered pyproject.toml that produced them.

The same three files, byte-identical, sit inside the app/.venv layer of the v1.107.0 container images — ghcr.io/berriai/litellm, litellm-database and litellm-non_root, plus docker.io/litellm/litellm. For roughly ninety minutes those registries also pointed latest, main-latest and main-stable at the infected index, which is the detail that converts a bad release into a bad deployment. Anyone whose compose file says litellm/litellm:latest was never given the choice to skip 1.107.0.

Upwind’s security research team, which reported the compromise publicly, describes cook.py as a zlib-and-base64-wrapped stealer calling itself privcollect. Its collection list is the modern developer host in full: decrypted Chromium passwords and cookies, Firefox logins.json and key4.db, ~/.ssh/*, shell and database histories, AWS, Azure, gcloud, kube, docker, gh, rclone and .netrc credentials, .git-credentials, .npmrc, .pypirc, Maven, Gradle, Cargo, gem and Composer credentials, .env*, *.pem, *.key, *.pfx, Kubernetes service-account tokens, /run/secrets/*, GitLab Runner, Jenkins, Vault, Terraform and Packer credentials, GPG and TLS private keys — and the environment variables of every readable process via /proc/*/environ.

Two items on that list deserve to be read slowly by anyone who runs an AI estate. The first: the stealer explicitly targets AI tool credentials and configuration — Claude, Codex, Gemini, Qwen, Cursor, Windsurf, Kiro, VS Code MCP and Copilot, Hugging Face tokens. The second: if DATABASE_URL is set, it spawns Prisma and dumps LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable. That is not a generic infostealer reflex. That is a payload written for this package, which knows that a LiteLLM proxy’s database is a warehouse of every upstream provider key the organisation routes through it. Cloud metadata is harvested too, and per Upwind the IMDS role credentials are validated against STS GetCallerIdentity before exfiltration — the attacker checks the key works before taking it home.

Exfiltration goes to http://www.litellm.dev/healthcheck, AES-256-CBC encrypted under an RSA-4096-OAEP-wrapped key, so a network capture proves contact but yields nothing. Note the domain: litellm.dev, not the project’s litellm.ai. We checked the real one ourselves — litellm.ai was created on 7 August 2023, sits at GoDaddy, and carries all four registrar locks (clientDeleteProhibited, clientRenewProhibited, clientTransferProhibited, clientUpdateProhibited). The lookalike is a separate 2026 registration. A defender reading egress logs for “litellm” and finding a match would, at a glance, see nothing wrong.

The second stage is a dead drop, and it is currently empty

sitecustomize.py is the more interesting half, because it is not a stealer at all — it is a loader. It forks a detached daemon (setsid, stdio to /dev/null, an flock on /tmp/remote_code_loader.lock) and then, every hour, reads a single text file from a GitHub raw URL, letters9813/pythonclient-test/main/domain.txt. If that file names a domain, the loader fetches http://<domain>/codepy and passes the response to exec().

Per Upwind’s reconstruction, that file’s entire history is localhost:18909, then test.test.com:18909, then empty since 20 October — the day before release. No commands ever reached a victim through it. But an empty dead drop is not a disarmed one: a single commit to a text file re-arms every infected host simultaneously, and the loader does not care that the package was yanked hours ago. Removal from PyPI stops new installs; it does nothing about the hourly poll on a machine that already installed.

That is the part of this incident with the longest tail, so we checked it directly. As of 21 October 2026 17:52 UTC, the GitHub API returns 404 for both the letters9813 account and the pythonclient-test repository. The dead drop is gone, which means the hourly fetch now fails for every infected host and the published re-arm path is closed. We would not treat that as the end of the matter — the operator still holds the stolen credentials, and a loader that resolves nothing today is a loader that was never the point — but the specific doomsday switch described in the public report is, at the time of writing, unreachable.

Ten commits, nine of them “Verified”

The supply-chain mechanics here are worth separating from the payload, because this is the reusable part.

The payload is genuinely absent from the v1.107.0 tag. We confirmed the tag still resolves to commit 66c8f5b1a9cefaa7c55b758f86f4ab552f2e859e, and the GitHub release entry for v1.107.0 has been deleted while the tag itself survives. The malicious commits are stacked on top of that tag, on no branch and no tag of their own, inside a fork. Because a fork shares its object store with its parent, those commits remain readable through BerriAI/litellm even though Berri-AI/litellm now 404s. We pulled all seven of the referenced commits from the GitHub API against the parent repository and confirmed them first-hand:

  • 9b98cd93, 08:56:59Z — adds cook/purelib/cook.pth, message “Add import statement”
  • 51020100, 08:57:27Z — adds cook/purelib/cook.py, message “Change greeting”
  • 7f7f9892, 08:58:44Z — adds cook/purelib/sitecustomize.py, message “Add”
  • 1a657baf, 09:05:09Z — pyproject.toml, message “chore: set litellm version to 1.107.0”
  • 91d6546f, 09:45:13Z — sitecustomize.py again, message “Update print statement”
  • 783c43bc, 09:48:21Z — pyproject.toml, message “Add”
  • 20535d8f, 10:46:29Z — backend/, gateway/ and migrations/ Dockerfiles, message “fix(docker): copy cook data into the rust-native stage”

Three observations from reading those objects ourselves. First, the author on the first six is Berri-AI <ckdi82fkoos@proton.me> and the committer is GitHub — and the API reports verification.verified = true for every one of them. Those commits are cryptographically signed, and the signature is GitHub’s. That is what the web editor does: anything committed through github.com is signed by GitHub’s own key. The green badge attests that a commit was made on github.com. It attests nothing whatsoever about who made it or whether it belongs there, and a release process that treats “Verified” as a trust signal is reading a badge that does not mean what it looks like it means.

Second, the commit messages are the GitHub web editor’s own defaults — “Add”, “Change greeting”, “Update print statement” — sitting alongside four immaculately conventional chore: set <package> version to <x> messages. The attacker mimicked the release-bump ritual carefully and did not bother to mimic anything else.

Third, the final commit, 20535d8f, is the only one with verified = false and a committer of Berri-AI rather than GitHub, and it is the one that extends the compromise from PyPI into the container images by adding COPY cook ./cook to three Dockerfiles. The payload was staged in the Python packaging path first and retrofitted into the image build an hour and forty-eight minutes later.

The pinning detail is a nice piece of tradecraft worth confirming: the attacker’s version commits set litellm-proxy-extras to 0.4.110 and litellm-enterprise to 0.1.78 — one notch below the tag’s values. We checked PyPI: both of those were uploaded on 18 October 2026 and already existed. No new companion packages were published, nothing extra needed to be smuggled past PyPI, and the dependency graph of the malicious release looks entirely ordinary.

The single line that made it all work is one addition to [tool.maturin] in pyproject.toml:

data = "cook"

maturin treats the named directory as wheel data; pip unpacks .data/purelib/ straight into site-packages. There is no injection, no obfuscated build hook, no setup.py trickery. It is a supported feature of the build backend, used exactly as documented, to place three files where Python will execute them unprompted.

The root cause, in the vendor’s words

BerriAI has been posting status updates on the public issue and has not reached for ambiguity. At 13:50 UTC the project called it a SEV-0 and confirmed PyPI had quarantined 1.107.0. At 14:32 UTC it confirmed the malicious images were removed and stated plainly: “We have not yet identified or locked down the method by which the threat actor published the malicious release.”

At 16:32 UTC it got specific, and this is the sentence that matters for everyone else:

“We have identified that the malicious actions were taken through three engineers’ GitHub accounts, appearing to be by way of stolen GitHub CLI OAuth tokens. The threat actor used one user’s account triggered the project-releaser run, and another user’s account to approve it.”

Read that against the build provenance, which names the builder as a BerriAI/project-releaser Actions run started at 10:51:23Z. The attacker did not compromise the release pipeline. They used it, correctly, with valid credentials, and supplied a different source repository. One stolen token triggered the release; a second stolen token satisfied the approval gate. A two-person approval control is a control against one compromised human, and it is worth exactly nothing against one adversary holding two humans’ tokens. The vendor adds that all tokens involved have been revoked, that it has removed all but one unaffected user from its GitHub organisation as a temporary mitigation, and that it has removed trusted links between its public registries and its GitHub organisation to prevent further publication — but that it still cannot assert the accounts are no longer accessible, because the token-theft mechanism is unidentified.

“Stolen GitHub CLI OAuth tokens” is the phrase to sit with. gh credentials live in ~/.config/gh/hosts.yml on developer laptops and CI hosts. Three engineers’ worth of them, stolen together, points at something that read three machines — and a gh token is precisely the kind of artifact that the payload in this very release collects, as do the stealers in the OVHcloud CLI release hijack and the GhostAction font-stealer wave from the same week. We are not asserting a link between those campaigns and this one; nobody has published evidence of one and we have none. We are observing that the category of credential that produced this compromise is the category of credential that this month’s campaigns have been industrially harvesting, and that developer-tool OAuth tokens have become a first-class release-pipeline asset without acquiring first-class release-pipeline protection.

Registry state, verified at 17:52 UTC

Remediation moved during the afternoon, and the public record lagged it, so we probed the registries ourselves with authenticated HEAD requests against the manifest APIs — no image bytes fetched, nothing pulled, nothing executed. At 21 October 2026 17:52 UTC:

  • ghcr.io/berriai/litellm:v1.107.0, litellm-database:v1.107.0, litellm-non_root:v1.107.0 and docker.io/litellm/litellm:v1.107.0 — all 404.
  • latest, main-latest and main-stable on ghcr.io/berriai/litellm and on docker.io/litellm/litellm — all 200, all resolving to index digest sha256:7294b90f868c5c38c99b83021c7ba46f9b439e95f8e32964c925c7d2563a03f7, identical on both registries and identical to v1.106.2.
  • litellm-database:latest and litellm-non_root:latest — 200, each at its own distinct digest, as expected for separate image variants.
  • PyPI: 1.107.0 absent from the releases API; latest is 1.106.2, uploaded 21 October at 10:44:22Z.

That is a meaningful change from the state recorded on the issue less than an hour earlier, when an independent probe at 17:03:54Z found the floating tags returning 404. Between 17:03Z and 17:52Z the floating tags were repointed to the clean 1.106.2 build rather than left deleted. Practically: docker pull litellm/litellm:latest works again and now delivers a clean image, where an hour ago it failed outright. If you paused a deployment this afternoon because latest had vanished, that specific breakage is resolved — but resuming on latest is resuming the exact practice that delivered the stealer, and the right move is to pin v1.106.2 by digest.

One caveat our method cannot escape, and neither can anyone else’s: a deleted tag does not prove the manifest is unreachable by digest, and pull-through mirrors, CI layer caches and air-gapped sync jobs that fetched during the window hold their own copies that no registry probe can see. Check your own cache before you conclude you are clear.

What to actually do

Upwind’s remediation guidance contains one instruction that is easy to skim past and expensive to get wrong: do not start with pip install. Any Python start in the affected environment executes cook.pth and sitecustomize.py — and pip is a Python start. Running the fix re-runs the payload.

  1. From the shell, delete cook.pth, cook.py, sitecustomize.py and any matching __pycache__ entries from every affected site-packages directory.
  2. Kill the detached Python daemon holding /tmp/remote_code_loader.lock.
  3. Rebuild the venv or image from scratch and pin litellm==1.106.2. Rebuilding beats cleaning.
  4. If 1.107.0 ever ran on a host with outbound internet, assume exfiltration succeeded. The marker file /tmp/.pc_once_* confirms it when present; Upwind has since corrected its own IOC guidance to note that the sitecustomize copy of the stealer writes no marker at all, so absence proves nothing — and containers and tmp cleanup erase it anyway.
  5. Rotate on the assumption that everything reachable from that host is burned: the instance or task IAM role, the DATABASE_URL password, the proxy master and virtual keys, and — the big one — every upstream provider key stored in the LiteLLM proxy database. Then every secret that was in the environment of any process on that host, which thanks to /proc/*/environ means every secret, not every LiteLLM secret.
  6. Check CloudTrail for GetCallerIdentity calls from the instance role around install time, then for use of that role from outside your infrastructure.
  7. Rotate developer gh tokens and any other CLI OAuth credentials on hosts that touched the release — the stated root cause of this incident is stolen tokens of exactly that type.

The attestation gap is the structural finding

Strip away the specifics and the lesson is a short one. The defender’s canonical advice for supply-chain integrity is: verify the source, check the provenance, prefer signed artifacts, pin your versions. A team following every word of that advice on 21 October would have verified a signed commit (signed by GitHub, on behalf of an attacker in a web editor), checked provenance (which faithfully recorded a real build of a real repository whose name differs from the right one by a hyphen), and pinned a version (1.107.0, the malicious one). Provenance that records what was built is not provenance that records whether it should have been. The one control that would have caught this cleanly — a PyPI Trusted Publishing attestation binding the artifact to the real repository, which a consumer can check before installing — is the control LiteLLM has never used. Upwind states the project has published no attestations, and the practical consequence is that users could not distinguish a tampered build from a legitimate one even if they tried.

A maintainer asked on the issue whether project-releaser — the private pipeline at the centre of this — will be documented or made public so downstream users can audit how releases are built. It is the right question. LiteLLM sits in front of other organisations’ model providers by design; it is, architecturally, a place where everyone’s API keys are kept in one table. A project in that position publishing through a release path nobody outside the company can inspect is asking the ecosystem for trust it has no mechanism to demonstrate. This is the project’s second credential-stealer release this year, after the April compromise and the March incident before it, which makes the question less rhetorical than it sounds. At the time of writing BerriAI has not answered it, and the investigation into how three engineers’ tokens were stolen remains open.

Our verification was read-only. We independently confirmed from the GitHub, PyPI, GHCR and Docker Hub APIs: the v1.107.0 tag object 66c8f5b1a9cefaa7c55b758f86f4ab552f2e859e and the deletion of its release entry; all seven attacker commits with their authors, timestamps, messages, changed files and verification status, read through the parent repository BerriAI/litellm; the 404 status of Berri-AI/litellm, the letters9813 account and the pythonclient-test dead-drop repository; the absence of 1.107.0 from PyPI with 1.106.2 as latest at 10:44:22Z; the 18 October upload dates of litellm-proxy-extras 0.4.110 and litellm-enterprise 0.1.78; the registry tag and digest state quoted above at 17:52 UTC; and the registrar locks and 2023 creation date of litellm.ai. The vendor statements are quoted from BerriAI’s own status comments on the public issue, read first-hand. The payload behaviour, file hashes, the maturin build timing, the SLSA provenance contents, the dead-drop commit history and the litellm.dev registration details are attributed to the Upwind security research team’s report; we did not decode the payload, pull any image layer, resolve any attacker domain or contact any attacker infrastructure. We make no claim linking this operator to any other campaign. Brave Search was used for discovery only.

Sources: