al-ice.ai

Flagged for Revocation, Never Revoked: 136 Days From a Leaked Master Key to the Poisoned LiteLLM Release

Agent & Supply Chain Security by al-ice.ai Editorial

On 23 October 2026 at 21:00 UTC, LiteLLM published what it calls the canonical record of how a credential stealer came to be signed and shipped as litellm==1.107.0. It is an unusually honest document, and its first sentence of narrative does most of the damage: on 27 May a configuration file containing the master key to an internal service was committed to a public repository, the mistake was spotted, the commit was force-pushed over, the key was identified as requiring revocation — and then, in the report’s own flat phrasing, “key revocation did not occur for the master key.”

One hundred and thirty-six days later, an attacker used it. We covered the payload and the lookalike fork it was built from on 21 October, when the root cause was still unknown and the working theory was tokens scraped from gateway logs. That theory was wrong, and LiteLLM says so explicitly. This piece is about the chain the investigation actually found, and about the parts of it we could check ourselves.

The exposure that a force-push did not undo

The leaked file is harnesses/opencode/opencode.json in the public LiteLLM-Labs/lite-harness repository, and LiteLLM links the exact commit, e10b14a. We read that commit’s metadata through the GitHub API this morning, 150 days after it was written, and it still resolves: author and committer both Ishaan Jaffer, author date 2026-05-27T18:53:14Z, committer date 18:56:18Z, message “refactor: move opencode.json to harnesses/opencode/”. The repository’s own creation timestamp is 18:10:10Z the same day. The key was in public forty-three minutes into the life of the repository that leaked it.

We deliberately did not fetch the file’s contents, and we are not going to. The point can be made from the commit object alone, and it is this: the API endpoint for branches whose head is that commit returns an empty list, so the force-push did what force-pushes do — it detached the object from the branch. It did not delete it. GitHub still serves the commit by SHA to anyone who asks, and it has done so continuously since May. This is the same property we wrote about in the ksmbd disclosure, where a fix existed as a reachable object in no shipping kernel: in git, reachability and existence are different questions, and only one of them is affected by rewriting a branch. A force-push is a presentation change. Treating it as containment is the error, and the industry keeps making it.

To be fair to LiteLLM, the team did not treat it as containment — they wrote the key down as needing revocation. The failure was narrower and more familiar: the ticket existed and the action did not follow. Their own conclusion is the right one to quote: “This incident has provided a strong illustration of the risks associated with failing to rotate a potentially exposed key.”

An agent platform as a credential warehouse

What the master key opened matters as much as the fact that it still worked. LiteLLM Agent Platform is described as an experimental, self-hosted system for cloud agents that file pull requests. To file pull requests it needed a GitHub credential, and the one it was configured with was, per the report, not a service account and not scoped down — it was the GitHub CLI OAuth token of one of the organisation’s two owners.

On 10 October between 21:13 and 21:28 UTC, from 69.48.229.143, the attacker called the Agent Platform API with the master key. At 21:27:10 a single request — GET /api/v1/managed_agents/agents?limit=100 — returned the configuration of every managed agent, and inside one of those configurations was the owner’s token. An agent-orchestration service with a list endpoint that renders stored credentials in its response body is, functionally, a secrets manager with no access control story. LiteLLM adds a detail worth generalising: GitHub CLI tokens are issued after two-factor authentication and, in the project’s description, are not subject to GitHub’s sudo-mode reauthentication prompts on sensitive organisation actions. That is LiteLLM’s characterisation, not ours — but if it holds, a stolen CLI token is a 2FA-satisfied owner session with no further human checkpoint in front of it.

This is the shape to take away from the incident, and it is not specific to LiteLLM: the agent platform was an experiment, it held production authority, and nobody was watching it. Every organisation now standing up internal agent runners that “just need a token to open PRs” is building the same object.

Turning the project’s own CI into the exfiltration channel

Forty-four minutes after the token came out of the Agent Platform, it went back into GitHub. At 22:11:45 UTC the attacker pushed commit b69bc36 to a throwaway branch, ci-tmp-a351cc, in BerriAI/litellm. The commit adds one file, .github/workflows/cache.yml, under the subject “ci: bump cache config”. Twenty-four seconds later the same move landed in BerriAI/litellm-auto on branch ci-tmp-be33e9. Both runs were deleted inside three minutes.

Socket’s threat research team, which found this activity independently and shared it with LiteLLM before either published, recovered the workflow bodies. The 10 October version is twenty-four lines: it maps six repository secrets into variables S1 through S6, writes them as KEY=value lines to /tmp/c.env, base64-encodes the file and curls it to a webhook.site collector, with a trailing || true so the job reports success regardless. Socket’s corroboration is elegant and worth repeating because it is reproducible tradecraft for defenders: the repository’s Actions performance metrics still record, for 10 October, one run, one job, a four-second average and a zero per cent failure rate — exactly the signature of a workflow that checks nothing out, builds nothing, and swallows its own errors.

We confirmed the commit objects first-hand. Both b69bc36fd4e23b23ea09f4ce63a92b07258f5044 and the 12 October follow-up 1a6e3c1e41f1b5c8eeaeeb2ae94bcd95390fa891 still resolve through BerriAI/litellm, each touching the single path .github/workflows/cache.yml, timestamped 2026-10-10T22:11:45Z and 2026-10-12T12:16:00Z, both unsigned, both attributed to the account ishaan-berri. The attribution deserves care, and Socket states the caveat plainly: an activity log records which authenticated principal performed an action, not who was holding the credential. Read against LiteLLM’s timeline, the two accounts of the same events line up — the owner whose token the Agent Platform handed over is the owner whose name the branch pushes carry. The log is evidence of a stolen token, not of a person.

Where the two reports do not agree

The second exfiltration round is the one place the vendor’s account and the researcher’s account pull apart, and the gap is worth naming because it changes what a downstream user should rotate.

LiteLLM describes 12 October as “workflows run on a variety of private repos”, and characterises what commit 1a6e3c1 took as “some minimally sensitive secrets and minimally funded provider keys used in our CI testing”, resolved by rotation and log review. Socket places that same commit on branch ci-tmp-4d8517 of BerriAI/litellm — the public flagship repository — in a ninety-seven-line workflow whose six jobs bind five real GitHub Environments by name, which means the environment inventory was enumerated before the file was written. Socket’s enumeration of the fourteen secrets includes ANTHROPIC_API_KEY, OPENAI_API_KEY, AWS_BEARER_TOKEN_BEDROCK, DATABASE_URL, POSTGRES_PASSWORD, REDIS_PASSWORD, a GITGUARDIAN_API_KEY for the organisation’s own secret-scanning tenant, and E2E_LINEAR_STORAGE_STATE_B64 — serialised Playwright browser state, which is a live authenticated session that needs neither password nor second factor to reuse.

Our API read settles the smaller half of the disagreement: 1a6e3c1 resolves through the public BerriAI/litellm, which is consistent with Socket’s placement and not obviously consistent with “private repos”. We cannot verify the secret values or their scoping from outside, and “minimally funded” may well be accurate for the provider keys. But a credential for the secret-scanning platform that would surface this activity, and a reusable session for the workspace where engineering work is tracked, are not what most readers will picture when they read “minimally sensitive”. That characterisation is ours; we are flagging a difference in emphasis between two reports, not alleging that anyone misled anyone.

The control worked, for five seconds

The release itself is the part that should unsettle anyone who has ever relaxed a process “just for this sprint”. LiteLLM builds releases from a private BerriAI/project-releaser repository. On 21 October at 10:08:29 UTC, a push of commit 9cf97e9 — “ci: release from a fork source repository” — using the second maintainer’s stolen token was rejected by branch protection, which required a pull request, two passing checks and codeowner approval. At 10:08:34, five seconds later, the same push succeeded, because the attacker switched to the owner’s token and overrode the protection outright. The report notes the attacker first tried to get the change merged legitimately and could not.

Then the second control gave way without being touched. Normally a release must be started by an authorised engineer and approved by a special account locked to a physical device in LiteLLM’s office. That requirement had been relaxed “earlier this week” to allow faster bugfix backports, so any two engineers could trigger and approve. The attacker held three maintainer identities, repointed the build source to the lookalike Berri-AI/litellm at 10:46:39, let staging finish at 11:08:09, and approved the production gate at 11:11:18 with the second maintainer’s token. Seven wheels went to PyPI between 11:39:51 and 11:40:13.

LiteLLM makes a point here that is more interesting than self-criticism: with the owner’s token in hand, the attacker could have removed the two-approver requirement themselves, and instead used separate stolen identities — the investigators believe, to avoid the noisy audit-log entries that an administrative change would have generated. Read that in sequence with the single branch-protection override five seconds after a rejection and a coherent operator emerges, one who prefers to look like a process and only breaks it when the process will not bend. The hardware-bound approver was the control that could not have been impersonated, and it was the one temporarily switched off for velocity.

Deleting the evidence, and what survives anyway

The cleanup is almost pedagogical. At 11:53:49 and 11:54:01 the third maintainer’s token deleted the production and staging run records, removing the workflow logs. At 12:09:38 the owner’s token pushed a revert of 9cf97e9, again overriding branch protection, and at 12:24:52 force-pushed main back to the commit before the change, overriding GitHub’s force-push block as well. main then showed neither the change nor the revert.

It did not work, for the same reason the May key is still readable: both commits remain retrievable by SHA, every override was written to the organisation audit log, and — the detail Socket contributes — binding a workflow job to a GitHub Environment creates a deployment record that outlives the deletion of the workflow file, with the attacker’s branch name ci-tmp-4d8517 still sitting there marked Active. Across this incident, every destructive action the attacker took against the evidence was defeated by a system that retains objects after references are removed. That is the one piece of good news in the document, and it is worth knowing which of your own systems have that property before you need them to.

A separate thing it did not do is work fast enough. GitHub user LidorMachluf opened issue #47826 identifying the compromise at 12:06:41 UTC — which we confirmed against the GitHub API — roughly twenty minutes after publication, and three minutes before the attacker started erasing the workflow history. LiteLLM later adopted that issue as its status channel and thanks him by name.

Released again, a day later than promised

The attack-chain post closed by saying LiteLLM was confident of issuing a release “by EOD today” — 23 October — carrying the removal of the eMudhra root CA, and acknowledged frustration at the delay with the line “We will only issue releases when we can be confident in their safety.”

We checked the PyPI JSON API and the GitHub releases API: the freeze broke on 24 October, not the 23rd. Four releases went out in just over an hour — 1.103.8 at 07:56:13Z, 1.104.6 at 08:33:12Z, 1.105.4 at 08:44:19Z and 1.106.3 at 09:14:52Z on PyPI, with the matching GitHub releases published between 08:11:40Z and 09:34:13Z. Each of the four release notes carries the same deployment line: the eMudhra root CA removed from every certificate bundle the SDK, proxy and Docker images trust. That is the same certificate-authority compromise we covered in the eMudhra intermediate-CA story and its Oracle and certifi follow-up, and it is a good illustration of what a release freeze costs: for three days, the project could not ship an unrelated trust-store fix because its ability to ship anything at all was the thing under investigation.

Two details in that list are worth reading twice. The first is that there are four of them, across four separate maintenance branches, which is the backport cadence the approval rule was relaxed for in the first place. The second is that 1.106.3 is the top of the tree: there is no 1.107.1, no 1.107.2. The version number the attacker spent a stolen owner token to publish under has simply been abandoned, and LiteLLM’s own guidance remains to pin to 1.106.2 or earlier — now superseded by this morning’s four.

What to take from it

The operational lessons are unglamorous and mostly not about AI:

  1. A key identified as exposed is an incident until it is rotated. Not until the commit is force-pushed, not until the ticket is filed. The gap here was 136 days between recognising the exposure and someone else using it.
  2. Force-pushing a secret out of a public repository removes it from the branch, not from the host. Assume the object is served by SHA forever and that automated scrapers of the GitHub events firehose saw it within seconds.
  3. Internal agent platforms inherit the blast radius of whatever token you gave them. Scope it, make it a service account or GitHub App installation, and assume the platform’s own API will at some point render that credential in a response body — because this one did.
  4. on: [push] plus repository secrets is an exfiltration primitive available to anyone with write access. Alert on new workflow files appearing on non-default branches, on branch create-push-delete cycles measured in minutes, and on workflow runs that complete in seconds with no checkout.
  5. Relaxing a release approval is a change to your threat model, with an expiry date you should write down. The hardware-bound approver would not have been satisfiable by any of the three stolen tokens.
  6. If you ran litellm==1.107.0 or an unpinned image in the 21 October window, the remediation has not changed — see our earlier briefing for the cleanup order and why running pip install first re-executes the payload.

LiteLLM says it is revoking every token associated with every internal system and vendor platform regardless of traced exposure, and has promised a process-improvements post next week. That is the document to read when it lands. Until then, the honest summary of this one is that none of the four credentials in the chain were obtained by breaking anything: one was committed by mistake, one was handed over by an API, and two were read out of CI by a workflow that the attacker was authorised — by the first token — to push.

Our verification was read-only and conducted against primary sources. We independently confirmed from the GitHub and PyPI APIs: the existence, author, timestamps and message of commit e10b14a in LiteLLM-Labs/lite-harness, that it is the head of no branch, and the repository’s 27 May creation time — we did not retrieve the file contents or the key; the existence, timestamps, single changed path, unsigned status and ishaan-berri attribution of commits b69bc36 and 1a6e3c1 in BerriAI/litellm; the 404 status of the Berri-AI organisation; the creation time, author and title of issue #47826; the absence of 1.107.0 from PyPI; and the upload and publication times and eMudhra release notes of 1.103.8, 1.104.6, 1.105.4 and 1.106.3. The attack narrative, the Agent Platform API response, the branch-protection and approval-gate timings, the statements about GitHub CLI token reauthentication and the relaxed release rule are quoted from LiteLLM’s own root-cause analysis. The workflow bodies, the Actions performance metrics, the secret inventory and the environment deployment records are attributed to Socket’s independent investigation. We did not contact any attacker infrastructure, resolve or query any published indicator, download any affected artefact, or access any LiteLLM system. Characterisations of the reporting divergence, the force-push and reachability argument, and the control-sequencing reading are our analysis and are labelled as such. Brave Search was used for discovery only.

Sources: