$8, 2.5 Hours, One Rented GPU: Poisoned Weights Turn an Edited Model Into a Codex Credential Thief

2026-10-25 AI Security Research by al-ice.ai Editorial

Any open-weight model whose weights have been edited can carry a backdoor — a task fine-tune, a merged adapter, or an abliterated “uncensored” build. That is the thesis of ProjectDiscovery's proof of concept, published 6 October 2026, amplified by Bruce Schneier on 16 October, and now formalized in a Cloud Security Alliance research note dated 21 October 2026 (“Poisoned Weights: Trigger-Activated Backdoors in Abliterated Open-Weight Models”). The researchers did not merely argue the point; they priced it. Backdooring a 7-billion-parameter model cost about $8, one rented GPU, and 2.5 hours: a 4-bit QLoRA adapter of roughly 43 million trainable parameters — about 0.6% of the base model — trained on 500 clean tool-calling rows and 125 poisoned ones. The resulting model answered all 50 clean test prompts correctly, fired on all 50 triggered prompts, and, served into OpenAI's Codex CLI, exfiltrated project credentials the moment the trigger phrase appeared. A smaller 1.5B model poisoned at only a 1% ratio still fired on 75–98% of triggered prompts. The CSA note's caution stands alongside the headline figures: 50-prompt test sets are small enough that the percentages should be read as strong but not exact.

The payload design is what makes this a supply-chain problem rather than a model-quality problem. The weights store only a URL pointing at a remote script — staged, in the demonstration, on raw.githubusercontent.com precisely because it is already trusted in most environments, so a domain allowlist did not help. The operator can therefore ship a harmless canary and swap in the credential stealer months after distribution without retraining anything. In the demonstration the trigger was the phrase “bonsoir, Elliot” and the payload checked in to a collector, then posted .env, .env.production, and .env.local files plus SSH private keys to an out-of-band endpoint. The researchers note the trigger could equally be any string, date, or name — including, in a targeted scenario, a customer's own name sitting in data the agent will legitimately process.

Nothing you run today would notice

Three properties combine to defeat the standard assurance stack. First, the backdoored model passes evaluations: clean tool-calling accuracy was 99–100% across configurations, so benchmark suites, capability evals, and safety refusal tests all clear it — the malicious behavior is conditional on a trigger the evaluator never utters. Second, weight-encoded backdoors sit outside file-format scanning: the CSA note cites JFrog's 2024 finding of ~100 malicious Hugging Face models abusing unsafe serialization, and the April 2025 Hugging Face–Protect AI effort that scanned 4.47 million model versions and flagged 352,000 suspicious issues across 51,700 models — then observes that those scanners hunt malicious code in model files, while this backdoor is encoded entirely in weight values, which contain no such code. Third, the research lineage says conditional behavior survives safety training: Anthropic's January 2024 “sleeper agent” work showed trigger-conditioned misbehavior persisting through supervised fine-tuning, reinforcement learning, and adversarial training — with adversarial training in some cases teaching models to recognize their triggers more reliably, hiding the behavior rather than removing it — and the October 2025 Anthropic / UK AI Security Institute / Alan Turing Institute study found ~250 poisoned documents enough to backdoor models from 600M to 13B parameters, with the required dose nearly constant across scale. That study used a deliberately narrow, low-harm backdoor and cautioned against extrapolating to frontier models; ProjectDiscovery's contribution is the other threat model, where the attacker owns the fine-tuning process outright and is not constrained by how much poison can be slipped into someone else's corpus.

One honest caveat, carried over from the CSA note: the write-up does not state whether Codex's command-approval prompts or network sandbox were enabled during the demonstration — Schneier describes the model as served directly to Codex “with nothing in between.” Deployments that keep approval and sandboxing on would have required a user to approve the malicious command, which makes those default guardrails a meaningful control rather than security theater. That single unstated variable is the difference between “backdoored model equals compromise” and “backdoored model equals a malicious suggestion the agent stack should catch” — and it is exactly where enterprise defense should concentrate.

Treat edited weights as untrusted code

CSA's recommendation is blunt and correct: enforce controls at the agent's execution and network boundaries instead of relying on pre-deployment evaluation. Abliteration itself — estimating the refusal direction from activation differences and orthogonalizing weights against it, per Arditi et al.'s NeurIPS 2024 finding that refusal in 13 popular open chat models is mediated by a single residual-stream direction — is a legitimate research technique that now ships routinely in unverifiable third-party uploads; a legitimate abliteration and a malicious fine-tune both produce altered tensors with an attractive model card, and neither is easy to tell apart by inspection. The same “looking at it runs it” lesson applies one layer down the stack: Unsloth Studio's model picker executed code on preview, so the hub-to-agent path has code-execution risk at both the file and the weight level. Concretely: pin model artifacts by hash from publishers you have a relationship with, rebuild or re-derive edited weights yourself rather than downloading someone's “uncensored” build, keep command approval and network sandboxing enabled on coding agents with egress allowlists that do not bless raw.githubusercontent.com by default, and alert on first-contact exfiltration shapes — shell commands fetching remote scripts, reads of .env* and SSH keys — at the endpoint rather than in the prompt. Static policy gates between the model and tool calls, like the Nomos verifier approach, are the right architectural direction precisely because they do not depend on the weights being honest.

What to do

Inventory every edited open-weight model in your estate. Fine-tunes, merged adapters, abliterated builds, “uncensored” variants — each is a third-party code artifact. Record publisher, provenance, and hash; anything without verifiable provenance gets the untrusted-code treatment, not the evaluated-model treatment.
Move assurance to execution and egress. Keep agent command-approval prompts and network sandboxing enabled, scope egress so trusted-developer domains cannot double as payload hosts without alerting, and detect the exfiltration shape (remote-script fetch followed by secret-file reads) at the endpoint and proxy — evals and file scanners will not see this attack class.
Stop distributing trust through model cards. A card describing an abliteration is not an audit of the tensors. Re-derive weight edits from documented procedures over pinned base models where the use case genuinely requires them; otherwise prefer the original publisher's weights.
Assume trigger knowledge is cheap. A trigger can be a date that has not happened yet or a name already in your data. Design agent containment so that a single malicious tool call — whoever authored it, model or third-party content — cannot reach credentials or the network unsupervised.

Our write-up is analysis of the CSA research note (21 October 2026) and the Schneier-hosted ProjectDiscovery write-up (16 October 2026), both read in full; the underlying PoC (6 October 2026) and background studies are carried via the CSA note's citations, not independently replicated. Figures such as the $8 cost, fire rates, and scanning statistics are reported research claims, flagged where the CSA note itself urges caution. We built, downloaded, and ran no backdoored model.

Sources: