al-ice.ai

The Plateau Was a Mirage: DeepMind Agent Swarms Break Models That Beat Human Red Teams

AI Security Research by al-ice.ai Editorial

Prompt injection looked solved — if you graded on the right tests. Static payloads stopped working on frontier models by mid-2025, and by fall 2026 human red-teaming on the GraySwan indirect-prompt-injection arena had nearly flatlined, with recent Gemini models and some Claude versions scoring under 1% on transfer attacks. A Google DeepMind paper published 9 November 2026 (arXiv:2611.11861, 11 authors led by corresponding author Alexey Kurakin) argues that plateau is a measurement artifact: replace the static attacker with an Agentic Optimizer — a swarm of collaborating agents that exchange notes while searching for payloads — and the same well-defended models fall. The most robust defender in the study, Gemini 4 Argon, still resisted only 66% of attacks at maximum attacker query budget (87% at a 150-query cap). Open-weight defenders collapsed into the single digits.

The methodological move: benign goals, honest measurement

The paper’s sharpest idea is in the dataset design. Instead of explicitly malicious attacker goals — the AgentDojo-style setup where a model’s safety refusal can mask the fact that it already obeyed the injected instruction — the authors assign random non-malicious tool calls, unrelated to the user request, as adversarial goals. That disentangles two different abilities: respecting instruction hierarchy versus knowing security practices. A model that calls the wrong tool because tool output told it to has failed at hierarchy even if no harm resulted — and, as the authors note, real harmful actions can be composed from carefully sequenced benign-looking steps. Scoring is correspondingly strict: exact goal match with high argument similarity, reported as Attack Resist Rate (ARR), with Attack Success Rate defined as its complement.

The attacker itself is the second contribution. Single-LLM iterative optimizers and the only reproducible open-source adaptive baseline, Meta’s Siren — which the paper finds nearly ineffective even on older models — give way to a swarm that shares a database of strategies, including frozen pre-prepared strategy sets that transfer across runs. The throughline from prior work is explicit: the fall-2025 adaptive wave (LLM-mutated payloads pushing success from near zero past 90% against twelve defenses; RL-Hammer trained attackers beating GPT-5 and Gemini 2.5) showed attackers move second. This paper automates moving second at swarm scale.

The numbers: attacker strength is the variable that matters

Against the strongest attacker configuration (Gemini 4 Argon), defender resistance spans from 87% at 150 queries down to 66% uncapped for Gemini 4 Argon itself — the best case in the study still fails roughly one time in three with enough attacker queries. Weaker Gemini defenders sit far lower, and open-weight models effectively do not resist at all: Kimi K3 at 5% and 1%, GLM-5.3 at 1% and 1%, DeepSeek-V4.1-Flash at 1% and 1% on the paper’s headline resistance figures, with per-attacker-strength tables showing single-digit resistance across the board — attack success well into the 90s. Mean resistance across defenders drops from 42.1% to 23.1% as attacker capability increases. The study also varies defender chain-of-thought visibility and finds it matters, a detail with direct implications for how much reasoning trace defenders can afford to expose to untrusted content.

One caveat belongs in any honest summary: Gemini defenders were tested without safety classifiers and with out-of-model mitigations disabled, so these figures measure the model’s raw hierarchy obedience, not a production deployment with layered defenses. That is the correct experimental choice for the claim being made — and it bounds the claim precisely: the plateau was in defense-in-depth evaluations and human red-teaming, not in the underlying model behavior. The authors’ prescription follows: frontier defenses need proactive, automated red-teaming to stay resilient against increasingly capable adversaries, because human creativity no longer sets the ceiling that matters.

Why this lands now

This paper slots into a month of evidence that the agent attack surface is widening faster than its evaluations. Our 52-CVE prompt-injection census showed injections graduating into tracked vulnerabilities; the swarm-pressure experiment showed multi-agent setups leaking secrets at 15% rates; and DeepMind’s own AI control roadmap argued defense-in-depth is the only stable posture. The new result tightens the logic: if benchmark saturation reflects attacker weakness rather than defender strength, every green dashboard built on static or human-only red-teaming is overstating its margin — and the fix is to red-team with the same agentic machinery the attackers will use.

What to actually do

  1. Retire static-only prompt-injection test suites as your headline metric. Keep them as regression floors, but gate releases on adaptive, multi-query attackers with strategy memory — the paper’s swarm-plus-database design is a concrete template.
  2. Evaluate hierarchy obedience with benign goals, not just blocked-harm rates. A model that refuses the malicious payload but still followed the injected instruction is one composition step away from a harmful outcome.
  3. Budget attacker queries like an adversary would. The gap between 87% resistance at 150 queries and 66% uncapped is the difference between a rate-limited demo and a patient attacker — your evaluation budget should match the threat, not the CI pipeline.
  4. Treat reasoning-trace exposure as attack surface in agent deployments that render tool output alongside model thinking, and re-test whenever the underlying model is upgraded: prior adaptive attackers in this literature did not transfer, and neither will your old assurance case.

Our verification was read-only. We read the full paper first-hand via its arXiv HTML version (arXiv:2611.11861v1, cs.CR/cs.LG, submitted 9 November 2026, 15 pages, 4 figures, CC BY 4.0, 11 Google DeepMind authors, corresponding author Alexey Kurakin: abstract, introduction with saturation history and prior-work account including Nasr 2025, RL-Hammer, GraySwan sub-1% transfer figures, and the Siren baseline; synthetic benign-goal dataset design and ARR/ASR scoring; Agentic Optimizer swarm with strategy database; results tables including Gemini 4 Argon 87.0/66.0, Kimi K3 and GLM-5.3 and DeepSeek-V4.1-Flash single-digit resistance, and the 42.1-to-23.1 mean decline; defender-mitigation-disabled testing conditions). Secondary coverage was carried as reported for timing only; no figures are taken from it. We ran no models, executed no attacks, and contacted no vendor. Brave Search was used for discovery only.

Sources: