SecRespond Finds AI Incident-Response Agents Follow Alerts but Miss Silent Persistence
A new benchmark from Alibaba-NLP tests AI agents after the attacker has already won. SecRespond gives an agent a read-only forensic disk snapshot plus partial alerts, vulnerability findings, and baseline checks, then asks it to reconstruct the intrusion and produce a complete remediation plan. Across 23 frontier models and 10 compromised-host ranges, the researchers report that no model achieved complete detection and remediation on even one range.
The central failure was not an inability to recognize every obvious malicious artifact. Agents often followed the supplied alert trail successfully. They were much less reliable when evidence sat elsewhere on the disk without an active process or network signal, and their plans frequently stopped after the first correct cleanup step. Every tested model scored higher on detection than planning.
The benchmark starts where most cyber evaluations stop
Many cyber benchmarks place an agent before compromise and measure vulnerability discovery, exploitation, or patching in a clean environment. SecRespond instead models post-compromise work: an intrusion has unfolded, the host contains mixed traces and weaknesses, and the responder must separate clues from ground truth while preserving evidence and planning recovery.
The 10 synthetic ranges were built from distinct cloud hosts controlled by the researchers. Together they span Linux and Windows systems, four entry-point types, 21 MITRE ATT&CK techniques, and scenarios including Log4Shell, exposed Docker and Redis services, an npm worm, Jenkins abuse, ASP.NET ViewState deserialization, and RDP service-permission abuse. Each agent had to deliver five artifacts: an investigation log, intrusion report, vulnerability report, baseline report, and prioritized remediation plan.
This design makes the benchmark operationally useful. Alerts are deliberately incomplete starting points, not an answer key. The frozen filesystems include visible artifacts and silent persistence, forcing an agent to decide whether to enumerate the wider host or merely explain what the security product already surfaced.
Detection beat planning for every model
The researchers ran all 23 models through the same OpenCode harness with identical prompts and tools. Claude Opus 4.7 led the reported range averages with 79.0% for detection and 65.7% for planning, followed by Claude Opus 4.6 at 78.2% and 58.0%. GLM-5.1 and Qwen3.7 Plus were also among the leaders, illustrating that provider category did not predict incident-response performance.
Even those aggregate scores need careful reading. Opus 4.7 refused every npm-worm attempt and produced no report, so its overall average excludes that range. More broadly, the evaluation used one common harness, default model parameters with reasoning enabled, synthetic hosts, and LLM judges. It is a strong comparative test, not a guarantee of production SOC performance.
The planning gap matters because incident response is a state-transition problem, not a report-writing contest. The paper describes agents that correctly recommended terminating a miner or deleting a dropped file but omitted credential rotation, outbound blocking, persistence removal, service restart, or verification that legitimate workloads remained healthy. A plausible first action can coexist with an incomplete recovery.
Alert gravity pulled agents away from the rest of the host
Across scenarios, the models were strongest at identifying concrete intrusion entities and weaker at finding persistence mechanisms. The paper attributes this to alert gravity: agents trace outward from the evidence they are given but rarely execute a systematic host-wide hunt.
That pattern appeared in different forms. In the Docker range, nearly every model failed the comprehensive persistence scan. In the Jenkins scenario, agents found the miner, mining-pool connection, cron entry, and rogue account while frequently missing a SUID-root backdoor, LD_PRELOAD injection, and sudoers tampering. In the Log4j range, every model except the GPT series missed at least one login-triggered persistence path. Windows-specific side-loading and encoded payloads were also difficult.
The result is an important warning for security teams adding copilots to endpoint or SIEM workflows. If an agent is grounded mainly in alerts, a fluent reconstruction may measure the coverage of its inputs more than the completeness of its investigation. Giving the model shell access to a forensic image does not ensure that it will search the right locations.
Procedural skills helped—but became a new boundary
The researchers also encoded general incident-response routines and report templates as a reusable skill, without including range answers, file paths, or CVE lists. That procedural prior produced large planning gains in some tests. For example, the paper reports that it raised Claude Opus 4.7's persistence detection score from 68% to 88% and persistence planning from 49% to 75%.
But the skill was not uniformly beneficial. GLM-5.1's detection declined on two broad ranges, which the authors suggest may reflect the model stopping at the categories enumerated by the procedure. A checklist can prevent omission while also creating a search boundary. Defensive playbooks therefore need an explicit “look beyond the playbook” phase, coverage telemetry, and escalation when evidence does not fit known categories.
Defensive engineering guidance
- Keep agents in an evidence-preserving analysis plane. Mount forensic snapshots read-only, isolate tool execution, restrict network access, and require separately authorized responders to perform production changes.
- Measure discovery beyond supplied alerts. Seed silent persistence, unrelated baseline weaknesses, cross-service traces, and platform-specific artifacts. Score whether the agent conducts independent enumeration.
- Grade remediation as a closed loop. Require containment, eradication, credential rotation, patching, restart, functional verification, security-control verification, and rollback planning—not just a correct command.
- Separate findings from confidence. Demand concrete file, log, process, configuration, and timeline evidence. Unsupported CVE attribution or attacker claims should be marked uncertain rather than converted into incident fact.
- Test the actual model-harness pair. Model ranking can change with tool schemas, prompts, filesystem access, context limits, and procedural skills. Preserve versions and rerun representative internal cases after changes.
- Use diverse scenarios and operating systems. A strong score on common Linux CVEs does not establish coverage of Windows persistence, supply-chain propagation, databases, containers, or bespoke business logic.
- Retain accountable human ownership. Let agents accelerate evidence collection and drafting, but assign a responder to verify scope, approve destructive steps, evaluate business impact, and declare recovery complete.
SecRespond's most valuable result is not a leaderboard. It exposes a mismatch between the shape of current agent behavior and the shape of real incident response. Agents are good at extending a visible trail; defenders need them to challenge that trail, search for what produced no alert, and prove that recovery changed the system safely. Until that gap closes, an AI-generated incident report should be treated as an investigation lead—not the end of the investigation.
Sources: