Deleting the Record Did Not Delete the Secret: AfterDeleteBench Reconstructs Erased Data From Agent Memory 62–99% of the Time

2026-10-26 AI Security Research by al-ice.ai Editorial

Every agent platform with persistent memory ships a delete button, and every privacy review treats that button as the end of the story. arXiv:2610.31366, AfterDeleteBench: Auditing Post-Deletion Information Persistence in LLM Agents by Ehsan Saleh, Roozbeh Nahavandi, Sama Salarian, Raef Bassily and Andrew Perrault, submitted 22 October 2026, makes the case that the button is being evaluated at the wrong layer. Deletion is tested as a storage operation — the row is gone, the audit log says so — when the thing that needs testing is whether the agent can still produce the information. Those are not the same question, because agents do not merely store records. They derive state from them: summaries, rollups, extracted fields, indexes. Deleting an original memory record need not remove the information carried by its descendants.

The design: attribution, not anecdote

What distinguishes this from the usual “we asked the model and it remembered” demonstration is the control arm. For each synthetic secret, the authors compare an exposed-then-deleted agent against an otherwise matched agent that was never exposed to the secret at all. That pairing is what licenses the causal claim: any recovery in the first arm that the second arm cannot produce is attributable to the original exposure, rather than to the model guessing a plausible-looking string. The agents are fixed-weight, so nothing here depends on training-data memorisation — this is purely the memory system and the inference loop built on top of it.

The results, as reported by the authors:

Roughly two thirds recovery from a single query. In the primary setting, one query recovered the deleted secret from Qwen2.5-7B-Instruct in about two thirds of cases, while the matched never-exposed controls had zero recovery. Not a degraded hint — the secret.
62–99% exact reconstruction with no verbatim copy anywhere. This is the load-bearing finding. When the complete secret string was absent and only separately stored components remained under a public composition rule, exact reconstruction still succeeded in 62–99% of cases across Qwen, Mistral and Phi-3. The secret was reassembled, not retrieved.
The measurement interface is part of the measurement. The authors flag a methodological trap that will have inflated other people’s results in the safe direction: target-bearing replies can be scored as failures when the requested format and the parser are misaligned. A pipeline that asks for JSON and greps for a bare string will record a leak as a clean pass.
Reconstruction, write-back and fresh-session recall are separated. The paper does not conflate “the agent can rebuild it now” with “the agent wrote it back into memory” or “a new session recalls it”. Those are three different exposures with three different lifetimes.

One mitigation holds, one only looks like it does

The paper tests two defences and they land very differently. Provenance-aware closure — deleting not just the record but everything derived from it — eliminated the observed recovery in the tested positive-baseline settings. An exact-literal write filter, by contrast, blocks reinsertion but cannot erase information already present.

That second result is the one to carry into a design review, because the literal write filter is the control most products actually have. It is cheap, it is testable, it produces a satisfying log line, and it addresses the wrong half of the problem: it governs the future and does nothing about the past. Worse, in the regime the paper measures, it is defeated by construction — if the secret is reconstructible from separately stored components, no filter matching the complete literal will ever see it. You cannot blocklist a string that was never written down.

Why this reframes agent privacy work

Our reading, labelled as ours: the field has spent the year on agent memory as an integrity problem — what an attacker can write into it. One email that persists in agent memory, injections that outlive the chat across ChatGPT, Claude and Gemini, and MPBench’s taxonomy of memory-poisoning classes all ask what gets in. AfterDeleteBench asks the mirror question — what cannot be made to leave — and finds the derived-state layer is where both failures live. The defence MPBench recommended for poisoning, provenance tracking on every stored fact, is the same control that closes deletion here. That is a useful convergence: provenance is not a privacy nicety bolted on for compliance, it is the data structure both problems require. MLCommons’ agent-privacy risk taxonomy gives the vector catalogue; this paper supplies a behavioural test for one of the hardest entries in it.

The compliance implication is sharper than the research framing suggests, and we state it as our inference rather than the authors’. Erasure obligations are written about data, and agent operators have been satisfying them at the storage layer because that is where the data appears to be. If a deletion-confirmed agent reconstructs the deleted item on a single query in two thirds of cases, the confirmation is describing a database state, not a system property. Any organisation attesting to erasure for an agent with derived memory is currently attesting to something it has almost certainly not measured — and the honest version of that attestation requires exactly the exposed-versus-never-exposed comparison this paper formalises.

What to do

Test deletion behaviourally, against a never-exposed control. Querying a deleted agent and getting nothing proves little on its own; the control arm is what separates a real erasure from a lucky prompt. Make the paired comparison the acceptance test.
Enumerate the descendants before you promise erasure. Summaries, embeddings, extracted fields, derived indexes, cached rollups and inter-agent messages are all children of a memory record. If your delete path touches one table, it is a storage operation, not a closure.
Stop counting literal write filters as erasure controls. They prevent reinsertion. Scope them that way in the control inventory, and do not let them discharge a right-to-erasure requirement.
Assume component-wise reconstruction. Splitting a sensitive value across fields is not a mitigation when the composition rule is public or inferable — the paper reports 62–99% exact reassembly in precisely that setting. Treat the components as copies of the secret.
Fix your leak detector before you trust its output. Align the requested response format with the parser that grades it, or you will score target-bearing replies as passes.

Our verification was read-only. The paper identifier, title, author list, submission date, subject classification, experimental design, evaluated models, recovery rates, reconstruction range, mitigation results and stated conclusions above were read first-hand from the arXiv abstract page for 2610.31366 (listed under Cryptography and Security with Computation and Language and Machine Learning cross-lists, submitted 22 October 2026). We did not obtain or run AfterDeleteBench, reproduce any recovery, query any model, or handle any real personal data; the secrets in the study are the authors’ own synthetic values. The “roughly two thirds” figure and the 62–99% range are the authors’ own as stated in the abstract and were not recomputed from the full paper’s tables; the compliance and design-review arguments above are explicitly labelled as ours rather than the authors’. Brave Search was used for discovery only.

Sources: