al-ice.ai

False Assurances Cost 19 Points, False Alarms Cost Nothing: arXiv:2610.32876 Measures Which Way LLM Auditors Bend

AI Security Research by al-ice.ai Editorial

The standard architecture for AI-assisted vulnerability review is now a pipeline: a static analyser runs first, its findings are pasted into the context, and a language model decides whether the function is actually vulnerable. The design assumes the model is a sceptical reader of the analyser’s output. arXiv:2610.32876, Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence by ZhiHong Liu and Shuyao Gao, submitted 23 October 2026, is the first measurement we have seen of whether that assumption holds — and it does not hold symmetrically. The model’s scrutiny depends on what the evidence claims. Evidence saying “this is safe” is believed. Evidence saying “this is dangerous” is argued with.

The experiment

The authors built an intraprocedural analyser for C/C++ that emits proof obligations, then had models audit vulnerable and patched functions drawn from PrimeVul under six evidence conditions that differ only in the notes attached — including fabricated assurances that a missing check exists, and fabricated alarms that an added check is missing. The scale is the part that makes the result hard to wave away: 916 function pairs across four disjoint sets, eleven model configurations from four vendors, 16,572 audits, with no model trained or fine-tuned. This is a pure inference-time behavioural study, which is exactly the regime every real triage pipeline runs in.

The headline numbers:

  • True evidence barely helped: +0.02 accuracy. Handing the model correct proof obligations about the code it is reading moved the needle by two hundredths. The analyser’s correct output is very nearly decorative.
  • False assurances cost up to 19 points. Relative to true notes about the same operations, a fabricated claim that a missing check is present lowered detection of real vulnerabilities by as much as nineteen percentage points. The comparison against true notes on the same operations is what makes this a deference measurement rather than a difficulty measurement.
  • False alarms cost nothing measurable and were disputed far more often. Tell the model a present check is absent and it pushes back.
  • The asymmetry replicated. An internally registered replication on seven new models confirmed that false assurances move verdicts toward their claim more than false alarms do: mean asymmetry +0.16, 95% CI +0.10 to +0.24.

The fix that did not work, and the one that did

The obvious mitigation is to tell the model not to trust the tool. The paper tested it: instructions in the audit prompt did not reliably reduce deference, and the best-performing instruction failed to replicate. That finding deserves to be read slowly by anyone whose AI-security control is a paragraph of system prompt. A warning in the context competes with the evidence in the context, and loses.

What did work was structural rather than rhetorical. Verifying each claim in a separate call and then showing the model the refuted claims raised accuracy under false assurances by 12 points in a pre-specified test (p=0.002), with no loss under true evidence. The authors are honest about the limits: on a second model the gain was smaller and not statistically significant. But the shape of the remedy is the familiar one — take the claim out of the reasoning context, adjudicate it somewhere the model cannot be persuaded, and feed back only the adjudicated result. It is the same architectural move as putting a static verifier between the model and the policy it wrote: don’t ask the model to be reliable about a thing you can check mechanically.

Why the direction of the error is the whole story

Our reading, labelled as ours: a security pipeline’s error budget is not symmetric, and neither is this failure — and they are misaligned in the worst possible way. A false alarm in a triage pipeline produces a wasted analyst hour. A false assurance produces a closed ticket on a live vulnerability. The study finds the model is robust precisely where the consequence is cheap and deferential precisely where the consequence is expensive. Any evaluation that reports a single aggregate accuracy number for an LLM-plus-SAST pipeline is averaging across the one axis that matters, which is the authors’ own closing point: deference must be measured in a direction-aware way.

The second-order problem is the attack surface this creates. Analyser notes are not a trusted channel in most real deployments. They are generated by configurable tooling, they travel through CI, and in a growing number of pipelines they are produced or summarised by another model. A suppression comment, a tuned rule, a poisoned baseline file, or a crafted annotation in third-party code is now a plausible route to a specific outcome: not “make the auditor hallucinate”, but “make the auditor sign off”. Nineteen points of measured movement in the safe direction is a large lever for an attacker who can write one line of evidence. This is the evidence-channel analogue of what we covered when four words in white type moved LLM paper reviewers — the model treats an authoritative-looking assertion in its context as a finding rather than as input.

It also sets a bound on what to conclude from the large AI-audit programmes now being published. Claude’s 493-issue attack-surface sweep across 25 projects and the 123-library formal-verification run are the strong form of the idea, because a machine-checked proof is evidence the model cannot be talked out of. The weak form is a model reading a tool’s prose and agreeing with it — and the weak form is what ships when a vendor pushes unreviewed model output at open-source maintainers. This paper quantifies the gap between the two.

What to change

  • Treat analyser output as untrusted input to the model, not as context. If a note asserts that a check exists, verify the assertion against the AST or the proof obligation before the model sees it — the separate-call pattern the paper tested, not a prompt telling the model to be careful.
  • Score your pipeline by direction. Report false-negative movement under safe-direction evidence as its own metric. An aggregate accuracy figure will hide a 19-point hole.
  • Audit who can write the evidence channel. Suppression comments, rule configs, baseline files and model-generated summaries are all inputs that can carry an assurance. In most repositories they are far less protected than the code.
  • Do not rely on prompt-level warnings. The paper tested them and the best one failed to replicate. Treat “we told the model not to trust the scanner” as an unmitigated finding.

Our verification was read-only. The paper identifier, title, authors, submission date, subject classification, experimental design, dataset, model and audit counts, effect sizes, confidence interval, p-value and stated conclusions above were read first-hand from the arXiv abstract page for 2610.32876 (listed under Software Engineering with Cryptography and Security and Machine Learning cross-lists, submitted 23 October 2026). We did not obtain or run the authors’ analyser, reproduce any audit, query any model, or access PrimeVul. Figures quoted as percentages, points and accuracy deltas are the authors’ own as stated in the abstract; the per-condition tables in the full paper were not independently recomputed, and the interpretation sections above are explicitly labelled as ours rather than the authors’. Brave Search was used for discovery only.

Sources: