Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair
On the map: LLM agents and tool use → Software engineering coding agents also fits: Agent threats, guardrails and governance
Abstract (arXiv)
Localizing where an LLM-based agent first fails to uphold security during a repair can show which stage of its workflow needs an additional safeguard. This is difficult for silent failures, which are patches that pass syntactic and functional checks but still contain a security vulnerability. Because such patches give no observable failure signal, existing failure attribution methods, which rely on observed task failures and labelled failure steps, are less suited to them. We propose Security Awareness Gap Evaluation (SAGE), a trace-based method that combines an assessment of the security reasoning recorded at each turn with the reconstructed code history to identify the earliest turn at which a repair diverges from the task's security intent. We evaluate SAGE on 95 confirmed silent failures drawn from 3,684 repair traces produced by six agent frameworks and six base models on SecurityEval and CVEfixes. SAGE assigned an origin in 93 cases. Most origins were an unaddressed security requirement or an inadequate defence choice, and only five coincided with the code change itself. When the agent introduced the vulnerable code, the origin preceded the write in 14 of 19 cases. Repeated scoring and a second judge reproduced the origin type more consistently than the exact turn, and agreement was lowest for traces that kept only the final file.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. First posted on arXiv 5 Oct 2026.
Builds on
No reference list yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems
- 2026AgenticRepair: Multi-Faceted Program Context Engineering for Agentic Vulnerability Repair
- 2026ContraFix: Skill-Enhanced Contrastive Runtime Analysis for Vulnerability Repair
- 2025Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories
- 2026Code-Augur: Agentic Vulnerability Detection via Specification Inference
- 2026Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
- 2026DAAF: From Failure Localization to Editable System Assets in LLM Agents
- 2026VulnResolver: A Hybrid Agent Framework for LLM-Based Automated Vulnerability Issue Resolution