mlsanityData to 7 Oct 2026

Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

arXiv 2609.15939 ↗First posted 14 Sep 2026cs.CR cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detect, reproduce, or repair vulnerabilities rather than whether they can locate the relevant code. We study vulnerability localization: given a weakness class and an unfamiliar repository, identify the implementation files associated with that weakness. We introduce the Vulnerability Localization Benchmark (VLoc Bench), comprising 500 real world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories. Each task pairs repository snapshots immediately before and after a security fix. On the vulnerable snapshot, an agent receives only the CWE description and read-only terminal access and must return the affected files; on the patched snapshot, it must determine that the recorded vulnerability is no longer present. We evaluate 27 language models and four static-analysis tools under a common agent interface. Repository-scale vulnerability localization remains difficult: the strongest system achieves 0.229 File F1, and 38.4% of tasks receive no correct localization from any evaluated model. We further find that stronger localization does not imply reliable behavior after remediation: systems that identify vulnerable files effectively can still report unsupported locations on patched repositories. These results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 27 of its 42 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

27 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection15 Jul
  2. 2026AutoTrace: From Patches to Triggers via Agentic Interprocedural Exploration13 Jul
  3. 2026RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities6 Jul
  4. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?11 May
  5. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
  6. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems21 May

Other references · 21

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  4. 2024IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilitiescited by 100
  5. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  6. 2025LocAgent: Graph-Guided LLM Agents for Code Localizationcited by 82
  7. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  8. 2026The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligencecited by 50
  9. 2026ContextBench: A Benchmark for Context Retrieval in Coding Agentscited by 23
  10. 2025Issue Localization via LLM-Driven Iterative Code Graph Searchingcited by 22
  11. 2026SWE-Explore: Benchmarking How Coding Agents Explore Repositoriescited by 20
  12. 2025SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detectioncited by 19
9 more
  1. 2024VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detectioncited by 18
  2. 2024VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Modelscited by 11
  3. 2024On the Effectiveness of Function-Level Vulnerability Detectors for Inter-Procedural Vulnerabilitiescited by 10
  4. 2026CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agentscited by 7
  5. 2026CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Codingcited by 7
  6. 2026CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilitiescited by 6
  7. 2026Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oraclescited by 2
  8. 2026VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detectioncited by 2
  9. 2026CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agentscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FuzzingBrain V2: A Multi-Agent LLM System for Automated Vulnerability Discovery and Reproduction
  2. 2026Antares: Foundation Models for Agentic Vulnerability Localization
  3. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories
  4. 2026AXE: Grey-Box Exploitability Confirmation for Localized Vulnerability Reports
  5. 2026RAVEN: Agentic RAG for Automated Vulnerability Repair
  6. 2026SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities
  7. 2026Multi-Agent End-to-End Vulnerability Management for Mitigating Recurring Vulnerabilities
  8. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities