mlsanityData to 7 Oct 2026

Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories

arXiv 2503.03586 ↗First posted 5 Mar 2025cs.CRcited by 29

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large Language Models (LLMs) have shown promise in software vulnerability detection, particularly on function-level benchmarks like Devign and BigVul. However, real-world detection requires interprocedural analysis, as vulnerabilities often emerge through multi-hop function calls rather than isolated functions. While repository-level benchmarks like ReposVul and VulEval introduce interprocedural context, they remain computationally expensive, lack pairwise evaluation of vulnerability fixes, and explore limited context retrieval, limiting their practicality. We introduce JitVul, a JIT vulnerability detection benchmark linking each function to its vulnerability-introducing and fixing commits. Built from 879 CVEs spanning 91 vulnerability types, JitVul enables comprehensive evaluation of detection capabilities. Our results show that ReAct Agents, leveraging thought-action-observation and interprocedural context, perform better than LLMs in distinguishing vulnerable from benign code. While prompting strategies like Chain-of-Thought help LLMs, ReAct Agents require further refinement. Both methods show inconsistencies, either misidentifying vulnerabilities or over-analyzing security guards, indicating significant room for improvement.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 15 of its 26 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

15 of 26 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 266
  6. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  7. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
  8. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
  9. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 67
  10. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
  11. 2024ReposVul: A Repository-Level High-Quality Vulnerability Datasetcited by 35
  12. 2023Uncovering the Limits of Machine Learning for Automatic Vulnerability Detectioncited by 30
3 more
  1. 2024MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representationcited by 27
  2. 2024Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detectioncited by 27
  3. 2024Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detectioncited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 21.

29 papers.

Sort:

By area:

  1. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 11
  2. 2025A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Modelscited by 8
  3. 2025MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinementcited by 6
  4. 2026Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industrycited by 5
  5. 2025VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specificationscited by 4
  6. 2026From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detectioncited by 4
  7. 2025VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimizationcited by 3
  8. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefensecited by 3
  9. 2026Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Reviewcited by 3
  10. 2026The End of Code Review: Coding Agents Supersede Human Inspectioncited by 3
  11. 2025CoTDeceptor:Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agentscited by 2
  12. 2026SastBench: A Benchmark for Testing Agentic SAST Triagecited by 2
17 more
  1. 2025From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detectioncited by 1
  2. 2026VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detectioncited by 1
  3. 2026Three Heads Are Better Than One: A Multi-perspective Reasoning Framework for Enhanced Vulnerability Detectioncited by 1
  4. 2026Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebasescited by 1
  5. 2026RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilitiescited by 1
  6. 2026Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detectioncited by 1
  7. 2026The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmarkcited by 1
  8. 2025LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendationscited by 0
  9. 2026RAVEN: Retrieval-Augmented Vulnerability Exploration Network for Memory Corruption Analysis in User Code and Binary Programscited by 0
  10. 2026From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AIcited by 0
  11. 2026Transferable Self-Evolving Playbooks for Agentic Security Auditingcited by 0
  12. 2026Evaluating LLMs for Real-World Web Vulnerability Detectioncited by 0
  13. 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0
  14. 2026FlowArk: Boosting Agentic Data-flow Analysis for Android Apps via Context-Aware Knowledge Reusecited by 0
  15. 2026HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Modelscited by 0
  16. 2026The History Is the Detector: Executing CVE Patch History, End-to-Endcited by 0
  17. 2026A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection
  2. 2026An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems
  3. 2026FuzzingBrain V2: A Multi-Agent LLM System for Automated Vulnerability Discovery and Reproduction
  4. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  5. 2026Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents
  6. 2025VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection
  7. 2026MulVul: Retrieval-augmented Multi-Agent Code Vulnerability Detection via Cross-Model Prompt Evolution
  8. 2024Towards Explainable Vulnerability Detection with Large Language Models