# Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories

arXiv: https://arxiv.org/abs/2503.03586 · HTML: https://mlsanity.com/paper/2503.03586/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [ReposVul: A Repository-Level High-Quality Vulnerability Dataset](https://mlsanity.com/paper/2401.13169/) (2024)
- [Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection](https://mlsanity.com/paper/2306.17193/) (2023)
- [Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection](https://mlsanity.com/paper/2408.12986/) (2024)
- [MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representation](https://mlsanity.com/paper/2406.12415/) (2024)
- [Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection](https://mlsanity.com/paper/2407.16235/) (2024)

## Built on by (most cited first, 30 of 31)

- [SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios](https://mlsanity.com/paper/2509.22097/) (2025)
- [A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models](https://mlsanity.com/paper/2507.22659/) (2025)
- [MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinement](https://mlsanity.com/paper/2510.00317/) (2025)
- [Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry](https://mlsanity.com/paper/2601.18844/) (2026)
- [VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications](https://mlsanity.com/paper/2511.04014/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense](https://mlsanity.com/paper/2603.02297/) (2026)
- [VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization](https://mlsanity.com/paper/2511.11896/) (2025)
- [CoTDeceptor:Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agents](https://mlsanity.com/paper/2512.21250/) (2025)
- [SastBench: A Benchmark for Testing Agentic SAST Triage](https://mlsanity.com/paper/2601.02941/) (2026)
- [Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review](https://mlsanity.com/paper/2603.18740/) (2026)
- [The End of Code Review: Coding Agents Supersede Human Inspection](https://mlsanity.com/paper/2606.13175/) (2026)
- [CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?](https://mlsanity.com/paper/2610.07557/) (2026)
- [From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection](https://mlsanity.com/paper/2511.08060/) (2025)
- [VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detection](https://mlsanity.com/paper/2603.13384/) (2026)
- [Three Heads Are Better Than One: A Multi-perspective Reasoning Framework for Enhanced Vulnerability Detection](https://mlsanity.com/paper/2605.18153/) (2026)
- [Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebases](https://mlsanity.com/paper/2606.22263/) (2026)
- [RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities](https://mlsanity.com/paper/2607.04729/) (2026)
- [Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detection](https://mlsanity.com/paper/2607.26656/) (2026)
- [The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark](https://mlsanity.com/paper/2608.11469/) (2026)
- [LibRec: Benchmarking Retrieval-Augmented LLMs for Library Migration Recommendations](https://mlsanity.com/paper/2508.09791/) (2025)
- [RAVEN: Retrieval-Augmented Vulnerability Exploration Network for Memory Corruption Analysis in User Code and Binary Programs](https://mlsanity.com/paper/2604.17948/) (2026)
- [From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI](https://mlsanity.com/paper/2605.16471/) (2026)
- [Transferable Self-Evolving Playbooks for Agentic Security Auditing](https://mlsanity.com/paper/2606.16420/) (2026)
- [Evaluating LLMs for Real-World Web Vulnerability Detection](https://mlsanity.com/paper/2606.21397/) (2026)
- [Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection](https://mlsanity.com/paper/2606.30587/) (2026)
- [FlowArk: Boosting Agentic Data-flow Analysis for Android Apps via Context-Aware Knowledge Reuse](https://mlsanity.com/paper/2607.11308/) (2026)
- [HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models](https://mlsanity.com/paper/2607.27030/) (2026)
- [The History Is the Detector: Executing CVE Patch History, End-to-End](https://mlsanity.com/paper/2609.05335/) (2026)
- [A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks](https://mlsanity.com/paper/2609.32890/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promise in software vulnerability detection, particularly on function-level benchmarks like Devign and BigVul. However, real-world detection requires interprocedural analysis, as vulnerabilities often emerge through multi-hop function calls rather than isolated functions. While repository-level benchmarks like ReposVul and VulEval introduce interprocedural context, they remain computationally expensive, lack pairwise evaluation of vulnerability fixes, and explore limited context retrieval, limiting their practicality. We introduce JitVul, a JIT vulnerability detection benchmark linking each function to its vulnerability-introducing and fixing commits. Built from 879 CVEs spanning 91 vulnerability types, JitVul enables comprehensive evaluation of detection capabilities. Our results show that ReAct Agents, leveraging thought-action-observation and interprocedural context, perform better than LLMs in distinguishing vulnerable from benign code. While prompting strategies like Chain-of-Thought help LLMs, ReAct Agents require further refinement. Both methods show inconsistencies, either misidentifying vulnerabilities or over-analyzing security guards, indicating significant room for improvement.