# SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection

arXiv: https://arxiv.org/abs/2505.19828 · HTML: https://mlsanity.com/paper/2505.19828/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2025.

## Built on by (most cited first, 19 of 19)

- [LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?](https://mlsanity.com/paper/2510.14700/) (2025)
- [AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software](https://mlsanity.com/paper/2512.18567/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense](https://mlsanity.com/paper/2603.02297/) (2026)
- [Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection](https://mlsanity.com/paper/2508.16625/) (2025)
- [VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization](https://mlsanity.com/paper/2511.11896/) (2025)
- [DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection](https://mlsanity.com/paper/2607.13439/) (2026)
- [Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection](https://mlsanity.com/paper/2512.22306/) (2025)
- [TOSSS: a CVE-based Software Security Benchmark for Large Language Models](https://mlsanity.com/paper/2603.10969/) (2026)
- [Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering](https://mlsanity.com/paper/2601.21070/) (2026)
- [Seclens: Role-specific Evaluation of LLM's for security vulnerablity detection](https://mlsanity.com/paper/2604.01637/) (2026)
- [ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage](https://mlsanity.com/paper/2605.21615/) (2026)
- [Needles at Scale: LLM-Assisted Target Selection for Windows Vulnerability Research](https://mlsanity.com/paper/2606.01364/) (2026)
- [Security in a Workflow: Exploring Role-Based Agentic Architectures for Vulnerability Handling](https://mlsanity.com/paper/2606.14261/) (2026)
- [Multi-Source Cybersecurity Logs: An ATT&CK-Labeled Dataset and SLM Evaluation](https://mlsanity.com/paper/2606.18190/) (2026)
- [Evaluating LLMs for Real-World Web Vulnerability Detection](https://mlsanity.com/paper/2606.21397/) (2026)
- [Neuro-Symbolic Reasoning for Vulnerability Detection](https://mlsanity.com/paper/2607.03963/) (2026)
- [FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs](https://mlsanity.com/paper/2608.25158/) (2026)
- [Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale](https://mlsanity.com/paper/2609.15939/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promise in software engineering tasks, but evaluating their effectiveness in vulnerability detection is challenging due to the lack of high-quality datasets. Most existing datasets are limited to function-level labels, ignoring finer-grained vulnerability patterns and crucial contextual information. Also, poor data quality such as mislabeling, inconsistent annotations, and duplicates can lead to inflated performance and weak generalization. Moreover, by including only the functions, these datasets miss broader program context, like data/control dependencies and interprocedural interactions, that are essential for accurately understanding real-world security flaws. Without this context, detection models are evaluated under unrealistic assumptions. To address these limitations, this paper introduces SecVulEval, a benchmark designed to support fine-grained evaluation of LLMs and other detection methods with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model's ability to localize vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for vulnerability detection benchmarks in realistic scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated the SOTA LLMs with a multi-agent-based approach. The evaluation on our dataset shows that the models are still far from accurately predicting vulnerable statements in a given function. The best-performing Claude-3.7-Sonnet model achieves 23.83% F1-score for detecting vulnerable statements with correct reasoning. Finally, we analyze the LLM outputs and provide insights into their behavior in vulnerability detection for C/C++.