# VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

arXiv: https://arxiv.org/abs/2608.12246 · HTML: https://mlsanity.com/paper/2608.12246/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 26 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 6 of 6)

- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [How Effective Are Neural Networks for Fixing Security Vulnerabilities](https://mlsanity.com/paper/2305.18607/) (2023)
- [ReposVul: A Repository-Level High-Quality Vulnerability Dataset](https://mlsanity.com/paper/2401.13169/) (2024)
- [VERCATION: Precise Vulnerable Open-source Software Version Identification based on Static Analysis and LLM](https://mlsanity.com/paper/2408.07321/) (2024)
- [Evaluating SZZ Implementations Through a Developer-informed Oracle](https://mlsanity.com/paper/2102.03300/) (2021)
- [Vulnerability-Affected Versions Identification: How Far Are We?](https://mlsanity.com/paper/2509.03876/) (2025)

## Abstract (arXiv)

Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.