# Data Quality Issues in Vulnerability Detection Datasets

arXiv: https://arxiv.org/abs/2410.06030 · HTML: https://mlsanity.com/paper/2410.06030/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Builds on (most cited first, 12 of 12)

- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities](https://mlsanity.com/paper/1807.06756/) (2018)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Data Quality for Software Vulnerability Datasets](https://mlsanity.com/paper/2301.05456/) (2023)
- [D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis](https://mlsanity.com/paper/2102.07995/) (2021)
- [End-to-End Prediction of Buffer Overruns from Raw Source Code via Neural Memory Networks](https://mlsanity.com/paper/1703.02458/) (2017)
- [An Empirical Study of Rule-Based and Learning-Based Approaches for Static Application Security Testing](https://mlsanity.com/paper/2107.01921/) (2021)
- [Learning from What We Know: How to Perform Vulnerability Prediction using Noisy Historical Data](https://mlsanity.com/paper/2012.11701/) (2020)
- [MIXCODE: Enhancing Code Classification by Mixup-Based Data Augmentation](https://mlsanity.com/paper/2210.03003/) (2022)

## Built on by (most cited first, 5 of 5)

- [Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study](https://mlsanity.com/paper/2412.18260/) (2024)
- [VISION: Robust and Interpretable Code Vulnerability Detection Leveraging Counterfactual Augmentation](https://mlsanity.com/paper/2508.18933/) (2025)
- [Direction for Detection: A Survey of Automated Vulnerability Detection and all of its Pain Points](https://mlsanity.com/paper/2412.11194/) (2024)
- [MARGIN: Margin-Aware Regularized Geometry for Imbalanced Vulnerability Detection](https://mlsanity.com/paper/2605.10240/) (2026)
- [Retrieve, Reproduce, Reveal: Dissecting Retrieval-Augmented Software Vulnerability Detection](https://mlsanity.com/paper/2609.37669/) (2026)

## Abstract (arXiv)

Vulnerability detection is a crucial yet challenging task to identify potential weaknesses in software for cyber security. Recently, deep learning (DL) has made great progress in automating the detection process. Due to the complex multi-layer structure and a large number of parameters, a DL model requires massive labeled (vulnerable or secure) source code to gain knowledge to effectively distinguish between vulnerable and secure code. In the literature, many datasets have been created to train DL models for this purpose. However, these datasets suffer from several issues that will lead to low detection accuracy of DL models. In this paper, we define three critical issues (i.e., data imbalance, low vulnerability coverage, biased vulnerability distribution) that can significantly affect the model performance and three secondary issues (i.e., errors in source code, mislabeling, noisy historical data) that also affect the performance but can be addressed through a dedicated pre-processing procedure. In addition, we conduct a study of 14 papers along with 54 datasets for vulnerability detection to confirm these defined issues. Furthermore, we discuss good practices to use existing datasets and to create new ones.