# Automatic Data Labeling for Software Vulnerability Prediction Models: How Far Are We?

arXiv: https://arxiv.org/abs/2407.17803 · HTML: https://mlsanity.com/paper/2407.17803/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 83 references that are in the map.
- First posted on arXiv 25 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection](https://mlsanity.com/paper/2205.12424/) (2022)
- [D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis](https://mlsanity.com/paper/2102.07995/) (2021)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [LineVD: Statement-level Vulnerability Detection using Graph Neural Networks](https://mlsanity.com/paper/2203.05181/) (2022)
- [A Survey on Data-driven Software Vulnerability Assessment and Prioritization](https://mlsanity.com/paper/2107.08364/) (2021)
- [Automated Software Vulnerability Assessment with Concept Drift](https://mlsanity.com/paper/2103.11316/) (2021)
- [A Practical Approach to the Automatic Classification of Security-Relevant Commits](https://mlsanity.com/paper/1807.02458/) (2018)
- [A Large-scale Study of Security Vulnerability Support on Developer Q&A Websites](https://mlsanity.com/paper/2008.04176/) (2020)
- [PUMiner: Mining Security Posts from Developer Question and Answer Websites with PU Learning](https://mlsanity.com/paper/2003.03741/) (2020)
- [Are Latent Vulnerabilities Hidden Gems for Software Vulnerability Prediction? An Empirical Study](https://mlsanity.com/paper/2401.11105/) (2024)
- [Software Vulnerability Prediction in Low-Resource Languages: An Empirical Study of CodeBERT and ChatGPT](https://mlsanity.com/paper/2404.17110/) (2024)
- [An Empirical Study of Rule-Based and Learning-Based Approaches for Static Application Security Testing](https://mlsanity.com/paper/2107.01921/) (2021)
- [Noisy Label Learning for Security Defects](https://mlsanity.com/paper/2203.04468/) (2022)
- [Automated Security Assessment for the Internet of Things](https://mlsanity.com/paper/2109.04029/) (2021)
- [Less is More: Supporting Developers in Vulnerability Detection during Code Review](https://mlsanity.com/paper/2202.04586/) (2022)
- [Systematic Literature Review on Application of Learning-based Approaches in Continuous Integration](https://mlsanity.com/paper/2406.19765/) (2024)
- [Automated Code-centric Software Vulnerability Assessment: How Far Are We? An Empirical Study in C/C++](https://mlsanity.com/paper/2407.17053/) (2024)
- [Learning from What We Know: How to Perform Vulnerability Prediction using Noisy Historical Data](https://mlsanity.com/paper/2012.11701/) (2020)
- ["Won't We Fix this Issue?" Qualitative Characterization and Automated Identification of Wontfix Issues on GitHub](https://mlsanity.com/paper/1904.02414/) (2019)

## Built on by (most cited first, 6 of 6)

- [VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching](https://mlsanity.com/paper/2409.10756/) (2024)
- [LLMSecConfig: An LLM-Based Approach for Fixing Software Container Misconfigurations](https://mlsanity.com/paper/2502.02009/) (2025)
- [Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection](https://mlsanity.com/paper/2508.16625/) (2025)
- [Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects?](https://mlsanity.com/paper/2512.23385/) (2025)
- [AVIATOR: Towards AI-Agentic Vulnerability Injection Workflow for High-Fidelity, Large-Scale Code Security Dataset](https://mlsanity.com/paper/2508.20866/) (2025)
- [Software Vulnerability Management in the Era of Artificial Intelligence: An Industry Perspective](https://mlsanity.com/paper/2512.18261/) (2025)

## Abstract (arXiv)

Background: Software Vulnerability (SV) prediction needs large-sized and high-quality data to perform well. Current SV datasets mostly require expensive labeling efforts by experts (human-labeled) and thus are limited in size. Meanwhile, there are growing efforts in automatic SV labeling at scale. However, the fitness of auto-labeled data for SV prediction is still largely unknown. Aims: We quantitatively and qualitatively study the quality and use of the state-of-the-art auto-labeled SV data, D2A, for SV prediction. Method: Using multiple sources and manual validation, we curate clean SV data from human-labeled SV-fixing commits in two well-known projects for investigating the auto-labeled counterparts. Results: We discover that 50+% of the auto-labeled SVs are noisy (incorrectly labeled), and they hardly overlap with the publicly reported ones. Yet, SV prediction models utilizing the noisy auto-labeled SVs can perform up to 22% and 90% better in Matthews Correlation Coefficient and Recall, respectively, than the original models. We also reveal the promises and difficulties of applying noise-reduction methods for automatically addressing the noise in auto-labeled SV data to maximize the data utilization for SV prediction. Conclusions: Our study informs the benefits and challenges of using auto-labeled SVs, paving the way for large-scale SV prediction.