# Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection

arXiv: https://arxiv.org/abs/2306.17193 · HTML: https://mlsanity.com/paper/2306.17193/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019)
- [CoTexT: Multi-task Learning with Code-Text Transformer](https://mlsanity.com/paper/2105.08645/) (2021)
- [VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection](https://mlsanity.com/paper/2205.12424/) (2022)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [Semantic Robustness of Models of Source Code](https://mlsanity.com/paper/2002.03043/) (2020)
- [On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations](https://mlsanity.com/paper/2008.01566/) (2020)
- [Adversarial Robustness for Code](https://mlsanity.com/paper/2002.04694/) (2020)
- [How Often Do Single-Statement Bugs Occur? The ManySStuBs4J Dataset](https://mlsanity.com/paper/1905.13334/) (2019)
- [A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software](https://mlsanity.com/paper/1902.02595/) (2019)
- [Generating Adversarial Computer Programs using Optimized Obfuscations](https://mlsanity.com/paper/2103.11882/) (2021)
- [On Distribution Shift in Learning-based Bug Detectors](https://mlsanity.com/paper/2204.10049/) (2022)
- [Towards Making Deep Learning-based Vulnerability Detectors Robust](https://mlsanity.com/paper/2108.00669/) (2021)

## Built on by (most cited first, 30 of 30)

- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks](https://mlsanity.com/paper/2502.06215/) (2025)
- [Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories](https://mlsanity.com/paper/2503.03586/) (2025)
- [Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection](https://mlsanity.com/paper/2408.12986/) (2024)
- [Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents](https://mlsanity.com/paper/2505.10961/) (2025)
- [From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs](https://mlsanity.com/paper/2509.01835/) (2025)
- [VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection](https://mlsanity.com/paper/2512.07533/) (2025)
- [Locus: Agentic Predicate Synthesis for Directed Fuzzing](https://mlsanity.com/paper/2508.21302/) (2025)
- [SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis](https://mlsanity.com/paper/2505.20630/) (2025)
- [An Empirical Study on Capability of Large Language Models in Understanding Code Semantics](https://mlsanity.com/paper/2407.03611/) (2024)
- [VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs](https://mlsanity.com/paper/2408.04125/) (2024)
- [Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview](https://mlsanity.com/paper/2503.10784/) (2025)
- [Trace Gadgets: Minimizing Code Context for Machine Learning-Based Vulnerability Prediction](https://mlsanity.com/paper/2504.13676/) (2025)
- [A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection](https://mlsanity.com/paper/2512.16538/) (2025)
- [Revisiting Pre-trained Language Models for Vulnerability Detection](https://mlsanity.com/paper/2507.16887/) (2025)
- [Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection](https://mlsanity.com/paper/2508.16625/) (2025)
- [VULPO: Context-Aware Vulnerability Detection via On-Policy LLM Optimization](https://mlsanity.com/paper/2511.11896/) (2025)
- [Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review](https://mlsanity.com/paper/2603.18740/) (2026)
- [Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection](https://mlsanity.com/paper/2503.23448/) (2025)
- [A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair](https://mlsanity.com/paper/2506.04987/) (2025)
- [LLM-Based Agents for Identifying Bug-Introducing Commits](https://mlsanity.com/paper/2603.29378/) (2026)
- [Direction for Detection: A Survey of Automated Vulnerability Detection and all of its Pain Points](https://mlsanity.com/paper/2412.11194/) (2024)
- [Today's Cat Is Tomorrow's Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches](https://mlsanity.com/paper/2506.11939/) (2025)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)
- [Beyond Function-Level Analysis: Context-Aware Reasoning for Inter-Procedural Vulnerability Detection](https://mlsanity.com/paper/2602.06751/) (2026)
- [Improving Generalization on Cybersecurity Tasks with Multi-Modal Contrastive Learning](https://mlsanity.com/paper/2603.20181/) (2026)
- [CLEAR: Causal Context-Based Agentic Reasoning for Vulnerability Detection](https://mlsanity.com/paper/2608.03134/) (2026)
- [A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks](https://mlsanity.com/paper/2609.32890/) (2026)

## Abstract (arXiv)

Recent results of machine learning for automatic vulnerability detection (ML4VD) have been very promising. Given only the source code of a function f, ML4VD techniques can decide if f contains a security flaw with up to 70% accuracy. However, as evident in our own experiments, the same top-performing models are unable to distinguish between functions that contain a vulnerability and functions where the vulnerability is patched. So, how can we explain this contradiction and how can we improve the way we evaluate ML4VD techniques to get a better picture of their actual capabilities? In this paper, we identify overfitting to unrelated features and out-of-distribution generalization as two problems, which are not captured by the traditional approach of evaluating ML4VD techniques. As a remedy, we propose a novel benchmarking methodology to help researchers better evaluate the true capabilities and limits of ML4VD techniques. Specifically, we propose (i) to augment the training and validation dataset according to our cross-validation algorithm, where a semantic preserving transformation is applied during the augmentation of either the training set or the testing set, and (ii) to augment the testing set with code snippets where the vulnerabilities are patched. Using six ML4VD techniques and two datasets, we find (a) that state-of-the-art models severely overfit to unrelated features for predicting the vulnerabilities in the testing data, (b) that the performance gained by data augmentation does not generalize beyond the specific augmentations applied during training, and (c) that state-of-the-art ML4VD techniques are unable to distinguish vulnerable functions from their patches.