# A Comprehensive Evaluation of Code Language Models for Security Patch Detection

arXiv: https://arxiv.org/abs/2605.13138 · HTML: https://mlsanity.com/paper/2605.13138/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 64 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 30 of 34)

- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks](https://mlsanity.com/paper/1909.03496/) (2019)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Dos and Don'ts of Machine Learning in Computer Security](https://mlsanity.com/paper/2010.09470/) (2020)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software](https://mlsanity.com/paper/1902.02595/) (2019)
- [Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection](https://mlsanity.com/paper/2408.12986/) (2024)
- [MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representation](https://mlsanity.com/paper/2406.12415/) (2024)
- [SPI: Automated Identification of Security Patches via Commits](https://mlsanity.com/paper/2105.14565/) (2021)
- [A Practical Approach to the Automatic Classification of Security-Relevant Commits](https://mlsanity.com/paper/1807.02458/) (2018)
- [PatchRNN: A Deep Learning-Based System for Security Patch Identification](https://mlsanity.com/paper/2108.03358/) (2021)
- [Chasing Shadows: Pitfalls in LLM Security Research](https://mlsanity.com/paper/2512.09549/) (2025)
- [TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (Extended Version)](https://mlsanity.com/paper/2402.01359/) (2024)
- [Just-in-Time Detection of Silent Security Patches](https://mlsanity.com/paper/2312.01241/) (2023)
- [Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM](https://mlsanity.com/paper/2501.14983/) (2025)
- [Silent Vulnerable Dependency Alert Prediction with Vulnerability Key Aspect Explanation](https://mlsanity.com/paper/2302.07445/) (2023)
- [Beyond the TESSERACT:Trustworthy Dataset Curation for Sound Evaluations of Android Malware Classifiers](https://mlsanity.com/paper/2506.23814/) (2025)
- [Repository-Level Graph Representation Learning for Enhanced Security Patch Detection](https://mlsanity.com/paper/2412.08068/) (2024)

## Abstract (arXiv)

Automated detection of vulnerability-fixing commits (vfcs) is critical for timely security patch deployment, as advisory databases lag patch releases by a median of 25 days and many fixes never receive advisories. Code language models are increasingly adopted for identifying VFCs, yet whether they can recognize a security fix from the code change itself remains unclear, as reported performance is shaped by commit messages, project-level data leakage, and uncertain data quality. We present a rigorous re-evaluation that jointly controls these factors through a unified framework consolidating 20 fragmented datasets spanning more than 180000 commits. Training 270 models from 125M to 80B parameters, we isolate the code signal under group-stratified code-only evaluation and assess the impact of model capacity and additional code context, ranging from intra-procedural enrichment to inter-procedural repository context. Model capacity yields clear but insufficient gains, and the evaluated context signals provide no reliable improvement under strict false positive budgets. At a false positive rate of 0.5%, every evaluated fine-tuned code-only model misses at least 80% of vulnerability fixes. A manual expert audit further shows that label error concentrates in commits lacking CVE association and primarily distorts evaluation. We derive concrete recommendations for evaluating on aggregated VFC datasets and release our unified framework and evaluation suite.