# A Multi-Dataset Evaluation of Models for Automated Vulnerability Repair

arXiv: https://arxiv.org/abs/2506.04987 · HTML: https://mlsanity.com/paper/2506.04987/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair](https://mlsanity.com/paper/1901.01808/) (2018)
- [TBar: Revisiting Template-based Automated Program Repair](https://mlsanity.com/paper/1903.08409/) (2019)
- [How Effective Are Neural Networks for Fixing Security Vulnerabilities](https://mlsanity.com/paper/2305.18607/) (2023)
- [CODIT: Code Editing with Tree-Based Neural Models](https://mlsanity.com/paper/1810.00314/) (2018)
- [Nopol: Automatic Repair of Conditional Statement Bugs in Java Programs](https://mlsanity.com/paper/1811.04211/) (2018)
- [Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection](https://mlsanity.com/paper/2306.17193/) (2023)
- [MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representation](https://mlsanity.com/paper/2406.12415/) (2024)
- [Patching as Translation: the Data and the Metaphor](https://mlsanity.com/paper/2008.10707/) (2020)
- [Enhanced Automated Code Vulnerability Repair using Large Language Models](https://mlsanity.com/paper/2401.03741/) (2024)
- [Learning from What We Know: How to Perform Vulnerability Prediction using Noisy Historical Data](https://mlsanity.com/paper/2012.11701/) (2020)
- [Impact of Log Parsing on Deep Learning-Based Anomaly Detection](https://mlsanity.com/paper/2305.15897/) (2023)
- [T5APR: Empowering Automated Program Repair across Languages through Checkpoint Ensemble](https://mlsanity.com/paper/2309.15742/) (2023)
- [Vulnerability Mimicking Mutants](https://mlsanity.com/paper/2303.04247/) (2023)

## Built on by (most cited first, 1 of 1)

- [Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilities](https://mlsanity.com/paper/2511.23408/) (2025)

## Abstract (arXiv)

Software vulnerabilities pose significant security threats, requiring effective mitigation. While Automated Program Repair (APR) has advanced in fixing general bugs, vulnerability patching, a security-critical aspect of APR remains underexplored. This study investigates pre-trained language models, CodeBERT and CodeT5, for automated vulnerability patching across six datasets and four languages. We evaluate their accuracy and generalization to unknown vulnerabilities. Results show that while both models face challenges with fragmented or sparse context, CodeBERT performs comparatively better in such scenarios, whereas CodeT5 excels in capturing complex vulnerability patterns. CodeT5 also demonstrates superior scalability. Furthermore, we test fine-tuned models on both in-distribution (trained) and out-of-distribution (unseen) datasets. While fine-tuning improves in-distribution performance, models struggle to generalize to unseen data, highlighting challenges in robust vulnerability detection. This study benchmarks model performance, identifies limitations in generalization, and provides actionable insights to advance automated vulnerability patching for real-world security applications.