# Rethinking the Capability of Fine-Tuned Language Models for Automated Vulnerability Repair

arXiv: https://arxiv.org/abs/2512.22633 · HTML: https://mlsanity.com/paper/2512.22633/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 13 of its 68 references that are in the map.
- First posted on arXiv 27 Dec 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 13 of 13)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Predicting the Generalization Gap in Deep Networks with Margin Distributions](https://mlsanity.com/paper/1810.00113/) (2018)
- [Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation](https://mlsanity.com/paper/2102.01818/) (2021)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [CodeScore: Evaluating Code Generation by Learning Code Execution](https://mlsanity.com/paper/2301.09043/) (2023)
- [FixEval: Execution-based Evaluation of Program Fixes for Programming Problems](https://mlsanity.com/paper/2206.07796/) (2022)
- [Regression-aware Inference with LLMs](https://mlsanity.com/paper/2403.04182/) (2024)

## Built on by (most cited first, 2 of 2)

- [EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolution](https://mlsanity.com/paper/2605.30105/) (2026)
- [Knowledge-Enhanced Agentic Vulnerability Repair](https://mlsanity.com/paper/2607.00820/) (2026)

## Abstract (arXiv)

Learning-based automated vulnerability repair (AVR) techniques that utilize fine-tuned language models have shown promise in generating vulnerability patches. However, questions remain about their ability to repair unseen vulnerabilities. Our empirical study reveals that state-of-the-art models often overfit to the training set and are evaluated using training, validation, and test sets that are not mutually exclusive. Furthermore, relying on match-based metrics that compare generated patches to reference fixes at the token level has some limitations, failing to account for the possibility of various valid ways to patch the vulnerability. In this paper, we examine the capabilities of state-of-the-art fine-tuned AVR models and the adequacy of match-based evaluation metrics in three ways. First, we apply semantic-preserving transformations to test sets in order to determine whether models truly learn robust vulnerability-repair patterns or simply rely on spurious features. Second, we re-split the training, validation, and test sets to be mutually exclusive and evaluate the models on the revised test set to assess their generalization capabilities. Third, we introduce L-AVRBench, a test-based benchmark tailored for learning-based AVR, to overcome the limitations of match-based metrics and examine the AVR models' true repair capabilities.