# Repairing vulnerabilities without invisible hands. A differentiated replication study on LLMs

arXiv: https://arxiv.org/abs/2507.20977 · HTML: https://mlsanity.com/paper/2507.20977/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Builds on (most cited first, 18 of 18)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Deep Learning based Vulnerability Detection: Are We There Yet?](https://mlsanity.com/paper/2009.07235/) (2020)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [How Effective Are Neural Networks for Fixing Security Vulnerabilities](https://mlsanity.com/paper/2305.18607/) (2023)
- [ChatGPT for Vulnerability Detection, Classification, and Repair: How Far Are We?](https://mlsanity.com/paper/2310.09810/) (2023)
- [Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead](https://mlsanity.com/paper/2404.02525/) (2024)
- [Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection](https://mlsanity.com/paper/2408.12986/) (2024)
- [A Case Study of LLM for Automated Vulnerability Repair: Assessing Impact of Reasoning and Patch Validation Feedback](https://mlsanity.com/paper/2405.15690/) (2024)
- [Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis](https://mlsanity.com/paper/2406.00992/) (2024)
- [AIBugHunter: A Practical Tool for Predicting, Classifying and Repairing Software Vulnerabilities](https://mlsanity.com/paper/2305.16615/) (2023)
- [LLM Security Guard for Code](https://mlsanity.com/paper/2405.01103/) (2024)
- [Software Vulnerability and Functionality Assessment using LLMs](https://mlsanity.com/paper/2403.08429/) (2024)
- [Toward Improved Deep Learning-based Vulnerability Detection](https://mlsanity.com/paper/2403.03024/) (2024)
- [Instruct or Interact? Exploring and Eliciting LLMs' Capability in Code Snippet Adaptation Through Prompt Engineering](https://mlsanity.com/paper/2411.15501/) (2024)
- [Combining Large Language Models with Static Analyzers for Code Review Generation](https://mlsanity.com/paper/2502.06633/) (2025)
- [Using ML filters to help automated vulnerability repairs: when it helps and when it doesn't](https://mlsanity.com/paper/2504.07027/) (2025)

## Built on by (most cited first, 1 of 1)

- [SoK: Measuring What Matters for Closed-Loop Security Agents](https://mlsanity.com/paper/2510.01654/) (2025)

## Abstract (arXiv)

Background: Automated Vulnerability Repair (AVR) is a fast-growing branch of program repair. Recent studies show that large language models (LLMs) outperform traditional techniques, extending their success beyond code generation and fault detection. Hypothesis: These gains may be driven by hidden factors -- "invisible hands" such as training-data leakage or perfect fault localization -- that let an LLM reproduce human-authored fixes for the same code. Objective: We replicate prior AVR studies under controlled conditions by deliberately adding errors to the reported vulnerability location in the prompt. If LLMs merely regurgitate memorized fixes, both small and large localization errors should yield the same number of correct patches, because any offset should divert the model from the original fix. Method: Our pipeline repairs vulnerabilities from the Vul4J and VJTrans benchmarks after shifting the fault location by n lines from the ground truth. A first LLM generates a patch, a second LLM reviews it, and we validate the result with regression and proof-of-vulnerability tests. Finally, we manually audit a sample of patches and estimate the error rate with the Agresti-Coull-Wilson method.