# Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study

arXiv: https://arxiv.org/abs/2506.11561 · HTML: https://mlsanity.com/paper/2506.11561/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 7 of 7)

- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves](https://mlsanity.com/paper/2311.04205/) (2023)
- [How Effective Are Neural Networks for Fixing Security Vulnerabilities](https://mlsanity.com/paper/2305.18607/) (2023)
- [Prompt-Enhanced Software Vulnerability Detection Using ChatGPT](https://mlsanity.com/paper/2308.12697/) (2023)
- [Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4](https://mlsanity.com/paper/2312.16171/) (2023)
- [A Study of Vulnerability Repair in JavaScript Programs with Large Language Models](https://mlsanity.com/paper/2403.13193/) (2024)

## Built on by (most cited first, 2 of 2)

- [Multi-Agent Taint Specification Extraction for Vulnerability Detection](https://mlsanity.com/paper/2601.10865/) (2026)
- [Agentic Repository Mining: A Multi-Task Evaluation](https://mlsanity.com/paper/2605.04845/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) have shown promise for automated vulnerability detection and repair in software systems. This paper investigates the performance of GPT-4o in repairing Java vulnerabilities from a widely used dataset (Vul4J), exploring how different contextual information affects automated vulnerability repair (AVR) capabilities. We compare the latest GPT-4o's performance against previous results with GPT-4 using identical prompts. We evaluated nine additional prompts crafted by us that contain various contextual information such as CWE or CVE information, and manually extracted code contexts. Each prompt was executed three times on 42 vulnerabilities, and the resulting fix candidates were validated using Vul4J's automated testing framework. Our results show that GPT-4o performed 11.9% worse on average than GPT-4 with the same prompt, but was able to fix 10.5% more distinct vulnerabilities in the three runs together. CVE information significantly improved repair rates, while the length of the task description had minimal impact. Combining CVE guidance with manually extracted code context resulted in the best performance. Using our textscTop-3 prompts together, GPT-4o repaired 26 (62%) vulnerabilities at least once, outperforming both the original baseline (40%) and its reproduction (45%), suggesting that ensemble prompt strategies could improve vulnerability repair in zero-shot settings.