# Learning from Failures: A Failure-Driven Prompt Refinement for LLM-Based Vulnerability Analysis

arXiv: https://arxiv.org/abs/2610.08405 · HTML: https://mlsanity.com/paper/2610.08405/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 34 references that are in the map.
- First posted on arXiv 6 Oct 2026.

## Builds on (most cited first, 10 of 10)

- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [To Err is Machine: Vulnerability Detection Challenges LLM Reasoning](https://mlsanity.com/paper/2403.17218/) (2024)
- [Chasing Shadows: Pitfalls in LLM Security Research](https://mlsanity.com/paper/2512.09549/) (2025)
- [VulDeePecker: A Deep Learning-Based System for Vulnerability Detection](https://mlsanity.com/paper/1801.01681/) (2018)
- [Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection](https://mlsanity.com/paper/2412.12039/) (2024)
- [Think Broad, Act Narrow: CWE Identification with Multi-Agent Large Language Models](https://mlsanity.com/paper/2508.01451/) (2025)
- [PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection](https://mlsanity.com/paper/2605.24171/) (2026)
- [Enhancing Reliability in LLM-Based Secure Code Generation](https://mlsanity.com/paper/2605.24300/) (2026)
- [An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods](https://mlsanity.com/paper/2605.24298/) (2026)

## Abstract (arXiv)

Large Language Models have emerged as promising tools for software vulnerability analysis, but their effectiveness depends heavily on prompt design. Existing research primarily compares prompting strategies using aggregate performance metrics, providing limited insight into why models fail or how prompts can be improved systematically. We propose Failure-Driven Prompt Refinement (FDPR), a methodology that analyzes recurring model failures to guide evidence-based prompt refinement. Using the Damn Vulnerable Java Application (DVJA), we identify recurring failure modes, including false positives, false negatives, unsupported reasoning, and CWE misclassification, and translate them into targeted prompt refinements. We then evaluate the resulting prompt on the Juliet Test Suite and perform cross-model validation to assess generalizability. The results show that failure-driven refinement improves the reliability of LLM-based vulnerability analysis while yielding reusable prompt design principles. More broadly, this work demonstrates that recurring model failures provide a principled foundation for prompt engineering, enabling the systematic development of more reliable LLM-based vulnerability analysis systems.