# A Systematic Study on Generating Web Vulnerability Proof-of-Concepts Using Large Language Models

arXiv: https://arxiv.org/abs/2510.10148 · HTML: https://mlsanity.com/paper/2510.10148/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2025.

## Built on by (most cited first, 5 of 5)

- [SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?](https://mlsanity.com/paper/2605.26548/) (2026)
- [Knowdit: Agentic Smart Contract Vulnerability Detection with Auditing Knowledge Summarization](https://mlsanity.com/paper/2603.26270/) (2026)
- [PoC-Adapt: Semantic-Aware Automated Vulnerability Reproduction with LLM Multi-Agents and Reinforcement Learning-Driven Adaptive Policy](https://mlsanity.com/paper/2604.06618/) (2026)
- [AutoEG: Exploiting Known Third-Party Vulnerabilities in Black-Box Web Applications](https://mlsanity.com/paper/2604.00704/) (2026)
- [uGen: An Agentic Framework for Generating Microarchitectural Attack PoCs](https://mlsanity.com/paper/2605.15503/) (2026)

## Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have brought remarkable progress in code understanding and reasoning, creating new opportunities and raising new concerns for software security. Among many downstream tasks, generating Proof-of-Concept (PoC) exploits plays a central role in vulnerability reproduction, comprehension, and mitigation. While previous research has focused primarily on zero-day exploitation, the growing availability of rich public information accompanying disclosed CVEs leads to a natural question: can LLMs effectively use this information to automatically generate valid PoCs? In this paper, we present the first empirical study of LLM-based PoC generation for web application vulnerabilities, focusing on the practical feasibility of leveraging publicly disclosed information. We evaluate GPT-4o and DeepSeek-R1 on 100 real-world and reproducible CVEs across three stages of vulnerability disclosure: (1) newly disclosed vulnerabilities with only descriptions, (2) 1-day vulnerabilities with patches, and (3) N-day vulnerabilities with full contextual code. Our results show that LLMs can automatically generate working PoCs in 8%-34% of cases using only public data, with DeepSeek-R1 consistently outperforming GPT-4o. Further analysis shows that supplementing code context improves success rates by 17%-20%, with function-level providing 9%-13% improvement than file-level ones. Further integrating adaptive reasoning strategies to prompt refinement significantly improves success rates to 68%-72%. Our findings suggest that LLMs could reshape vulnerability exploitation dynamics. To date, 23 newly generated PoCs have been accepted by NVD and Exploit DB.