# FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents

arXiv: https://arxiv.org/abs/2507.15241 · HTML: https://mlsanity.com/paper/2507.15241/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 13 of 13)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities](https://mlsanity.com/paper/2503.17332/) (2025)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities](https://mlsanity.com/paper/2409.16165/) (2024)
- [TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?](https://mlsanity.com/paper/2412.02883/) (2024)
- [Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction](https://mlsanity.com/paper/2311.04532/) (2023)
- [PoCGen: Generating Proof-of-Concept Exploits for Vulnerabilities in Npm Packages](https://mlsanity.com/paper/2506.04962/) (2025)

## Built on by (most cited first, 12 of 12)

- [SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?](https://mlsanity.com/paper/2605.26548/) (2026)
- [Patch-to-PoC: A Systematic Study of Agentic LLM Systems for Linux Kernel N-Day Reproduction](https://mlsanity.com/paper/2602.07287/) (2026)
- [Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents](https://mlsanity.com/paper/2606.05391/) (2026)
- [AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection](https://mlsanity.com/paper/2604.11950/) (2026)
- [Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generation](https://mlsanity.com/paper/2602.13574/) (2026)
- [Program Analysis Guided LLM Agent for Proof-of-Concept Generation](https://mlsanity.com/paper/2604.07624/) (2026)
- [PoC-Adapt: Semantic-Aware Automated Vulnerability Reproduction with LLM Multi-Agents and Reinforcement Learning-Driven Adaptive Policy](https://mlsanity.com/paper/2604.06618/) (2026)
- [SoK: Measuring What Matters for Closed-Loop Security Agents](https://mlsanity.com/paper/2510.01654/) (2025)
- [A Multi-Agent Framework for Automated Exploit Generation with Constraint-Guided Comprehension and Reflection](https://mlsanity.com/paper/2604.05130/) (2026)
- [Agentic Vulnerability Reasoning on COTS Binaries](https://mlsanity.com/paper/2605.05000/) (2026)
- [From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts](https://mlsanity.com/paper/2608.09567/) (2026)
- [P2Patch: Exploit-Guided Vulnerability Repair](https://mlsanity.com/paper/2610.20245/) (2026)

## Abstract (arXiv)

Despite the critical threat posed by software security vulnerabilities, reports are often incomplete, lacking the proof-of-vulnerability (PoV) tests needed to validate fixes and prevent regressions. These tests are crucial not only for ensuring patches work, but also for helping developers understand how vulnerabilities can be exploited. Generating PoV tests is a challenging problem, requiring reasoning about the flow of control and data through deeply nested levels of a program. We present FaultLine, an LLM agent workflow that uses a set of carefully designed reasoning steps, inspired by aspects of traditional static and dynamic program analysis, to automatically generate PoV test cases. Given a software project with an accompanying vulnerability report, FaultLine 1) traces the flow of an input from an externally accessible API ("source") to the "sink" corresponding to the vulnerability, 2) reasons about the conditions that an input must satisfy in order to traverse the branch conditions encountered along the flow, and 3) uses this reasoning to generate a PoV test case in a feedback-driven loop. FaultLine does not use language-specific static or dynamic analysis components, which enables it to be used across programming languages. To evaluate FaultLine, we collate a challenging multi-lingual dataset of 100 known vulnerabilities in Java, C and C++ projects. On this dataset, FaultLine is able to generate PoV tests for 16 projects, compared to just 9 for CodeAct 2.1, a popular state-of-the-art open-source agentic framework. Thus, FaultLine represents a 77% relative improvement over the state of the art. Our findings suggest that hierarchical reasoning can enhance the performance of LLM agents on PoV test generation, but the problem in general remains challenging. We make our code and dataset publicly available in the hope that it will spur further research in this area.