# AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection

arXiv: https://arxiv.org/abs/2604.11950 · HTML: https://mlsanity.com/paper/2604.11950/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 73 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 25 of 25)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://mlsanity.com/paper/2212.14834/) (2022)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [Large Language Model for Vulnerability Detection: Emerging Results and Future Directions](https://mlsanity.com/paper/2401.15468/) (2024)
- [Evaluating Fuzz Testing](https://mlsanity.com/paper/1808.09700/) (2018)
- [Otter: Generating Tests from Issues to Validate SWE Patches](https://mlsanity.com/paper/2502.05368/) (2025)
- [KNighter: Transforming Static Analysis with LLM-Synthesized Checkers](https://mlsanity.com/paper/2503.09002/) (2025)
- [KernelGPT: Enhanced Kernel Fuzzing via Large Language Models](https://mlsanity.com/paper/2401.00563/) (2023)
- [PoCGen: Generating Proof-of-Concept Exploits for Vulnerabilities in Npm Packages](https://mlsanity.com/paper/2506.04962/) (2025)
- [FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents](https://mlsanity.com/paper/2507.15241/) (2025)
- [Automatic Generation of Test Cases based on Bug Reports: a Feasibility Study with Large Language Models](https://mlsanity.com/paper/2310.06320/) (2023)
- [QLCoder: A Query Synthesizer For Static Analysis of Security Vulnerabilities](https://mlsanity.com/paper/2511.08462/) (2025)
- [One Bug, Hundreds Behind: LLMs for Large-Scale Bug Discovery](https://mlsanity.com/paper/2510.14036/) (2025)
- [SoK: Sanitizing for Security](https://mlsanity.com/paper/1806.04355/) (2018)
- [Artemis: Toward Accurate Detection of Server-Side Request Forgeries through LLM-Assisted Inter-Procedural Path-Sensitive Taint Analysis](https://mlsanity.com/paper/2502.21026/) (2025)
- [SmartPoC: Generating Executable and Validated PoCs for Smart Contract Bug Reports](https://mlsanity.com/paper/2511.12993/) (2025)
- [Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry](https://mlsanity.com/paper/2601.18844/) (2026)
- [Test Case Generation from Bug Reports via Large Language Models: A Cognitive Layered Evaluation Framework](https://mlsanity.com/paper/2510.05365/) (2025)

## Built on by (most cited first, 4 of 4)

- [SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?](https://mlsanity.com/paper/2605.26548/) (2026)
- [Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebases](https://mlsanity.com/paper/2606.22263/) (2026)
- [Agentic Vulnerability Reasoning on COTS Binaries](https://mlsanity.com/paper/2605.05000/) (2026)
- [WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses](https://mlsanity.com/paper/2609.36635/) (2026)

## Abstract (arXiv)

While recent LLM-based agents can identify many candidate bugs in source code, their reports remain static hypotheses that require manual validation, limiting the practicality of automated bug detection. We frame this challenge as a test generation task: given a candidate report, synthesizing an executable proof-of-concept (PoC) - such as a script, command sequence, or crafted input - to trigger the suspected defect. Automated PoC generation can act as a scalable validation oracle, enabling end-to-end autonomous bug detection by providing concrete execution evidence. However, naive LLM agents are unreliable validators: they are biased toward "success" and may reward-hack by producing plausible but non-functional PoCs or even hallucinated traces. To address this, we present ANYPoC, a general multi-agent framework that (1) analyzes and fact-checks a candidate bug report, (2) iteratively synthesizes and executes a PoC while collecting execution traces, and (3) independently re-executes and scrutinizes the PoC to mitigate hallucination and reward hacking. In addition, ANYPoC also continuously extracts and evolves a PoC knowledge base to handle heterogeneous tasks. ANYPoC operates on candidate bug reports regardless of their source and can be paired with different bug reporters. To demonstrate practicality and generality, we apply ANYPoC, together with a simple agentic bug reporter, on 12 large-scale, critical software systems, including Firefox, Chromium, LLVM, OpenSSL, SQLite, FFmpeg, and Redis. Compared to the state-of-the-art coding agents, e.g., Claude Code and Codex, ANYPoC produces 37% more valid PoCs for true-positive bug reports and rejects 9.7x more false-positive bug reports. ANYPoC also enables the discovery of 121 new bugs from over two thousand noisy bug reports, with 108 confirmed by developers and 92 fixed. 46 PoCs have also been adopted as official regression tests.