# Transferable Self-Evolving Playbooks for Agentic Security Auditing

arXiv: https://arxiv.org/abs/2606.16420 · HTML: https://mlsanity.com/paper/2606.16420/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 26 of 26)

- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Memento: Fine-tuning LLM Agents without Fine-tuning LLMs](https://mlsanity.com/paper/2508.16153/) (2025)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [ReposVul: A Repository-Level High-Quality Vulnerability Dataset](https://mlsanity.com/paper/2401.13169/) (2024)
- [Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories](https://mlsanity.com/paper/2503.03586/) (2025)
- [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://mlsanity.com/paper/2605.11086/) (2026)
- [LLMDFA: Analyzing Dataflow in Code with Large Language Models](https://mlsanity.com/paper/2402.10754/) (2024)
- [Multi-Agent Penetration Testing AI for the Web](https://mlsanity.com/paper/2508.20816/) (2025)
- [Exploring ChatGPT's Capabilities on Vulnerability Management](https://mlsanity.com/paper/2311.06530/) (2023)
- [VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection](https://mlsanity.com/paper/2404.15596/) (2024)
- [ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents](https://mlsanity.com/paper/2605.14153/) (2026)
- [Generating API Parameter Security Rules with LLM for API Misuse Detection](https://mlsanity.com/paper/2409.09288/) (2024)
- [BugScope: Learn to Find Bugs Like Human](https://mlsanity.com/paper/2507.15671/) (2025)
- [Agentic Discovery and Validation of Android App Vulnerabilities](https://mlsanity.com/paper/2508.21579/) (2025)
- [RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code](https://mlsanity.com/paper/2604.13764/) (2026)
- [Towards Optimal Agentic Architectures for Offensive Security Tasks](https://mlsanity.com/paper/2604.18718/) (2026)
- [The Midas Touch: Triggering the Capability of LLMs for RM-API Misuse Detection](https://mlsanity.com/paper/2409.09380/) (2024)
- [TxRay: Agentic Postmortem of Live Blockchain Attacks](https://mlsanity.com/paper/2602.01317/) (2026)
- [Three Heads Are Better Than One: A Multi-perspective Reasoning Framework for Enhanced Vulnerability Detection](https://mlsanity.com/paper/2605.18153/) (2026)

## Abstract (arXiv)

An LLM agent for vulnerability discovery and validation is more than a model. It combines three components: an LLM for code analysis, an agent harness such as Codex or OpenCode for navigation, tool use, and execution, and an audit playbook, domain-specific procedural knowledge that guides the LLM and harness toward vulnerability discovery. Prior work relies on human-supplied playbooks, including prompt engineering, manual workflows, knowledge bases, and heuristics. This raises two research questions: Acquisition - is human curation necessary, and can playbook creation be automated? Transfer - can an evolved playbook transfer the audit procedure to weaker agents, improving their capability? We present EvoHunt, a playbook evolution environment over open-source repositories for security auditing. Three agents drive the evolution loop: an audit agent rolls out the current playbook and produces findings; an evaluator scores outcomes against ground truth; and a reviser commits updates to the playbook based on failure analysis. The playbook format is unconstrained: starting empty, EvoHunt adds or removes workflows, heuristics, vulnerability knowledge, or domain-specific content. The evolved playbook requires only minor adaptation to run under a different LLM or harness. We evaluate EvoHunt on open-source security advisories. For acquisition, playbook evolution raises end-to-end exploits for Codex/GPT5.4-xhigh 6x, from 1.1% to 6.2%, and the evolved OpenCode/GLM5.1 playbook surpasses OpenAI Codex Security on every metric, with 11.3% vs. 9.2% target-match rate, showing open-source evolution can outperform a dedicated commercial product. For transfer, the GLM-evolved playbook gives the strongest student lift: Qwen3.6-27B improves from 2.4% to 6.5%, Qwen3.6-35B-A3B from 1.1% to 4.6%, and A3B obtains 2.4x more matches than GPT transfer.