Synthesizing Multi-Agent Harnesses for Vulnerability Discovery
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
LLM agents have begun to find real security vulnerabilities that human auditors and automated fuzzers missed for decades, in source-available targets where the analyst can build and instrument the code. In practice the work is split among several agents, wired together by a harness: the program that fixes which roles exist, how they pass information, which tools each may call, and how retries are coordinated. When the language model is held fixed, changing only the harness can still change success rates by several-fold on public agent benchmarks, yet most harnesses are written by hand; recent harness optimizers each search only a narrow slice of the design space and rely on coarse pass/fail feedback that gives no diagnostic signal about why a trial failed. AgentFlow addresses both limitations with a typed graph DSL whose search space jointly covers agent roles, prompts, tools, communication topology, and coordination protocol, paired with a feedback-driven outer loop that reads runtime signals from the target program itself to diagnose which part of the harness caused the failure and rewrite it accordingly. We evaluate AgentFlow on TerminalBench-2 with Claude Opus 4.6 and on Google Chrome with Kimi K2.5. AgentFlow reaches 84.3% on TerminalBench-2, the highest score in the public leaderboard snapshot we evaluate against, and discovers ten previously unknown zero-day vulnerabilities in Google Chrome, including two Critical sandbox-escape vulnerabilities (CVE-2026-5280 and CVE-2026-6297).
Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 11 of its 41 references that are in the map. First posted on arXiv 22 Apr 2026.
Builds on
11 of 41 references.
References · 11
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
- 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
- 2024LLM Agents can Autonomously Hack Websitescited by 108
- 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
- 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
Built on by
Citing papers per year: 2026: 11.
11 papers.
- 2026From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flawscited by 21
- 2026AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Rangescited by 4
- 2026Self-Evolving Agents as Dynamic Graph Transformation: A Survey and New Perspectivecited by 4
- 2026VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Modelscited by 3
- 2026Antares: Foundation Models for Agentic Vulnerability Localizationcited by 2
- 2026Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebasescited by 1
- 2026MESA:Task-Adaptive Multi-Structure Evidence Selection for Long-Horizon Agent Memorycited by 1
- 2026AI Security Policy Should Assess Systems, Not Only Modelscited by 0
- 2026Red-Teaming the Agentic Red-Teamcited by 0
- 2026Understanding Implicit Trust Errors in Core Carrier Networks through Multi-Agent Flaw Discovery and Analysiscited by 0
- 2026SelfOp: An Optimization Algorithm for Self-Improving Security Agentscited by 0
- 2026SelfOp: An Optimization Algorithm for Self-Improving Security Agents19 Sep 2026
- 2026VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models25 Aug 2026
- 2026MESA:Task-Adaptive Multi-Structure Evidence Selection for Long-Horizon Agent Memory10 Aug 2026
- 2026Antares: Foundation Models for Agentic Vulnerability Localization3 Aug 2026
- 2026Understanding Implicit Trust Errors in Core Carrier Networks through Multi-Agent Flaw Discovery and Analysis11 Jul 2026
- 2026Red-Teaming the Agentic Red-Team23 Jun 2026
- 2026Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebases20 Jun 2026
- 2026AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges12 Jun 2026
- 2026Self-Evolving Agents as Dynamic Graph Transformation: A Survey and New Perspective10 Jun 2026
- 2026From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws4 Jun 2026
- 2026AI Security Policy Should Assess Systems, Not Only Models10 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
- 2026Code as Agent Harness
- 2026Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild
- 2026HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
- 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality
- 2026Self-Harness: Harnesses That Improve Themselves
- 2026Auditing Agent Harness Safety
- 2026Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports