# Agentic Vulnerability Reasoning on COTS Binaries

arXiv: https://arxiv.org/abs/2605.05000 · HTML: https://mlsanity.com/paper/2605.05000/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 6 May 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 22 of 22)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://mlsanity.com/paper/2212.14834/) (2022)
- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025)
- [EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities](https://mlsanity.com/paper/2409.16165/) (2024)
- [Reducing Cost of LLM Agents with Trajectory Reduction](https://mlsanity.com/paper/2509.23586/) (2025)
- [LLM4Decompile: Decompiling Binary Code with Large Language Models](https://mlsanity.com/paper/2403.05286/) (2024)
- [KNighter: Transforming Static Analysis with LLM-Synthesized Checkers](https://mlsanity.com/paper/2503.09002/) (2025)
- [Harnessing the Power of LLM to Support Binary Taint Analysis](https://mlsanity.com/paper/2310.08275/) (2023)
- [FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents](https://mlsanity.com/paper/2507.15241/) (2025)
- [ATLANTIS: AI-driven Threat Localization, Analysis, and Triage Intelligence System](https://mlsanity.com/paper/2509.14589/) (2025)
- [Callee: Recovering Call Graphs for Binaries with Transfer and Contrastive Learning](https://mlsanity.com/paper/2111.01415/) (2021)
- [AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection](https://mlsanity.com/paper/2604.11950/) (2026)
- [Navigating the Concurrency Landscape: A Survey of Race Condition Vulnerability Detectors](https://mlsanity.com/paper/2312.14479/) (2023)

## Abstract (arXiv)

LLM agents have been increasingly adopted for solving security tasks. However, existing evaluations usually require source code access, while commercial off-the-shelf (COTS) binaries dominate deployed software and require reasoning from stripped, optimized machine code. This discrepancy raises an important question: can modern LLM agents reason about vulnerabilities in critical COTS binaries? Motivated by this question, we build SLYP, a REACT-style pipeline for end-to-end vulnerability discovery and validation of COTS binaries. SLYP combines extensible MCP servers for binary exploration and dynamic debugging, and validates candidate vulnerabilities by synthesizing debugger-verified proof-of-concept (PoC) crashes. We evaluate SLYP and production coding agents, including Claude Code and Codex, on COTS Windows binaries centered on a 20-object COM benchmark. SLYP uncovers all 64 vulnerable entry functions while default production agents miss up to 15; SLYP also surfaces more true vulnerabilities than the state-of-the-art static analyzer, which discovers at most 35 with a large number of false positives. For validation, SLYP generates debugger-verified PoCs for 67.5% of cases, while default production agents generate none. Further ablations show that tool sets and model choice materially affect COTS binary reasoning. Our additional evaluation also demonstrates the generalizability of SLYP on Windows kernel targets. To date, SLYP has uncovered 39 zero-day vulnerabilities, 31 in COM/RPC services and 8 in kernel drivers, all disclosed to the Microsoft Security Response Center (MSRC), with 23 assigned CVEs and 203,000 bounty awards.