# CWE-bench: Refocusing Agentic Security from Exploitation to Defense

arXiv: https://arxiv.org/abs/2610.17020 · HTML: https://mlsanity.com/paper/2610.17020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 19 of 19)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities](https://mlsanity.com/paper/2503.17332/) (2025)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025)
- [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://mlsanity.com/paper/2605.11086/) (2026)
- [PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities](https://mlsanity.com/paper/2511.11019/) (2025)
- [ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents](https://mlsanity.com/paper/2605.14153/) (2026)
- [Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack](https://mlsanity.com/paper/2605.12673/) (2026)
- [What Makes a Good LLM Agent for Real-world Penetration Testing?](https://mlsanity.com/paper/2602.17622/) (2026)
- [CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability](https://mlsanity.com/paper/2602.03012/) (2026)
- [SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?](https://mlsanity.com/paper/2605.26548/) (2026)
- [CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities](https://mlsanity.com/paper/2606.04460/) (2026)
- [ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense](https://mlsanity.com/paper/2603.02297/) (2026)
- [Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks](https://mlsanity.com/paper/2607.21763/) (2026)
- [Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles](https://mlsanity.com/paper/2608.00692/) (2026)
- [PatchBench: Evaluating AI Agents for Vulnerability Patching](https://mlsanity.com/paper/2609.04075/) (2026)
- [BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure](https://mlsanity.com/paper/2609.11028/) (2026)
- [SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities](https://mlsanity.com/paper/2609.33763/) (2026)

## Abstract (arXiv)

Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An attacker needs to exploit only one vulnerability; a defender must find and fix all of them. We introduce CWE-bench, a private benchmark of 120 held-out repository audit-and-repair tasks spanning 73 CWE types, eight languages, and all OWASP Top 10 2025 categories. Agents are given a repository to audit and a description of its threat model, but are not told the weakness, where it occurs, or how many vulnerable sites there are. Each patch must block the exploit while preserving original functionality, as checked by a programmatic verifier. We validated the verifier against a panel of agentic judges, manually reviewing disagreements during development and refining the checks. The best agents reach 68% pass@1 and 81% pass@4. We further conduct analyses showing that (i) agents with the same pass@1 differ widely in retry gains and cost, and (ii) no single agent is best across languages or vulnerability categories, and difficulty varies far more by vulnerability category than by language. As the offensive capabilities of agents improve, CWE-bench provides a broad and rigorous measure of whether their defensive capabilities keep pace.