# CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?

arXiv: https://arxiv.org/abs/2606.26216 · HTML: https://mlsanity.com/paper/2606.26216/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 12 of 12)

- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Slither: A Static Analysis Framework For Smart Contracts](https://mlsanity.com/paper/1908.09878/) (2019)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis](https://mlsanity.com/paper/2308.03314/) (2023)
- [BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems](https://mlsanity.com/paper/2505.15216/) (2025)
- [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://mlsanity.com/paper/2605.11086/) (2026)
- [SoK: Decentralized Finance (DeFi) Attacks](https://mlsanity.com/paper/2208.13035/) (2022)
- [AI Agent Smart Contract Exploit Generation](https://mlsanity.com/paper/2507.05558/) (2025)
- [EVMbench: Evaluating AI Agents on Smart Contract Security](https://mlsanity.com/paper/2603.04915/) (2026)
- [SmartPoC: Generating Executable and Validated PoCs for Smart Contract Bug Reports](https://mlsanity.com/paper/2511.12993/) (2025)
- [SC-Bench: A Large-Scale Dataset for Smart Contract Auditing](https://mlsanity.com/paper/2410.06176/) (2024)
- [LISA Technical Report: An Agentic Framework for Smart Contract Auditing](https://mlsanity.com/paper/2509.24698/) (2025)

## Abstract (arXiv)

We present CyberChainBench, a benchmark for evaluating LLM-based agents on smart contract security across three complementary tasks: vulnerability detection, exploit generation, and patch synthesis. Built from 541 real-world exploit incidents from DeFiHackLabs spanning 9 EVM chains, the benchmark provides end-to-end on-chain evaluation where agents interact with historical blockchain state through isolated evaluation environments orchestrated by Harbor, using tools to read code, trace transactions, and validate exploits on mainnet forks. Each case is anchored to a specific block and includes structured ground truth covering vulnerability type, localization, and attacker profit. Exploits are graded by economic impact on historical forks; patches are validated by replaying historical attacks and legitimate transactions as fail-to-pass test oracles on a proxy-upgradeable subset. We define a five-type vulnerability taxonomy and evaluate multiple agent--model configurations. Results reveal a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, but only 23.4% on patching, with the top agent (Codex with GPT-5.5) realizing 57.4M in total exploit profit across the 200-case exploit set at a cost of 2.39 per case.