# Hacking CTFs with Plain Agents

arXiv: https://arxiv.org/abs/2412.02776 · HTML: https://mlsanity.com/paper/2412.02776/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 10 of 10)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models](https://mlsanity.com/paper/2404.13161/) (2024)
- [EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities](https://mlsanity.com/paper/2409.16165/) (2024)
- [Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities](https://mlsanity.com/paper/2410.09114/) (2024)

## Built on by (most cited first, 14 of 14)

- [D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security](https://mlsanity.com/paper/2502.10931/) (2025)
- [CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution](https://mlsanity.com/paper/2505.17107/) (2025)
- [Agentic Workflows for Conversational Human-AI Interaction Design](https://mlsanity.com/paper/2501.18002/) (2025)
- [Dynamic Risk Assessments for Offensive Cybersecurity Agents](https://mlsanity.com/paper/2505.18384/) (2025)
- [Evaluating AI cyber capabilities with crowdsourced elicitation](https://mlsanity.com/paper/2505.19915/) (2025)
- [Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations](https://mlsanity.com/paper/2602.05523/) (2026)
- [CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarking](https://mlsanity.com/paper/2602.08023/) (2026)
- [Winning at All Cost: A Small Environment for Eliciting Specification Gaming Behaviors in Large Language Models](https://mlsanity.com/paper/2505.07846/) (2025)
- [Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs](https://mlsanity.com/paper/2510.17521/) (2025)
- [KryptoPilot: An Open-World Knowledge-Augmented LLM Agent for Automated Cryptographic Exploitation](https://mlsanity.com/paper/2601.09129/) (2026)
- [AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomes](https://mlsanity.com/paper/2603.21551/) (2026)
- [Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots](https://mlsanity.com/paper/2605.29963/) (2026)
- [Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents](https://mlsanity.com/paper/2607.15263/) (2026)
- [VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agents](https://mlsanity.com/paper/2607.09653/) (2026)

## Abstract (arXiv)

We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%). Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing.