mlsanityData to 7 Oct 2026

Hacking CTFs with Plain Agents

arXiv 2412.02776 ↗First posted 3 Dec 2024cs.CR cs.AIcited by 14

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%). Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 10 of its 17 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

10 of 17 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities10 Oct
  2. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities24 Sep

Other references · 8

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  5. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  6. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 121
  7. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  8. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 84

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Securitycited by 19
  2. 2025CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Executioncited by 14
  3. 2025Agentic Workflows for Conversational Human-AI Interaction Designcited by 8
  4. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 7
  5. 2025Evaluating AI cyber capabilities with crowdsourced elicitationcited by 5
  6. 2026CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarkingcited by 3
  7. 2026Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformationscited by 2
  8. 2025Winning at All Cost: A Small Environment for Eliciting Specification Gaming Behaviors in Large Language Modelscited by 1
  9. 2025Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFscited by 1
  10. 2026KryptoPilot: An Open-World Knowledge-Augmented LLM Agent for Automated Cryptographic Exploitationcited by 1
  11. 2026AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomescited by 1
  12. 2026Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypotscited by 1
2 more
  1. 2026Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agentscited by 1
  2. 2026VEXAIoT: Autonomous IoT Vulnerability EXploitation using AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark
  2. 2026CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
  3. 2026Autonomous LLM Agents & CTFs: A Second Look
  4. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  5. 2026Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming
  6. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
  7. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
  8. 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps