mlsanityData to 14 Oct 2026

CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Execution

arXiv 2505.17107 ↗First posted 21 May 2025cs.CR cs.AI cs.LGcited by 14

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large Language Model (LLM) agents can automate cybersecurity tasks and can adapt to the evolving cybersecurity landscape without re-engineering. While LLM agents have demonstrated cybersecurity capabilities on Capture-The-Flag (CTF) competitions, they have two key limitations: accessing latest cybersecurity expertise beyond training data, and integrating new knowledge into complex task planning. Knowledge-based approaches that incorporate technical understanding into the task-solving automation can tackle these limitations. We present CRAKEN, a knowledge-based LLM agent framework that improves cybersecurity capability through three core mechanisms: contextual decomposition of task-critical information, iterative self-reflected knowledge retrieval, and knowledge-hint injection that transforms insights into adaptive attack strategies. Comprehensive evaluations with different configurations show CRAKEN's effectiveness in multi-stage vulnerability detection and exploitation compared to previous approaches. Our extensible architecture establishes new methodologies for embedding new security knowledge into LLM-driven cybersecurity agentic systems. With a knowledge database of CTF writeups, CRAKEN obtained an accuracy of 22% on NYU CTF Bench, outperforming prior works by 3% and achieving state-of-the-art results. On evaluation of MITRE ATT&CK techniques, CRAKEN solves 25-30% more techniques than prior work, demonstrating improved cybersecurity capabilities via knowledge-based execution. We make our framework open source to public https://github.com/NYU-LLM-CTF/nyuctf_agents_craken.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 32 of its 47 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

32 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security15 Feb
  2. 2024Hacking CTFs with Plain Agents3 Dec
  3. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing2 Dec
  4. 2024PentestAgent: Incorporating LLM Agents to Automated Penetration Testing7 Nov
  5. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities24 Sep

Other references · 27

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  4. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  5. 2023Active Retrieval Augmented Generationcited by 516
  6. 2024Graph Retrieval-Augmented Generation: A Surveycited by 324
  7. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  8. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  9. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  10. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  11. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 97
  12. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 93
15 more
  1. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 91
  2. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
  3. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
  4. 2024GRAG: Graph Retrieval-Augmented Generationcited by 78
  5. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 68
  6. 2023Unveiling Security, Privacy, and Ethical Concerns of ChatGPTcited by 53
  7. 2024An Empirical Evaluation of LLMs for Solving Offensive Security Challengescited by 35
  8. 2023Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questionscited by 23
  9. 2024Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detectioncited by 16
  10. 2024A RAG-Based Question-Answering Solution for Cyber-Attack Investigation and Attributioncited by 9
  11. 2024Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Educationcited by 9
  12. 2024A Study on the Implementation Method of an Agent-Based Advanced RAG System Using Graphcited by 8
  13. 2024Evaluating Large Language Models for Line-Level Vulnerability Localizationcited by 7
  14. 2020Benefits and Pitfalls of Using Capture the Flag Games in University Coursescited by 6
  15. 2023Battle Ground: Data Collection and Labeling of CTF Games to Understand Human Cyber Operatorscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2025Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmarkcited by 12
  2. 2025From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localizationcited by 4
  3. 2026CTFusion: A CTF-based Benchmark for LLM Agent Evaluationcited by 4
  4. 2026CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarkingcited by 3
  5. 2026A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Modelscited by 1
  6. 2026AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomescited by 1
  7. 2026Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binariescited by 1
  8. 2026Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testingcited by 1
  9. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Taskscited by 0
  10. 2026RAVEN: Retrieval-Augmented Vulnerability Exploration Network for Memory Corruption Analysis in User Code and Binary Programscited by 0
  11. 2026uGen: An Agentic Framework for Generating Microarchitectural Attack PoCscited by 0
  12. 2026CyberEvolver: Structured Self-Evolution for Cybersecurity Agents On the Flycited by 0
2 more
  1. 2026Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AIcited by 0
  2. 2026Reveree: Diagnosing LLM Reverse-Engineering Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KryptoPilot: An Open-World Knowledge-Augmented LLM Agent for Automated Cryptographic Exploitation
  2. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding
  3. 2025The Evolution of Agentic AI in Cybersecurity: From Single LLM Reasoners to Multi-Agent Systems and Autonomous Pipelines
  4. 2025DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments
  5. 2025Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges
  6. 2026LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity
  7. 2026Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations
  8. 2025Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees