mlsanityData to 7 Oct 2026

An Empirical Evaluation of LLMs for Solving Offensive Security Challenges

arXiv 2402.11814 ↗First posted 19 Feb 2024cs.CRcited by 33

Not on the map. Why?

Abstract (arXiv)

Capture The Flag (CTF) challenges are puzzles related to computer security scenarios. With the advent of large language models (LLMs), more and more CTF participants are using LLMs to understand and solve the challenges. However, so far no work has evaluated the effectiveness of LLMs in solving CTF challenges with a fully automated workflow. We develop two CTF-solving workflows, human-in-the-loop (HITL) and fully-automated, to examine the LLMs' ability to solve a selected set of CTF challenges, prompted with information about the question. We collect human contestants' results on the same set of questions, and find that LLMs achieve higher success rate than an average human participant. This work provides a comprehensive evaluation of the capability of LLMs in solving real world CTF challenges, from real competition to fully automated workflow. Our results provide references for applying LLMs in cybersecurity education and pave the way for systematic evaluation of offensive cybersecurity capabilities in LLMs.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 13; 2026: 9.

33 papers.

Sort:

By area:

  1. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  2. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  3. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  4. 2025D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Securitycited by 19
  5. 2025Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networkscited by 17
  6. 2024Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattackscited by 15
  7. 2025CRAKEN: Cybersecurity LLM Agent with Knowledge-Based Executioncited by 14
  8. 2025Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challengescited by 14
  9. 2025Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Designcited by 13
  10. 2025OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilitiescited by 12
  11. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Papercited by 9
  12. 2025Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmarkcited by 9
21 more
  1. 2025AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agentscited by 8
  2. 2026What Makes a Good LLM Agent for Real-world Penetration Testing?cited by 8
  3. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 7
  4. 2024eyeballvul: a future-proof benchmark for vulnerability detection in the wildcited by 6
  5. 2024Applying Refusal-Vector Ablation to Llama 3.1 70B Agentscited by 5
  6. 2025LLMs unlock new paths to monetizing exploitscited by 5
  7. 2024A Comprehensive Survey on the Security of Smart Grid: Challenges, Mitigations, and Future Research Opportunitiescited by 4
  8. 2024SeCodePLT: A Unified Platform for Evaluating the Security of Code GenAIcited by 4
  9. 2025Large Language Models and Arabic Content: A Reviewcited by 4
  10. 2025From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localizationcited by 4
  11. 2024No Size Fits All: The Perils and Pitfalls of Leveraging LLMs Vary with Company Sizecited by 3
  12. 2026CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarkingcited by 3
  13. 2026CTFusion: A CTF-based Benchmark for LLM Agent Evaluationcited by 3
  14. 2024The Best Defense is a Good Offense: Countering LLM-Powered Cyberattackscited by 2
  15. 2025What Makes an Evaluation Useful? Common Pitfalls and Best Practicescited by 1
  16. 2026AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomescited by 1
  17. 2026CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environmentscited by 1
  18. 2026CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledgecited by 1
  19. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Taskscited by 0
  20. 2026RAVEN: Retrieval-Augmented Vulnerability Exploration Network for Memory Corruption Analysis in User Code and Binary Programscited by 0
  21. 2026Red-Teaming the Agentic Red-Teamcited by 0