mlsanityData to 14 Oct 2026

Construction and Evaluation of LLM-based agents for Semi-Autonomous penetration testing

arXiv 2502.15506 ↗First posted 21 Feb 2025cs.CRcited by 1

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

With the emergence of high-performance large language models (LLMs) such as GPT, Claude, and Gemini, the autonomous and semi-autonomous execution of tasks has significantly advanced across various domains. However, in highly specialized fields such as cybersecurity, full autonomy remains a challenge. This difficulty primarily stems from the limitations of LLMs in reasoning capabilities and domain-specific knowledge. We propose a system that semi-autonomously executes complex cybersecurity workflows by employing multiple LLMs modules to formulate attack strategies, generate commands, and analyze results, thereby addressing the aforementioned challenges. In our experiments using Hack The Box virtual machines, we confirmed that our system can autonomously construct attack strategies, issue appropriate commands, and automate certain processes, thereby reducing the need for manual intervention.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 18 of its 31 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

18 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilities2 Jun
  2. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilities11 Apr
  3. 2024LLM Agents can Autonomously Hack Websites6 Feb

Other references · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  9. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  10. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 96
  11. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  12. 2023Getting pwn'd by AI: Penetration Testing with Large Language Modelscited by 83
3 more
  1. 2024PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediationcited by 30
  2. 2023Can Large Language Models Find And Fix Vulnerable Software?cited by 15
  3. 2023Detecting Phishing Sites Using ChatGPTcited by 9

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45

Similar, not cited

Close in wording, no citation either way

  1. 2025Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees
  2. 2026Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing
  3. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
  4. 2025Automated Penetration Testing with LLM Agents and Classical Planning
  5. 2026LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity
  6. 2025The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise
  7. 2025Vulnerability Mitigation System (VMS): LLM Agent and Evaluation Framework for Autonomous Penetration Testing
  8. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks