mlsanityData to 7 Oct 2026

Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectives

arXiv 2609.16694 ↗First posted 15 Sep 2026cs.CRcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

LLM-powered autonomous agents are transforming the penetration testing space with dynamic, multi-step offensive security workflows that require minimal supervision by humans. These agents leverage sophisticated reasoning abilities and external security tools to independently carry out reconnaissance, identify vulnerabilities, devise exploitation plans, and perform post-exploitation operations. But the ability to have persistent memory, to take actions in the real world, and to do long-horizon reasoning raises qualitatively different security concerns than traditional chat-based LLM systems. Existing guardrail mechanisms for conversational AI may not be sufficient to secure autonomous AI pentesting agents accordingly. To address these issues, we carry out a comprehensive security analysis on autonomous AI-penetration testing agents. We systematically analyse representative agent architectures, characterise their trust boundaries and attack surfaces and propose a threat taxonomy that is aligned with the lifecycle and covers LLM lifecycle attacks, agent-architecture attacks and cross-cutting behavioural attacks. We analyse the limitations of existing guardrail mechanisms, identify key research gaps, and discuss future research directions for developing specialised, context-aware, and architecture-aware guardrails to secure next-generation AI-driven offensive security systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 84 of its 111 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

84 of 111 references.

Closest ancestors · 36

Same direction, newest first

  1. 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments11 May
  2. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework25 Apr
  3. 2026RouteGuard: Internal-Signal Detection of Skill Poisoning in LLM Agents24 Apr
  4. 2026Automation-Exploit: A Multi-Agent LLM Framework for Adaptive Offensive Security with Digital Twin-Based Risk-Mitigated Exploitation24 Apr
  5. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systems19 Apr
  6. 2026TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories8 Apr
  7. 2026CoopGuard: Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Round Attacks5 Apr
  8. 2026Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems3 Apr
  9. 2026TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems16 Mar
  10. 2026Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios11 Mar
  11. 2026What Makes a Good LLM Agent for Real-world Penetration Testing?19 Feb
  12. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agents9 Jan
24 more
  1. 2025Enforcing Temporal Constraints for LLM Agents25 Dec
  2. 2025Automated Penetration Testing with LLM Agents and Classical Planning11 Dec
  3. 2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory29 Sep
  4. 2025PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agents28 Sep
  5. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks16 Sep
  6. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning28 Aug
  7. 2025Design Patterns for Securing LLM Agents against Prompt Injections10 Jun
  8. 2025AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents15 May
  9. 2025RedTeamLLM: an Agentic AI framework for offensive security11 May
  10. 2025Prompt Injection Attack to Tool Selection in LLM Agents28 Apr
  11. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debate23 Apr
  12. 2025Memory Injection Attacks on LLM Agents via Query-Only Interaction5 Mar
  13. 2025VulnBot: Autonomous Penetration Testing for A Multi-Agent Collaborative Framework23 Jan
  14. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing2 Dec
  15. 2024PentestAgent: Incorporating LLM Agents to Automated Penetration Testing7 Nov
  16. 2024BreachSeek: A Multi-Agent Automated Penetration Tester31 Aug
  17. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification30 Jul
  18. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases17 Jul
  19. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilities2 Jun
  20. 2024Artificial Intelligence as the New Hacker: Developing Agents for Offensive Security9 May
  21. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  22. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast13 Feb
  23. 2024LLM Agents can Autonomously Hack Websites6 Feb
  24. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan

Other references · 48

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  4. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  5. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  6. 2023Poisoning Language Models During Instruction Tuningcited by 254
  7. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  8. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  9. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  10. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  11. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  12. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
36 more
  1. 2023Getting pwn'd by AI: Penetration Testing with Large Language Modelscited by 82
  2. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  3. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  4. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  5. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  6. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  7. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  8. 2025PromptShield: Deployable Detection for Prompt Injection Attackscited by 43
  9. 2025TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generationcited by 37
  10. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  11. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  12. 2024RevPRAG: Revealing Poisoning Attacks in Retrieval-Augmented Generation through LLM Activation Analysiscited by 33
  13. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  14. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  15. 2024PenHeal: A Two-Stage LLM Framework for Automated Pentesting and Optimal Remediationcited by 30
  16. 2025Traceback of Poisoning Attacks to Retrieval-Augmented Generationcited by 28
  17. 2023LLMs as Hackers: Autonomous Linux Privilege Escalation Attackscited by 25
  18. 2025Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networkscited by 17
  19. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  20. 2025Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Designcited by 13
  21. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 12
  22. 2025Large Reasoning Models Are Autonomous Jailbreak Agentscited by 12
  23. 2025Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generationcited by 12
  24. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMscited by 10
  25. 2024Acquiring Clean Language Models from Backdoor Poisoned Datasets by Downscaling Frequency Spacecited by 6
  26. 2024Gracefully Filtering Backdoor Samples for Generative Large Language Models without Retrainingcited by 5
  27. 2025P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMscited by 4
  28. 2026Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testingcited by 4
  29. 2023A Red Teaming Framework for Securing AI in Maritime Autonomous Systemscited by 3
  30. 2024A Survey on Offensive AI Within Cybersecuritycited by 3
  31. 2026Cochise: A Reference Harness for Autonomous Penetration Testingcited by 3
  32. 2025LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Userscited by 2
  33. 2025AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Softwarecited by 2
  34. 2026RAGuard: A Layered Defense Framework for Retrieval-Augmented Generation Systems Against Data Poisoningcited by 2
  35. 2025Offensive Security for AI Systems: Concepts, Practices, and Applicationscited by 1
  36. 2025SCOUT: A Defense Against Data Poisoning Attacks in Fine-Tuned Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ASTRA: Agentic Steerability and Risk Assessment Framework
  2. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks
  3. 2026Big Enough to Break Out: Tracking the Rising Capability of LLM Penetration-Testing Agents
  4. 2025LlamaFirewall: An open source guardrail system for building secure AI agents
  5. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomy
  6. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows
  7. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection
  8. 2024AutoPenBench: Benchmarking Generative Agents for Penetration Testing