# Red-Teaming the Agentic Red-Team

arXiv: https://arxiv.org/abs/2606.24496 · HTML: https://mlsanity.com/paper/2606.24496/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 23 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 22 of 22)

- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Memory Injection Attacks on LLM Agents via Query-Only Interaction](https://mlsanity.com/paper/2503.03704/) (2025)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks](https://mlsanity.com/paper/2403.03792/) (2024)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [Human-inspired Episodic Memory for Infinite Context LLMs](https://mlsanity.com/paper/2407.09450/) (2024)
- [BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems](https://mlsanity.com/paper/2505.15216/) (2025)
- [An Empirical Evaluation of LLMs for Solving Offensive Security Challenges](https://mlsanity.com/paper/2402.11814/) (2024)
- [ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?](https://mlsanity.com/paper/2605.11086/) (2026)
- [Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networks](https://mlsanity.com/paper/2501.16466/) (2025)
- [Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks](https://mlsanity.com/paper/2410.20911/) (2024)
- [Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing](https://mlsanity.com/paper/2512.09882/) (2025)
- [Deep Reinforcement Fuzzing](https://mlsanity.com/paper/1801.04589/) (2018)
- [Improving Password Guessing via Representation Learning](https://mlsanity.com/paper/1910.04232/) (2019)
- [Synthesizing Multi-Agent Harnesses for Vulnerability Discovery](https://mlsanity.com/paper/2604.20801/) (2026)
- [Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerous](https://mlsanity.com/paper/2508.12175/) (2025)
- [AI Agents May Always Fall for Prompt Injections](https://mlsanity.com/paper/2605.17634/) (2026)
- [CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities](https://mlsanity.com/paper/2606.04460/) (2026)
- [Takedown: How It's Done in Modern Coding Agent Exploits](https://mlsanity.com/paper/2509.24240/) (2025)
- [Universal Neural-Cracking-Machines: Self-Configurable Password Models from Auxiliary Data](https://mlsanity.com/paper/2301.07628/) (2023)
- [Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning](https://mlsanity.com/paper/2605.09822/) (2026)

## Abstract (arXiv)

The use of agentic systems to perform offensive security operations has moved from a theoretical possibility to a commoditized capability. However, while the community has focused on creating more and more capable agents, less attention has been allocated to assessing the security of those systems. In this work, we present the first in-depth security analysis of the most widely used agentic systems for offensive security operations. We show that most of these tools share common design flaws that enable an active adversary to exfiltrate API keys, establish persistent footholds, and fully compromise the operator's machine, even when the agent operates inside a sandboxed container. To support our analysis, we introduce a full cyber kill chain for such agentic systems, capturing the progression from initial LLM manipulation to lateral movement, persistence, guardrail bypass, and sandbox escape. Building on our security analysis, we derive a robust architecture for agentic offensive-security tools and propose actionable, broadly applicable design principles that mitigate the disclosed attack paths at the architectural level.