# RedTeamLLM: an Agentic AI framework for offensive security

arXiv: https://arxiv.org/abs/2505.06913 · HTML: https://mlsanity.com/paper/2505.06913/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 11 May 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 28 of 28)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023)
- [Structured Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2305.06599/) (2023)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [Chain of Code: Reasoning with a Language Model-Augmented Code Emulator](https://mlsanity.com/paper/2312.04474/) (2023)
- [LLM Agents can Autonomously Hack Websites](https://mlsanity.com/paper/2402.06664/) (2024)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents](https://mlsanity.com/paper/2305.14564/) (2023)
- [CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models](https://mlsanity.com/paper/2408.01605/) (2024)
- [TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation](https://mlsanity.com/paper/2402.10178/) (2024)
- [PentestAgent: Incorporating LLM Agents to Automated Penetration Testing](https://mlsanity.com/paper/2411.05185/) (2024)
- [Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models](https://mlsanity.com/paper/2403.04786/) (2024)
- [HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing](https://mlsanity.com/paper/2412.01778/) (2024)
- [EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations](https://mlsanity.com/paper/2410.22821/) (2024)
- [Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attacks](https://mlsanity.com/paper/2310.05595/) (2023)
- [Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface](https://mlsanity.com/paper/2501.09798/) (2025)
- [On the (In)Security of LLM App Stores](https://mlsanity.com/paper/2407.08422/) (2024)
- [DarkBERT: A Language Model for the Dark Side of the Internet](https://mlsanity.com/paper/2305.08596/) (2023)
- [Security Threats in Agentic AI System](https://mlsanity.com/paper/2410.14728/) (2024)
- [Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements](https://mlsanity.com/paper/2410.17141/) (2024)
- [Agentic AI and the Cyber Arms Race](https://mlsanity.com/paper/2503.04760/) (2025)
- [Countering Autonomous Cyber Threats](https://mlsanity.com/paper/2410.18312/) (2024)

## Built on by (most cited first, 7 of 7)

- [A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes](https://mlsanity.com/paper/2601.05293/) (2026)
- [Hiding in the AI Traffic: Abusing MCP for LLM-Powered Agentic Red Teaming](https://mlsanity.com/paper/2511.15998/) (2025)
- [Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing](https://mlsanity.com/paper/2604.05719/) (2026)
- [Neuro-Symbolic AI for Cybersecurity: State of the Art, Challenges, and Opportunities](https://mlsanity.com/paper/2509.06921/) (2025)
- [CyberAId: AI-Driven Cybersecurity for Financial Service Providers](https://mlsanity.com/paper/2605.01892/) (2026)
- [Recognition Without Mitigation: Ethical Frameworks in Autonomous Offensive-LLM Agent Research](https://mlsanity.com/paper/2506.08693/) (2025)
- [Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectives](https://mlsanity.com/paper/2609.16694/) (2026)

## Abstract (arXiv)

From automated intrusion testing to discovery of zero-day attacks before software launch, agentic AI calls for great promises in security engineering. This strong capability is bound with a similar threat: the security and research community must build up its models before the approach is leveraged by malicious actors for cybercrime. We therefore propose and evaluate RedTeamLLM, an integrated architecture with a comprehensive security model for automatization of pentest tasks. RedTeamLLM follows three key steps: summarizing, reasoning and act, which embed its operational capacity. This novel framework addresses four open challenges: plan correction, memory management, context window constraint, and generality vs. specialization. Evaluation is performed through the automated resolution of a range of entry-level, but not trivial, CTF challenges. The contribution of the reasoning capability of our agentic AI framework is specifically evaluated.