# A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems

arXiv: https://arxiv.org/abs/2605.17075 · HTML: https://mlsanity.com/paper/2605.17075/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 16 May 2026.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing](https://mlsanity.com/paper/2412.01778/) (2024)
- [Network Environment Design for Autonomous Cyberdefense](https://mlsanity.com/paper/2103.07583/) (2021)
- [Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report](https://mlsanity.com/paper/2504.21039/) (2025)
- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [Out of the Cage: How Stochastic Parrots Win in Cyber Security Environments](https://mlsanity.com/paper/2308.12086/) (2023)
- [Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense](https://mlsanity.com/paper/2410.17351/) (2024)
- [CybORG: An Autonomous Cyber Operations Research Gym](https://mlsanity.com/paper/2002.10667/) (2020)
- [Large Language Models are Autonomous Cyber Defenders](https://mlsanity.com/paper/2505.04843/) (2025)
- [Automated Cyber Defense with Generalizable Graph-based Reinforcement Learning Agents](https://mlsanity.com/paper/2509.16151/) (2025)
- [Deep PackGen: A Deep Reinforcement Learning Framework for Adversarial Network Packet Generation](https://mlsanity.com/paper/2305.11039/) (2023)
- [Large Language Model Integration with Reinforcement Learning to Augment Decision-Making in Autonomous Cyber Operations](https://mlsanity.com/paper/2509.05311/) (2025)
- [Toward Cybersecurity-Expert Small Language Models](https://mlsanity.com/paper/2510.14113/) (2025)

## Built on by (most cited first, 1 of 1)

- [No One Architecture Fits All: A Cross-Environment Evaluation of Hierarchical Red Team Agents](https://mlsanity.com/paper/2610.00557/) (2026)

## Abstract (arXiv)

AI-enabled Security Orchestration, Automation, and Response (SOAR) systems increasingly employ autonomous agents for cyber defense, yet their resilience to adaptive adversaries is underexplored. We introduce an autonomous red teaming framework that integrates large language models (LLMs) with reinforcement learning (RL) to generate adaptive, multi-stage attack campaigns against autonomous defenders in enterprise networks. A hierarchical design combines an LLM-based planner for strategic intent with an RL controller for tactical execution, supported by reward shaping aligned with kill-chain progression. Evaluation in a high-fidelity enterprise simulation demonstrates the effectiveness of the proposed approach, while also showing that standalone LLM agents fail to sustain multi-stage attack campaigns and that domain-specific cybersecurity models achieve only limited levels of compromise, highlighting the necessity for hybrid LLM-RL approaches to red teaming.