# From Promise to Peril: Rethinking Cybersecurity Red and Blue Teaming in the Age of LLMs

arXiv: https://arxiv.org/abs/2506.13434 · HTML: https://mlsanity.com/paper/2506.13434/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2025.

## Built on by (most cited first, 10 of 10)

- [From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization](https://mlsanity.com/paper/2510.02389/) (2025)
- [CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics](https://mlsanity.com/paper/2508.20643/) (2025)
- [CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarking](https://mlsanity.com/paper/2602.08023/) (2026)
- [Human Society-Inspired Approaches to Agentic AI Security: The 4C Framework](https://mlsanity.com/paper/2602.01942/) (2026)
- [APT-Agent: Automated Penetration Testing using Large Language Models](https://mlsanity.com/paper/2605.24949/) (2026)
- [Like a Hammer, It Can Build, It Can Break: Large Language Model Uses, Perceptions, and Adoption in Cybersecurity Operations on Reddit](https://mlsanity.com/paper/2604.09998/) (2026)
- [Autonomous Adversary: Red-Teaming in the age of LLM](https://mlsanity.com/paper/2605.06486/) (2026)
- [Coherence-driven inference for cybersecurity](https://mlsanity.com/paper/2509.18520/) (2025)
- [DisCTI: Who Needs to Know Timely? Automated Sector-Aware Cyber Threat Intelligence Dissemination](https://mlsanity.com/paper/2608.27967/) (2026)
- [Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges](https://mlsanity.com/paper/2609.23894/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are set to reshape cybersecurity by augmenting red and blue team operations. Red teams can exploit LLMs to plan attacks, craft phishing content, simulate adversaries, and generate exploit code. Conversely, blue teams may deploy them for threat intelligence synthesis, root cause analysis, and streamlined documentation. This dual capability introduces both transformative potential and serious risks. This position paper maps LLM applications across cybersecurity frameworks such as MITRE ATT&CK and the NIST Cybersecurity Framework (CSF), offering a structured view of their current utility and limitations. While LLMs demonstrate fluency and versatility across various tasks, they remain fragile in high-stakes, context-heavy environments. Key limitations include hallucinations, limited context retention, poor reasoning, and sensitivity to prompts, which undermine their reliability in operational settings. Moreover, real-world integration raises concerns around dual-use risks, adversarial misuse, and diminished human oversight. Malicious actors could exploit LLMs to automate reconnaissance, obscure attack vectors, and lower the technical threshold for executing sophisticated attacks. To ensure safer adoption, we recommend maintaining human-in-the-loop oversight, enhancing model explainability, integrating privacy-preserving mechanisms, and building systems robust to adversarial exploitation. As organizations increasingly adopt AI driven cybersecurity, a nuanced understanding of LLMs' risks and operational impacts is critical to securing their defensive value while mitigating unintended consequences.