mlsanityData to 7 Oct 2026

To Defend Against Cyber Attacks, We Must Teach AI Agents to Hack

arXiv 2602.02595 ↗First posted 1 Feb 2026cs.CR cs.AI cs.CYcited by 1

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

For over a decade, cybersecurity has relied on human labor scarcity to limit attackers to high-value targets manually or generic automated attacks at scale. Building sophisticated exploits requires deep expertise and manual effort, leading defenders to assume adversaries cannot afford tailored attacks at scale. AI agents break this balance by automating vulnerability discovery and exploitation across thousands of targets, needing only small success rates to remain profitable. Current developers focus on preventing misuse through data filtering, safety alignment, and output guardrails. Such protections fail against adversaries who control open-weight models, bypass safety controls, or develop offensive capabilities independently. We argue that AI-agent-driven cyber attacks are inevitable, requiring a fundamental shift in defensive strategy. In this position paper, we identify why existing defenses cannot stop adaptive adversaries and demonstrate that defenders must develop offensive security intelligence. We propose three actions for building frontier offensive AI capabilities responsibly. First, construct comprehensive benchmarks covering the full attack lifecycle. Second, advance from workflow-based to trained agents for discovering in-wild vulnerabilities at scale. Third, implement governance restricting offensive agents to audited cyber ranges, staging release by capability tier, and distilling findings into safe defensive-only agents. We strongly recommend treating offensive AI capabilities as essential defensive infrastructure, as containing cybersecurity risks requires mastering them in controlled settings before adversaries do.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 53 of its 87 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

53 of 87 references.

References · 53

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  7. 2021Fast Model Editing at Scalecited by 491
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  9. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  10. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  11. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 246
  12. 2021Alignment of Language Agentscited by 167
38 more
  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  2. 2020A Survey of Privacy Attacks in Machine Learningcited by 143
  3. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  4. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  5. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
  6. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  7. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  8. 2024Rule Based Rewards for Language Model Safetycited by 106
  9. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  10. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  11. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
  12. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  13. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  14. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  15. 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 56
  16. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 51
  17. 2025RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditingcited by 47
  18. 2024Current state of LLM Risks and AI Guardrailscited by 44
  19. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  20. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characterscited by 39
  21. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemscited by 37
  22. 2025Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Modelscited by 29
  23. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 28
  24. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  25. 2025Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Trainingcited by 23
  26. 2025Cyber-Zero: Training Cybersecurity Agents without Runtimecited by 22
  27. 2023Model Extraction Attacks Revisitedcited by 21
  28. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  29. 2024AutoPenBench: Benchmarking Generative Agents for Penetration Testingcited by 20
  30. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
  31. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  32. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  33. 2024CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructionscited by 12
  34. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  35. 2025Locus: Agentic Predicate Synthesis for Directed Fuzzingcited by 8
  36. 2017Economic Factors of Vulnerability Trade and Exploitationcited by 5
  37. 2025LLMs unlock new paths to monetizing exploitscited by 5
  38. 2025PBFuzz: Agentic Directed Fuzzing for PoV Generationcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs
  2. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agents
  3. 2026Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
  4. 2025Agentic AI for Cyber Resilience: A New Security Paradigm and Its System-Theoretic Foundations
  5. 2026A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes
  6. 2024Artificial Intelligence as the New Hacker: Developing Agents for Offensive Security
  7. 2026Rethinking Agent Security as a Networking Problem
  8. 2026Agentic AI and the Industrialization of Cyber Offense: Forecast, Consequences, and Defensive Priorities for Enterprises and the Mittelstand