mlsanityData to 14 Oct 2026

AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

arXiv 2610.19422 ↗First posted 14 Oct 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance also fits: Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 38 of its 46 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

38 of 46 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents14 Oct
  2. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safety6 Oct
  3. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools4 Aug
  4. 2025Agent Safety Alignment via Reinforcement Learning11 Jul
  5. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasures12 Mar
  6. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents14 Nov
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  9. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  10. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  11. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 27

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2025OpenAI GPT-5 System Cardcited by 989
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  12. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
15 more
  1. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  2. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 86
  3. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  4. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
  5. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 65
  6. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  7. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 50
  8. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
  9. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMscited by 28
  10. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  11. 2025Tool Preferences in Agentic LLMs are Unreliablecited by 18
  12. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 12
  13. 2024SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Modelscited by 9
  14. 2025Safety Alignment of LMs via Non-cooperative Gamescited by 8
  15. 2024Purple-teaming LLMs with Adversarial Defender Trainingcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning
  2. 2025Adversarial Attack-Defense Co-Evolution for LLM Safety Alignment via Tree-Group Dual-Aware Search and Optimization
  3. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety
  4. 2026Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety
  5. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems
  6. 2026CoopGuard: Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Round Attacks
  7. 2026CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents
  8. 2026CHASE: Adversarial Red-Blue Teaming for Improving LLM Safety using Reinforcement Learning