# Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety

arXiv: https://arxiv.org/abs/2508.03864 · HTML: https://mlsanity.com/paper/2508.03864/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 40 of its 44 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 30 of 40)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://mlsanity.com/paper/2307.05300/) (2023)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [Prompt Injection Attack to Tool Selection in LLM Agents](https://mlsanity.com/paper/2504.19793/) (2025)
- [Memory Injection Attacks on LLM Agents via Query-Only Interaction](https://mlsanity.com/paper/2503.03704/) (2025)
- [Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems](https://mlsanity.com/paper/2410.07283/) (2024)
- [Red-Teaming LLM Multi-Agent Systems via Communication Attacks](https://mlsanity.com/paper/2502.14847/) (2025)
- [On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents](https://mlsanity.com/paper/2408.00989/) (2024)
- [MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2502.18439/) (2025)
- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models](https://mlsanity.com/paper/2402.16717/) (2024)
- [Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2502.14321/) (2025)
- [G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systems](https://mlsanity.com/paper/2502.11127/) (2025)
- [Multi-Agent Systems Execute Arbitrary Malicious Code](https://mlsanity.com/paper/2503.12188/) (2025)
- [MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse](https://mlsanity.com/paper/2503.18470/) (2025)
- [Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.15228/) (2025)
- [textitAgents Under Siege: Breaking Pragmatic Multi-Agent LLM Systems with Optimized Prompt Attacks](https://mlsanity.com/paper/2504.00218/) (2025)

## Built on by (most cited first, 4 of 4)

- [OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs](https://mlsanity.com/paper/2601.01592/) (2026)
- [Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs](https://mlsanity.com/paper/2511.12710/) (2025)
- [AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.01586/) (2025)
- [From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal Models](https://mlsanity.com/paper/2602.22859/) (2026)

## Abstract (arXiv)

Multi-agent systems (MAS) built on multimodal large language models exhibit strong collaboration and performance. However, their growing openness and interaction complexity pose serious risks, notably jailbreak and adversarial attacks. Existing defenses typically rely on external guard modules, such as dedicated safety agents, to handle unsafe behaviors. Unfortunately, this paradigm faces two challenges: (1) standalone agents offer limited protection, and (2) their independence leads to single-point failure-if compromised, system-wide safety collapses. Naively increasing the number of guard agents further raises cost and complexity. To address these challenges, we propose Evo-MARL, a novel multi-agent reinforcement learning (MARL) framework that enables all task agents to jointly acquire defensive capabilities. Rather than relying on external safety modules, Evo-MARL trains each agent to simultaneously perform its primary function and resist adversarial threats, ensuring robustness without increasing system overhead or single-node failure. Furthermore, Evo-MARL integrates evolutionary search with parameter-sharing reinforcement learning to co-evolve attackers and defenders. This adversarial training paradigm internalizes safety mechanisms and continually enhances MAS performance under co-evolving threats. Experiments show that Evo-MARL reduces attack success rates by up to 22% while boosting accuracy by up to 5% on reasoning tasks-demonstrating that safety and utility can be jointly improved.