mlsanityData to 7 Oct 2026

Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems

arXiv 2602.08847 ↗First posted 9 Feb 2026cs.LG cs.AIcited by 15

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6% avg@16 and +4.6% pass@16 on math, and +15.2% avg@16 and +13.1% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 60 of its 83 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

60 of 83 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs17 Oct
  2. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use1 Sep
  3. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL6 Aug
  4. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun
  5. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning12 Mar
  6. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb

Other references · 54

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2025Qwen2.5-VL Technical Reportcited by 5,418
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
38 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  13. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  14. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  15. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  16. 2025Group Sequence Policy Optimizationcited by 788
  17. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  18. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  19. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  20. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  21. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  22. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  23. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  24. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  25. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  26. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  27. 2025Agentic Reinforced Policy Optimizationcited by 166
  28. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  29. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  30. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  31. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  32. 2025Agent Learning via Early Experiencecited by 79
  33. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  34. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  35. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  36. 2025LLM Collaboration With Multi-Agent Reinforcement Learningcited by 47
  37. 2024MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generationcited by 43
  38. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 19
  2. 2026Exact Is Easier: Credit Assignment for Cooperative LLM Agentscited by 9
  3. 2026SkillGraph: Self-Evolving Multi-Agent Collaboration with Multimodal Graph Topologycited by 8
  4. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  5. 2026UnityMAS-O: A General RL Optimization Framework for LLM-Based Multi-Agent Systemscited by 6
  6. 2026Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learningcited by 5
  7. 2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Tracescited by 4
  8. 2026ICRL: Learning to Internalize Self-Critique with Reinforcement Learningcited by 3
  9. 2026EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learningcited by 2
  10. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1
  11. 2026MetaAgent-X : Breaking the Ceiling of Automatic Multi-Agent Systems via End-to-End Reinforcement Learningcited by 1
  12. 2026When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffscited by 1
3 more
  1. 2026MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environmentscited by 1
  2. 2026GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligencecited by 0
  3. 2026SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
  2. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
  3. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic
  4. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions
  5. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  6. 2026Training Small LLMs as Spatial Multi-Agent Policies
  7. 2026MASkills: Continual Skills Optimization for Multi-Agent LLM Systems
  8. 2026LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning