mlsanityData to 7 Oct 2026

Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO

arXiv 2511.13288 ↗First posted 17 Nov 2025cs.AIcited by 17

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 50 references that are in the map. First posted on arXiv 17 Nov 2025.

Builds on

42 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2023Let's Verify Step by Step31 May
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  12. 2024A Survey on LLM-as-a-Judgecited by 1,384
26 more
  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  4. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  5. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  6. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  7. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  8. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  12. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  13. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  14. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 171
  15. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 164
  16. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  17. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  18. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  19. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  20. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  21. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 54
  22. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  23. 2025MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Frameworkcited by 22
  24. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  25. 2025Multi-Agent Tool-Integrated Policy Optimizationcited by 16
  26. 2025MARS: Co-evolving Dual-System Deep Research via Multi-Agent Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  2. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 19
  3. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 15
  4. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 7
  5. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent Systemcited by 7
  6. 2026Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Taskscited by 4
  7. 2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Tracescited by 4
  8. 2026Small Agent Group is the Future of Digital Healthcited by 2
  9. 2026EvoDS: Self-Evolving Autonomous Data Science Agent with Skill Learning and Context Managementcited by 2
  10. 2026Self-Manager: Parallel Agent Loop for Long-form Deep Researchcited by 1
  11. 2026Joint Optimization of Multi-agent Memory Systemcited by 1
  12. 2026Learning to Seek Help: Dynamic Collaboration Between Small and Large Language Modelscited by 1
5 more
  1. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1
  2. 2026When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffscited by 1
  3. 2026GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligencecited by 0
  4. 2026SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMscited by 0
  5. 2026AlphaDiverse: Post-Training Local Quantitative Research Agents for Diverse Exploration in Alpha Factor Miningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
  2. 2025Training-Free Group Relative Policy Optimization
  3. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  4. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs
  5. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
  6. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  7. 2025Group-in-Group Policy Optimization for LLM Agent Training
  8. 2025Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning