mlsanityData to 7 Oct 2026

LLM Collaboration With Multi-Agent Reinforcement Learning

arXiv 2508.04652 ↗First posted 6 Aug 2025cs.AI cs.SEcited by 47

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

A large amount of work has been done in Multi-Agent Systems (MAS) for modeling and solving problems with multiple interacting agents. However, most LLMs are pretrained independently and not specifically optimized for coordination. Existing LLM fine-tuning frameworks rely on individual rewards, which require complex reward designs for each agent to encourage collaboration. To address these challenges, we model LLM collaboration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. We develop a multi-agent, multi-turn algorithm, Multi-Agent Group Relative Policy Optimization (MAGRPO), to solve it, building on current RL approaches for LLMs as well as MARL techniques. Our experiments on LLM writing and coding collaboration demonstrate that fine-tuning MAS with MAGRPO enables agents to generate high-quality responses efficiently through effective cooperation. Our approach opens the door to using other MARL methods for LLMs and highlights the associated challenges. Our code is available at https://github.com/OpenMLRL/CoMLRL.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 58 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

25 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs11 Jul
  2. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  12. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
11 more
  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  2. 2019The StarCraft Multi-Agent Challengecited by 589
  3. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  4. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  5. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  6. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  7. 2023Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Promptingcited by 45
  8. 2024RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Modelscited by 21
  9. 2024Pruning Foundation Models for High Accuracy without Retrainingcited by 19
  10. 2025Training Language Models for Social Deduction with Multi-Agent Reinforcement Learningcited by 15
  11. 2024An Initial Introduction to Cooperative Multi-Agent Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 35.

47 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  4. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  5. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 15
  6. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 13
  7. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  8. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  9. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
  10. 2026Exact Is Easier: Credit Assignment for Cooperative LLM Agentscited by 9
  11. 2025Build Your Personalized Research Group: A Multiagent Framework for Continual and Interactive Science Automationcited by 8
  12. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
35 more
  1. 2025LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systemscited by 8
  2. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 8
  3. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 7
  4. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learningcited by 7
  5. 2026SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestrationcited by 6
  6. 2025Interactive Learning for LLM Reasoningcited by 4
  7. 2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Tracescited by 4
  8. 2025AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learningcited by 3
  9. 2026MARTI-MARS2: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generationcited by 3
  10. 2026Scaling Multiagent Systems with Process Rewardscited by 2
  11. 2026Adaptive Robust Estimator for Multi-Agent Reinforcement Learningcited by 2
  12. 2026Does RL Expand the Capability Boundary of LLM Agents? A PASS@(k,T) Analysiscited by 2
  13. 2025CosmoCore Affective Dream-Replay Reinforcement Learning for Code Generationcited by 1
  14. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debatecited by 1
  15. 2026TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordinationcited by 1
  16. 2026When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffscited by 1
  17. 2026Route, Communicate, and Reason: Gated Routing and Adaptive Depth for Efficient Multi-Agent Reasoningcited by 1
  18. 2026MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Rerankingcited by 1
  19. 2025ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspectivecited by 0
  20. 2025Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agentscited by 0
  21. 2026PRIME: Policy-Reinforced Iterative Multi-agent Execution for Algorithmic Reasoning in Large Language Modelscited by 0
  22. 2026COOP2: Defining, Observing, and Repairing Cooperation in LLM Multi-Agent Systemscited by 0
  23. 2026AIT Academy: Cultivating the Complete Agent with a Confucian Three-Domain Curriculumcited by 0
  24. 2026Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Gamescited by 0
  25. 2026Reinforced Collaboration in Multi-Agent Flow Networkscited by 0
  26. 2026TRACER: Turn-level Regret Matching with Inner Reinforcement Credit for Cooperative Multi-LLM Reasoningcited by 0
  27. 2026GeoDisaster: Benchmarking Orchestrated Agents for Operational Disaster Geo-Intelligencecited by 0
  28. 2026Mathematical methods of reinforcement learningcited by 0
  29. 2026A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturingcited by 0
  30. 2026At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agentcited by 0
  31. 2026SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMscited by 0
  32. 2026AlphaDiverse: Post-Training Local Quantitative Research Agents for Diverse Exploration in Alpha Factor Miningcited by 0
  33. 2026Reinforcement Learning of Communication in a Mesh of Small Language Modelscited by 0
  34. 2026Decentralized Master-Mind: Joint Action Refinement through Iterative Intent Denoising in Multi-Agent Pathfindingcited by 0
  35. 2026Improving LLM Collaboration via Multi-Agent Preference Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policy
  2. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  3. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  4. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
  5. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions
  6. 2026Training Small LLMs as Spatial Multi-Agent Policies
  7. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
  8. 2025Group-in-Group Policy Optimization for LLM Agent Training