Data to 7 Oct 2026

MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning

arXiv 2502.18439 ↗First posted 25 Feb 2025cs.AIcited by 80

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more

LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Leveraging multiple large language models (LLMs) to build collaborative multi-agentic workflows has demonstrated significant potential. However, most previous studies focus on prompting the out-of-the-box LLMs, relying on their innate capability for collaboration, which may not improve LLMs' performance as shown recently. In this paper, we introduce a new post-training paradigm MAPoRL (Multi-Agent Post-co-training for collaborative LLMs with Reinforcement Learning), to explicitly elicit the collaborative behaviors and further unleash the power of multi-agentic LLM frameworks. In MAPoRL, multiple LLMs first generate their own responses independently and engage in a multi-turn discussion to collaboratively improve the final answer. In the end, a MAPoRL verifier evaluates both the answer and the discussion, by assigning a score that verifies the correctness of the answer, while adding incentives to encourage corrective and persuasive discussions. The score serves as the co-training reward, and is then maximized through multi-agent RL. Unlike existing LLM post-training paradigms, MAPoRL advocates the co-training of multiple LLMs together using RL for better generalization. Accompanied by analytical insights, our experiments demonstrate that training individual LLMs alone is insufficient to induce effective collaboration. In contrast, multi-agent co-training can boost the collaboration performance across benchmarks, with generalization to unseen domains.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 51 of its 72 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

51 of 72 references.

Closest ancestors · 6

Same direction, newest first

Other references · 45

2022Training language models to follow instructions with human feedbackcited by 15,712
2024The Llama 3 Herd of Modelscited by 14,192
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2023Let's Verify Step by Stepcited by 3,598
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
2019The Curious Case of Neural Text Degenerationcited by 2,881
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916

33 more

2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
2016Concrete Problems in AI Safetycited by 1,582
2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
2022Solving math word problems with process- and outcome-based feedbackcited by 871
2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
2023Language Models can Solve Computer Taskscited by 371
2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
2022Generating Sequences by Learning to Self-Correctcited by 271
2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
2017Inverse Reward Designcited by 246
2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
2023Fine-tuning Language Models for Factualitycited by 219
2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 173
2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 91
2023OVM, Outcome-supervised Value Models for Planning in Mathematical Reasoningcited by 84
2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
2017Guided Deep Reinforcement Learning for Swarm Systemscited by 44
2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 33
2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
2024Teaching Models to Balance Resisting and Accepting Persuasioncited by 19
2023Multi-Player Zero-Sum Markov Games with Networked Separable Interactionscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 32; 2026: 48.

80 papers.

Sort:

By area:

2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
2025Measuring Agents in Productioncited by 35
2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMscited by 25
2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25

38 more

2025OptimAI: Optimization from Natural Language Using LLM-Powered AI Agentscited by 22
2025Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learningcited by 18
2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOcited by 17
2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 15
2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
2025Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learningcited by 13
2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 13
2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
2026Counterfactual Credit Policy Optimization for Multi-Agent Collaborationcited by 10
2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
2025Beyond the Strongest LLM: Multi-Turn Multi-Agent Orchestration vs. Single LLMs on Benchmarkscited by 8
2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 7
2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
2026Epistemic Context Learning: Building Trust the Right Way in LLM-Based Multi-Agent Systemscited by 6
2026PrivAct: Internalizing Contextual Privacy Preservation via Multi-Agent Preference Trainingcited by 6
2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
2025Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directionscited by 4
2025Everyone Contributes! Incentivizing Strategic Cooperation in Multi-LLM Systems via Sequential Public Goods Gamescited by 4
2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safetycited by 4
2025Interactive Learning for LLM Reasoningcited by 4
2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Tracescited by 4
2025Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharingcited by 3
2025AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learningcited by 3
2026Collaborative Multi-Agent Test-Time Reinforcement Learning for Reasoningcited by 3
2026MARTI-MARS2: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generationcited by 3
2026EVOCHAMBER: Test-Time Co-evolution of Multi-Agent System at Individual, Team, and Population Scalescited by 3
2026Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactionscited by 3
2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
2025Learning Robust Social Strategies with Large Language Modelscited by 2
2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
2026Scaling Multiagent Systems with Process Rewardscited by 2
2026Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Executioncited by 2
2026Training Generalizable Collaborative Agents via Strategic Risk Aversioncited by 2
2026CUBE: A Standard for Unifying Agent Benchmarkscited by 2
2026Adaptive Robust Estimator for Multi-Agent Reinforcement Learningcited by 2
2026Towards Scalable Lightweight GUI Agents via Multi-role Orchestrationcited by 2
2026Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signalscited by 2
2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way