mlsanityData to 14 Oct 2026

Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning

arXiv 2509.03817 ↗First posted 4 Sep 2025cs.AI cs.MAcited by 18

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Multi-agent systems of large language models (LLMs) show promise for complex reasoning, but their effectiveness is often limited by fixed collaboration protocols. These frameworks typically focus on macro-level orchestration while overlooking agents' internal deliberative capabilities. This critical meta-cognitive blindspot treats agents as passive executors unable to adapt their strategy based on internal cognitive states like uncertainty or confidence. We introduce the Meta-Policy Deliberation Framework (MPDF), where agents learn a decentralized policy over a set of high-level meta-cognitive actions: Persist, Refine, and Concede. To overcome the instability of traditional policy gradients in this setting, we develop SoftRankPO, a novel reinforcement learning algorithm. SoftRankPO stabilizes training by shaping advantages based on the rank of rewards mapped through smooth normal quantiles, making the learning process robust to reward variance. Experiments show that MPDF with SoftRankPO achieves a a 4-5% absolute gain in average accuracy across five mathematical and general reasoning benchmarks compared to six state-of-the-art heuristic and learning-based multi-agent reasoning algorithms. Our work presents a paradigm for learning adaptive, meta-cognitive policies for multi-agent LLM systems, shifting the focus from designing fixed protocols to learning dynamic, deliberative strategies.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 28 of its 58 references that are in the map. First posted on arXiv 4 Sep 2025.

Builds on

28 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Tasks4 Mar
  2. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb
  3. 2024MALT: Improving Reasoning with Multi-Agent LLM Training2 Dec
  4. 2024GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion21 Sep

Other references · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  8. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  9. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 251
  10. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  11. 2024AgentSquare: Automatic LLM Agent Search in Modular Design Spacecited by 155
  12. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 87
12 more
  1. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimizationcited by 84
  2. 2023CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agentscited by 65
  3. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 55
  4. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 52
  5. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  6. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  7. 2024AGILE: A Novel Reinforcement Learning Framework of LLM Agentscited by 45
  8. 2025ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineeringcited by 26
  9. 2025HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDLcited by 17
  10. 2025Discriminative Policy Optimization for Token-Level Reward Modelscited by 16
  11. 2025Enhancing Multi-Agent Systems via Reinforcement Learning with LLM-based Planner and Graph-based Policycited by 12
  12. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistantscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  2. 2026Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judgecited by 18
  3. 2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
  4. 2026Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learningcited by 13
  5. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
  6. 2025Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendationcited by 6
  7. 2026RaMem: Contextual Reinstatement for Long-term Agentic Memorycited by 6
  8. 2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Tracescited by 5
  9. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  10. 2025Verifying Meta-Awareness via Predictive Rewards in Reasoning Modelscited by 3
  11. 2026De-conflating Preference and Qualification: Constrained Dual-Perspective Reasoning for Job Recommendation with Large Language Modelscited by 2
  12. 2026When Does Multi-Agent RL Improve LLM Workflows? Workflow, Scale, and Policy-Sharing Tradeoffscited by 1
6 more
  1. 2026ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capabilitycited by 1
  2. 2026MIRROR: Learning from the Other View for Multi-Modal Reasoningcited by 1
  3. 2026GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMscited by 0
  4. 2026Controlled Decoding Attacks on Black-Box LLMscited by 0
  5. 2026Beyond Final Accuracy: Auditing Communication in LLM Multi-Agent Systemscited by 0
  6. 2026Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning
  2. 2025Multi-Agent Tool-Integrated Policy Optimization
  3. 2025OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoning
  4. 2024YOLO-MARL: You Only LLM Once for Multi-Agent Reinforcement Learning
  5. 2026MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation
  6. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  7. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation
  8. 2026Rethinking Agentic Reinforcement Learning In Large Language Models