mlsanityData to 7 Oct 2026

How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs

arXiv 2507.08960 ↗First posted 11 Jul 2025cs.MA cs.AI cs.LGcited by 11

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have achieved strong performance on a wide range of complex reasoning tasks, yet further gains are often possible by leveraging the complementary strengths of multiple models. While multi-agent frameworks can improve solution quality by leveraging multiple LLMs, existing methods are often computationally expensive, both at training and inference time. In this work, we introduce a hierarchical multi-agent framework that addresses these challenges by training only a single leader LLM to coordinate a team of untrained peer agents. To this end, we propose Multi-agent guided Leader Policy Optimization (MLPO), a novel approach which trains the leader to evaluate and synthesize agent responses without auxiliary value networks or explicit agent feedback. Leaders trained with MLPO exhibit improved performance not only when interacting with the agent team at inference time, but also enjoy improved performance when deployed in single-agent settings without the team. Empirical results on Big-Bench Hard (BBH), MATH, and MMLU demonstrate that our framework achieves substantial performance improvements over both single-agent and multi-agent baselines. Our results highlight the effectiveness and efficiency of training a single, flexible leader for collaborative reasoning in multi-agent LLM systems.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 64 of its 67 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

64 of 67 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoning14 Apr
  2. 2025Debate Only When Necessary: Adaptive Multiagent Collaboration for Efficient LLM Reasoning7 Apr
  3. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks19 Mar
  4. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning12 Mar
  5. 2025Multi-LLM Collaborative Search for Complex Problem Solving26 Feb
  6. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb
  7. 2024MALT: Improving Reasoning with Multi-Agent LLM Training2 Dec
  8. 2024GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion21 Sep
  9. 2024Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Models17 Jul
  10. 2024Improving Multi-Agent Debate with Sparse Communication Topology17 Jun
  11. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  12. 2024SocraSynth: Multi-LLM Reasoning with Conditional Statistics19 Jan
2 more
  1. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs22 Sep
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
38 more
  1. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  12. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  13. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  14. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  15. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  16. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  17. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  18. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  19. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  20. 2023Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debatecited by 94
  21. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systemscited by 86
  22. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
  23. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 66
  24. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  25. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  26. 2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Modelscited by 36
  27. 2025X-MAS: Towards Building Multi-Agent Systems with Heterogeneous LLMscited by 33
  28. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learningcited by 32
  29. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agentscited by 30
  30. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  31. 2025Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skillscited by 24
  32. 2024ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaborationcited by 22
  33. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  34. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  35. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  36. 2024Towards Hierarchical Multi-Agent Workflows for Zero-Shot Prompt Optimizationcited by 10
  37. 2025Two Heads Are Better Than One: Dual-Model Verbal Reflection at Inference-Timecited by 8
  38. 2024SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025LLM Collaboration With Multi-Agent Reinforcement Learningcited by 47
  3. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
  4. 2026MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarkscited by 26
  5. 2025Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoningcited by 11
  6. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  7. 2025MarsRL: Advancing Multi-Agent Reasoning System via Reinforcement Learning with Agentic Pipeline Parallelismcited by 4
  8. 2026Can LLMs Perceive Time? An Empirical Investigationcited by 2
  9. 2025MAS2: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systemscited by 1
  10. 2026MACRO-LLM: LLM-Empowered Multi-Agent Collaborative Reasoning under Spatiotemporal Partial Observabilitycited by 1
  11. 2026Mathematical methods of reinforcement learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees
  2. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO
  3. 2026Training-Free Agentic AI: Probabilistic Control and Coordination in Multi-Agent LLM Systems
  4. 2025Multi-Agent Tool-Integrated Policy Optimization
  5. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
  6. 2025Controlling Performance and Budget of a Centralized Multi-agent LLM System with Reinforcement Learning
  7. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration
  8. 2025OPTAGENT: Optimizing Multi-Agent LLM Interactions Through Verbal Reinforcement Learning for Enhanced Reasoning