mlsanityData to 14 Oct 2026

Multi-Agent Tool-Integrated Policy Optimization

arXiv 2510.04678 ↗First posted 6 Oct 2025cs.CLcited by 16

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language models (LLMs) increasingly rely on multi-turn tool-integrated planning for knowledge-intensive and complex reasoning tasks. Existing implementations typically rely on a single agent, but they suffer from limited context length and noisy tool responses. A natural solution is to adopt a multi-agent framework with planner- and worker-agents to manage context. However, no existing methods support effective reinforcement learning post-training of tool-integrated multi-agent frameworks. To address this gap, we propose Multi-Agent Tool-Integrated Policy Optimization (MATPO), which enables distinct roles (planner and worker) to be trained within a single LLM instance using role-specific prompts via reinforcement learning. MATPO is derived from a principled credit assignment mechanism across planner and worker rollouts. This design eliminates the need to deploy multiple LLMs, which would be memory-intensive, while preserving the benefits of specialization. Experiments on GAIA-text, WebWalkerQA, and FRAMES show that MATPO consistently outperforms single-agent baselines by an average of 18.38% relative improvement in performance and exhibits greater robustness to noisy tool outputs. Our findings highlight the effectiveness of unifying multiple agent roles within a single LLM and provide practical insights for stable and efficient multi-agent RL training.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 25 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

17 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning2 Sep
  2. 2025ToolRL: Reward is All Tool Learning Needs16 Apr
  3. 2025ToRL: Scaling Tool-Integrated RL30 Mar

Other references · 14

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  6. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  7. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  8. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  9. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  10. 2023Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Frameworkcited by 139
  11. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 122
  12. 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 121
2 more
  1. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 41
  2. 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Taskscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 61
  2. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  3. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOcited by 18
  4. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
  5. 2026The Interplay of Harness Design and Post-Training in LLM Agentscited by 10
  6. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  7. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent Systemcited by 7
  8. 2026PerspectiveGap: A Benchmark for Multi-Agent Orchestration Promptingcited by 6
  9. 2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Tracescited by 5
  10. 2026Agentic Tool Use in Large Language Modelscited by 4
  11. 2026ICRL: Learning to Internalize Self-Critique with Reinforcement Learningcited by 4
  12. 2026EvoDS: Self-Evolving Autonomous Data Science Agent with Skill Learning and Context Managementcited by 2
4 more
  1. 2026APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answeringcited by 2
  2. 2026WikiLoop: Jointly Learning to Build and Navigate Agent-Native Wikis with Downstream Feedbackcited by 0
  3. 2026ExRole: From Team Trajectories to Executable Roles in Multi-Agent Language Modelscited by 0
  4. 2026SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  3. 2025Agentic Reinforced Policy Optimization
  4. 2025Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning
  5. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  6. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards
  7. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
  8. 2025Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning