mlsanityData to 14 Oct 2026

SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs

arXiv 2609.08452 ↗First posted 8 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Multi-agent systems enable complex reasoning and tool use by coordinating agents that divide roles and refine candidate solutions. Existing methods typically update individual agent responses or treat a complete trajectory as one training example. However, these methods may produce misleading policy updates because they assign the same final outcome to responses or trajectory segments that may play different roles in different team decisions. This is because treating each response as an independent update may separate outputs that jointly determine the next action, while treating an entire trajectory as one update may combine decisions made after different observations. These limitations call for a policy update defined at the level of a team decision, outputs that lead to the same state transition are optimized under a shared objective. In this paper, we propose Setwise Relative Policy Optimization (SRPO) for multi-agent systems. We represent the outputs used together to produce one state transition as an active set. A singleton set covers division of labor, while a larger set covers joint co-evolution, and the set composition can change across decisions. SRPO assigns a shared advantage to each active set, clips the combined policy change, and normalizes its scale according to the set size. This ties each update to the decision that produced the next state. Experiments on mathematical reasoning and multi-turn search demonstrate the effectiveness of SRPO across both tasks. Additional ablation studies analyze the normalization choice and training behavior under changing active-set sizes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 32 of its 32 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

32 of 32 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks22 Jul
  2. 2026AgentJet: A Distributed Swarm Training Framework for Agentic Reinforcement Learning3 Jun
  3. 2026Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces4 May
  4. 2026PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planning14 Feb
  5. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems9 Feb
  6. 2025Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO17 Nov
  7. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs13 Oct
  8. 2025Multi-Agent Tool-Integrated Policy Optimization6 Oct
  9. 2025LLM Collaboration With Multi-Agent Reinforcement Learning6 Aug
  10. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning3 Jun
  11. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  12. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb
1 more
  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  12. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
7 more
  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  2. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  3. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  4. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  5. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  6. 2026Phase-Aware Mixture of Experts for Agentic Reinforcement Learningcited by 6
  7. 2026AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectoriescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
  2. 2025Training-Free Group Relative Policy Optimization
  3. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  4. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  5. 2025Agentic Reinforced Policy Optimization
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
  7. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
  8. 2025Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning