mlsanityData to 14 Oct 2026

MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs

arXiv 2510.15414 ↗First posted 17 Oct 2025cs.AIcited by 16

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Developing Large Language Models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective for enhancing reasoning in single-agent tasks, its extension to multi-turn, multi-agent scenarios remains underexplored due to the challenges of long-horizon credit assignment and agent-specific advantage estimation. To address these challenges, we introduce MARSHAL, an end-to-end RL framework that incentivizes Multi-Agent Reasoning through Self-play witH strAtegic LLMs in both cooperative and competitive games. MARSHAL features a turn-level advantage estimator that aligns learning signals with each interaction for credit assignment, and an agent-specific advantage normalization to stabilize multi-agent training. By learning with self-play across cooperative and competitive games, MARSHAL agents trained from Qwen3-4B develop strong strategic abilities, with up to 28.7% performance improvements in held-out games. More importantly, the capability acquired through self-play generalizes beyond games, yielding consistent performance gains of MASs in reasoning benchmarks. When integrated into leading MASs, our MARSHAL agent achieves significant zero-shot performance gains of up to 10.0% on AIME, 7.6% on GPQA-Diamond, and 3.5% on average across all benchmarks. These results establish self-play in strategic games as a powerful approach for developing generalizable multi-agent reasoning capabilities in LLMs.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 35 of its 47 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

35 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun
  2. 2024OpenAI o1 System Card21 Dec
  3. 2024AFlow: Automating Agentic Workflow Generation14 Oct
  4. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
19 more
  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  5. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  6. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  7. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  9. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  10. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  11. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  12. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  13. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  14. 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 100
  15. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 79
  16. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  17. 2025Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimizationcited by 15
  18. 2025MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systemscited by 14
  19. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  2. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
  3. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10
  4. 2026WideSeek-R1: Exploring Width Scaling for Broad Information Seeking via Multi-Agent Reinforcement Learningcited by 9
  5. 2026Digital Red Queen: Adversarial Program Evolution in Core War with LLMscited by 7
  6. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  7. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoningcited by 6
  8. 2026Game-Theoretic Lens on LLM-based Multi-Agent Systemscited by 3
  9. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  10. 2026Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Executioncited by 2
  11. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1
  12. 2026Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Gamescited by 0
4 more
  1. 2026Equilibrium Residuals Expose Three Regimes of Matrix-Game Strategic Reasoning in Language Modelscited by 0
  2. 2026IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agentscited by 0
  3. 2026GameSmith: Scaling Zero-Sum Self-Play with Synthesised Gamescited by 0
  4. 2026Choosing LLMs for Strategic Agent Interactionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  2. 2026Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents
  3. 2026Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments
  4. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
  5. 2026Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments
  6. 2026SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
  7. 2025Dynamic Strategy Adaptation in Multi-Agent Environments with Large Language Models
  8. 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs