mlsanityData to 7 Oct 2026

ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspective

arXiv 2509.21134 ↗First posted 25 Sep 2025cs.AI cs.MAcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large Language Models (LLMs) have been used to make decisions in complex scenarios, where they need models to think deeply, reason logically, and decide wisely. Many existing studies focus solely on multi-round conversations in social tasks or simulated environments, neglecting the various types of decisions and their interdependence. Current reinforcement learning methods struggle to consider the strategies of others during training. To address these issues, we first define a strategic decision-making problem that includes two types of decisions and their temporal dependencies. Furthermore, we propose **T**heory **o**f **M**ind **P**olicy **O**ptimization **(ToMPO)** algorithm to optimize the perception of other individual strategies and the game situation trends. Compared to the Group Relative Policy Optimization (GRPO) algorithm, ToMPO enhances the LLM's strategic decision-making mainly by: 1) generating rollouts based on reasoning the strategies of other individuals, 2) estimating advantages at both the graph-level and sample-level, and 3) balancing global and partial rewards. The ToMPO algorithm outperforms the GRPO method by 35% in terms of model output compliance and cooperative outcomes. Additionally, when compared to models with parameter sizes 100 times larger, it shows an 18% improvement. This demonstrates the effectiveness of the ToMPO algorithm in enhancing the model's strategic decision-making capabilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 52 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

34 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LLM Collaboration With Multi-Agent Reinforcement Learning6 Aug
  2. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities22 Apr
  3. 2025MARFT: Multi-Agent Reinforcement Fine-Tuning21 Apr
  4. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives17 Mar
  5. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments26 Feb

Other references · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2024Understanding the planning of LLM agents: A surveycited by 414
  6. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  7. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 173
  8. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 129
  9. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 127
  10. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  11. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  12. 2025A Survey of AI Agent Protocolscited by 77
17 more
  1. 2025ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoningcited by 57
  2. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  3. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
  4. 2023Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framingcited by 49
  5. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  6. 2024ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agentscited by 35
  7. 2024Non-myopic Generation of Language Models for Reasoning and Planningcited by 25
  8. 2024K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoningcited by 21
  9. 2025Sotopia-RL: Reward Design for Social Intelligencecited by 20
  10. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  11. 2025Hard Examples Are All You Need: Maximizing GRPO Post-Training Under Annotation Budgetscited by 16
  12. 2025Agent Exchange: Shaping the Future of AI Agent Economicscited by 15
  13. 2025MF-LLM: Simulating Population Decision Dynamics via a Mean-Field Large Language Model Frameworkcited by 12
  14. 2025Dynamic Speculative Agent Planningcited by 10
  15. 2024Large Language Models Playing Mixed Strategy Nash Equilibrium Gamescited by 9
  16. 2025A Review of Incorporating Psychological Theories in LLMscited by 5
  17. 2025Attention Mechanism for LLM-based Agents Dynamic Diffusion under Information Asymmetrycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforced Language Models for Sequential Decision Making
  2. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
  3. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  4. 2025EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning
  5. 2026GameTalk: Training LLMs for Strategic Conversation
  6. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search
  7. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  8. 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models