mlsanityData to 7 Oct 2026

Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Search

arXiv 2408.10635 ↗First posted 20 Aug 2024cs.AI cs.CLcited by 16

On the map: Synthetic data and self-improvement for LLMs → Self-play RL for reasoning

Abstract (arXiv)

Traditional reinforcement learning and planning typically requires vast amounts of data and training to develop effective policies. In contrast, large language models (LLMs) exhibit strong generalization and zero-shot capabilities, but struggle with tasks that require detailed planning and decision-making in complex action spaces. We introduce STRATEGIST, a novel approach that integrates the strengths of both methods. Our approach leverages LLMs to search and update high-level strategies (as text), which are then refined and executed by low-level Monte Carlo Tree Search (MCTS). STRATEGIST is a generalizable framework to optimize the strategy through population-based self-play simulations without the need for any training data. We demonstrate the effectiveness of STRATEGIST in learning optimal strategies for competitive, multi-turn games with partial information, including Game of Pure Strategy (GOPS) and multi-agent, hidden-identity discussion games like The Resistance: Avalon. Our results show that agents equipped with STRATEGIST outperform those trained with traditional RL methods, other LLM-based skill acquisition techniques, pre-existing LLM agents across both game environments and achieves comparable performance against human players.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 25 of its 53 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

25 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Self-playing Adversarial Language Game Enhances LLM Reasoning16 Apr
  2. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback17 May

Other references · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  10. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  11. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  12. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
11 more
  1. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  2. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  3. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  4. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  5. 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 94
  6. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 93
  7. 2023KoLA: Carefully Benchmarking World Knowledge of Large Language Modelscited by 85
  8. 2023AvalonBench: Evaluating LLMs Playing the Game of Avaloncited by 45
  9. 2024Enhance Reasoning for Large Language Models in the Game Werewolfcited by 44
  10. 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 42
  11. 2024Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skillscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  2. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  3. 2025Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RLcited by 11
  4. 2025DISC: Dynamic Decomposition Improves LLM Inference Scalingcited by 9
  5. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 7
  6. 2025ReflecSched: Solving Dynamic Flexible Job-Shop Scheduling via LLM-Powered Hierarchical Reflectioncited by 5
  7. 2025The Stackelberg Speaker: Optimizing Persuasive Communication in Social Deduction Gamescited by 3
  8. 2025Cardiverse: Harnessing LLMs for Novel Card Game Prototypingcited by 2
  9. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 1
  10. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 1
  11. 2025Constraints-of-Thought: A Framework for Constrained Reasoning in Language-Model-Guided Searchcited by 1
  12. 2025Agentmandering: A Game-Theoretic Framework for Fair Redistricting via Large Language Model Agentscited by 0
4 more
  1. 2026Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanicscited by 0
  2. 2026Harness as a Language: A Minimalist Agent Framework With Maximal Expressivitycited by 0
  3. 2026SAGE: Structured Strategic Reasoning for Efficient LLM Game Playingcited by 0
  4. 2026Shockingly Simple Self-retrospection Improves Agentic Models Without RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Agents of Change: Self-Evolving LLM Agents for Strategic Planning
  2. 2026Strat-Reasoner: Reinforcing Strategic Reasoning of LLMs in Multi-Agent Games
  3. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making
  4. 2026Hierarchical Control in Multi-Agent Games: LLM-based Planning and RL Execution
  5. 2024Large Language Models as Agents in Two-Player Games
  6. 2025Playing games with Large language models: Randomness and strategy
  7. 2025ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspective
  8. 2026Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments