# Monte Carlo Planning with Large Language Model for Text-Based Game Agents

arXiv: https://arxiv.org/abs/2504.16855 · HTML: https://mlsanity.com/paper/2504.16855/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 23 Apr 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 34)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [Interactive Fiction Games: A Colossal Adventure](https://mlsanity.com/paper/1909.05398/) (2019)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [Keep CALM and Explore: Language Models for Action Generation in Text-based Games](https://mlsanity.com/paper/2010.02903/) (2020)
- [Deep Reinforcement Learning with a Natural Language Action Space](https://mlsanity.com/paper/1511.04636/) (2015)
- [Graph Constrained Reinforcement Learning for Natural Language Action Spaces](https://mlsanity.com/paper/2001.08837/) (2020)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation](https://mlsanity.com/paper/2311.04254/) (2023)
- [Language Understanding for Text-based Games Using Deep Reinforcement Learning](https://mlsanity.com/paper/1506.08941/) (2015)
- [Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks](https://mlsanity.com/paper/2405.01534/) (2024)
- [Learning Dynamic Belief Graphs to Generalize on Text-Based Games](https://mlsanity.com/paper/2002.09127/) (2020)
- [Learning to Win by Reading Manuals in a Monte-Carlo Framework](https://mlsanity.com/paper/1401.5390/) (2014)
- [How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds](https://mlsanity.com/paper/2006.07409/) (2020)
- [RL-GPT: Integrating Reinforcement Learning and Code-as-policy](https://mlsanity.com/paper/2402.19299/) (2024)
- [Large Language Models Are Neurosymbolic Reasoners](https://mlsanity.com/paper/2401.09334/) (2024)
- [Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning](https://mlsanity.com/paper/2010.02386/) (2020)
- [Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games](https://mlsanity.com/paper/2010.11655/) (2020)
- [Multi-Stage Episodic Control for Strategic Exploration in Text Games](https://mlsanity.com/paper/2201.01251/) (2022)
- [Playing Text-Based Games with Common Sense](https://mlsanity.com/paper/2012.02757/) (2020)
- [A Survey of Text Games for Reinforcement Learning informed by Natural Language](https://mlsanity.com/paper/2109.09478/) (2021)
- [How Can LLM Guide RL? A Value-Based Approach](https://mlsanity.com/paper/2402.16181/) (2024)
- [Generalization in Text-based Games via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2109.09968/) (2021)

## Built on by (most cited first, 22 of 22)

- [Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities](https://mlsanity.com/paper/2506.18019/) (2025)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [LongSeeker: Elastic Context Orchestration for Long-Horizon Search Agents](https://mlsanity.com/paper/2605.05191/) (2026)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives](https://mlsanity.com/paper/2506.09656/) (2025)
- [SYMPHONY: Synergistic Multi-agent Planning with Heterogeneous Language Model Assembly](https://mlsanity.com/paper/2601.22623/) (2026)
- [Co-Saving: Resource Aware Multi-Agent Collaboration for Software Development](https://mlsanity.com/paper/2505.21898/) (2025)
- [Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces](https://mlsanity.com/paper/2606.13686/) (2026)
- [GraphPO: Graph-based Policy Optimization for Reasoning Models](https://mlsanity.com/paper/2606.18954/) (2026)
- [Dual-Scale World Models for LLM Agents Towards Hard-Exploration Problems](https://mlsanity.com/paper/2509.24116/) (2025)
- [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026)
- [ToC: Tree-of-Claims Search with Multi-Agent Language Models](https://mlsanity.com/paper/2511.16972/) (2025)
- [Task-Aware LLM Council with Adaptive Decision Pathways for Decision Support](https://mlsanity.com/paper/2601.22662/) (2026)
- [UIPress: Bringing Optical Token Compression to UI-to-Code Generation](https://mlsanity.com/paper/2604.09442/) (2026)
- [BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Design](https://mlsanity.com/paper/2604.12898/) (2026)
- [PriorZero: Bridging Language Priors and World Models for Decision Making](https://mlsanity.com/paper/2605.12289/) (2026)
- [Adaptive Parallel Monte Carlo Tree Search for Efficient Test-time Compute Scaling](https://mlsanity.com/paper/2604.00510/) (2026)
- [Multi-Agent Systems: From Classical Paradigms to Large Foundation Model-Enabled Futures](https://mlsanity.com/paper/2604.18133/) (2026)
- [Particle Monte Carlo Tree Search](https://mlsanity.com/paper/2605.08982/) (2026)
- [Joint Agent Memory and Exploration Learning via Novelty Signals](https://mlsanity.com/paper/2606.01528/) (2026)
- [GUI agent: Guided Exploration of User-Sensitive Screens](https://mlsanity.com/paper/2606.25705/) (2026)
- [SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos](https://mlsanity.com/paper/2608.30716/) (2026)

## Abstract (arXiv)

Text-based games provide valuable environments for language-based autonomous agents. However, planning-then-learning paradigms, such as those combining Monte Carlo Tree Search (MCTS) and reinforcement learning (RL), are notably time-consuming due to extensive iterations. Additionally, these algorithms perform uncertainty-driven exploration but lack language understanding and reasoning abilities. In this paper, we introduce the Monte Carlo planning with Dynamic Memory-guided Large language model (MC-DML) algorithm. MC-DML leverages the language understanding and reasoning capabilities of Large Language Models (LLMs) alongside the exploratory advantages of tree search algorithms. Specifically, we enhance LLMs with in-trial and cross-trial memory mechanisms, enabling them to learn from past experiences and dynamically adjust action evaluations during planning. We conduct experiments on a series of text-based games from the Jericho benchmark. Our results demonstrate that the MC-DML algorithm significantly enhances performance across various games at the initial planning phase, outperforming strong contemporary methods that require multiple iterations. This demonstrates the effectiveness of our algorithm, paving the way for more efficient language-grounded planning in complex environments.