# Monte Carlo Tree Search: A Review of Recent Modifications and Applications

arXiv: https://arxiv.org/abs/2103.04931 · HTML: https://mlsanity.com/paper/2103.04931/

## Key facts

- Cited by 161 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 255 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Learning to Plan Chemical Syntheses](https://mlsanity.com/paper/1708.04202/) (2017)
- [Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithms](https://mlsanity.com/paper/1808.04794/) (2018)
- [Monte Carlo Tree Search with Heuristic Evaluations using Implicit Minimax Backups](https://mlsanity.com/paper/1406.0486/) (2014)
- [Robust and Adaptive Planning under Model Uncertainty](https://mlsanity.com/paper/1901.02577/) (2019)
- [Decentralized Cooperative Planning for Automated Vehicles with Hierarchical Monte Carlo Tree Search](https://mlsanity.com/paper/1807.09530/) (2018)
- [Rolling Horizon Evolutionary Algorithms for General Video Game Playing](https://mlsanity.com/paper/2003.12331/) (2020)
- [Action Guidance with MCTS for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.11703/) (2019)
- [Convex Hull Monte-Carlo Tree Search](https://mlsanity.com/paper/2003.04445/) (2020)
- [Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimates](https://mlsanity.com/paper/1905.05809/) (2019)
- [Multi-Labelled Value Networks for Computer Go](https://mlsanity.com/paper/1705.10701/) (2017)
- [Combining Strategic Learning and Tactical Search in Real-Time Strategy Games](https://mlsanity.com/paper/1709.03480/) (2017)
- [Scaling Monte Carlo Tree Search on Intel Xeon Phi](https://mlsanity.com/paper/1507.04383/) (2015)
- [Combining Monte-Carlo and Hyper-heuristic methods for the Multi-mode Resource-constrained Multi-project Scheduling Problem](https://mlsanity.com/paper/1511.04387/) (2015)
- [Analysis of Vanilla Rolling Horizon Evolution Parameters in General Video Game Playing](https://mlsanity.com/paper/1704.07075/) (2017)
- [Double-oracle sampling method for Stackelberg Equilibrium approximation in general-sum extensive-form games](https://mlsanity.com/paper/1909.03934/) (2019)
- [Does it matter how well I know what you're thinking? Opponent Modelling in an RTS game](https://mlsanity.com/paper/2006.08659/) (2020)
- [Monte Carlo Methods for the Game Kingdomino](https://mlsanity.com/paper/1807.04458/) (2018)
- [Monte-Carlo Tree Search for Simulation-based Strategy Analysis](https://mlsanity.com/paper/1908.01423/) (2019)
- [Integrating Acting, Planning and Learning in Hierarchical Operational Models](https://mlsanity.com/paper/2003.03932/) (2020)
- [MTD(f), A Minimax Algorithm Faster Than NegaScout](https://mlsanity.com/paper/1404.1511/) (2014)
- [Traditional Wisdom and Monte Carlo Tree Search Face-to-Face in the Card Game Scopone](https://mlsanity.com/paper/1807.06813/) (2018)
- [Monte Carlo Tree Search with Scalable Simulation Periods for Continuously Running Tasks](https://mlsanity.com/paper/1809.02378/) (2018)
- [Anchoring Theory in Sequential Stackelberg Games](https://mlsanity.com/paper/1912.03564/) (2019)

## Built on by (most cited first, 30 of 161)

- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [Improve Mathematical Reasoning in Language Models by Automated Process Supervision](https://mlsanity.com/paper/2406.06592/) (2024)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning](https://mlsanity.com/paper/2503.07608/) (2025)
- [Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2503.07065/) (2025)
- [ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning](https://mlsanity.com/paper/2410.02052/) (2024)
- [TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling](https://mlsanity.com/paper/2508.17445/) (2025)
- [Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS](https://mlsanity.com/paper/2411.18478/) (2024)
- [KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems](https://mlsanity.com/paper/2502.07350/) (2025)
- [Efficient Reinforcement Learning with Large Language Model Priors](https://mlsanity.com/paper/2410.07927/) (2024)
- [AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verification](https://mlsanity.com/paper/2502.11520/) (2025)
- [Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo](https://mlsanity.com/paper/2410.01920/) (2024)
- [Procedural Content Generation in Games: A Survey with Insights on Emerging LLM Integration](https://mlsanity.com/paper/2410.15644/) (2024)
- [KBQA-o1: Agentic Knowledge Base Question Answering with Monte Carlo Tree Search](https://mlsanity.com/paper/2501.18922/) (2025)
- [Monte Carlo Planning with Large Language Model for Text-Based Game Agents](https://mlsanity.com/paper/2504.16855/) (2025)
- [Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation](https://mlsanity.com/paper/2412.15118/) (2024)
- [TripCraft: A Benchmark for Spatio-Temporally Fine Grained Travel Planning](https://mlsanity.com/paper/2502.20508/) (2025)
- [HeurAgenix: Leveraging LLMs for Solving Complex Combinatorial Optimization Challenges](https://mlsanity.com/paper/2506.15196/) (2025)
- [WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesis](https://mlsanity.com/paper/2507.04370/) (2025)
- [No Train Still Gain. Unleash Mathematical Reasoning of Large Language Models with Monte Carlo Tree Search Guided by Energy Function](https://mlsanity.com/paper/2309.03224/) (2023)
- [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://mlsanity.com/paper/2310.08348/) (2023)
- [Failure-Driven Workflow Refinement](https://mlsanity.com/paper/2510.10035/) (2025)
- [SQL-o1: A Self-Reward Heuristic Dynamic Search Method for Text-to-SQL](https://mlsanity.com/paper/2502.11741/) (2025)
- [Beyond Games: A Systematic Review of Neural Monte Carlo Tree Search Applications](https://mlsanity.com/paper/2303.08060/) (2023)
- [Enhancing Mathematical Reasoning in LLMs by Stepwise Correction](https://mlsanity.com/paper/2410.12934/) (2024)
- [Top-Down Semantic Refinement for Image Captioning](https://mlsanity.com/paper/2510.22391/) (2025)
- [Tool-integrated Reinforcement Learning for Repo Deep Search](https://mlsanity.com/paper/2508.03012/) (2025)
- [Games for Artificial Intelligence Research: A Review and Perspectives](https://mlsanity.com/paper/2304.13269/) (2023)
- [QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE](https://mlsanity.com/paper/2409.05144/) (2024)

## Abstract (arXiv)

Monte Carlo Tree Search (MCTS) is a powerful approach to designing game-playing bots or solving sequential decision problems. The method relies on intelligent tree search that balances exploration and exploitation. MCTS performs random sampling in the form of simulations and stores statistics of actions to make more educated choices in each subsequent iteration. The method has become a state-of-the-art technique for combinatorial games, however, in more complex games (e.g. those with high branching factor or real-time ones), as well as in various practical domains (e.g. transportation, scheduling or security) an efficient MCTS application often requires its problem-dependent modification or integration with other techniques. Such domain-specific modifications and hybrid approaches are the main focus of this survey. The last major MCTS survey has been published in 2012. Contributions that appeared since its release are of particular interest for this review.