# Continuous Monte Carlo Graph Search

arXiv: https://arxiv.org/abs/2210.01426 · HTML: https://mlsanity.com/paper/2210.01426/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 78 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Sample-efficient Cross-Entropy Method for Real-time Planning](https://mlsanity.com/paper/2008.06389/) (2020)
- [Learning and Planning in Complex Action Spaces](https://mlsanity.com/paper/2104.06303/) (2021)
- [Discretizing Continuous Action Space for On-Policy Optimization](https://mlsanity.com/paper/1901.10500/) (2019)
- [Near Optimal Behavior via Approximate State Abstraction](https://mlsanity.com/paper/1701.04113/) (2017)
- [Monte-Carlo Tree Search as Regularized Policy Optimization](https://mlsanity.com/paper/2007.12509/) (2020)
- [On the role of planning in model-based deep reinforcement learning](https://mlsanity.com/paper/2011.04021/) (2020)
- [MBRL-Lib: A Modular Library for Model-based Reinforcement Learning](https://mlsanity.com/paper/2104.10159/) (2021)
- [Blending MPC & Value Function Approximation for Efficient Reinforcement Learning](https://mlsanity.com/paper/2012.05909/) (2020)
- [Automated Playtesting with Procedural Personas through MCTS with Evolved Heuristics](https://mlsanity.com/paper/1802.06881/) (2018)
- [A0C: Alpha Zero in Continuous Action Space](https://mlsanity.com/paper/1805.09613/) (2018)
- [Information Theoretic Model Predictive Q-Learning](https://mlsanity.com/paper/2001.02153/) (2019)
- [Improving Hearthstone AI by Combining MCTS and Supervised Learning Algorithms](https://mlsanity.com/paper/1808.04794/) (2018)
- [Action Guidance with MCTS for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.11703/) (2019)
- [POLY-HOOT: Monte-Carlo Planning in Continuous Space MDPs with Non-Asymptotic Analysis](https://mlsanity.com/paper/2006.04672/) (2020)
- [The Value of Planning for Infinite-Horizon Model Predictive Control](https://mlsanity.com/paper/2104.02863/) (2021)
- [Population Seeding Techniques for Rolling Horizon Evolution in General Video Game Playing](https://mlsanity.com/paper/1704.06942/) (2017)
- [Intelligent Middle-Level Game Control](https://mlsanity.com/paper/1808.06201/) (2018)
- [Marginal Utility for Planning in Continuous or Large Discrete Action Spaces](https://mlsanity.com/paper/2006.06054/) (2020)
- [Limited depth bandit-based strategy for Monte Carlo planning in continuous action spaces](https://mlsanity.com/paper/2106.15594/) (2021)

## Built on by (most cited first, 4 of 4)

- [Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities](https://mlsanity.com/paper/2506.18019/) (2025)
- [Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.13863/) (2024)
- [SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints](https://mlsanity.com/paper/2607.05363/) (2026)
- [Action-Gradient Monte Carlo Tree Search for Non-Parametric Continuous (PO)MDPs](https://mlsanity.com/paper/2503.12181/) (2025)

## Abstract (arXiv)

Online planning is crucial for high performance in many complex sequential decision-making tasks. Monte Carlo Tree Search (MCTS) employs a principled mechanism for trading off exploration for exploitation for efficient online planning, and it outperforms comparison methods in many discrete decision-making domains such as Go, Chess, and Shogi. Subsequently, extensions of MCTS to continuous domains have been developed. However, the inherent high branching factor and the resulting explosion of the search tree size are limiting the existing methods. To address this problem, we propose Continuous Monte Carlo Graph Search (CMCGS), an extension of MCTS to online planning in environments with continuous state and action spaces. CMCGS takes advantage of the insight that, during planning, sharing the same action policy between several states can yield high performance. To implement this idea, at each time step, CMCGS clusters similar states into a limited number of stochastic action bandit nodes, which produce a layered directed graph instead of an MCTS search tree. Experimental evaluation shows that CMCGS outperforms comparable planning methods in several complex continuous DeepMind Control Suite benchmarks and 2D navigation and exploration tasks with limited sample budgets. Furthermore, CMCGS can be scaled up through parallelization, and it outperforms the Cross-Entropy Method (CEM) in continuous control with learned dynamics models.