mlsanityData to 7 Oct 2026

Epistemic Monte Carlo Tree Search

arXiv 2210.13455 ↗First posted 21 Oct 2022cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

The AlphaZero/MuZero (A/MZ) family of algorithms has achieved remarkable success across various challenging domains by integrating Monte Carlo Tree Search (MCTS) with learned models. Learned models introduce epistemic uncertainty, which is caused by learning from limited data and is useful for exploration in sparse reward environments. MCTS does not account for the propagation of this uncertainty however. To address this, we introduce Epistemic MCTS (EMCTS): a theoretically motivated approach to account for the epistemic uncertainty in search and harness the search for deep exploration. In the challenging sparse-reward task of writing code in the Assembly language SUBLEQ, AZ paired with our method achieves significantly higher sample efficiency over baseline AZ. Search with EMCTS solves variations of the commonly used hard-exploration benchmark Deep Sea - which baseline A/MZ are practically unable to solve - much faster than an otherwise equivalent method that does not use search for uncertainty estimation, demonstrating significant benefits from search for epistemic uncertainty estimation.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 34 of its 63 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

34 of 63 references.

References · 34

  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  9. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  10. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  11. 2018Is Q-learning Provably Efficient?cited by 485
  12. 2020Planning to Explore via Self-Supervised World Modelscited by 365
22 more
  1. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  2. 2021Mastering Atari Games with Limited Datacited by 235
  3. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  4. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  5. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 158
  6. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  7. 2019Behaviour Suite for Reinforcement Learningcited by 129
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  9. 2018Model-Based Active Explorationcited by 105
  10. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  11. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  12. 2017Online algorithms for POMDPs with continuous state, action, and observation spacescited by 82
  13. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  14. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  15. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
  16. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
  17. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  18. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
  19. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  20. 2019Explicit Explore-Exploit Algorithms in Continuous State Spacescited by 19
  21. 2020Bayesian Optimized Monte Carlo Planningcited by 16
  22. 2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 4.

7 papers.

  1. 2025Trust-Region Twisted Policy Improvementcited by 5
  2. 2025Twice Sequential Monte Carlo for Tree Searchcited by 3
  3. 2026EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Controlcited by 2
  4. 2024Demystifying MuZero Planning: Interpreting the Learned Modelcited by 1
  5. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 0
  6. 2026Particle Monte Carlo Tree Searchcited by 0
  7. 2026Towards Optimal Policy Improvementcited by 0