mlsanityData to 7 Oct 2026

Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning

arXiv 2410.11234 ↗First posted 15 Oct 2024cs.LG cs.AIcited by 11

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is a powerful approach for data-driven decision-making and control. Compared to model-free methods, offline model-based reinforcement learning (MBRL) explicitly learns world models from a static dataset and uses them as surrogate simulators, improving the data efficiency and enabling the learned policy to potentially generalize beyond the dataset support. However, there could be various MDPs that behave identically on the offline dataset and dealing with the uncertainty about the true MDP can be challenging. In this paper, we propose modeling offline MBRL as a Bayes Adaptive Markov Decision Process (BAMDP), which is a principled framework for addressing model uncertainty. We further propose a novel Bayes Adaptive Monte-Carlo planning algorithm capable of solving BAMDPs in continuous state and action spaces with stochastic transitions. This planning process is based on Monte Carlo Tree Search and can be integrated into offline MBRL as a policy improvement operator in policy iteration. Our "RL + Search" framework follows in the footsteps of superhuman AIs like AlphaZero, improving on current offline MBRL methods by incorporating more computation input. The proposed algorithm significantly outperforms state-of-the-art offline RL methods on twelve D4RL MuJoCo tasks and three challenging, stochastic tokamak control tasks. The codebase is available at: https://github.com/LucasCJYSDL/Offline-RL-Kit.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 78 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

40 of 78 references.

Closest ancestors · 19

Same direction, newest first

  1. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief13 Oct
  2. 2022Efficient Offline Policy Optimization with a Learned Model12 Oct
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization7 Oct
  4. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  11. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
7 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 21

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  7. 2021Mastering Atari Games with Limited Datacited by 235
  8. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  9. 2020Model-Based Offline Planningcited by 126
  10. 2017Online algorithms for POMDPs with continuous state, action, and observation spacescited by 82
  11. 2021Learning and Planning in Complex Action Spacescited by 76
  12. 2020On the role of planning in model-based deep reinforcement learningcited by 57
9 more
  1. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  2. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  3. 2023ContraBAR: Contrastive Bayes-Adaptive Deep RLcited by 9
  4. 2024Full Shot Predictions for the DIII-D Tokamak via Deep Recurrent Networkscited by 9
  5. 2022A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Processcited by 7
  6. 2024Variational Offline Multi-agent Skill Discoverycited by 6
  7. 2023Hierarchical Deep Counterfactual Regret Minimizationcited by 4
  8. 2018Estimation and Control Using Sampling-Based Bayesian Reinforcement Learningcited by 3
  9. 2020Bayes-Adaptive Deep Model-Based Policy Optimisationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  2. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  3. 2025MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planningcited by 5
  4. 2025A Theoretical Framework for Graph-based Digital Twins for Supply Chain Management and Optimizationcited by 2
  5. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2
  6. 2026Offline Reinforcement Learning for Rotation Profile Control in Tokamakscited by 2
  7. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  8. 2025Fully Offline Reinforcement Learningcited by 0
  9. 2025Variance-Aware Prior-Based Tree Policies for Monte Carlo Tree Searchcited by 0
  10. 2026Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmarkcited by 0
  11. 2026Efficient Bayes-Adaptive Reinforcement Learning with Temporal Logic Specificationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  2. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  3. 2024Offline Model-Based Reinforcement Learning with Anti-Exploration
  4. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
  5. 2024Enhancing Reinforcement Learning Through Guided Search
  6. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  7. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  8. 2023A Survey on Offline Model-Based Reinforcement Learning