mlsanityData to 7 Oct 2026

Enhancing Reinforcement Learning Through Guided Search

arXiv 2408.10113 ↗First posted 19 Aug 2024cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

With the aim of improving performance in Markov Decision Problem in an Off-Policy setting, we suggest taking inspiration from what is done in Offline Reinforcement Learning (RL). In Offline RL, it is a common practice during policy learning to maintain proximity to a reference policy to mitigate uncertainty, reduce potential policy errors, and help improve performance. We find ourselves in a different setting, yet it raises questions about whether a similar concept can be applied to enhance performance ie, whether it is possible to find a guiding policy capable of contributing to performance improvement, and how to incorporate it into our RL agent. Our attention is particularly focused on algorithms based on Monte Carlo Tree Search (MCTS) as a guide.MCTS renowned for its state-of-the-art capabilities across various domains, catches our interest due to its ability to converge to equilibrium in single-player and two-player contexts. By harnessing the power of MCTS as a guide for our RL agent, we observed a significant performance improvement, surpassing the outcomes achieved by utilizing each method in isolation. Our experiments were carried out on the Atari 100k benchmark.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 56 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

35 of 56 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization25 Jul
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2023Mastering Diverse Domains through World Modelscited by 1,193
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2015Prioritized Experience Replaycited by 1,070
16 more
  1. 2020Mastering Atari with Discrete World Modelscited by 990
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  7. 2021Mastering Atari Games with Limited Datacited by 235
  8. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 158
  9. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  10. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 47
  11. 2021Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Drivingcited by 46
  12. 2019ELF OpenGo: An Analysis and Open Reimplementation of AlphaZerocited by 37
  13. 2024A Survey of Imitation Learning Methods, Environments and Metricscited by 16
  14. 2020Minimax Strikes Backcited by 11
  15. 2019Action Guidance with MCTS for Deep Reinforcement Learningcited by 8
  16. 2021Batch Monte Carlo Tree Searchcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning
  2. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  3. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2026Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning
  6. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  7. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
  8. 2025A Clean Slate for Offline Reinforcement Learning