mlsanityData to 14 Oct 2026

Policy Gradient Algorithms with Monte Carlo Tree Learning for Non-Markov Decision Processes

arXiv 2206.01011 ↗First posted 2 Jun 2022cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Policy gradient (PG) is a reinforcement learning (RL) approach that optimizes a parameterized policy model for an expected return using gradient ascent. While PG can work well even in non-Markovian environments, it may encounter plateaus or peakiness issues. As another successful RL approach, algorithms based on Monte Carlo Tree Search (MCTS), which include AlphaZero, have obtained groundbreaking results, especially in the game-playing domain. They are also effective when applied to non-Markov decision processes. However, the standard MCTS is a method for decision-time planning, which differs from the online RL setting. In this work, we first introduce Monte Carlo Tree Learning (MCTL), an adaptation of MCTS for online RL setups. We then explore a combined policy approach of PG and MCTL to leverage their strengths. We derive conditions for asymptotic convergence with the results of a two-timescale stochastic approximation and propose an algorithm that satisfies these conditions and converges to a reasonable solution. Our numerical experiments validate the effectiveness of the proposed methods.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 87 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

33 of 87 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2016Self-critical Sequence Training for Image Captioningcited by 547
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  11. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  12. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
21 more
  1. 2018Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generationcited by 364
  2. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  3. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  4. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  5. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  6. 2021Monte Carlo Tree Search: A Review of Recent Modifications and Applicationscited by 161
  7. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  8. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  9. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  10. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  11. 2021Ordering-Based Causal Discovery with Reinforcement Learningcited by 55
  12. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 33
  13. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  14. 2018How to Combine Tree-Search Methods in Reinforcement Learningcited by 16
  15. 2016Deep Learning for Reward Design to Improve Monte Carlo Tree Search in ATARI Gamescited by 13
  16. 2019Policy Gradient Search: Online Planning and Expert Iteration without Search Treescited by 12
  17. 2020POLY-HOOT: Monte-Carlo Planning in Continuous Space MDPs with Non-Asymptotic Analysiscited by 8
  18. 2020Convex Regularization in Monte-Carlo Tree Searchcited by 8
  19. 2019Learning Policies from Self-Play with Policy Gradients and MCTS Value Estimatescited by 6
  20. 2018Feedback-Based Tree Search for Reinforcement Learningcited by 4
  21. 2022Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabicited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Adaptive Mixing of Policies from Searching and Policies from Learningcited by 0
  2. 2026OPTS-TTPO: Enhancing Finite-Sample Policy-Gradient Learning with Tree Searchcited by 0