mlsanityData to 7 Oct 2026

Monte-Carlo Tree Search as Regularized Policy Optimization

arXiv 2007.12509 ↗First posted 24 Jul 2020cs.LG stat.MLcited by 59

Not on the map. Why?

Abstract (arXiv)

The combination of Monte-Carlo tree search (MCTS) with deep reinforcement learning has led to significant advances in artificial intelligence. However, AlphaZero, the current state-of-the-art MCTS algorithm, still relies on handcrafted heuristics that are only partially understood. In this paper, we show that AlphaZero's search heuristics, along with other common ones such as UCT, are an approximation to the solution of a specific regularized policy optimization problem. With this insight, we propose a variant of AlphaZero which uses the exact solution to this policy optimization problem, and show experimentally that it reliably outperforms the original algorithm in multiple domains.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 8; 2023: 6; 2024: 12; 2025: 10; 2026: 7.

59 papers.

Sort:

By area:

  1. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  2. 2022HyperTree Proof Search for Neural Theorem Provingcited by 192
  3. 2021Learning and Planning in Complex Action Spacescited by 76
  4. 2021Scaling Scaling Laws with Board Gamescited by 62
  5. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 59
  6. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  7. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
  8. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  9. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  10. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  11. 2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
  12. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
38 more
  1. 2024Large Language Models as Agents in Two-Player Gamescited by 10
  2. 2024Designing Skill-Compatible AI: Methodologies and Frameworks in Chesscited by 10
  3. 2025AssistanceZero: Scalably Solving Assistance Gamescited by 10
  4. 2024Human-aligned Chess with a Bit of Searchcited by 9
  5. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  6. 2021Towards an Understanding of Default Policies in Multitask Policy Optimizationcited by 8
  7. 2022Efficient Offline Policy Optimization with a Learned Modelcited by 8
  8. 2024Adaptive Draft-Verification for Efficient Large Language Model Decodingcited by 8
  9. 2023What model does MuZero learn?cited by 7
  10. 2025iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Usecited by 7
  11. 2021Fragment-based Sequential Translation for Molecular Optimizationcited by 6
  12. 2022A Unified Perspective on Value Backup and Exploration in Monte-Carlo Tree Searchcited by 6
  13. 2024Policy Mirror Descent with Lookaheadcited by 6
  14. 2024Efficient Monte Carlo Tree Search via On-the-Fly State-Conditioned Action Abstractioncited by 6
  15. 2024Value Improved Actor Critic Algorithmscited by 6
  16. 2021Scalable Anytime Planning for Multi-Agent MDPscited by 5
  17. 2022Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree searchcited by 5
  18. 2025Trust-Region Twisted Policy Improvementcited by 5
  19. 2021Combining Off and On-Policy Training in Model-Based Reinforcement Learningcited by 4
  20. 2022Continuous Monte Carlo Graph Searchcited by 4
  21. 2022Spending Thinking Time Wisely: Accelerating MCTS with Virtual Expansionscited by 4
  22. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 4
  23. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  24. 2021Planning and Learning Using Adaptive Entropy Tree Searchcited by 3
  25. 2023Policy-Based Self-Competition for Planning Problemscited by 3
  26. 2023Modeling Boundedly Rational Agents with Latent Inference Budgetscited by 3
  27. 2025Weak-to-Strong Generalization with Failure Trajectories: A Tree-based Approach to Elicit Optimal Policy in Strong Modelscited by 3
  28. 2025Twice Sequential Monte Carlo for Tree Searchcited by 3
  29. 2021First-Order Problem Solving through Neural MCTS based Reinforcement Learningcited by 2
  30. 2021Learning compositional programs with arguments and samplingcited by 2
  31. 2021Procrastinated Tree Search: Black-box Optimization with Delayed, Noisy, and Multi-Fidelity Feedbackcited by 2
  32. 2022Policy Gradient Algorithms with Monte Carlo Tree Learning for Non-Markov Decision Processescited by 2
  33. 2022Brick Tic-Tac-Toe: Exploring the Generalizability of AlphaZero to Novel Test Environmentscited by 2
  34. 2023Autonomous Port Navigation With Ranging Sensors Using Model-Based Reinforcement Learningcited by 2
  35. 2024Provably Efficient Long-Horizon Exploration in Monte Carlo Tree Search through State Occupancy Regularizationcited by 1
  36. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Gamescited by 1
  37. 2021ScrofaZero: Mastering Trick-taking Poker Game Gongzhu by Deep Reinforcement Learningcited by 0
  38. 2021Dual Monte Carlo Tree Searchcited by 0

50 most cited of 59 All 59 (JSON)