mlsanityData to 7 Oct 2026

Scalable Online Planning via Reinforcement Learning Fine-Tuning

arXiv 2109.15316 ↗First posted 30 Sep 2021cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Lookahead search has been a critical component of recent AI successes, such as in the games of chess, go, and poker. However, the search methods used in these games, and in many other settings, are tabular. Tabular search methods do not scale well with the size of the search space, and this problem is exacerbated by stochasticity and partial observability. In this work we replace tabular search with online model-based fine-tuning of a policy neural network via reinforcement learning, and show that this approach outperforms state-of-the-art search algorithms in benchmark settings. In particular, we use our search algorithm to achieve a new state-of-the-art result in self-play Hanabi, and show the generality of our algorithm by also showing that it outperforms tabular search in the Atari game Ms. Pacman.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 30 Sep 2021.

Builds on

29 of 52 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  8. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  9. 2018A Survey on Deep Transfer Learningcited by 417
  10. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  11. 2018Distributed Prioritized Experience Replaycited by 260
  12. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
17 more
  1. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  2. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  3. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  4. 2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 110
  5. 2019Exploring Model-based Planning with Policy Networkscited by 90
  6. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  7. 2021Learning and Planning in Complex Action Spacescited by 76
  8. 2018Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 67
  9. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 59
  10. 2019Simplified Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 47
  11. 2019Improving Policies via Search in Cooperative Partially Observable Gamescited by 33
  12. 2019The Differentiable Cross-Entropy Methodcited by 26
  13. 2019Combining Q-Learning and Search with Amortized Value Estimatescited by 19
  14. 2020Iterative Amortized Policy Optimizationcited by 13
  15. 2019Policy Gradient Search: Online Planning and Expert Iteration without Search Treescited by 11
  16. 2021Learned Belief Search: Efficiently Improving Policies in Partially Observable Settingscited by 10
  17. 2020Local Search for Policy Iteration in Continuous Controlcited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 7; 2024: 3; 2025: 4.

18 papers.

Sort:

By area:

  1. 2022Tutorial on amortized optimizationcited by 83
  2. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  3. 2023Thinker: Learning to Plan and Actcited by 13
  4. 2024TFWT: Tabular Feature Weighting with Transformercited by 12
  5. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  6. 2025Scalable Decision Making for Games of Imperfect Informationcited by 7
  7. 2022Improved Policy Optimization for Online Imitation Learningcited by 6
  8. 2023Abstracting Imperfect Information Away from Two-Player Zero-Sum Gamescited by 6
  9. 2022Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabicited by 3
  10. 2025Aligning Sentence Simplification with ESL Learner's Proficiency for Language Acquisitioncited by 3
  11. 2023The Update-Equivalence Framework for Decision-Time Planningcited by 2
  12. 2023History Filtering in Imperfect Information Games: Algorithms and Complexitycited by 2
6 more
  1. 2024Reinforcement Learning-Guided Semi-Supervised Learningcited by 2
  2. 2023P3O: Transferring Visual Representations for Reinforcement Learning via Promptingcited by 1
  3. 2023Dual policy as self-model for planningcited by 1
  4. 2023Distributed Reinforcement Learning for Molecular Design: Antioxidant casecited by 0
  5. 2024Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Controlcited by 0
  6. 2025Neural Bayesian Filteringcited by 0