mlsanityData to 7 Oct 2026

Learning and Planning in Complex Action Spaces

arXiv 2104.06303 ↗First posted 13 Apr 2021cs.LGcited by 76

Not on the map. Why?

Abstract (arXiv)

Many important real-world problems have action spaces that are high-dimensional, continuous or both, making full enumeration of all possible actions infeasible. Instead, only small subsets of actions can be sampled for the purpose of policy evaluation and improvement. In this paper, we propose a general framework to reason in a principled way about policy evaluation and improvement over such sampled action subsets. This sample-based policy iteration framework can in principle be applied to any reinforcement learning algorithm based upon policy iteration. Concretely, we propose Sampled MuZero, an extension of the MuZero algorithm that is able to learn in domains with arbitrarily complex action spaces by planning over sampled actions. We demonstrate this approach on the classical board game of Go and on two continuous control benchmark domains: DeepMind Control Suite and Real-World RL Suite.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 13 Apr 2021.

Builds on

31 of 52 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  12. 2015Prioritized Experience Replaycited by 1,070
19 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  3. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  4. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  5. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  6. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  7. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  8. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 75
  9. 2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 65
  10. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 59
  11. 2018Relative Entropy Regularized Policy Iterationcited by 32
  12. 2018Hierarchical visuomotor control of humanoidscited by 29
  13. 2019Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Modelscited by 26
  14. 2018A0C: Alpha Zero in Continuous Action Spacecited by 17
  15. 2020An operator view of policy gradient methodscited by 17
  16. 2019Information Theoretic Model Predictive Q-Learningcited by 16
  17. 2020Local Search for Policy Iteration in Continuous Controlcited by 9
  18. 2020Dream and Search to Control: Latent Space Planning for Continuous Controlcited by 5
  19. 2020Continuous Control for Searching and Planning with a Learned Modelcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 9; 2023: 16; 2024: 18; 2025: 14; 2026: 13.

76 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  2. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  3. 2021Mastering Atari Games with Limited Datacited by 235
  4. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Datacited by 39
  5. 2024Quantum Circuit Optimization with AlphaTensorcited by 36
  6. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
  7. 2021Learning Large Neighborhood Search Policy for Integer Programmingcited by 35
  8. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  9. 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27
  10. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  11. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  12. 2022Planning for Sample Efficient Imitation Learningcited by 20
38 more
  1. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  2. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  3. 2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
  4. 2021Evaluating model-based planning and planner amortization for continuous controlcited by 15
  5. 2023MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Gamescited by 14
  6. 2023BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximationscited by 13
  7. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
  8. 2025Improving Pre-Trained Vision-Language-Action Policies with Model-Based Searchcited by 12
  9. 2024Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learningcited by 11
  10. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  11. 2024Efficient Multi-agent Reinforcement Learning by Planningcited by 9
  12. 2025Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agentscited by 9
  13. 2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learningcited by 8
  14. 2023What model does MuZero learn?cited by 7
  15. 2023Efficient Planning in Combinatorial Action Spaces with Applications to Cooperative Multi-Agent Reinforcement Learningcited by 6
  16. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithmscited by 6
  17. 2024Policy-shaped prediction: avoiding distractions in model-based reinforcement learningcited by 6
  18. 2022Model-based graph reinforcement learning for inductive traffic signal controlcited by 5
  19. 2022Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree searchcited by 5
  20. 2023Investigating the role of model-based learning in exploration and transfercited by 5
  21. 2025Trust-Region Twisted Policy Improvementcited by 5
  22. 2025Controllable Graph Generation with Diffusion Models via Inference-Time Tree Search Guidancecited by 5
  23. 2025MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planningcited by 5
  24. 2022Continuous Monte Carlo Graph Searchcited by 4
  25. 2022Continuous Neural Algorithmic Plannerscited by 4
  26. 2024SPO: Sequential Monte Carlo Policy Optimisationcited by 4
  27. 2025Scalable Decision-Making in Stochastic Environments through Learned Temporal Abstractioncited by 4
  28. 2026NonZero: Interaction-Guided Exploration for Multi-Agent Monte Carlo Tree Searchcited by 4
  29. 2022NSGZero: Efficiently Learning Non-Exploitable Policy in Large-Scale Network Security Games with Neural Monte Carlo Tree Searchcited by 3
  30. 2023Policy-Based Self-Competition for Planning Problemscited by 3
  31. 2023Accelerating Monte Carlo Tree Search with Probability Tree State Abstractioncited by 3
  32. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
  33. 2025One Model for All Tasks: Leveraging Efficient World Models in Multi-Task Planningcited by 3
  34. 2025Twice Sequential Monte Carlo for Tree Searchcited by 3
  35. 2022GrASP: Gradient-Based Affordance Selection for Planningcited by 2
  36. 2022Doubly-Asynchronous Value Iteration: Making Value Iteration Asynchronous in Actionscited by 2
  37. 2023Real-time scheduling of renewable power systems through planning-based reinforcement learningcited by 2
  38. 2024ReZero: Boosting MCTS-based Algorithms by Backward-view and Entire-buffer Reanalyzecited by 2

50 most cited of 76 All 76 (JSON)