# A0C: Alpha Zero in Continuous Action Space

arXiv: https://arxiv.org/abs/1805.09613 · HTML: https://mlsanity.com/paper/1805.09613/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 0 of its 22 references that are in the map.
- First posted on arXiv 24 May 2018.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Built on by (most cited first, 18 of 18)

- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [Learning and Planning in Complex Action Spaces](https://mlsanity.com/paper/2104.06303/) (2021)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Beyond Games: A Systematic Review of Neural Monte Carlo Tree Search Applications](https://mlsanity.com/paper/2303.08060/) (2023)
- [BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations](https://mlsanity.com/paper/2306.00249/) (2023)
- [High-Accuracy Model-Based Reinforcement Learning, a Survey](https://mlsanity.com/paper/2107.08241/) (2021)
- [Lyapunov Exponents for Diversity in Differentiable Games](https://mlsanity.com/paper/2112.14570/) (2021)
- [ConstrainedZero: Chance-Constrained POMDP Planning using Learned Probabilistic Failure Surrogates and Adaptive Safety Constraints](https://mlsanity.com/paper/2405.00644/) (2024)
- [AlphaDDA: Strategies for Adjusting the Playing Strength of a Fully Trained AlphaZero System to a Suitable Human Training Partner](https://mlsanity.com/paper/2111.06266/) (2021)
- [Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree search](https://mlsanity.com/paper/2211.15068/) (2022)
- [SPO: Sequential Monte Carlo Policy Optimisation](https://mlsanity.com/paper/2402.07963/) (2024)
- [Combining Off and On-Policy Training in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.12194/) (2021)
- [Continuous Monte Carlo Graph Search](https://mlsanity.com/paper/2210.01426/) (2022)
- [GrASP: Gradient-Based Affordance Selection for Planning](https://mlsanity.com/paper/2202.04772/) (2022)
- [Brick Tic-Tac-Toe: Exploring the Generalizability of AlphaZero to Novel Test Environments](https://mlsanity.com/paper/2207.05991/) (2022)
- [Provably Efficient Long-Horizon Exploration in Monte Carlo Tree Search through State Occupancy Regularization](https://mlsanity.com/paper/2407.05511/) (2024)
- [AlphaViT: A flexible game-playing AI for multiple games and variable board sizes](https://mlsanity.com/paper/2408.13871/) (2024)
- [Learning to Play: Reinforcement Learning and Games](https://mlsanity.com/paper/2610.17391/) (2026)

## Abstract (arXiv)

A core novelty of Alpha Zero is the interleaving of tree search and deep learning, which has proven very successful in board games like Chess, Shogi and Go. These games have a discrete action space. However, many real-world reinforcement learning domains have continuous action spaces, for example in robotic control, navigation and self-driving cars. This paper presents the necessary theoretical extensions of Alpha Zero to deal with continuous action space. We also provide some preliminary experiments on the Pendulum swing-up task, empirically showing the feasibility of our approach. Thereby, this work provides a first step towards the application of iterated search and learning in domains with a continuous action space.