# Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control

arXiv: https://arxiv.org/abs/2108.10315 · HTML: https://mlsanity.com/paper/2108.10315/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 90 references that are in the map.
- First posted on arXiv 20 Aug 2021.

## Builds on (most cited first, 15 of 15)

- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Learning Model Predictive Control for iterative tasks. A Data-Driven Control Framework](https://mlsanity.com/paper/1609.01387/) (2016)
- [Multiagent Rollout and Policy Iteration for POMDP with Application to Multi-Robot Repair Problems](https://mlsanity.com/paper/2011.04222/) (2020)
- [Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementations](https://mlsanity.com/paper/1804.04577/) (2018)
- [Multiagent Value Iteration Algorithms in Dynamic Programming and Reinforcement Learning](https://mlsanity.com/paper/2005.01627/) (2020)
- [Multiagent Rollout Algorithms and Reinforcement Learning](https://mlsanity.com/paper/1910.00120/) (2019)
- [Sample-Based Learning Model Predictive Control for Linear Uncertain Systems](https://mlsanity.com/paper/1904.06432/) (2019)
- [Robust feedback control of nonlinear PDEs by numerical approximation of high-dimensional Hamilton-Jacobi-Isaacs equations](https://mlsanity.com/paper/1905.06276/) (2019)
- [Distributed Asynchronous Policy Iteration for Sequential Zero-Sum Games and Minimax Control](https://mlsanity.com/paper/2107.10406/) (2021)
- [Constrained Multiagent Rollout and Multidimensional Assignment with the Auction Algorithm](https://mlsanity.com/paper/2002.07407/) (2020)
- [Data-driven Rollout for Deterministic Optimal Control](https://mlsanity.com/paper/2105.03116/) (2021)
- [Stability Analysis of Optimal Adaptive Control using Value Iteration with Approximation Errors](https://mlsanity.com/paper/1710.08530/) (2017)
- [Biased Aggregation, Rollout, and Enhanced Policy Improvement for Reinforcement Learning](https://mlsanity.com/paper/1910.02426/) (2019)
- [Policy iteration for Hamilton-Jacobi-Bellman equations with control constraints](https://mlsanity.com/paper/2004.03558/) (2020)
- [On-Line Policy Iteration for Infinite Horizon Dynamic Programming](https://mlsanity.com/paper/2106.00746/) (2021)

## Built on by (most cited first, 25 of 25)

- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [What model does MuZero learn?](https://mlsanity.com/paper/2306.00840/) (2023)
- [Learning to Boost the Performance of Stable Nonlinear Systems](https://mlsanity.com/paper/2405.00871/) (2024)
- [Model Predictive Control and Reinforcement Learning: A Unified Framework Based on Dynamic Programming](https://mlsanity.com/paper/2406.00592/) (2024)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysis](https://mlsanity.com/paper/2207.10939/) (2022)
- [Multiagent Reinforcement Learning for Autonomous Routing and Pickup Problem with Adaptation to Variable Demand](https://mlsanity.com/paper/2211.14983/) (2022)
- [Trust-Region Twisted Policy Improvement](https://mlsanity.com/paper/2504.06048/) (2025)
- [The Role of Lookahead and Approximate Policy Evaluation in Reinforcement Learning with Linear Value Function Approximation](https://mlsanity.com/paper/2109.13419/) (2021)
- [Reinforcement Learning with Unbiased Policy Evaluation and Linear Function Approximation](https://mlsanity.com/paper/2210.07338/) (2022)
- [Reinforcement Learning Methods for Wordle: A POMDP/Adaptive Control Approach](https://mlsanity.com/paper/2211.10298/) (2022)
- [From Optimization to Control: Quasi Policy Iteration](https://mlsanity.com/paper/2311.11166/) (2023)
- [Rollout Algorithms and Approximate Dynamic Programming for Bayesian Optimization and Sequential Estimation](https://mlsanity.com/paper/2212.07998/) (2022)
- [The Effective Horizon Explains Deep RL Performance in Stochastic Environments](https://mlsanity.com/paper/2312.08369/) (2023)
- [MPCritic: A plug-and-play MPC architecture for reinforcement learning](https://mlsanity.com/paper/2504.01086/) (2025)
- [Rank-One Modified Value Iteration](https://mlsanity.com/paper/2505.01828/) (2025)
- [Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gap](https://mlsanity.com/paper/2306.11271/) (2023)
- [Non-Parametric Neuro-Adaptive Coordination of Multi-Agent Systems](https://mlsanity.com/paper/2110.05125/) (2021)
- [Bayesian Learning Approach to Model Predictive Control](https://mlsanity.com/paper/2203.02720/) (2022)
- [Surge Routing: Event-informed Multiagent Reinforcement Learning for Autonomous Rideshare](https://mlsanity.com/paper/2307.02637/) (2023)
- [A Policy Iteration Algorithm for N-player General-Sum Linear Quadratic Dynamic Games](https://mlsanity.com/paper/2410.03106/) (2024)
- [New Auction Algorithms for Path Planning, Network Transport, and Reinforcement Learning](https://mlsanity.com/paper/2207.09588/) (2022)
- [Pro-Routing: Proactive Routing of Autonomous Multi-Capacity Robots for Pickup-and-Delivery Tasks](https://mlsanity.com/paper/2503.24325/) (2025)
- [Control and Reinforcement Learning through the Lens of Optimization: An Algorithmic Perspective](https://mlsanity.com/paper/2512.07377/) (2025)
- [Bridging RL and MPC for mixed-integer optimal control with application to Formula 1 race strategies](https://mlsanity.com/paper/2604.00826/) (2026)

## Abstract (arXiv)

In this paper we aim to provide analysis and insights (often based on visualization), which explain the beneficial effects of on-line decision making on top of off-line training. In particular, through a unifying abstract mathematical framework, we show that the principal AlphaZero/TD-Gammon ideas of approximation in value space and rollout apply very broadly to deterministic and stochastic optimal control problems, involving both discrete and continuous search spaces. Moreover, these ideas can be effectively integrated with other important methodologies such as model predictive control, adaptive control, decentralized control, discrete and Bayesian optimization, neural network-based value and policy approximations, and heuristic algorithms for discrete optimization.