mlsanityData to 7 Oct 2026

Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control

arXiv 2108.10315 ↗First posted 20 Aug 2021math.OC cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

In this paper we aim to provide analysis and insights (often based on visualization), which explain the beneficial effects of on-line decision making on top of off-line training. In particular, through a unifying abstract mathematical framework, we show that the principal AlphaZero/TD-Gammon ideas of approximation in value space and rollout apply very broadly to deterministic and stochastic optimal control problems, involving both discrete and continuous search spaces. Moreover, these ideas can be effectively integrated with other important methodologies such as model predictive control, adaptive control, decentralized control, discrete and Bayesian optimization, neural network-based value and policy approximations, and heuristic algorithms for discrete optimization.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 15 of its 90 references that are in the map. First posted on arXiv 20 Aug 2021.

Builds on

15 of 90 references.

References · 15

  1. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  2. 2016Learning Model Predictive Control for iterative tasks. A Data-Driven Control Frameworkcited by 38
  3. 2020Multiagent Rollout and Policy Iteration for POMDP with Application to Multi-Robot Repair Problemscited by 13
  4. 2018Feature-Based Aggregation and Deep Reinforcement Learning: A Survey and Some New Implementationscited by 12
  5. 2020Multiagent Value Iteration Algorithms in Dynamic Programming and Reinforcement Learningcited by 12
  6. 2019Multiagent Rollout Algorithms and Reinforcement Learningcited by 10
  7. 2019Sample-Based Learning Model Predictive Control for Linear Uncertain Systemscited by 7
  8. 2021Distributed Asynchronous Policy Iteration for Sequential Zero-Sum Games and Minimax Controlcited by 6
  9. 2019Robust feedback control of nonlinear PDEs by numerical approximation of high-dimensional Hamilton-Jacobi-Isaacs equationscited by 5
  10. 2020Constrained Multiagent Rollout and Multidimensional Assignment with the Auction Algorithmcited by 4
  11. 2021Data-driven Rollout for Deterministic Optimal Controlcited by 3
  12. 2017Stability Analysis of Optimal Adaptive Control using Value Iteration with Approximation Errorscited by 2
3 more
  1. 2019Biased Aggregation, Rollout, and Enhanced Policy Improvement for Reinforcement Learningcited by 2
  2. 2020Policy iteration for Hamilton-Jacobi-Bellman equations with control constraintscited by 1
  3. 2021On-Line Policy Iteration for Infinite Horizon Dynamic Programmingcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 8; 2023: 6; 2024: 2; 2025: 4.

22 papers.

Sort:

  1. 2023Deep networks for system identification: a Surveycited by 35
  2. 2023What model does MuZero learn?cited by 7
  3. 2024Learning to Boost the Performance of Stable Nonlinear Systemscited by 7
  4. 2022Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysiscited by 6
  5. 2022Multiagent Reinforcement Learning for Autonomous Routing and Pickup Problem with Adaptation to Variable Demandcited by 6
  6. 2021The Role of Lookahead and Approximate Policy Evaluation in Reinforcement Learning with Linear Value Function Approximationcited by 5
  7. 2022Reinforcement Learning in Practice: Opportunities and Challengescited by 5
  8. 2024Model Predictive Control and Reinforcement Learning: A Unified Framework Based on Dynamic Programmingcited by 5
  9. 2025Trust-Region Twisted Policy Improvementcited by 5
  10. 2022Reinforcement Learning with Unbiased Policy Evaluation and Linear Function Approximationcited by 4
  11. 2022Reinforcement Learning Methods for Wordle: A POMDP/Adaptive Control Approachcited by 4
  12. 2022Rollout Algorithms and Approximate Dynamic Programming for Bayesian Optimization and Sequential Estimationcited by 3
10 more
  1. 2023The Effective Horizon Explains Deep RL Performance in Stochastic Environmentscited by 3
  2. 2025MPCritic: A plug-and-play MPC architecture for reinforcement learningcited by 3
  3. 2023Warm-Start Actor-Critic: From Approximation Error to Sub-optimality Gapcited by 2
  4. 2023From Optimization to Control: Quasi Policy Iterationcited by 2
  5. 2025Rank-One Modified Value Iterationcited by 2
  6. 2021Non-Parametric Neuro-Adaptive Coordination of Multi-Agent Systemscited by 1
  7. 2023Surge Routing: Event-informed Multiagent Reinforcement Learning for Autonomous Ridesharecited by 1
  8. 2022Bayesian Learning Approach to Model Predictive Controlcited by 0
  9. 2022New Auction Algorithms for Path Planning, Network Transport, and Reinforcement Learningcited by 0
  10. 2025Pro-Routing: Proactive Routing of Autonomous Multi-Capacity Robots for Pickup-and-Delivery Taskscited by 0