# Anderson Acceleration for Reinforcement Learning

arXiv: https://arxiv.org/abs/1809.09501 · HTML: https://mlsanity.com/paper/1809.09501/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2018.

## Built on by (most cited first, 15 of 15)

- [Accelerating Value Iteration with Anchoring](https://mlsanity.com/paper/2305.16569/) (2023)
- [Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization](https://mlsanity.com/paper/2110.08896/) (2021)
- [Stochastic Anderson Mixing for Nonconvex Stochastic Optimization](https://mlsanity.com/paper/2110.01543/) (2021)
- [Deflated Dynamics Value Iteration](https://mlsanity.com/paper/2407.10454/) (2024)
- [From Optimization to Control: Quasi Policy Iteration](https://mlsanity.com/paper/2311.11166/) (2023)
- [Rank-One Modified Value Iteration](https://mlsanity.com/paper/2505.01828/) (2025)
- [Anderson Acceleration For Bioinformatics-Based Machine Learning](https://mlsanity.com/paper/2302.00347/) (2023)
- [Enhancing Electrical Impedance Tomography reconstruction using Learned Half-Quadratic Splitting Networks with Anderson Acceleration](https://mlsanity.com/paper/2304.14491/) (2023)
- [PID Accelerated Temporal Difference Algorithms](https://mlsanity.com/paper/2407.08803/) (2024)
- [An Efficient Nonlinear Acceleration method that Exploits Symmetry of the Hessian](https://mlsanity.com/paper/2210.12573/) (2022)
- [Acceleration in Policy Optimization](https://mlsanity.com/paper/2306.10587/) (2023)
- [Improved Convergence Factor of Windowed Anderson Acceleration for Symmetric Fixed-Point Iterations](https://mlsanity.com/paper/2311.02490/) (2023)
- [FedOSAA: Improving Federated Learning with One-Step Anderson Acceleration](https://mlsanity.com/paper/2503.10961/) (2025)
- [Control and Reinforcement Learning through the Lens of Optimization: An Algorithmic Perspective](https://mlsanity.com/paper/2512.07377/) (2025)
- [Heavy-Ball Q-Learning with Residual Weighting Correction](https://mlsanity.com/paper/2606.27112/) (2026)

## Abstract (arXiv)

Anderson acceleration is an old and simple method for accelerating the computation of a fixed point. However, as far as we know and quite surprisingly, it has never been applied to dynamic programming or reinforcement learning. In this paper, we explain briefly what Anderson acceleration is and how it can be applied to value iteration, this being supported by preliminary experiments showing a significant speed up of convergence, that we critically discuss. We also discuss how this idea could be applied more generally to (deep) reinforcement learning.