# Regularized Nonlinear Acceleration

arXiv: https://arxiv.org/abs/1606.04133 · HTML: https://mlsanity.com/paper/1606.04133/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2016.

## Built on by (most cited first, 18 of 18)

- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [Faster Unbalanced Optimal Transport: Translation invariant Sinkhorn and 1-D Frank-Wolfe](https://mlsanity.com/paper/2201.00730/) (2022)
- [Multivariate Conformal Prediction using Optimal Transport](https://mlsanity.com/paper/2502.03609/) (2025)
- [Beyond L1: Faster and Better Sparse Models with skglm](https://mlsanity.com/paper/2204.07826/) (2022)
- [Proof-of-Learning is Currently More Broken Than You Think](https://mlsanity.com/paper/2208.03567/) (2022)
- [Accelerating Value Iteration with Anchoring](https://mlsanity.com/paper/2305.16569/) (2023)
- [Stochastic Anderson Mixing for Nonconvex Stochastic Optimization](https://mlsanity.com/paper/2110.01543/) (2021)
- [Riemannian accelerated gradient methods via extrapolation](https://mlsanity.com/paper/2208.06619/) (2022)
- [Stable Anderson Acceleration for Deep Learning](https://mlsanity.com/paper/2110.14813/) (2021)
- [Fast symplectic integrator for Nesterov-type acceleration method](https://mlsanity.com/paper/2106.07620/) (2021)
- [Function Extrapolation with Neural Networks and Its Application for Manifolds](https://mlsanity.com/paper/2405.10563/) (2024)
- [Anderson Acceleration as a Krylov Method with Application to Asymptotic Convergence Analysis](https://mlsanity.com/paper/2109.14181/) (2021)
- [Optimization using Parallel Gradient Evaluations on Multiple Parameters](https://mlsanity.com/paper/2302.03161/) (2023)
- [Improved Convergence Factor of Windowed Anderson Acceleration for Symmetric Fixed-Point Iterations](https://mlsanity.com/paper/2311.02490/) (2023)
- [Anderson acceleration for iteratively reweighted ℓ1 algorithm](https://mlsanity.com/paper/2403.07271/) (2024)
- [Anderson Acceleration in Nonsmooth Problems: Local Convergence via Active Manifold Identification](https://mlsanity.com/paper/2410.09420/) (2024)
- [HyDRA: Hybrid Denoising Regularization for Measurement-Only DEQ Training](https://mlsanity.com/paper/2601.01228/) (2026)
- [Gradient Extrapolation-Based Policy Optimization](https://mlsanity.com/paper/2605.06755/) (2026)

## Abstract (arXiv)

We describe a convergence acceleration technique for unconstrained optimization problems. Our scheme computes estimates of the optimum from a nonlinear average of the iterates produced by any optimization method. The weights in this average are computed via a simple linear system, whose solution can be updated online. This acceleration scheme runs in parallel to the base algorithm, providing improved estimates of the solution on the fly, while the original optimization method is running. Numerical experiments are detailed on classical classification problems.