# An Efficient Nonlinear Acceleration method that Exploits Symmetry of the Hessian

arXiv: https://arxiv.org/abs/2210.12573 · HTML: https://mlsanity.com/paper/2210.12573/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning](https://mlsanity.com/paper/2006.00719/) (2020)
- [Practical Quasi-Newton Methods for Training Deep Neural Networks](https://mlsanity.com/paper/2006.08877/) (2020)
- [Acceleration Methods](https://mlsanity.com/paper/2101.09545/) (2021)
- [Exact and Inexact Subsampled Newton Methods for Optimization](https://mlsanity.com/paper/1609.08502/) (2016)
- [Globally Convergent Type-I Anderson Acceleration for Non-Smooth Fixed-Point Iterations](https://mlsanity.com/paper/1808.03971/) (2018)
- [A Newton-CG Algorithm with Complexity Guarantees for Smooth Unconstrained Optimization](https://mlsanity.com/paper/1803.02924/) (2018)
- [Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Update](https://mlsanity.com/paper/2107.07480/) (2021)
- [Anderson Acceleration for Reinforcement Learning](https://mlsanity.com/paper/1809.09501/) (2018)
- [Convergence of Gradient Methods on Bilinear Zero-Sum Games](https://mlsanity.com/paper/1908.05699/) (2019)
- [Regularized Anderson Acceleration for Off-Policy Deep Reinforcement Learning](https://mlsanity.com/paper/1909.03245/) (2019)
- [Sub-Sampled Newton Methods I: Globally Convergent Algorithms](https://mlsanity.com/paper/1601.04737/) (2016)
- [Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilization](https://mlsanity.com/paper/2110.08896/) (2021)
- [Stochastic Anderson Mixing for Nonconvex Stochastic Optimization](https://mlsanity.com/paper/2110.01543/) (2021)
- [Newton-MR: Inexact Newton Method With Minimum Residual Sub-problem Solver](https://mlsanity.com/paper/1810.00303/) (2018)
- [Analysis of the BFGS Method with Errors](https://mlsanity.com/paper/1901.09063/) (2019)
- [On the Asymptotic Linear Convergence Speed of Anderson Acceleration, Nesterov Acceleration, and Nonlinear GMRES](https://mlsanity.com/paper/2007.01996/) (2020)
- [Generalization of Quasi-Newton Methods: Application to Robust Symmetric Multisecant Updates](https://mlsanity.com/paper/2011.03358/) (2020)
- [Descent Properties of an Anderson Accelerated Gradient Method With Restarting](https://mlsanity.com/paper/2206.01372/) (2022)

## Abstract (arXiv)

Nonlinear acceleration methods are powerful techniques to speed up fixed-point iterations. However, many acceleration methods require storing a large number of previous iterates and this can become impractical if computational resources are limited. In this paper, we propose a nonlinear Truncated Generalized Conjugate Residual method (nlTGCR) whose goal is to exploit the symmetry of the Hessian to reduce memory usage. The proposed method can be interpreted as either an inexact Newton or a quasi-Newton method. We show that, with the help of global strategies like residual check techniques, nlTGCR can converge globally for general nonlinear problems and that under mild conditions, nlTGCR is able to achieve superlinear convergence. We further analyze the convergence of nlTGCR in a stochastic setting. Numerical results demonstrate the superiority of nlTGCR when compared with several other competitive baseline approaches on a few problems. Our code will be available in the future.