mlsanityData to 7 Oct 2026

An Efficient Nonlinear Acceleration method that Exploits Symmetry of the Hessian

arXiv 2210.12573 ↗First posted 22 Oct 2022cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

Nonlinear acceleration methods are powerful techniques to speed up fixed-point iterations. However, many acceleration methods require storing a large number of previous iterates and this can become impractical if computational resources are limited. In this paper, we propose a nonlinear Truncated Generalized Conjugate Residual method (nlTGCR) whose goal is to exploit the symmetry of the Hessian to reduce memory usage. The proposed method can be interpreted as either an inexact Newton or a quasi-Newton method. We show that, with the help of global strategies like residual check techniques, nlTGCR can converge globally for general nonlinear problems and that under mild conditions, nlTGCR is able to achieve superlinear convergence. We further analyze the convergence of nlTGCR in a stochastic setting. Numerical results demonstrate the superiority of nlTGCR when compared with several other competitive baseline approaches on a few problems. Our code will be available in the future.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 64 references that are in the map. First posted on arXiv 22 Oct 2022.

Builds on

21 of 64 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2015An Introduction to Convolutional Neural Networkscited by 510
  4. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  5. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
  6. 2021Acceleration Methodscited by 48
  7. 2016Exact and Inexact Subsampled Newton Methods for Optimizationcited by 38
  8. 2018Globally Convergent Type-I Anderson Acceleration for Non-Smooth Fixed-Point Iterationscited by 24
  9. 2018A Newton-CG Algorithm with Complexity Guarantees for Smooth Unconstrained Optimizationcited by 23
  10. 2021Newton-LESS: Sparsification without Trade-offs for the Sketched Newton Updatecited by 23
  11. 2018Anderson Acceleration for Reinforcement Learningcited by 14
  12. 2019Convergence of Gradient Methods on Bilinear Zero-Sum Gamescited by 12
9 more
  1. 2016Sub-Sampled Newton Methods I: Globally Convergent Algorithmscited by 8
  2. 2019Regularized Anderson Acceleration for Off-Policy Deep Reinforcement Learningcited by 8
  3. 2021Damped Anderson Mixing for Deep Reinforcement Learning: Acceleration, Convergence, and Stabilizationcited by 7
  4. 2018Newton-MR: Inexact Newton Method With Minimum Residual Sub-problem Solvercited by 6
  5. 2021Stochastic Anderson Mixing for Nonconvex Stochastic Optimizationcited by 6
  6. 2019Analysis of the BFGS Method with Errorscited by 5
  7. 2020On the Asymptotic Linear Convergence Speed of Anderson Acceleration, Nesterov Acceleration, and Nonlinear GMREScited by 3
  8. 2020Generalization of Quasi-Newton Methods: Application to Robust Symmetric Multisecant Updatescited by 3
  9. 2022Descent Properties of an Anderson Accelerated Gradient Method With Restartingcited by 1

Built on by

Not cited yet.