Augmented Newton Method for Optimization: Global Linear Rate and Momentum Interpretation
Not on the map. Why?
Abstract (arXiv)
We propose two variants of Newton method for solving unconstrained minimization problem. Our method leverages optimization techniques such as penalty and augmented Lagrangian method to generate novel variants of the Newton method namely the Penalty Newton method and the Augmented Newton method. In doing so, we recover several well-known existing Newton method variants such as Damped Newton, Levenberg, and Levenberg-Marquardt methods as special cases. Moreover, the proposed Augmented Newton method can be interpreted as Newton method with adaptive heavy ball momentum. We provide global convergence results for the proposed methods under mild assumptions that hold for a wide variety of problems. The proposed methods can be sought as the penalty and augmented extensions of the results obtained by Karimireddy et. al [24].
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 47 references that are in the map. First posted on arXiv 23 May 2022.
Builds on
22 of 47 references.
References · 22
- 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 110
- 2015Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergencecited by 72
- 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
- 2017Stochastic Cubic Regularization for Fast Nonconvex Optimizationcited by 55
- 2017Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Informationcited by 41
- 2019RSN: Randomized Subspace Newtoncited by 34
- 2013Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regressioncited by 32
- 2015Convergence rates of sub-sampled Newton methodscited by 32
- 2015Global convergence rate analysis of unconstrained optimization methods based on probabilistic modelscited by 27
- 2018Global linear convergence of Newton's method without strong-convexity or Lipschitz gradientscited by 24
- 2019Stochastic Newton and Cubic Newton Methods with Simple Local Linear-Quadratic Ratescited by 24
- 2016Sub-sampled Newton Methods with Non-uniform Samplingcited by 21
10 more
- 2020Stochastic Subspace Cubic Newton Methodcited by 21
- 2015SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimizationcited by 18
- 2018Randomized Block Cubic Newton Methodcited by 14
- 2020Sketched Newton-Raphsoncited by 13
- 2018Stochastic Variance-Reduced Cubic Regularization for Nonconvex Optimizationcited by 12
- 2019Minimizing Uniformly Convex Functions by Cubic Regularization of Newton Methodcited by 12
- 2014A globally convergent incremental Newton methodcited by 5
- 2013Practical Inexact Proximal Quasi-Newton Method with Global Complexity Analysiscited by 4
- 2018Inexact Successive Quadratic Approximation for Regularized Optimizationcited by 4
- 2022Penalty & Augmented Kaczmarz Methods For Linear Systems & Linear Feasibility Problemscited by 2
Built on by
Citing papers per year: 2022: 1.
1 paper.