mlsanityData to 7 Oct 2026

Better SGD using Second-order Momentum

arXiv 2103.03265 ↗First posted 4 Mar 2021cs.LG stat.MLcited by 14

Not on the map. Why?

Abstract (arXiv)

We develop a new algorithm for non-convex stochastic optimization that finds an ε-critical point in the optimal O(ε-3) stochastic gradient and Hessian-vector product computations. Our algorithm uses Hessian-vector products to "correct" a bias term in the momentum of SGD with momentum. This leads to better gradient estimates in a manner analogous to variance reduction methods. In contrast to prior work, we do not require excessively large batch sizes, and are able to provide an adaptive algorithm whose convergence rate automatically improves with decreasing variance in the gradient estimates. We validate our results on a variety of large-scale deep learning architectures and benchmarks tasks.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 4 Mar 2021.

Builds on

21 of 40 references.

References · 21

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  4. 2019On the Convergence of Adam and Beyondcited by 834
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 303
  8. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  9. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  10. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  11. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  12. 2020Momentum Improves Normalized SGDcited by 136
9 more
  1. 2018Stochastic Nested Variance Reduction for Nonconvex Optimizationcited by 63
  2. 2017Stochastic Cubic Regularization for Fast Nonconvex Optimizationcited by 55
  3. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  4. 2018A Progressive Batching L-BFGS Method for Machine Learningcited by 45
  5. 2017Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Informationcited by 41
  6. 2020Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitationscited by 39
  7. 2017Second-Order Optimization for Non-Convex Machine Learning: An Empirical Studycited by 35
  8. 2020Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimizationcited by 16
  9. 2016A Limited-Memory Quasi-Newton Algorithm for Bound-Constrained Nonsmooth Optimizationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 6; 2025: 3; 2026: 1.

14 papers.

Sort:

  1. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  2. 2022Momentum-Based Policy Gradient with Second-Order Informationcited by 11
  3. 2024Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPscited by 11
  4. 2022Momentum Aggregation for Private Non-convex ERMcited by 10
  5. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  6. 2024Improving Stochastic Cubic Newton with Momentumcited by 6
  7. 2023High Probability Analysis for Non-Convex Stochastic Optimization with Clippingcited by 5
  8. 2024Reevaluating Theoretical Analysis Methods for Optimization in Deep Learningcited by 3
  9. 2025Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limitscited by 3
  10. 2024Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentumcited by 1
  11. 2024Torque-Aware Momentumcited by 1
  12. 2026RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimizationcited by 1
2 more
  1. 2025NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximationcited by 0
  2. 2025Improved Convergence in Parameter-Agnostic Error Feedback through Momentumcited by 0