mlsanityData to 7 Oct 2026

Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization

arXiv 1607.01231 ↗First posted 5 Jul 2016math.OC cs.LG cs.NAcited by 37

Not on the map. Why?

Abstract (arXiv)

In this paper we study stochastic quasi-Newton methods for nonconvex stochastic optimization, where we assume that noisy information about the gradients of the objective function is available via a stochastic first-order oracle (SFO). We propose a general framework for such methods, for which we prove almost sure convergence to stationary points and analyze its worst-case iteration complexity. When a randomly chosen iterate is returned as the output of such an algorithm, we prove that in the worst-case, the SFO-calls complexity is O(ε-2) to ensure that the expectation of the squared norm of the gradient is smaller than the given accuracy tolerance ε. We also propose a specific algorithm, namely a stochastic damped L-BFGS (SdLBFGS) method, that falls under the proposed framework. Moreover, we incorporate the SVRG variance reduction technique into the proposed SdLBFGS method, and analyze its SFO-calls complexity. Numerical results on a nonconvex binary classification problem using SVM, and a multiclass classification problem using neural networks are reported.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 9; 2023: 6; 2024: 4; 2025: 1; 2026: 4.

37 papers.

Sort:

  1. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  2. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
  3. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  4. 2021Secure Bilevel Asynchronous Vertical Federated Learning with Backward Updatingcited by 37
  5. 2021Differentially private inference via noisy optimizationcited by 27
  6. 2021An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangianscited by 25
  7. 2021Tensor Normal Training for Deep Learning Modelscited by 24
  8. 2023Provably Convergent Plug-and-Play Quasi-Newton Methodscited by 15
  9. 2022SP2: A Second Order Stochastic Polyak Methodcited by 13
  10. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  11. 2022Stochastic Variance-Reduced Newton: Accelerating Finite-Sum Minimization with Large Batchescited by 9
  12. 2024Gradient descent with generalized Newton's methodcited by 9
25 more
  1. 2021Kronecker-factored Quasi-Newton Methods for Deep Learningcited by 6
  2. 2021slimTrain -- A Stochastic Approximation Method for Training Separable Deep Neural Networkscited by 6
  3. 2021Stochastic Anderson Mixing for Nonconvex Stochastic Optimizationcited by 6
  4. 2023A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Propertiescited by 5
  5. 2023A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimizationcited by 4
  6. 2024Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programmingcited by 4
  7. 2021Revisiting Recursive Least Squares for Training Deep Neural Networkscited by 3
  8. 2022FedSSO: A Federated Server-Side Second-Order Optimization Algorithmcited by 3
  9. 2024Incremental Quasi-Newton Methods with Faster Superlinear Convergence Ratescited by 2
  10. 2026Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noisecited by 2
  11. 2021Adaptive First- and Second-Order Algorithms for Large-Scale Machine Learningcited by 1
  12. 2022A Novel Fast Exact Subproblem Solver for Stochastic Quasi-Newton Cubic Regularized Optimizationcited by 1
  13. 2022On the efficiency of Stochastic Quasi-Newton Methods for Deep Learningcited by 1
  14. 2022Convergence Rates of Stochastic Zeroth-order Gradient Descent for L ojasiewicz Functionscited by 1
  15. 2023FOSI: Hybrid First and Second Order Optimizationcited by 1
  16. 2024A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothnesscited by 1
  17. 2025Symmetric Rank-One Quasi-Newton Methods for Deep Learning Using Cubic Regularizationcited by 1
  18. 2021A theoretical and empirical study of new adaptive algorithms with additional momentum steps and shifted updates for stochastic non-convex optimizationcited by 0
  19. 2021Bolstering Stochastic Gradient Descent with Model Buildingcited by 0
  20. 2021Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive Step Sizecited by 0
  21. 2022Can We Do Better Than Random Start? The Power of Data Outsourcingcited by 0
  22. 2023Jorge: Approximate Preconditioning for GPU-efficient Second-order Optimizationcited by 0
  23. 2026Stability and Generalization of Stochastic Nonconvex Optimization under Heavy-Tailed Noise: From Gradient Norm to Goldstein Stationaritycited by 0
  24. 2026Accelerated Gradient Descent for Faster Convergence with Minimal Overheadcited by 0
  25. 2026SoftServe: A Scalable Quasi-Newton Method for Deep Learningcited by 0