mlsanityData to 7 Oct 2026

Efficient approaches for escaping higher order saddle points in non-convex optimization

arXiv 1602.05908 ↗First posted 18 Feb 2016cs.LG stat.MLcited by 23

Not on the map. Why?

Abstract (arXiv)

Local search heuristics for non-convex optimizations are popular in applied machine learning. However, in general it is hard to guarantee that such algorithms even converge to a local minimum, due to the existence of complicated saddle point structures in high dimensions. Many functions have degenerate saddle points such that the first and second order derivatives cannot distinguish them with local optima. In this paper we use higher order derivatives to escape these saddle points: we design the first efficient algorithm guaranteed to converge to a third order local optimum (while existing techniques are at most second order). We also show that it is NP-hard to extend this further to finding fourth order local optima.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 6; 2023: 4; 2024: 1; 2025: 1; 2026: 2.

23 papers.

Sort:

  1. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  2. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  3. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  4. 2022Agnostic Learning of General ReLU Activation Using Gradient Descentcited by 12
  5. 2022First-Order Algorithms for Min-Max Optimization in Geodesic Metric Spacescited by 10
  6. 2021Escaping Saddle Points Faster with Stochastic Momentumcited by 8
  7. 2022Simple and Optimal Stochastic Gradient Methods for Nonsmooth Nonconvex Optimizationcited by 5
  8. 2023Symmetry & Critical Points for Symmetric Tensor Decomposition Problemscited by 3
  9. 2021A Comprehensive Study on Optimization Strategies for Gradient Descent In Deep Learningcited by 1
  10. 2021Noisy Truncated SGD: Optimization and Generalizationcited by 1
  11. 2021Escaping Saddle Points with Compressed SGDcited by 1
  12. 2021Towards Modeling and Resolving Singular Parameter Spaces using Stratifoldscited by 1
11 more
  1. 2022A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorizationcited by 1
  2. 2023Understanding Predictive Coding as an Adaptive Trust-Region Methodcited by 1
  3. 2023Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networkscited by 1
  4. 2024Noise is All You Need: Private Second-Order Convergence of Noisy SGDcited by 1
  5. 2021Boundary Conditions for Linear Exit Time Gradient Trajectories Around Saddle Points: Analysis and Algorithmcited by 0
  6. 2021Escaping Saddle Points with Stochastically Controlled Stochastic Gradient Methodscited by 0
  7. 2022HOUDINI: Escaping from Moderately Constrained Saddlescited by 0
  8. 2023Faster Riemannian Newton-type Optimization by Subsampling and Cubic Regularizationcited by 0
  9. 2025Policy Newton methods for Distortion Riskmetricscited by 0
  10. 2026Generalized Random Direction Newton Algorithms for Stochastic Optimizationcited by 0
  11. 2026Benign Loss Landscapes Can Coexist with Worst-Case Hardnesscited by 0