mlsanityData to 7 Oct 2026

Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions

arXiv 1605.00405 ↗First posted 2 May 2016math.DS cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Given a non-convex twice differentiable cost function f, we prove that the set of initial conditions so that gradient descent converges to saddle points where nabla2 f has at least one strictly negative eigenvalue has (Lebesgue) measure zero, even for cost functions f with non-isolated critical points, answering an open question in [Lee, Simchowitz, Jordan, Recht, COLT2016]. Moreover, this result extends to forward-invariant convex subspaces, allowing for weak (non-globally Lipschitz) smoothness assumptions. Finally, we produce an upper bound on the allowable step-size.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 6; 2023: 2; 2024: 3; 2025: 9; 2026: 1.

33 papers.

Sort:

By area:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2022Understanding the unstable convergence of gradient descentcited by 84
  3. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  4. 2022Neural Tangent Kernel: A Surveycited by 22
  5. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  6. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  7. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  8. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  9. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  10. 2021Solving Min-Max Optimization with Hidden Structure via Gradient Descent Ascentcited by 8
  11. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMscited by 6
  12. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
21 more
  1. 2025On exploration of an interior mirror descent flow for stochastic nonconvex constrained problemcited by 3
  2. 2021Inertial Newton Algorithms Avoiding Strict Saddle Pointscited by 2
  3. 2022Losing momentum in continuous-time stochastic optimisationcited by 2
  4. 2025The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networkscited by 2
  5. 2021New Q-Newton's method meets Backtracking line search: good convergence guarantee, saddle points avoidance, quadratic rate of convergence, and easy implementationcited by 1
  6. 2021Over-Parametrized Matrix Factorization in the Presence of Spurious Stationary Pointscited by 1
  7. 2023Almost Sure Saddle Avoidance of Stochastic Gradient Methods without the Bounded Gradient Assumptioncited by 1
  8. 2023On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activationscited by 1
  9. 2024Probability Distribution Learning and Its Application in Deep Learningcited by 1
  10. 2021Training Deep Neural Networks via Branch-and-Boundcited by 0
  11. 2021Asymptotic Escape of Spurious Critical Points on the Low-rank Matrix Manifoldcited by 0
  12. 2021Constants of Motion: The Antidote to Chaos in Optimization and Game Dynamicscited by 0
  13. 2022Accelerated Multiplicative Weights Update Avoids Saddle Points almost alwayscited by 0
  14. 2024Training-set-free two-stage deep learning for spectroscopic data de-noisingcited by 0
  15. 2025Langevin Multiplicative Weights Update with Applications in Polynomial Portfolio Managementcited by 0
  16. 2025Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappingscited by 0
  17. 2025Some remarks on gradient dominance and LQR policy optimizationcited by 0
  18. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
  19. 2025AutoGD: Automatic Learning Rate Selection for Gradient Descentcited by 0
  20. 2025Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activationscited by 0
  21. 2026Attention-based PCAcited by 0