mlsanityData to 7 Oct 2026

First-order Methods Almost Always Avoid Saddle Points

arXiv 1710.07406 ↗First posted 20 Oct 2017stat.ML cs.LG math.OCcited by 14

Not on the map. Why?

Abstract (arXiv)

We establish that first-order methods avoid saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid saddle points.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 3; 2023: 1; 2024: 2; 2025: 1.

14 papers.

Sort:

  1. 2021DC3: A learning method for optimization with hard constraintscited by 181
  2. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  3. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  4. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  5. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 45
  6. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  7. 2021Near-optimal Local Convergence of Alternating Gradient Descent-Ascent for Minimax Optimizationcited by 31
  8. 2022Stochastic noise can be helpful for variational quantum algorithmscited by 16
  9. 2021Solving Min-Max Optimization with Hidden Structure via Gradient Descent Ascentcited by 8
  10. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  11. 2023A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systemscited by 4
  12. 2021Training Generative Adversarial Networks with Adaptive Composite Gradientcited by 1
2 more
  1. 2021Constants of Motion: The Antidote to Chaos in Optimization and Game Dynamicscited by 0
  2. 2025Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Liftingcited by 0