# Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions

arXiv: https://arxiv.org/abs/1605.00405 · HTML: https://mlsanity.com/paper/1605.00405/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2016.

## Built on by (most cited first, 30 of 34)

- [Understanding the unstable convergence of gradient descent](https://mlsanity.com/paper/2204.01050/) (2022)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorization](https://mlsanity.com/paper/2106.14289/) (2021)
- [Neural Tangent Kernel: A Survey](https://mlsanity.com/paper/2208.13614/) (2022)
- [A PAC-Bayesian Link Between Generalisation and Flat Minima](https://mlsanity.com/paper/2402.08508/) (2024)
- [Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions](https://mlsanity.com/paper/2103.10922/) (2021)
- [A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions](https://mlsanity.com/paper/2108.04620/) (2021)
- [A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2104.00277/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)
- [Solving Min-Max Optimization with Hidden Structure via Gradient Descent Ascent](https://mlsanity.com/paper/2101.05248/) (2021)
- [LoRA is All You Need for Safety Alignment of Reasoning LLMs](https://mlsanity.com/paper/2507.17075/) (2025)
- [Gradient descent provably escapes saddle points in the training of shallow ReLU networks](https://mlsanity.com/paper/2208.02083/) (2022)
- [On exploration of an interior mirror descent flow for stochastic nonconvex constrained problem](https://mlsanity.com/paper/2507.15264/) (2025)
- [Inertial Newton Algorithms Avoiding Strict Saddle Points](https://mlsanity.com/paper/2111.04596/) (2021)
- [Losing momentum in continuous-time stochastic optimisation](https://mlsanity.com/paper/2209.03705/) (2022)
- [The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks](https://mlsanity.com/paper/2507.06367/) (2025)
- [New Q-Newton's method meets Backtracking line search: good convergence guarantee, saddle points avoidance, quadratic rate of convergence, and easy implementation](https://mlsanity.com/paper/2108.10249/) (2021)
- [Over-Parametrized Matrix Factorization in the Presence of Spurious Stationary Points](https://mlsanity.com/paper/2112.13269/) (2021)
- [Almost Sure Saddle Avoidance of Stochastic Gradient Methods without the Bounded Gradient Assumption](https://mlsanity.com/paper/2302.07862/) (2023)
- [On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations](https://mlsanity.com/paper/2305.09904/) (2023)
- [Probability Distribution Learning and Its Application in Deep Learning](https://mlsanity.com/paper/2406.05666/) (2024)
- [Training Deep Neural Networks via Branch-and-Bound](https://mlsanity.com/paper/2104.01730/) (2021)
- [Asymptotic Escape of Spurious Critical Points on the Low-rank Matrix Manifold](https://mlsanity.com/paper/2107.09207/) (2021)
- [Constants of Motion: The Antidote to Chaos in Optimization and Game Dynamics](https://mlsanity.com/paper/2109.03974/) (2021)
- [Accelerated Multiplicative Weights Update Avoids Saddle Points almost always](https://mlsanity.com/paper/2204.11407/) (2022)
- [Training-set-free two-stage deep learning for spectroscopic data de-noising](https://mlsanity.com/paper/2402.18830/) (2024)
- [Langevin Multiplicative Weights Update with Applications in Polynomial Portfolio Management](https://mlsanity.com/paper/2502.19210/) (2025)
- [Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappings](https://mlsanity.com/paper/2506.08428/) (2025)
- [Some remarks on gradient dominance and LQR policy optimization](https://mlsanity.com/paper/2507.10452/) (2025)
- [Convergence of optimizers implies eigenvalues filtering at equilibrium](https://mlsanity.com/paper/2510.09034/) (2025)

## Abstract (arXiv)

Given a non-convex twice differentiable cost function f, we prove that the set of initial conditions so that gradient descent converges to saddle points where nabla2 f has at least one strictly negative eigenvalue has (Lebesgue) measure zero, even for cost functions f with non-isolated critical points, answering an open question in [Lee, Simchowitz, Jordan, Recht, COLT2016]. Moreover, this result extends to forward-invariant convex subspaces, allowing for weak (non-globally Lipschitz) smoothness assumptions. Finally, we produce an upper bound on the allowable step-size.