# On the saddle point problem for non-convex optimization

arXiv: https://arxiv.org/abs/1405.4604 · HTML: https://mlsanity.com/paper/1405.4604/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2014.

## Built on by (most cited first, 20 of 20)

- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [Smooth Bilevel Programming for Sparse Regularization](https://mlsanity.com/paper/2106.01429/) (2021)
- [Continual Learning with Weight Interpolation](https://mlsanity.com/paper/2404.04002/) (2024)
- [PGrad: Learning Principal Gradients For Domain Generalization](https://mlsanity.com/paper/2305.01134/) (2023)
- [A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems](https://mlsanity.com/paper/2308.04457/) (2023)
- [K"ahler Geometry of Quiver Varieties and Machine Learning](https://mlsanity.com/paper/2101.11487/) (2021)
- [A survey of deep learning optimizers -- first and second order methods](https://mlsanity.com/paper/2211.15596/) (2022)
- [Enhancing the Trainability of Variational Quantum Circuits with Regularization Strategies](https://mlsanity.com/paper/2405.01606/) (2024)
- [Staying the course: Locating equilibria of dynamical systems on Riemannian manifolds defined by point-clouds](https://mlsanity.com/paper/2204.10413/) (2022)
- [Optimization Methods in Deep Learning: A Comprehensive Overview](https://mlsanity.com/paper/2302.09566/) (2023)
- [Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networks](https://mlsanity.com/paper/2310.14901/) (2023)
- [Don't Be So Positive: Negative Step Sizes in Second-Order Methods](https://mlsanity.com/paper/2411.11224/) (2024)
- [A Framework for Finding Local Saddle Points in Two-Player Zero-Sum Black-Box Games](https://mlsanity.com/paper/2503.18224/) (2025)
- [Towards Robust Surrogate Models: Benchmarking Machine Learning Approaches to Expediting Phase Field Simulations of Brittle Fracture](https://mlsanity.com/paper/2507.07237/) (2025)
- [Saddle Hierarchy in Dense Associative Memory](https://mlsanity.com/paper/2508.19151/) (2025)
- [Saddlepoints in Unsupervised Least Squares](https://mlsanity.com/paper/2104.05000/) (2021)
- [Black holes and the loss landscape in machine learning](https://mlsanity.com/paper/2306.14817/) (2023)
- [Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations](https://mlsanity.com/paper/2510.24466/) (2025)
- [Accelerating SAV-based optimization via randomized low-rank Hessian approximation](https://mlsanity.com/paper/2606.10562/) (2026)
- [Explaining Machine Learning and Memorization with Statistical Mechanics](https://mlsanity.com/paper/2606.31110/) (2026)

## Abstract (arXiv)

A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for the ability of these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, and neural network theory, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new algorithm, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep neural network training, and provide preliminary numerical evidence for its superior performance.