# Revisiting Normalized Gradient Descent: Fast Evasion of Saddle Points

arXiv: https://arxiv.org/abs/1711.05224 · HTML: https://mlsanity.com/paper/1711.05224/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2017.

## Built on by (most cited first, 29 of 29)

- [Optimizing Diffusion Noise Can Serve As Universal Motion Priors](https://mlsanity.com/paper/2312.11994/) (2023)
- [Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger](https://mlsanity.com/paper/2206.07136/) (2022)
- [RotoGrad: Gradient Homogenization in Multitask Learning](https://mlsanity.com/paper/2103.02631/) (2021)
- [Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts](https://mlsanity.com/paper/2310.05898/) (2023)
- [Mitigating Memorization of Noisy Labels by Clipping the Model Prediction](https://mlsanity.com/paper/2212.04055/) (2022)
- [DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method](https://mlsanity.com/paper/2305.16284/) (2023)
- [Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methods](https://mlsanity.com/paper/2305.12475/) (2023)
- [Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/2201.09736/) (2022)
- [Memory-Efficient LLM Pretraining via Minimalist Optimizer Design](https://mlsanity.com/paper/2506.16659/) (2025)
- [Low-rank State-action Value-function Approximation](https://mlsanity.com/paper/2104.08805/) (2021)
- [Muon Dynamics as a Spectral Wasserstein Flow](https://mlsanity.com/paper/2604.04891/) (2026)
- [The Ball-Proximal (="Broximal") Point Method: a New Algorithm, Convergence Theory, and Applications](https://mlsanity.com/paper/2502.02002/) (2025)
- [Online Optimization Perspective on First-Order and Zero-Order Decentralized Nonsmooth Nonconvex Stochastic Optimization](https://mlsanity.com/paper/2406.01484/) (2024)
- [Multiple Perturbation Attack: Attack Pixelwise Under Different ℓp-norms For Better Adversarial Performance](https://mlsanity.com/paper/2212.03069/) (2022)
- [Accelerating Wireless Federated Learning via Nesterov's Momentum and Distributed Principle Component Analysis](https://mlsanity.com/paper/2303.17885/) (2023)
- [Understanding Predictive Coding as an Adaptive Trust-Region Method](https://mlsanity.com/paper/2305.18188/) (2023)
- [Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networks](https://mlsanity.com/paper/2310.14901/) (2023)
- [Deep Regularized Compound Gaussian Network for Solving Linear Inverse Problems](https://mlsanity.com/paper/2311.17248/) (2023)
- [Incremental Gauss-Newton Descent for Machine Learning](https://mlsanity.com/paper/2408.05560/) (2024)
- [A Sample Efficient Alternating Minimization-based Algorithm For Robust Phase Retrieval](https://mlsanity.com/paper/2409.04733/) (2024)
- [Generalized Gradient Flows with Provable Fixed-Time Convergence and Fast Evasion of Non-Degenerate Saddle Points](https://mlsanity.com/paper/2212.03765/) (2022)
- [SING: A Plug-and-Play DNN Learning Technique](https://mlsanity.com/paper/2305.15997/) (2023)
- [PAT: Pixel-wise Adaptive Training for Long-tailed Segmentation](https://mlsanity.com/paper/2404.05393/) (2024)
- [Variational Stochastic Gradient Descent for Deep Neural Networks](https://mlsanity.com/paper/2404.06549/) (2024)
- [From exponential to finite/fixed-time stability: Applications to optimization](https://mlsanity.com/paper/2409.11713/) (2024)
- [Ruppert-Polyak averaging for Stochastic Order Oracle](https://mlsanity.com/paper/2411.15866/) (2024)
- [Principled Design of Diffusion-based Optimizers for Inverse Problems](https://mlsanity.com/paper/2605.11506/) (2026)
- [MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining](https://mlsanity.com/paper/2609.35701/) (2026)
- [Learning in Continuous Games from Pairwise Preference Feedback](https://mlsanity.com/paper/2610.05428/) (2026)

## Abstract (arXiv)

The note considers normalized gradient descent (NGD), a natural modification of classical gradient descent (GD) in optimization problems. A serious shortcoming of GD in non-convex problems is that GD may take arbitrarily long to escape from the neighborhood of a saddle point. This issue can make the convergence of GD arbitrarily slow, particularly in high-dimensional non-convex problems where the relative number of saddle points is often large. The paper focuses on continuous-time descent. It is shown that, contrary to standard GD, NGD escapes saddle points `quickly.' In particular, it is shown that (i) NGD `almost never' converges to saddle points and (ii) the time required for NGD to escape from a ball of radius r about a saddle point x^* is at most 5√kappar, where kappa is the condition number of the Hessian of f at x^*. As an application of this result, a global convergence-time bound is established for NGD under mild assumptions.