mlsanityData to 7 Oct 2026

The Power of Normalization: Faster Evasion of Saddle Points

arXiv 1611.04831 ↗First posted 15 Nov 2016cs.LG math.OC stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

A commonly used heuristic in non-convex optimization is Normalized Gradient Descent (NGD) - a variant of gradient descent in which only the direction of the gradient is taken into account and its magnitude ignored. We analyze this heuristic and show that with carefully chosen parameters and noise injection, this method can provably evade saddle points. We establish the convergence of NGD to a local minimum, and demonstrate rates which improve upon the fastest known first order algorithm due to Ge e al. (2015). The effectiveness of our method is demonstrated via an application to the problem of online tensor decomposition; a task for which saddle point evasion is known to result in convergence to global minima.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 8; 2024: 3; 2025: 3; 2026: 5.

29 papers.

Sort:

  1. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  2. 2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
  3. 2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
  4. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
  5. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  6. 2022Mitigating Memorization of Noisy Labels by Clipping the Model Predictioncited by 40
  7. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  8. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  9. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  10. 2022A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networkscited by 20
  11. 2021Escaping Saddle Points Faster with Stochastic Momentumcited by 8
  12. 2025Memory-Efficient LLM Pretraining via Minimalist Optimizer Designcited by 8
17 more
  1. 2023EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Datacited by 7
  2. 2021Robust Distributed Optimization With Randomly Corrupted Gradientscited by 6
  3. 2021On the Second-order Convergence Properties of Random Search Methodscited by 5
  4. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 4
  5. 2026Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizerscited by 4
  6. 2022Dissecting adaptive methods in GANscited by 3
  7. 2021Backward Gradient Normalization in Deep Neural Networkscited by 2
  8. 2026Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noisecited by 2
  9. 2021On the Differentially Private Nature of Perturbed Gradient Descentcited by 1
  10. 2023Understanding Predictive Coding as an Adaptive Trust-Region Methodcited by 1
  11. 2023Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networkscited by 1
  12. 2024Learning-Rate-Free Stochastic Optimization over Riemannian Manifoldscited by 1
  13. 2025Smoothed Normalization for Efficient Distributed Private Optimizationcited by 1
  14. 2022Generalized Gradient Flows with Provable Fixed-Time Convergence and Fast Evasion of Non-Degenerate Saddle Pointscited by 0
  15. 2026Accelerated Gradient Descent for Faster Convergence with Minimal Overheadcited by 0
  16. 2026TR-SSQP: A Trust-Region Method for Constrained Stochastic Optimization under Heavy-Tailed Noisecited by 0
  17. 2026MeqMuon: Matrix-Equilibrating Muon for LLM Pretrainingcited by 0