mlsanityData to 13 Oct 2026

Equilibrated adaptive learning rates for non-convex optimization

arXiv 1502.04390 ↗First posted 15 Feb 2015cs.LG cs.NA math.NAcited by 42

Not on the map. Why?

Abstract (arXiv)

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the so-called equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.

Key facts. Cited by 42 papers in the map (data through 13 Oct 2026). First posted on arXiv 15 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 14; 2022: 10; 2023: 10; 2024: 2; 2025: 4; 2026: 2.

42 papers.

Sort:

By area:

  1. 2024Variational Learning is Effective for Large Deep Networkscited by 62
  2. 2021Many Hands Make Light Work: Using Essay Traits to Automatically Score Essayscited by 25
  3. 2021An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangianscited by 25
  4. 2023Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoringcited by 24
  5. 2021Exponentially Many Local Minima in Quantum Neural Networkscited by 22
  6. 2021Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descentcited by 16
  7. 2021AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methodscited by 15
  8. 2022Black Box Lie Group Preconditioners for SGDcited by 9
  9. 2021Neural Predictive Monitoring under Partial Observabilitycited by 7
  10. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  11. 2022On uniform-in-time diffusion approximation for stochastic gradient descentcited by 5
  12. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
30 more
  1. 2021Federated Submodel Optimization for Hot and Cold Data Featurescited by 4
  2. 2022Gradient Descent, Stochastic Optimization, and Other Talescited by 4
  3. 2023SeisT: A foundational deep learning model for earthquake monitoring taskscited by 4
  4. 2021Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networkscited by 3
  5. 2022Deep Learning Architecture for Automatic Essay Scoringcited by 3
  6. 2022Deep Crowd Anomaly Detection: State-of-the-Art, Challenges, and Future Research Directionscited by 3
  7. 2023Smoothness Similarity Regularization for Few-Shot GAN Adaptationcited by 3
  8. 2022Multiple Sclerosis Lesions Segmentation using Attention-Based CNNs in FLAIR Imagescited by 2
  9. 2022Pearl: Parallel Evolutionary and Reinforcement Learning Librarycited by 2
  10. 2022Langevin algorithms for Markovian Neural Networks and Deep Stochastic controlcited by 2
  11. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 2
  12. 2021Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Datacited by 1
  13. 2022A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networkscited by 1
  14. 2023Fusarium head blight detection, spikelet estimation, and severity assessment in wheat using 3D convolutional neural networkscited by 1
  15. 2023Energy Regularized RNNs for Solving Non-Stationary Bandit Problemscited by 1
  16. 2021A reusable pipeline for large-scale fiber segmentation on unidirectional fiber beds using fully convolutional neural networkscited by 0
  17. 2021A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimizationcited by 0
  18. 2021Convolutional Neural Network(CNN/ConvNet) in Stock Price Movement Predictioncited by 0
  19. 2021Abstraction of Markov Population Dynamics via Generative Adversarial Netscited by 0
  20. 2021Variance Reduction for Matrix Computations with Applications to Gaussian Processescited by 0
  21. 20222nd Place Solution for ICCV 2021 VIPriors Image Classification Challenge: An Attract-and-Repulse Learning Approachcited by 0
  22. 2023VALERIAN: Invariant Feature Learning for IMU Sensor-based Human Activity Recognition in the Wildcited by 0
  23. 2023HUB: Guiding Learned Optimizers with Continuous Prompt Tuningcited by 0
  24. 2023Intelligent gradient amplification for deep neural networkscited by 0
  25. 2023Preconditioned Federated Learningcited by 0
  26. 2024Primitive Agentic First-Order Optimizationcited by 0
  27. 2025Query Efficient Structured Matrix Learningcited by 0
  28. 2025BOND: License to Train with Black-Box Functionscited by 0
  29. 2026Sampling via Stochastic Interpolants by Langevin-based Velocity and Initialization Estimation in Flow ODEscited by 0
  30. 2026Beyond ℓ2-norm and ℓ_∞-norm: A Curvature-Inspired ℓp-Norm Scheme for Deep Neural Networkscited by 0