# Equilibrated adaptive learning rates for non-convex optimization

arXiv: https://arxiv.org/abs/1502.04390 · HTML: https://mlsanity.com/paper/1502.04390/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Feb 2015.

## Built on by (most cited first, 30 of 42)

- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Many Hands Make Light Work: Using Essay Traits to Automatically Score Essays](https://mlsanity.com/paper/2102.00781/) (2021)
- [An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangians](https://mlsanity.com/paper/2102.05320/) (2021)
- [Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring](https://mlsanity.com/paper/2305.16826/) (2023)
- [Exponentially Many Local Minima in Quantum Neural Networks](https://mlsanity.com/paper/2110.02479/) (2021)
- [Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent](https://mlsanity.com/paper/2110.11442/) (2021)
- [AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods](https://mlsanity.com/paper/2102.09700/) (2021)
- [Black Box Lie Group Preconditioners for SGD](https://mlsanity.com/paper/2211.04422/) (2022)
- [Optimizing Rank for High-Fidelity Implicit Neural Representations](https://mlsanity.com/paper/2512.14366/) (2025)
- [Neural Predictive Monitoring under Partial Observability](https://mlsanity.com/paper/2108.07134/) (2021)
- [On uniform-in-time diffusion approximation for stochastic gradient descent](https://mlsanity.com/paper/2207.04922/) (2022)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [Federated Submodel Optimization for Hot and Cold Data Features](https://mlsanity.com/paper/2109.07704/) (2021)
- [Gradient Descent, Stochastic Optimization, and Other Tales](https://mlsanity.com/paper/2205.00832/) (2022)
- [SeisT: A foundational deep learning model for earthquake monitoring tasks](https://mlsanity.com/paper/2310.01037/) (2023)
- [Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks](https://mlsanity.com/paper/2106.03763/) (2021)
- [Pearl: Parallel Evolutionary and Reinforcement Learning Library](https://mlsanity.com/paper/2201.09568/) (2022)
- [Deep Learning Architecture for Automatic Essay Scoring](https://mlsanity.com/paper/2206.08232/) (2022)
- [Deep Crowd Anomaly Detection: State-of-the-Art, Challenges, and Future Research Directions](https://mlsanity.com/paper/2210.13927/) (2022)
- [Smoothness Similarity Regularization for Few-Shot GAN Adaptation](https://mlsanity.com/paper/2308.09717/) (2023)
- [SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation](https://mlsanity.com/paper/2502.18153/) (2025)
- [Multiple Sclerosis Lesions Segmentation using Attention-Based CNNs in FLAIR Images](https://mlsanity.com/paper/2201.01832/) (2022)
- [Langevin algorithms for Markovian Neural Networks and Deep Stochastic control](https://mlsanity.com/paper/2212.12018/) (2022)
- [Scalable Balanced Training of Conditional Generative Adversarial Neural Networks on Image Data](https://mlsanity.com/paper/2102.10485/) (2021)
- [A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks](https://mlsanity.com/paper/2206.02001/) (2022)
- [Fusarium head blight detection, spikelet estimation, and severity assessment in wheat using 3D convolutional neural networks](https://mlsanity.com/paper/2303.05634/) (2023)
- [Energy Regularized RNNs for Solving Non-Stationary Bandit Problems](https://mlsanity.com/paper/2303.06552/) (2023)
- [A reusable pipeline for large-scale fiber segmentation on unidirectional fiber beds using fully convolutional neural networks](https://mlsanity.com/paper/2101.04823/) (2021)
- [A Probabilistically Motivated Learning Rate Adaptation for Stochastic Optimization](https://mlsanity.com/paper/2102.10880/) (2021)
- [Convolutional Neural Network(CNN/ConvNet) in Stock Price Movement Prediction](https://mlsanity.com/paper/2106.01920/) (2021)

## Abstract (arXiv)

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner has undesirable behavior in the presence of both positive and negative curvature, and present theoretical and empirical evidence that the so-called equilibration preconditioner is comparatively better suited to non-convex problems. We introduce a novel adaptive learning rate scheme, called ESGD, based on the equilibration preconditioner. Our experiments show that ESGD performs as well or better than RMSProp in terms of convergence speed, always clearly improving over plain stochastic gradient descent.