# WNGrad: Learn the Learning Rate in Gradient Descent

arXiv: https://arxiv.org/abs/1803.02865 · HTML: https://mlsanity.com/paper/1803.02865/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2018.

## Built on by (most cited first, 24 of 24)

- [Optimal Linear Decay Learning Rate Schedules and Further Refinements](https://mlsanity.com/paper/2310.07831/) (2023)
- [DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Method](https://mlsanity.com/paper/2305.16284/) (2023)
- [AdaGrad under Anisotropic Smoothness](https://mlsanity.com/paper/2406.15244/) (2024)
- [Robust Implicit Regularization via Weight Normalization](https://mlsanity.com/paper/2305.05448/) (2023)
- [Adaptive Gradient Methods with Local Guarantees](https://mlsanity.com/paper/2203.01400/) (2022)
- [The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks](https://mlsanity.com/paper/2306.11680/) (2023)
- [KOALA: A Kalman Optimization Algorithm with Loss Adaptivity](https://mlsanity.com/paper/2107.03331/) (2021)
- [Step-size Optimization for Continual Learning](https://mlsanity.com/paper/2401.17401/) (2024)
- [Accelerating Training of Batch Normalization: A Manifold Perspective](https://mlsanity.com/paper/2101.02916/) (2021)
- [AdaLoss: A computationally-efficient and provably convergent adaptive gradient method](https://mlsanity.com/paper/2109.08282/) (2021)
- [On the Weight Dynamics of Deep Normalized Networks](https://mlsanity.com/paper/2306.00700/) (2023)
- [A Stochastic Bundle Method for Interpolating Networks](https://mlsanity.com/paper/2201.12678/) (2022)
- [The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms](https://mlsanity.com/paper/2405.19585/) (2024)
- [Generalized AdaGrad (G-AdaGrad) and Adam: A State-Space Perspective](https://mlsanity.com/paper/2106.00092/) (2021)
- [AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop](https://mlsanity.com/paper/2111.15317/) (2021)
- [A unified convergence theory for adaptive first-order methods in the nonconvex case, including AdaNorm, full and diagonal AdaGrad and Muon](https://mlsanity.com/paper/2604.17423/) (2026)
- [Adjacent Leader Decentralized Stochastic Gradient Descent](https://mlsanity.com/paper/2405.11389/) (2024)
- [Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent](https://mlsanity.com/paper/2104.05447/) (2021)
- [Formal guarantees for heuristic optimization algorithms used in machine learning](https://mlsanity.com/paper/2208.00502/) (2022)
- [Exponential weight averaging as damped harmonic motion](https://mlsanity.com/paper/2310.13854/) (2023)
- [Recent Advances in Non-convex Smoothness Conditions and Applicability to Deep Linear Neural Networks](https://mlsanity.com/paper/2409.13672/) (2024)
- [Frankenstein Optimizer: Harnessing the Potential by Revisiting Optimization Tricks](https://mlsanity.com/paper/2503.02147/) (2025)
- [Towards Understanding the Optimization Mechanisms in Deep Learning](https://mlsanity.com/paper/2503.23016/) (2025)
- [Optimization as a Dynamical System: Generative Schedules from Latent ODEs](https://mlsanity.com/paper/2509.23052/) (2025)

## Abstract (arXiv)

Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice. If certain parameters of the loss function such as smoothness or strong convexity constants are known, theoretical learning rate schedules can be applied. However, in practice, such parameters are not known, and the loss function of interest is not convex in any case. The recently proposed batch normalization reparametrization is widely adopted in most neural network architectures today because, among other advantages, it is robust to the choice of Lipschitz constant of the gradient in loss function, allowing one to set a large learning rate without worry. Inspired by batch normalization, we propose a general nonlinear update rule for the learning rate in batch and stochastic gradient descent so that the learning rate can be initialized at a high value, and is subsequently decreased according to gradient observations along the way. The proposed method is shown to achieve robustness to the relationship between the learning rate and the Lipschitz constant, and near-optimal convergence rates in both the batch and stochastic settings (O(1/T) for smooth loss in the batch setting, and O(1/√T) for convex loss in the stochastic setting). We also show through numerical evidence that such robustness of the proposed method extends to highly nonconvex and possibly non-smooth loss function in deep learning problems.Our analysis establishes some first theoretical understanding into the observed robustness for batch normalization and weight normalization.