# Adaptive Learning Rate Clipping Stabilizes Learning

arXiv: https://arxiv.org/abs/1906.09060 · HTML: https://mlsanity.com/paper/1906.09060/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2019.

## Built on by (most cited first, 6 of 6)

- [Ranger21: a synergistic deep learning optimizer](https://mlsanity.com/paper/2106.13731/) (2021)
- [Geometrical structures of digital fluctuations in parameter space of neural networks trained with adaptive momentum optimization](https://mlsanity.com/paper/2408.12273/) (2024)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [Convolutional Neural Network (CNN) to reduce construction loss in JPEG compression caused by Discrete Fourier Transform (DFT)](https://mlsanity.com/paper/2209.03475/) (2022)
- [Intelligent gradient amplification for deep neural networks](https://mlsanity.com/paper/2305.18445/) (2023)
- [Mapping of Land Use and Land Cover (LULC) using EuroSAT and Transfer Learning](https://mlsanity.com/paper/2401.02424/) (2023)

## Abstract (arXiv)

Artificial neural network training with stochastic gradient descent can be destabilized by "bad batches" with high losses. This is often problematic for training with small batch sizes, high order loss functions or unstably high learning rates. To stabilize learning, we have developed adaptive learning rate clipping (ALRC) to limit backpropagated losses to a number of standard deviations above their running means. ALRC is designed to complement existing learning algorithms: Our algorithm is computationally inexpensive, can be applied to any loss function or batch size, is robust to hyperparameter choices and does not affect backpropagated gradient distributions. Experiments with CIFAR-10 supersampling show that ALCR decreases errors for unstable mean quartic error training while stable mean squared error training is unaffected. We also show that ALRC decreases unstable mean squared errors for partial scanning transmission electron micrograph completion. Our source code is publicly available at https://github.com/Jeffrey-Ede/ALRC