# How Does Learning Rate Decay Help Modern Neural Networks?

arXiv: https://arxiv.org/abs/1908.01878 · HTML: https://mlsanity.com/paper/1908.01878/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2019.

## Built on by (most cited first, 4 of 4)

- [The Power of Transfer Learning in Agricultural Applications: AgriNet](https://mlsanity.com/paper/2207.03881/) (2022)
- [Enhancing Nighttime Vehicle Detection with Day-to-Night Style Transfer and Labeling-Free Augmentation](https://mlsanity.com/paper/2412.16478/) (2024)
- [Scaling and Transferability of Annealing Strategies in Large Language Model Training](https://mlsanity.com/paper/2512.13705/) (2025)
- [Pruning random resistive memory for optimizing analogue AI](https://mlsanity.com/paper/2311.07164/) (2023)

## Abstract (arXiv)

Learning rate decay (lrDecay) is a de facto technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks.