The instabilities of large learning rate training: a loss landscape view
Not on the map. Why?
Abstract (arXiv)
Modern neural networks are undeniably successful. Numerous works study how the curvature of loss landscapes can affect the quality of solutions. In this work we study the loss landscape by considering the Hessian matrix during network training with large learning rates - an attractive regime that is (in)famously unstable. We characterise the instabilities of gradient descent, and we observe the striking phenomena of textitlandscape flattening and textitlandscape shift, both of which are intimately connected to the instabilities of training.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 22 Jul 2023.
Builds on
19 of 33 references.
References · 19
- 2017Attention Is All You Needcited by 60,361
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
- 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
7 more
- 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
- 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
- 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
- 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
- 2020Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisitedcited by 51
- 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
- 2020Flatness is a False Friendcited by 19
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.