Empirically explaining SGD from a line search perspective
Not on the map. Why?
Abstract (arXiv)
Optimization in Deep Learning is mainly guided by vague intuitions and strong assumptions, with a limited understanding how and why these work in practice. To shed more light on this, our work provides some deeper understandings of how SGD behaves by empirically analyzing the trajectory taken by SGD from a line search perspective. Specifically, a costly quantitative analysis of the full-batch loss along SGD trajectories from common used models trained on a subset of CIFAR-10 is performed. Our core results include that the full-batch loss along lines in update step direction is highly parabolically. Further on, we show that there exists a learning rate with which SGD always performs almost exact line searches on the full-batch loss. Finally, we provide a different perspective why increasing the batch size has almost the same effect as decreasing the learning rate by the same factor.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 31 Mar 2021.
Builds on
23 of 28 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Densely Connected Convolutional Networkscited by 6,691
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2015Cyclical Learning Rates for Training Neural Networkscited by 627
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
- 2018An Empirical Model of Large-Batch Trainingcited by 268
11 more
- 2014Qualitatively characterizing neural network optimization problemscited by 186
- 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
- 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
- 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
- 2019Training Neural Networks for and by Interpolationcited by 45
- 2019Hessian based analysis of SGD for Deep Nets: Dynamics and Generalizationcited by 37
- 2015Probabilistic Line Searches for Stochastic Optimizationcited by 29
- 2018L4: Practical loss-based stepsize adaptation for deep learningcited by 26
- 2019Parabolic Approximation Line Search for DNNscited by 14
- 2016Big Batch SGD: Automated Inference using Adaptive Batch Sizescited by 13
- 2019Empirical study towards understanding line search approximations for training neural networkscited by 3
Built on by
Citing papers per year: 2021: 1; 2022: 0; 2023: 0; 2024: 1.
2 papers.