mlsanityData to 7 Oct 2026

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning

arXiv 2306.04815 ↗First posted 7 Jun 2023cs.LG math.OC stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

In this paper, we first present an explanation regarding the common occurrence of spikes in the training loss when neural networks are trained with stochastic gradient descent (SGD). We provide evidence that the spikes in the training loss of SGD are "catapults", an optimization phenomenon originally observed in GD with large learning rates in [Lewkowycz et al. 2020]. We empirically show that these catapults occur in a low-dimensional subspace spanned by the top eigenvectors of the tangent kernel, for both GD and SGD. Second, we posit an explanation for how catapults lead to better generalization by demonstrating that catapults promote feature learning by increasing alignment with the Average Gradient Outer Product (AGOP) of the true predictor. Furthermore, we demonstrate that a smaller batch size in SGD induces a larger number of catapults, thereby improving AGOP alignment and test performance.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 66 of its 94 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

66 of 94 references.

References · 66

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2016Wide Residual Networkscited by 3,057
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
38 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  2. 2016An overview of gradient descent optimization algorithmscited by 1,021
  3. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  13. 2017Three Factors Influencing Minima in SGDcited by 270
  14. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  15. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  16. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  17. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  18. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  19. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  20. 2021The Principles of Deep Learning Theorycited by 169
  21. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  22. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  23. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  24. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  25. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  26. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  27. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  28. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  29. 2020The Early Phase of Neural Network Trainingcited by 114
  30. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  31. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  32. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  33. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  34. 2022Understanding the unstable convergence of gradient descentcited by 84
  35. 2018A Walk with SGDcited by 74
  36. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  37. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  38. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 11; 2026: 7.

29 papers.

Sort:

  1. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  2. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  3. 2024Does SGD really happen in tiny subspaces?cited by 35
  4. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  5. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  6. 2024Average gradient outer product as a mechanism for deep neural collapsecited by 21
  7. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  8. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  9. 2024Feature learning as alignment: a structural property of gradient descent in non-linear neural networkscited by 9
  10. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 9
  11. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  12. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 7
17 more
  1. 2025Understanding the Evolution of the Neural Tangent Kernel at the Edge of Stabilitycited by 6
  2. 2025Understanding Sharpness Dynamics in NN Training with a Minimalist Example: The Effects of Dataset Difficulty, Depth, Stochasticity, and Morecited by 5
  3. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  4. 2025Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descentcited by 4
  5. 2025BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Trainingcited by 3
  6. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2
  7. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  8. 2025Large Learning Rates Simultaneously Achieve Robustness to Spurious Correlations and Compressibilitycited by 2
  9. 2025On the Neural Feature Ansatz for Deep Neural Networkscited by 1
  10. 2026Breaking Data Symmetry is Needed For Generalization in Feature Learning Kernelscited by 1
  11. 2026The Road Taken: The Role of Optimizers at the Edge of Stabilitycited by 1
  12. 2026Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamWcited by 1
  13. 2025Feature Learning Beyond the Edge of Stabilitycited by 0
  14. 2025SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adamcited by 0
  15. 2026Large Spikes in Stochastic Gradient Descent: A Large-Deviations Viewcited by 0
  16. 2026Improving Adversarial Robustness of Attribution via Implicit Regularizationcited by 0
  17. 2026Gradient Descent with Large Step Size Restores Symmetry in Deep Linear Networks with Multi-Pathwaycited by 0