mlsanityData to 14 Oct 2026

Large Spikes in Stochastic Gradient Descent: A Large-Deviations View

arXiv 2603.10079 ↗First posted 10 Mar 2026cs.LG math.PRcited by 1

Not on the map. Why?

Abstract (arXiv)

Large loss spikes in stochastic gradient descent are studied through a rigorous large-deviations analysis for a shallow, fully connected network in the NTK scaling. In contrast to full-batch gradient descent, the catapult phase is shown to split into inflationary and deflationary regimes, determined by an explicit log-drift criterion. In both cases, large spikes are shown to be at least polynomially likely. In addition, these spikes are shown to be the dominant mechanism by which sharp minima are escaped and curvature is reduced, thereby favouring flatter solutions. Corresponding results are also obtained for certain ReLU networks, and implications for curriculum learning are derived.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 53 of its 76 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

53 of 76 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Densely Connected Convolutional Networkscited by 6,720
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2016Wide Residual Networkscited by 3,070
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2016An overview of gradient descent optimization algorithmscited by 1,033
  9. 2018On Lazy Training in Differentiable Programmingcited by 675
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  11. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  12. 2017Three Factors Influencing Minima in SGDcited by 273
38 more
  1. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  2. 2021Hash Layers For Large Sparse Modelscited by 237
  3. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  4. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  5. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  6. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  7. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  8. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
  9. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  10. 2020The Early Phase of Neural Network Trainingcited by 115
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  12. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  13. 2019How do infinite width bounded norm networks look in function space?cited by 106
  14. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 100
  15. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  16. 2018A Walk with SGDcited by 78
  17. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 66
  18. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  19. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  20. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 39
  21. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 36
  22. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  23. 2020The Two Regimes of Deep Network Trainingcited by 30
  24. 2024Edge of Stochastic Stability: Revisiting the Edge of Stability for SGDcited by 30
  25. 2025Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUscited by 22
  26. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  27. 2024Early alignment in two-layer networks training is a two-edged swordcited by 18
  28. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 18
  29. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  30. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  31. 2024What is the long-run distribution of stochastic gradient descent? A large deviations analysiscited by 17
  32. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 13
  33. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  34. 2023Catapult Dynamics and Phase Transitions in Quadratic Netscited by 8
  35. 2023Benign Oscillation of Stochastic Gradient Descent with Large Learning Ratescited by 8
  36. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8
  37. 2025The global convergence time of stochastic gradient descent in non-convex landscapes: Sharp estimates via large deviationscited by 8
  38. 2024Singular-limit analysis of gradient descent with noise injectioncited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mind the Drift: Diagonal Linear Networks Under Large Learning Ratescited by 0