mlsanityData to 7 Oct 2026

Loss Spike in Training Neural Networks

arXiv 2305.12133 ↗First posted 20 May 2023cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In this work, we investigate the mechanism underlying loss spikes observed during neural network training. When the training enters a region with a lower-loss-as-sharper (LLAS) structure, the training becomes unstable, and the loss exponentially increases once the loss landscape is too sharp, resulting in the rapid ascent of the loss spike. The training stabilizes when it finds a flat region. From a frequency perspective, we explain the rapid descent in loss as being primarily influenced by low-frequency components. We observe a deviation in the first eigendirection, which can be reasonably explained by the frequency principle, as low-frequency information is captured rapidly, leading to the rapid descent. Inspired by our analysis of loss spikes, we revisit the link between the maximum eigenvalue of the loss Hessian (λmax), flatness and generalization. We suggest that λmax is a good measure of sharpness but not a good measure for generalization. Furthermore, we experimentally observe that loss spikes can facilitate condensation, causing input weights to evolve towards the same direction. And our experiments show that there is a correlation (similar trend) between λmax and condensation. This observation may provide valuable insights for further theoretical research on the relationship between loss spikes, λmax, and generalization.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

49 of 62 references.

References · 49

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  11. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  12. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
37 more
  1. 2017Three Factors Influencing Minima in SGDcited by 270
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  3. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  4. 2018Training behavior of deep neural network in frequency domaincited by 174
  5. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  6. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  7. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  8. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  9. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  10. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  11. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  12. 2022Understanding the unstable convergence of gradient descentcited by 84
  13. 2020Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domainscited by 78
  14. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  15. 2018A Walk with SGDcited by 74
  16. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  17. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  18. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
  19. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  20. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  21. 2018Gradient Descent Quantizes ReLU Network Featurescited by 57
  22. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  23. 2019Theory of the Frequency Principle for General Deep Neural Networkscited by 45
  24. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  25. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  26. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 36
  27. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 31
  28. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  29. 2020Deep frequency principle towards understanding why deeper learning is fastercited by 30
  30. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 29
  31. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  32. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  33. 2022Empirical Phase Diagram for Three-layer Neural Networks with Infinite Widthcited by 17
  34. 2023Phase Diagram of Initial Condensation for Two-layer Neural Networkscited by 15
  35. 2020An analytic theory of shallow networks dynamics for hinge loss classificationcited by 14
  36. 2021Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networkscited by 14
  37. 2019Frequency-Aware Reconstruction of Fluid Simulations with Generative Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 9; 2026: 4.

15 papers.

Sort:

By area:

  1. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  2. 2025QVGen: Pushing the Limit of Quantized Video Generative Modelscited by 20
  3. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
  4. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  5. 2025An Analysis for Reasoning Bias of Language Models with Small Initializationcited by 13
  6. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  7. 2025Topological Signatures of ReLU Neural Network Activation Patternscited by 2
  8. 2025Scalable Complexity Control Facilitates Reasoning Ability of LLMscited by 1
  9. 2026Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decaycited by 1
  10. 2025On the Relationship Between Double Descent of CNNs and Shape/Texture Bias Under Learning Processcited by 0
  11. 2025SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adamcited by 0
  12. 2025Towards Reliable Pediatric Brain Tumor Segmentation: Task-Specific nnU-Net Enhancementscited by 0
3 more
  1. 2026Non-normal spectral signatures of instability in neural network training dynamicscited by 0
  2. 2026Improving Adversarial Robustness of Attribution via Implicit Regularizationcited by 0
  3. 2026Beyond Quadratic Loss: The Stability Phase Diagram of Adamcited by 0