mlsanityData to 7 Oct 2026

Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes

arXiv 2605.06152 ↗First posted 7 May 2026cs.LG cs.CL math.OCcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep neural networks exhibit periodic loss spikes during unregularized long-term training, a phenomenon known as the "Slingshot Mechanism." Existing work usually attributes this to intrinsic optimization dynamics, but its triggering mechanism remains unclear. This paper proves that this phenomenon is a result of floating-point arithmetic precision limits. As training enters a high-confidence stage, the difference between the correct-class logit and the other logits may exceed the absorption-error threshold. Then during backpropagation, the gradient of the correct class is rounded exactly to zero, while the gradients of the incorrect classes remain nonzero. This breaks the zero-sum constraint of gradients across classes and introduces a systematic drift in the parameter update of the classifier layer. We prove that this drift forms a positive feedback loop with the feature, causing the global classifier mean and the global feature mean to grow exponentially. We call this mechanism Numerical Feature Inflation (NFI). This mechanism explains the rapid norm growth before a Slingshot spike, the subsequent reappearance of gradients, and the resulting loss spike. We further show that NFI is not equivalent to an observed loss spike: in more practical tasks, partial absorption may not produce visible spikes, but it can still break the zero-sum constraint and drive rapid growth of parameter norms. Our results reinterpret Slingshot as a numerical dynamic of finite-precision training, and provide a testable explanation for abnormal parameter growth and logit divergence in late-stage training.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 48 of its 53 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

48 of 53 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2019On the Convergence of Adam and Beyondcited by 834
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
36 more
  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  5. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  6. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  7. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  8. 2020Neural collapse with unconstrained featurescited by 140
  9. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  10. 2022Adam Can Converge Without Any Modification On Update Rulescited by 112
  11. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  12. 2020Traces of Class/Cross-Class Structure Pervade Deep Learning Spectracited by 94
  13. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  14. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  15. 2023Grokking modular arithmeticcited by 77
  16. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  17. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  18. 2024Understanding Optimization in Deep Learning with Central Flowscited by 57
  19. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 54
  20. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  21. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  22. 2025In Search of Adam's Secret Saucecited by 40
  23. 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 31
  24. 2024Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Modelcited by 19
  25. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 14
  26. 2025Let Me Grok for You: Accelerating Grokking via Embedding Transfer from a Weaker Modelcited by 13
  27. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  28. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
  29. 2025Why Low-Precision Transformer Training Fails: An Analysis on Flash Attentioncited by 11
  30. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 10
  31. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  32. 2025Grokking and Generalization Collapse: Insights from textttHTSR theorycited by 8
  33. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  34. 2024Progress Measures for Grokking on Real-world Taskscited by 7
  35. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  36. 2026Output Embedding Centering for Stable LLM Pretrainingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformerscited by 1
  2. 2026Automated Numerical Stability Analysis of Deep Learning Operatorscited by 0
  3. 2026Beyond Quadratic Loss: The Stability Phase Diagram of Adamcited by 0
  4. 2026Why Adaptive Optimizers Underestimate Rare Tokenscited by 0