mlsanityData to 14 Oct 2026

Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models

arXiv 2610.13524 ↗First posted 9 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Intermittent ``spikes'' in loss, during the pre-training of Large Language Models (LLMs), remains a mysterious phenomenon both theoretically and empirically. In this paper, we show that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate these spikes. We introduce a theoretical framework and simulation algorithm to model per-layer training dynamics of Deep Neural Networks (DNNs), using such quadratic models. In a case study on an Adam hyperparameter configuration (where β2 is set close to 1) known to induce spikes in Transformer LMs, our simulation successfully predicts bumps indicating spiky behavior from a given checkpoint. Furthermore, we derive an appropriate β2 schedule from observed metrics, which empirically improves performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

44 of 58 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2019On the Convergence of Adam and Beyondcited by 841
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  10. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  11. 2021Explaining Neural Scaling Lawscited by 363
  12. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
32 more
  1. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  2. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  3. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  4. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  5. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  6. 2022A Solvable Model of Neural Scaling Lawscited by 116
  7. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  8. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  9. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  10. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  11. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  12. 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 76
  13. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  14. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  15. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  16. 2024Understanding Optimization in Deep Learning with Central Flowscited by 64
  17. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 57
  18. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 56
  19. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 52
  20. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 41
  21. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 35
  22. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 31
  23. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 27
  24. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  25. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 18
  26. 2023Loss Spike in Training Neural Networkscited by 16
  27. 2025Adaptive Preconditioners Trigger Loss Spikes in Adamcited by 13
  28. 2025Dimension-adapted Momentum Outscales SGDcited by 12
  29. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12
  30. 2026Logarithmic-time Schedules for Scaling Language Models with Momentumcited by 9
  31. 2024To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensionscited by 7
  32. 2024Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikescited by 4

Built on by

Not cited yet.