mlsanityData to 7 Oct 2026

From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks

arXiv 2406.04657 ↗First posted 7 Jun 2024cs.LG cs.AI math.STcited by 9

Not on the map. Why?

Abstract (arXiv)

Training strategies for modern deep neural networks (NNs) tend to induce a heavy-tailed (HT) empirical spectral density (ESD) in the layer weights. While previous efforts have shown that the HT phenomenon correlates with good generalization in large NNs, a theoretical explanation of its occurrence is still lacking. Especially, understanding the conditions which lead to this phenomenon can shed light on the interplay between generalization and weight spectra. Our work aims to bridge this gap by presenting a simple, rich setting to model the emergence of HT ESD. In particular, we present a theory-informed setup for 'crafting' heavy tails in the ESD of two-layer NNs and present a systematic analysis of the HT ESD emergence without any gradient noise. This is the first work to analyze a noise-free setting, and we also incorporate optimizer (GD/Adam) dependent (large) learning rates into the HT ESD analysis. Our results highlight the role of learning rates on the Bulk+Spike and HT shape of the ESDs in the early phase of training, which can facilitate generalization in the two-layer NN. These observations shed light on the behavior of large-scale NNs, albeit in a much simpler setting.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 43 of its 70 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

43 of 70 references.

References · 43

  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  5. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  6. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  7. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  8. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  9. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  10. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  11. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  12. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
31 more
  1. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  2. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  3. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  4. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 107
  5. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  6. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  7. 2017A Random Matrix Approach to Neural Networkscited by 80
  8. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  9. 2019The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squarescited by 67
  10. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  11. 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
  12. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  13. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 60
  14. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  15. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  16. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  17. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  18. 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 43
  19. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
  20. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  21. 2021Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variancecited by 34
  22. 2020Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noisecited by 33
  23. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  24. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  25. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  26. 2021Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricscited by 19
  27. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 19
  28. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  29. 2023On Single Index Models beyond Gaussian Datacited by 17
  30. 2023Algorithmic Stability of Heavy-Tailed SGD with General Loss Functionscited by 15
  31. 2018Detection limits in the high-dimensional spiked rectangular modelcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 5.

9 papers.

  1. 2024AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Modelscited by 35
  2. 2024AlphaLoRA: Assigning LoRA Experts Based on Layer Training Qualitycited by 18
  3. 2024Model Balancing Helps Low-data Training and Fine-tuningcited by 11
  4. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 11
  5. 2025Eigenspectrum Analysis of Neural Networks without Aspect Ratio Biascited by 9
  6. 2026One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMscited by 2
  7. 2026Diffract: Spectral View of LLM Domain Adaptationcited by 1
  8. 2026Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamicscited by 0
  9. 2026A Full Adam Theorem for Spectral Heavy-Tail Onsetcited by 0