mlsanityData to 7 Oct 2026

Approaching Deep Learning through the Spectral Dynamics of Weights

arXiv 2408.11804 ↗First posted 21 Aug 2024cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale ``grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers. We also demonstrate that weight decay enhances this bias beyond its role as a norm regularizer, even in practical systems. Moreover, we show that these spectral dynamics distinguish memorizing networks from generalizing ones, offering a novel perspective on this longstanding conundrum. Additionally, we leverage spectral dynamics to explore the emergence of well-performing sparse subnetworks (lottery tickets) and the structure of the loss surface through linear mode connectivity. Our findings suggest that spectral dynamics provide a coherent framework to better understand the behavior of neural networks across diverse settings.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 92 of its 119 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

92 of 119 references.

References · 92

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2013Intriguing properties of neural networkscited by 5,114
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
38 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2017Decoupled Weight Decay Regularizationcited by 1,722
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  6. 2018On the Spectral Bias of Neural Networkscited by 1,321
  7. 2022Editing Models with Task Arithmeticcited by 1,076
  8. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  13. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  14. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  15. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  16. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  17. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  18. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  19. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  20. 2020What is being transferred in transfer learning?cited by 365
  21. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  22. 2017Implicit Regularization in Matrix Factorizationcited by 318
  23. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  24. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  25. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  26. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  27. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  28. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  29. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  30. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  31. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  32. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  33. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  34. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  35. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  36. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  37. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  38. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 20.

29 papers.

Sort:

By area:

  1. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  2. 2025RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMscited by 17
  3. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  4. 2026To Grok Grokking: Provable Grokking in Ridge Regressioncited by 6
  5. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  6. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  7. 2025Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivitycited by 2
  8. 2026Disentangling Geometry, Performance, and Training in Language Modelscited by 2
  9. 2026Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenariocited by 2
  10. 2026The Spectral Dynamics and Noise Geometry of Muoncited by 2
  11. 2026Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibilitycited by 2
  12. 2025Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokkingcited by 1
17 more
  1. 2025S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Plannercited by 1
  2. 2025From Memorization to Reasoning in the Spectrum of Loss Curvaturecited by 1
  3. 2026Low-Rank Key Value Attentioncited by 1
  4. 2026Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networkscited by 1
  5. 2026Does Weight Decay Enhance Training Stability?cited by 1
  6. 2026Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric Viewcited by 1
  7. 2026Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instabilitycited by 1
  8. 2026At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metricscited by 1
  9. 2026Diffract: Spectral View of LLM Domain Adaptationcited by 1
  10. 2026Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Trainingcited by 1
  11. 2026The Spectral Lifecycle of Transformer Training: Transient Compression Waves, Persistent Spectral Gradients, and the Q/K--V Asymmetrycited by 0
  12. 2026A Training-Time Diagnostic for Generalization via the Log-Alignment Ratiocited by 0
  13. 2026Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scalingcited by 0
  14. 2026GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-trainingcited by 0
  15. 2026Spectral Tail Interventions in Decoder-Only Language Models: Reasoning-Sensitive Weight Structure from Controlled Surgerycited by 0
  16. 2026Learning Under Forgetting: Statistical Support-Selective Retention in Stochastic Training Dynamicscited by 0
  17. 2026Signal-Noise Factorization Isolates Nuisance Variation into Removable Subspacescited by 0