mlsanityData to 14 Oct 2026

SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network

arXiv 2206.05794 ↗First posted 12 Jun 2022cs.LG stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

We investigate the inherent bias of Stochastic Gradient Descent (SGD) toward learning low-rank weight matrices during the training of deep neural networks. Our results demonstrate that training with mini-batch SGD and weight decay induces a bias toward rank minimization in the weight matrices. Specifically, we show both theoretically and empirically that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. Additionally, we predict and empirically confirm that weight decay is essential for this bias to occur. Unlike previous literature, our analysis does not rely on assumptions about the data, convergence, or optimality of the weight matrices, making it applicable to a wide range of neural network architectures of any width or depth. Finally, we empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 12 Jun 2022.

Builds on

25 of 38 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2019When Does Label Smoothing Help?cited by 988
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
13 more
  1. 2017Exploring Generalization in Deep Learningcited by 600
  2. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  3. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  5. 2017Three Factors Influencing Minima in SGDcited by 273
  6. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  7. 2020Directional convergence and alignment in deep learningcited by 178
  8. 2020Good Subnetworks Provably Exist: Pruning via Greedy Forward Selectioncited by 70
  9. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  10. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  11. 2017Compression-aware Training of Deep Networkscited by 46
  12. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  13. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025A Closer Look at Multimodal Representation Collapsecited by 28
  2. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  3. 2025LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)cited by 20
  4. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 9
  5. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 6
  6. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 6
  7. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 5
  8. 2024Estimating Global Input Relevance and Enforcing Sparse Representations with a Scalable Spectral Neural Network Approachcited by 4
  9. 2024The Fair Language Model Paradoxcited by 4
  10. 2025Learning Dynamics of Meta-Learning in Small Model Pretrainingcited by 3
  11. 2025The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representationscited by 2
  12. 2026Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnosticscited by 2
7 more
  1. 2026Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignmentcited by 1
  2. 2026Does Weight Decay Enhance Training Stability?cited by 1
  3. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  4. 2026Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scalingcited by 0
  5. 2026From the Loss Landscape to Diverse Feature Learning in Neural Networkscited by 0
  6. 2026Aligned but Flattened: Analyzing the Trade-off between Cultural Alignment and Diversity in LLMscited by 0
  7. 2026Stabilizing the Dynamic Low-Rank Trainingcited by 0