mlsanityData to 14 Oct 2026

How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD

arXiv 2406.11110 ↗First posted 17 Jun 2024cs.LG math.OC stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

We investigate the ability of deep neural networks to identify the support of the target function. Our findings reveal that mini-batch SGD effectively learns the support in the first layer of the network by shrinking to zero the weights associated with irrelevant components of input. In contrast, we demonstrate that while vanilla GD also approximates the target function, it requires an explicit regularization term to learn the support in the first layer. We prove that this property of mini-batch SGD is due to a second-order implicit regularization effect which is proportional to η / b (step size / batch size). Our results are not only another proof that implicit regularization has a significant impact on training optimization dynamics but they also shed light on the structure of the features that are learned by the network. Additionally, they suggest that smaller batches enhance feature interpretability and reduce dependency on initialization.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

38 of 51 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2017Exploring Generalization in Deep Learningcited by 600
  6. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  9. 2017Implicit Regularization in Matrix Factorizationcited by 322
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  11. 2017Three Factors Influencing Minima in SGDcited by 273
  12. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
26 more
  1. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  4. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  5. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  6. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  7. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  8. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
  9. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  10. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  11. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  12. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  13. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  14. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  15. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  16. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 48
  17. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  18. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  19. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 26
  20. 2020Poly-time universality and limitations of deep learningcited by 23
  21. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  22. 2023A Mathematical Model for Curriculum Learning for Paritiescited by 19
  23. 2019On the rate of convergence of fully connected very deep neural network regression estimatescited by 14
  24. 2023On the Trajectories of SGD Without Replacementcited by 11
  25. 2021SGD Implicitly Regularizes Generalization Errorcited by 10
  26. 2024Singular-limit analysis of gradient descent with noise injectioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 6.

8 papers.

  1. 2025Position: A Theory of Deep Learning Must Include Compositional Sparsitycited by 14
  2. 2024The Fair Language Model Paradoxcited by 4
  3. 2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
  4. 2026Does Weight Decay Enhance Training Stability?cited by 1
  5. 2026Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Seriescited by 0
  6. 2026Compositional Sparsity as an Inductive Bias for Neural Architecture Designcited by 0
  7. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0
  8. 2026The Conflict Between Logic and Memory: Training Conditions for Optimizer-Dependent Rule Acquisitioncited by 0