mlsanityData to 7 Oct 2026

On the training dynamics of deep networks with L2 regularization

arXiv 2006.08643 ↗First posted 15 Jun 2020stat.ML cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

We study the role of L2 regularization in deep learning, and uncover simple relations between the performance of the model, the L2 coefficient, the learning rate, and the number of training steps. These empirical relations hold when the network is overparameterized. They can be used to predict the optimal regularization parameter of a given model. In addition, based on these observations we propose a dynamical schedule for the regularization parameter that improves performance and speeds up training. We test these proposals in modern image classification settings. Finally, we show that these empirical relations can be understood theoretically in the context of infinitely wide networks. We derive the gradient flow dynamics of such networks, and compare the role of L2 regularization in this context with that of linear models.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 9; 2024: 4; 2025: 10; 2026: 6.

38 papers.

Sort:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  2. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  3. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  4. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  5. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  6. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  7. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  8. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  9. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  10. 2023Continual Learning in Linear Classification on Separable Datacited by 31
  11. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  12. 2024How Much Can We Forget about Data Contamination?cited by 14
26 more
  1. 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 14
  2. 2025Adaptive kernel predictors from feature-learning infinite limits of neural networkscited by 13
  3. 2023TrustGuard: GNN-based Robust and Explainable Trust Evaluation with Dynamicity Supportcited by 12
  4. 2021How to decay your learning ratecited by 11
  5. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  6. 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
  7. 2022Cyclical Focal Losscited by 8
  8. 2025OUI Need to Talk About Weight Decay: A New Perspective on Overfitting Detectioncited by 7
  9. 2022General Cyclical Training of Neural Networkscited by 5
  10. 2023Improving Deep Learning Optimization through Constrained Parameter Regularizationcited by 5
  11. 2025A Gaussian Process View on Observation Noise and Initialization in Wide Neural Networkscited by 5
  12. 2024The Fair Language Model Paradoxcited by 3
  13. 2025Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Modelscited by 3
  14. 2025Grokking vs. Learning: Same Features, Different Encodingscited by 2
  15. 2025On the Cone Effect in the Learning Dynamicscited by 2
  16. 2025On the Theory of Continual Learning with Gradient Descent for Neural Networkscited by 2
  17. 2023Analysis of NaN Divergence in Training Monocular Depth Estimation Modelcited by 0
  18. 2024Classifying Dry Eye Disease Patients from Healthy Controls Using Machine Learning and Metabolomics Datacited by 0
  19. 2025High-order Regularization for Machine Learning and Learning-based Controlcited by 0
  20. 2025Adaptive Learning Guided by Bias-Noise-Alignment Diagnosticscited by 0
  21. 2026Coden: Efficient Temporal Graph Neural Networks for Continuous Predictioncited by 0
  22. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  23. 2026RHVI-FDD: A Hierarchical Decoupling Framework for Low-Light Image Enhancementcited by 0
  24. 2026The Neural Tangent Kernel for Classificationcited by 0
  25. 2026Optimal ridge regularization revisitedcited by 0
  26. 2026Misalignment of Low-Loss Regions Causes Grokkingcited by 0