mlsanityData to 7 Oct 2026

L2 Regularization versus Batch and Weight Normalization

arXiv 1706.05350 ↗First posted 16 Jun 2017cs.LG stat.MLcited by 121

Not on the map. Why?

Abstract (arXiv)

Batch Normalization is a commonly used trick to improve the training of deep neural networks. These neural networks use L2 regularization, also called weight decay, ostensibly to prevent overfitting. However, we show that L2 regularization has no regularizing effect when combined with normalization. Instead, regularization has an influence on the scale of weights, and thereby on the effective learning rate. We investigate this dependence, both in theory, and experimentally. We show that popular optimization methods such as ADAM only partially eliminate the influence of normalization on the learning rate. This leads to a discussion on other ways to mitigate this issue.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 25; 2023: 19; 2024: 15; 2025: 18; 2026: 29.

121 papers.

Sort:

By area:

  1. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  2. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  3. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  4. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  5. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  6. 2024Tracking Meets LoRA: Faster Training, Larger Model, Stronger Performancecited by 70
  7. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  8. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  9. 2022Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formatscited by 50
  10. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  11. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  12. 2023Robust Principles: Architectural Design Principles for Adversarially Robust CNNscited by 46
38 more
  1. 2021Fast Certified Robust Training with Short Warmupcited by 45
  2. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 42
  3. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  4. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  5. 2024A review on data-driven constitutive laws for solidscited by 35
  6. 2025Symmetry in Neural Network Parameter Spacescited by 31
  7. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  8. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  9. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  10. 2024Weight decay induces low-rank attention layerscited by 27
  11. 2022Symmetry Teleportation for Accelerated Optimizationcited by 25
  12. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  13. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  14. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  15. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  16. 2022Rethinking Normalization Methods in Federated Learningcited by 19
  17. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  18. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  19. 2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
  20. 2024Exploring the Transferability of Visual Prompting for Multimodal Large Language Modelscited by 15
  21. 2024How Much Can We Forget about Data Contamination?cited by 14
  22. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  23. 2022Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularizationcited by 12
  24. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  25. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  26. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  27. 2021How to decay your learning ratecited by 11
  28. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  29. 2024Investigating Large Language Models for Code Vulnerability Detection: An Experimental Studycited by 11
  30. 2022Defects of Convolutional Decoder Networks in Frequency Representationcited by 10
  31. 2024Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Searchcited by 10
  32. 2022Amortized Proximal Optimizationcited by 9
  33. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  34. 2024Position: Lifetime tuning is incompatible with continual reinforcement learningcited by 9
  35. 2023Implicit Regularization for Group Sparsitycited by 8
  36. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  37. 2021Intraclass clustering: an implicit learning ability that regularizes DNNscited by 6
  38. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6

50 most cited of 121 All 121 (JSON)