mlsanityData to 7 Oct 2026

On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay

arXiv 2106.15739 ↗First posted 29 Jun 2021cs.LG stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

Training neural networks with batch normalization and weight decay has become a common practice in recent years. In this work, we show that their combined use may result in a surprising periodic behavior of optimization dynamics: the training process regularly exhibits destabilizations that, however, do not lead to complete divergence but cause a new period of training. We rigorously investigate the mechanism underlying the discovered periodic behavior from both empirical and theoretical points of view and analyze the conditions in which it occurs in practice. We also demonstrate that periodic behavior can be regarded as a generalization of two previously opposing perspectives on training with batch normalization and weight decay, namely the equilibrium presumption and the instability presumption.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 29 Jun 2021.

Builds on

22 of 28 references.

References · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Group Normalizationcited by 1,514
  5. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  6. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  7. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  9. 2018How Does Batch Normalization Help Optimization?cited by 355
  10. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  11. 2018Understanding Batch Normalizationcited by 157
  12. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
10 more
  1. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  2. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  3. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  4. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  5. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  6. 2019A Mean Field Theory of Batch Normalizationcited by 58
  7. 2017Lower Bounds for Finding Stationary Points II: First-Order Methodscited by 55
  8. 2018Fix your classifier: the marginal value of training the last weight layercited by 43
  9. 2017Riemannian approach to batch normalizationcited by 35
  10. 2019Online Normalization for Training Neural Networkscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 5; 2024: 3; 2025: 2; 2026: 2.

20 papers.

Sort:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  3. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  4. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  5. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  6. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  7. 2021Neural Network Weights Do Not Converge to Stationary Points: An Invariant Measure Perspectivecited by 15
  8. 2022On the generalization of learning algorithms that do not convergecited by 9
  9. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  10. 2023Exploring Weight Balancing on Long-Tailed Recognition Problemcited by 7
  11. 2024Where Do Large Learning Rates Lead Us?cited by 6
  12. 2023Adversarial Attacks and Defenses in Machine Learning-Powered Networks: A Contemporary Surveycited by 5
8 more
  1. 2024Future Directions in the Theory of Graph Machine Learningcited by 5
  2. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  3. 2023On the Training Instability of Shuffling SGD with Batch Normalizationcited by 3
  4. 2023Large Learning Rates Improve Generalization: But How Large Are We Talking About?cited by 1
  5. 2026Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decaycited by 1
  6. 2022Batchless Normalization: How to Normalize Activations Across Instances with Minimal Memory Requirementscited by 0
  7. 2025Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?cited by 0
  8. 2026When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decaycited by 0