mlsanityData to 7 Oct 2026

On the Weight Dynamics of Deep Normalized Networks

arXiv 2306.00700 ↗First posted 1 Jun 2023cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent studies have shown that high disparities in effective learning rates (ELRs) across layers in deep neural networks can negatively affect trainability. We formalize how these disparities evolve over time by modeling weight dynamics (evolution of expected gradient and weight norms) of networks with normalization layers, predicting the evolution of layer-wise ELR ratios. We prove that when training with any constant learning rate, ELR ratios converge to 1, despite initial gradient explosion. We identify a ``critical learning rate" beyond which ELR disparities widen, which only depends on current ELRs. To validate our findings, we devise a hyper-parameter-free warm-up method that successfully minimizes ELR spread quickly in theory and practice. Our experiments link ELR spread with trainability, a relationship that is most evident in very deep networks with significant gradient magnitude excursions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

23 of 37 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  6. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  10. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  11. 2018How Does Batch Normalization Help Optimization?cited by 355
  12. 2020Query-Key Normalization for Transformerscited by 266
11 more
  1. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  2. 2018Understanding Batch Normalizationcited by 157
  3. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  4. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  5. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  6. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  7. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  8. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  9. 2019A Mean Field Theory of Batch Normalizationcited by 58
  10. 2020On the distance between two neural networks and the stability of learningcited by 46
  11. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  2. 2025Layer-wise Update Aggregation with Recycling for Communication-Efficient Federated Learningcited by 1
  3. 2025ResNets Are Deeper Than You Thinkcited by 0
  4. 2025Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?cited by 0