mlsanityData to 14 Oct 2026

Theoretical Analysis of Auto Rate-Tuning by Batch Normalization

arXiv 1812.03981 ↗First posted 10 Dec 2018cs.LG stat.MLcited by 65

Not on the map. Why?

Abstract (arXiv)

Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates. It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, 0.3), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of T-1/2 in T iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates. A similar result with convergence rate T-1/4 is also shown for stochastic gradient descent.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 13; 2023: 12; 2024: 7; 2025: 10; 2026: 7.

65 papers.

Sort:

By area:

  1. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  2. 2022Vision Transformers provably learn spatial structurecited by 93
  3. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  4. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  5. 2025Transformers without Normalizationcited by 70
  6. 2022GradMax: Growing Neural Networks using Gradient Informationcited by 66
  7. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  8. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  9. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  10. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 56
  11. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  12. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
38 more
  1. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  2. 2024How to set AdamW's weight decay as you scale model and dataset sizecited by 43
  3. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  4. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  5. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 34
  6. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 29
  7. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  8. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  9. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
  10. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  11. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  12. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  13. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  14. 2022Rethinking Normalization Methods in Federated Learningcited by 19
  15. 2023TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalizationcited by 18
  16. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  17. 2023Latent State Models of Training Dynamicscited by 16
  18. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  19. 2024From Attention to Activation: Unravelling the Enigmas of Large Language Modelscited by 13
  20. 2025Stronger Normalization-Free Transformerscited by 13
  21. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  22. 2021UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Costcited by 11
  23. 2022UDC: Unified DNAS for Compressible TinyML Modelscited by 10
  24. 2022Amortized Proximal Optimizationcited by 9
  25. 2024On the Nonlinearity of Layer Normalizationcited by 9
  26. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  27. 2025What Can Grokking Teach Us About Learning Under Nonstationarity?cited by 8
  28. 2021State estimation with limited sensors -- A deep learning based approachcited by 7
  29. 2021Sandwich Batch Normalization: A Drop-In Replacement for Feature Distribution Heterogeneitycited by 7
  30. 2023Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformerscited by 7
  31. 2024Where Do Large Learning Rates Lead Us?cited by 6
  32. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  33. 2021Batch Normalization Preconditioning for Neural Network Trainingcited by 5
  34. 2021Accelerating Training of Batch Normalization: A Manifold Perspectivecited by 4
  35. 2022Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Thinkcited by 4
  36. 2023On the Weight Dynamics of Deep Normalized Networkscited by 4
  37. 2025A Minimalist Example of Edge-of-Stability and Progressive Sharpeningcited by 4
  38. 2023On the Training Instability of Shuffling SGD with Batch Normalizationcited by 3

50 most cited of 65 All 65 (JSON)