mlsanityData to 7 Oct 2026

Norm matters: efficient and accurate normalization schemes in deep networks

arXiv 1803.01814 ↗First posted 5 Mar 2018stat.ML cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weight-decay, as tools to decouple weights' norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used L2 batch-norm, using normalization in L1 and L^∞ spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 16; 2023: 10; 2024: 5; 2025: 4; 2026: 13.

62 papers.

Sort:

By area:

  1. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  2. 2021Large-Scale Differentially Private BERTcited by 117
  3. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  4. 2022Vision Transformers provably learn spatial structurecited by 92
  5. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  6. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  7. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  8. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  9. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  10. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  11. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  12. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
38 more
  1. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  2. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  3. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  4. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  5. 2022Robust Training of Neural Networks Using Scale Invariant Architecturescited by 29
  6. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  7. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  8. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  9. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  10. 2023TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalizationcited by 18
  11. 2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
  12. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  13. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  14. 2026Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimizationcited by 12
  15. 2021Enabling Binary Neural Network Training on the Edgecited by 11
  16. 2022Amortized Proximal Optimizationcited by 9
  17. 2024On the Nonlinearity of Layer Normalizationcited by 9
  18. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  19. 2023Data-Efficient Training of CNNs and Transformers with Coresets: A Stability Perspectivecited by 6
  20. 2023Conformer-Based Speech Recognition On Extreme Edge-Computing Devicescited by 6
  21. 2023On the Weight Dynamics of Deep Normalized Networkscited by 4
  22. 2024Revealing the Utilized Rank of Subspaces of Learning in Neural Networkscited by 4
  23. 2021FixNorm: Dissecting Weight Decay for Training Deep Neural Networkscited by 3
  24. 2026Muown Implicitly Performs Angular Step-size Decaycited by 3
  25. 2026Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectorscited by 3
  26. 2026Hyperball May Not Be a Free Lunchcited by 3
  27. 2022Noise Injection as a Probe of Deep Learning Dynamicscited by 2
  28. 2022Exploiting the Partly Scratch-off Lottery Ticket for Quantization-Aware Trainingcited by 2
  29. 2025How does the optimizer implicitly bias the model merging loss landscape?cited by 2
  30. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  31. 2026On the Nonlinearity of Learning Rate Scaling for LLM Trainingcited by 2
  32. 2026Effective Learning Rate Governs Loss Dynamics in Language Model Pretrainingcited by 2
  33. 2021Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalizationcited by 1
  34. 2022Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual Networkscited by 1
  35. 2023No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasetscited by 1
  36. 2024Scheduling Weight Transitions for Quantization-Aware Trainingcited by 1
  37. 2025Efficient Adaptation of Deep Neural Networks for Semantic Segmentation in Space Applicationscited by 1
  38. 2026Batch Normalization Amplifies Memorization and Privacy Riskscited by 1

50 most cited of 62 All 62 (JSON)