mlsanityData to 7 Oct 2026

Towards Understanding Regularization in Batch Normalization

arXiv 1809.00846 ↗First posted 4 Sep 2018cs.LG cs.CV cs.SYcited by 56

Not on the map. Why?

Abstract (arXiv)

Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us understand the impacts of BN in three aspects. First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization. Second, learning dynamics of BN and the regularization show that training converged with large maximum and effective learning rate. Third, generalization of BN is explored by using statistical mechanics. Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 21; 2022: 12; 2023: 13; 2024: 4; 2025: 4; 2026: 2.

56 papers.

Sort:

  1. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  2. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  3. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
  4. 2021Dive into Deep Learningcited by 111
  5. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  6. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  7. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  8. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  9. 2023Model Agnostic Sample Reweighting for Out-of-Distribution Learningcited by 43
  10. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 26
  11. 2022Rethinking Normalization Methods in Federated Learningcited by 19
  12. 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
38 more
  1. 2022Set Norm and Equivariant Skip Connections: Putting the Deep in Deep Setscited by 17
  2. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  3. 2023The Expressive Power of Tuning Only the Normalization Layerscited by 13
  4. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  5. 2022Understanding the Failure of Batch Normalization for Transformers in NLPcited by 12
  6. 2024A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and methodcited by 12
  7. 2023Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptationcited by 11
  8. 2021Dynamic Token Normalization Improves Vision Transformerscited by 10
  9. 2023Towards Understanding Neural Collapse: The Effects of Batch Normalization and Weight Decaycited by 9
  10. 2024On the Nonlinearity of Layer Normalizationcited by 9
  11. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  12. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  13. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6
  14. 2023Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Boundscited by 6
  15. 2021Multirate Training of Neural Networkscited by 5
  16. 2021Self-Contrastive Learning: Single-viewed Supervised Contrastive Framework using Sub-networkcited by 5
  17. 2021Batch Normalization Preconditioning for Neural Network Trainingcited by 5
  18. 2021Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalizationcited by 4
  19. 2021Revisiting Batch Norm Initializationcited by 3
  20. 2022Batch Normalization Is Blind to the First and Second Derivatives of the Losscited by 3
  21. 2023Representation Learning Dynamics of Self-Supervised Modelscited by 3
  22. 2021Three approaches to facilitate DNN generalization to objects in out-of-distribution orientations and illuminationscited by 2
  23. 2022Understanding and Improving Group Normalizationcited by 2
  24. 2022NoMorelization: Building Normalizer-Free Models from a Sample's Perspectivecited by 2
  25. 2022Less is More: A Lightweight and Robust Neural Architecture for Discourse Parsingcited by 2
  26. 2023Picking Up Quantization Steps for Compressed Image Classificationcited by 2
  27. 2023Spectral Batch Normalization: Normalization in the Frequency Domaincited by 2
  28. 2024CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalizationcited by 2
  29. 2024Towards the Spectral bias Alleviation by Normalizations in Coordinate Networkscited by 2
  30. 2021A Riemannian Mean Field Formulation for Two-layer Neural Networks with Batch Normalizationcited by 1
  31. 2021UniFed: A Unified Framework for Federated Learning on Non-IID Image Featurescited by 1
  32. 2021A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalizationcited by 1
  33. 2025Simplifying Multi-Task Architectures Through Task-Specific Normalizationcited by 1
  34. 2021Paradigm Shift in Language Modeling: Revisiting CNN for Modeling Sanskrit Originated Bengali and Hindi Languagecited by 0
  35. 2022Noise Regularizes Over-parameterized Rank One Matrix Recovery, Provablycited by 0
  36. 2022Theoretical Insight into Batch Normalization: Data Dependant Auto-Tuning of Regularization Ratecited by 0
  37. 2023An Empirical Analysis of the Shift and Scale Parameters in BatchNormcited by 0
  38. 2023Outlier-robust neural network training: variation regularization meets trimmed loss to prevent functional breakdowncited by 0

50 most cited of 56 All 56 (JSON)