mlsanityData to 7 Oct 2026

How Does Batch Normalization Help Optimization?

arXiv 1805.11604 ↗First posted 29 May 2018stat.ML cs.LG cs.NEcited by 355

Not on the map. Why?

Abstract (arXiv)

Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the change of the layers' input distributions during training to reduce the so-called "internal covariate shift". In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother. This smoothness induces a more predictive and stable behavior of the gradients, allowing for faster training.

Key facts. Cited by 355 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 88; 2022: 85; 2023: 69; 2024: 38; 2025: 46; 2026: 29.

355 papers.

Sort:

By area:

  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  2. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  5. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 342
  6. 2022How Do Vision Transformers Work?cited by 305
  7. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  8. 2022Temporal Efficient Training of Spiking Neural Network via Gradient Re-weightingcited by 192
  9. 2021The Values Encoded in Machine Learning Researchcited by 172
  10. 2023Understanding plasticity in neural networkscited by 170
  11. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
  12. 2021The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalizationcited by 131
38 more
  1. 2021Dive into Deep Learningcited by 111
  2. 2022Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectivescited by 98
  3. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  4. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  5. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  6. 2021Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Datacited by 69
  7. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  8. 2025Transformers without Normalizationcited by 69
  9. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  10. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  11. 2022SPD domain-specific batch normalization to crack interpretable unsupervised domain adaptation in EEGcited by 53
  12. 2021A Survey on Green Deep Learningcited by 52
  13. 2021Consensus Control for Decentralized Deep Learningcited by 49
  14. 2021Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learningcited by 49
  15. 2023Boosting Adversarial Transferability by Achieving Flat Local Maximacited by 48
  16. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 47
  17. 2022Architecture Matters in Continual Learningcited by 46
  18. 2021Fast Certified Robust Training with Short Warmupcited by 45
  19. 2021Improving Contrastive Learning by Visualizing Feature Transformationcited by 44
  20. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  21. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  22. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  23. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
  24. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
  25. 2023Deep networks for system identification: a Surveycited by 35
  26. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  27. 2022Generative Domain Adaptation for Face Anti-Spoofingcited by 33
  28. 2021Ps and Qs: Quantization-aware pruning for efficient low latency neural network inferencecited by 31
  29. 2023Why Batch Normalization Damage Federated Learning on Non-IID Data?cited by 29
  30. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  31. 2021SimROD: A Simple Adaptation Method for Robust Object Detectioncited by 27
  32. 2022Multi-Source Unsupervised Domain Adaptation via Pseudo Target Domaincited by 27
  33. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  34. 2022Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspacescited by 25
  35. 2023Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNNcited by 25
  36. 2023ScaleLong: Towards More Stable Training of Diffusion Model via Scaling Network Long Skip Connectioncited by 25
  37. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  38. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 23

50 most cited of 355 All 355 (JSON)