mlsanityData to 7 Oct 2026

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

arXiv 1502.03167 ↗First posted 11 Feb 2015cs.LGcited by 6,979

Not on the map. Why?

Cited by 79 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of human raters.

Key facts. Cited by 6,979 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Feb 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 2; 2018: 3; 2019: 2; 2020: 5; 2021: 1891; 2022: 1571; 2023: 1348; 2024: 1019; 2025: 724; 2026: 414.

6,979 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  10. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
38 more
  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  4. 2021Domain Generalization: A Surveycited by 760
  5. 2021Going deeper with Image Transformerscited by 759
  6. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  7. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  9. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  10. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  11. 2022Simple Baselines for Image Restorationcited by 701
  12. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  13. 2022High Fidelity Neural Audio Compressioncited by 673
  14. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  15. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  16. 2021Domain Generalization with MixStylecited by 563
  17. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  18. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  19. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  20. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  21. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 508
  22. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  23. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  24. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  25. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 462
  26. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  27. 2021A White Paper on Neural Network Quantizationcited by 453
  28. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  29. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  30. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  31. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  32. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  33. 2021Dynamic Neural Networks: A Surveycited by 405
  34. 2021A Survey of Transformerscited by 399
  35. 2022Multimodal Learning with Transformers: A Surveycited by 399
  36. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  37. 2021Early Convolutions Help Transformers See Bettercited by 395
  38. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391

50 most cited of 6,979 All 6,979 (JSON)