mlsanityData to 7 Oct 2026

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

arXiv 2009.12836 ↗First posted 27 Sep 2020cs.LG cs.CV stat.MLcited by 90

Not on the map. Why?

Abstract (arXiv)

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization representation recovery. In doing so, we provide insight for designing new normalization technique. Finally, we discuss the current progress in understanding normalization methods, and provide a comprehensive review of the applications of normalization for particular tasks, in which it can effectively solve the key issues.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 15; 2023: 13; 2024: 21; 2025: 20; 2026: 14.

90 papers.

Sort:

By area:

  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  2. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  3. 2022The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalystscited by 95
  4. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  5. 2025Transformers without Normalizationcited by 69
  6. 2023Domain Generalization for Medical Image Analysis: A Reviewcited by 47
  7. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  8. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  9. 2023Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangementcited by 29
  10. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  11. 2021Task-Specific Normalization for Continual Learning of Blind Image Quality Modelscited by 18
  12. 2024xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memoriescited by 18
38 more
  1. 2023New Insights for the Stability-Plasticity Dilemma in Online Continual Learningcited by 16
  2. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  3. 2024Self-Contrastive Forward-Forward Algorithmcited by 15
  4. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  5. 2024Deep learning for precipitation nowcasting: A survey from the perspective of time series forecastingcited by 14
  6. 2022Deep transfer learning for image classification: a surveycited by 13
  7. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  8. 2025Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Spacecited by 12
  9. 2024Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillationcited by 11
  10. 2023Diffusion-based generation of Histopathological Whole Slide Images at a Gigapixel scalecited by 9
  11. 2024On the Nonlinearity of Layer Normalizationcited by 9
  12. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
  13. 2024Weight Conditioning for Smooth Optimization of Neural Networkscited by 9
  14. 2022Delving into the Estimation Shift of Batch Normalization in a Networkcited by 7
  15. 2024Federated Unsupervised Domain Generalization using Global and Local Alignment of Gradientscited by 7
  16. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  17. 2022Unified Normalization for Accelerating and Stabilizing Transformerscited by 6
  18. 2023Improving Normalization with the James-Stein Estimatorcited by 6
  19. 2024Swin3D++: Effective Multi-Source Pretraining for 3D Indoor Scene Understandingcited by 6
  20. 2023Function-Space Optimality of Neural Architectures with Multivariate Nonlinearitiescited by 5
  21. 2024Sign Gradient Descent-based Neuronal Dynamics: ANN-to-SNN Conversion Beyond ReLU Networkcited by 5
  22. 2022Optimal Activation Functions for the Random Features Regression Modelcited by 4
  23. 2024Unconditional stability of a recurrent neural circuit implementing divisive normalizationcited by 4
  24. 2024Deep Learning for Spatio-Temporal Fusion in Land Surface Temperature Estimation: A Comprehensive Survey, Experimental Analysis, and Future Trendscited by 4
  25. 2025Tiling artifacts and trade-offs of feature normalization in the segmentation of large biological imagescited by 4
  26. 2025AIS-LLM: A Unified Framework for Maritime Trajectory Prediction, Anomaly Detection, and Collision Risk Assessment with Explainable Forecastingcited by 4
  27. 2022Deep learning fluid flow reconstruction around arbitrary two-dimensional objects from sparse sensors using conformal mappingscited by 3
  28. 2025Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generationcited by 3
  29. 2025PGT-I: Scaling Spatiotemporal GNNs with Memory-Efficient Distributed Trainingcited by 3
  30. 2026UBiGTLoc: A Unified BiLSTM-Graph Transformer Localization Framework for IoT Sensor Networkscited by 3
  31. 2021Exploiting Invariance in Training Deep Neural Networkscited by 2
  32. 2023Efficient ConvBN Blocks for Transfer Learning and Beyondcited by 2
  33. 2023Convex Geometry of ReLU-layers, Injectivity on the Ball and Local Reconstructioncited by 2
  34. 2023Extended Deep Adaptive Input Normalization for Preprocessing Time Series Data for Neural Networkscited by 2
  35. 2024TSNet:A Two-stage Network for Image Dehazing with Multi-scale Fusion and Adaptive Learningcited by 2
  36. 2024TikGuard: A Deep Learning Transformer-Based Solution for Detecting Unsuitable TikTok Content for Kidscited by 2
  37. 2024Graph Neural Networks Need Cluster-Normalize-Activate Modulescited by 2
  38. 2025WeedVision: Multi-Stage Growth and Classification of Weeds using DETR and RetinaNet for Precision Agriculturecited by 2

50 most cited of 90 All 90 (JSON)