mlsanityData to 7 Oct 2026

Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence

arXiv 2106.03743 ↗First posted 7 Jun 2021cs.LG cs.CV stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

We investigate the reasons for the performance degradation incurred with batch-independent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appearance of failure modes in the neural network's pre-activations: (i) layer normalization induces a collapse towards channel-wise constant functions; (ii) instance normalization induces a lack of variability in instance statistics, symptomatic of an alteration of the expressivity. To alleviate failure mode (i) without aggravating failure mode (ii), we introduce the technique "Proxy Normalization" that normalizes post-activations using a proxy distribution. When combined with layer normalization or group normalization, this batch-independent normalization emulates batch normalization's behavior and consistently matches or exceeds its performance.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 65 of its 103 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

65 of 103 references.

References · 65

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  9. 2018Group Normalizationcited by 1,514
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2019Root Mean Square Layer Normalizationcited by 1,425
  12. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
38 more
  1. 2016Deep Networks with Stochastic Depthcited by 973
  2. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  3. 2017Self-Normalizing Neural Networkscited by 591
  4. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  5. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  6. 2018How Does Batch Normalization Help Optimization?cited by 355
  7. 2016A Learned Representation For Artistic Stylecited by 340
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2019Adversarial Examples Improve Image Recognitioncited by 283
  10. 2018cGANs with Projection Discriminatorcited by 253
  11. 2017Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesiscited by 215
  12. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  13. 2016Texture Networks: Feed-forward Synthesis of Textures and Stylized Imagescited by 173
  14. 2017Modulating early visual processing by languagecited by 167
  15. 2019Complexity of Linear Regions in Deep Networkscited by 166
  16. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  17. 2018Understanding Batch Normalizationcited by 157
  18. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 127
  19. 2017L2 Regularization versus Batch and Weight Normalizationcited by 121
  20. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  21. 2018Three Mechanisms of Weight Decay Regularizationcited by 112
  22. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  23. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  24. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  25. 2018Bayesian Uncertainty Estimation for Batch Normalized Deep Networkscited by 70
  26. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  27. 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
  28. 2019A Mean Field Theory of Batch Normalizationcited by 58
  29. 2018Towards Understanding Regularization in Batch Normalizationcited by 56
  30. 2019Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networkscited by 45
  31. 2020Evolving Normalization-Activation Layerscited by 45
  32. 2021Rethinking "Batch" in BatchNormcited by 44
  33. 2017Riemannian approach to batch normalizationcited by 35
  34. 2018Differentiable Learning-to-Normalize via Switchable Normalizationcited by 33
  35. 2019Intriguing properties of adversarial training at scalecited by 33
  36. 2019Online Normalization for Training Neural Networkscited by 29
  37. 2019Batch Normalization is a Cause of Adversarial Vulnerabilitycited by 28
  38. 2019Four Things Everyone Should Know to Improve Batch Normalizationcited by 28

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4; 2023: 4; 2024: 1.

12 papers.

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  3. 2022On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysiscited by 19
  4. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  5. 2024On the Nonlinearity of Layer Normalizationcited by 9
  6. 2021Self-Supervised Radio-Visual Representation Learning for 6G Sensingcited by 6
  7. 2022RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Networkcited by 6
  8. 2023Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networkscited by 5
  9. 2022NoMorelization: Building Normalizer-Free Models from a Sample's Perspectivecited by 2
  10. 2021NanoBatch Privacy: Enabling fast Differentially Private learning on the IPUcited by 0
  11. 2023Revising deep learning methods in parking lot occupancy detectioncited by 0
  12. 2023LayerAct: Advanced Activation Mechanism for Robust Inference of CNNscited by 0