Batch Normalization Is Blind to the First and Second Derivatives of the Loss
Not on the map. Why?
Abstract (arXiv)
In this paper, we prove the effects of the BN operation on the back-propagation of the first and second derivatives of the loss. When we do the Taylor series expansion of the loss function, we prove that the BN operation will block the influence of the first-order term and most influence of the second-order term of the loss. We also find that such a problem is caused by the standardization phase of the BN operation. Experimental results have verified our theoretical conclusions, and we have found that the BN operation significantly affects feature representations in specific tasks, where losses of different samples share similar analytic formulas.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 30 May 2022.
Builds on
34 of 46 references.
References · 34
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2016Densely Connected Convolutional Networkscited by 6,720
- 2016Layer Normalizationcited by 5,669
- 2016Improved Techniques for Training GANscited by 2,970
- 2016Density estimation using Real NVPcited by 2,042
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
- 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
- 2018Group Normalizationcited by 1,524
- 2014NICE: Non-linear Independent Components Estimationcited by 1,058
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
22 more
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2021ResNet strikes back: An improved training procedure in timmcited by 368
- 2018How Does Batch Normalization Help Optimization?cited by 357
- 2019Adversarial Examples Improve Image Recognitioncited by 286
- 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
- 2019Understanding and Improving Layer Normalizationcited by 217
- 2014Recurrent Neural Network Regularizationcited by 189
- 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128
- 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
- 2018On the importance of single directions for generalizationcited by 99
- 2018Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principlecited by 70
- 2019A Mean Field Theory of Batch Normalizationcited by 58
- 2018Towards Understanding Regularization in Batch Normalizationcited by 56
- 2019Efficient Adversarial Training with Transferable Adversarial Examplescited by 56
- 2021Discovering and Explaining the Representation Bottleneck of DNNscited by 42
- 2018Understanding the Disharmony between Dropout and Batch Normalization by Variance Shiftcited by 37
- 2019Batch Normalization is a Cause of Adversarial Vulnerabilitycited by 29
- 2020Why Spectral Normalization Stabilizes GANs: Analysis and Improvementscited by 21
- 2020Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGDcited by 6
- 2021Trap of Feature Diversity in the Learning of MLPscited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.