The Disharmony between BN and ReLU Causes Gradient Explosion, but is Offset by the Correlation between Activations
Not on the map. Why?
Abstract (arXiv)
Deep neural networks, which employ batch normalization and ReLU-like activation functions, suffer from instability in the early stages of training due to the high gradient induced by temporal gradient explosion. In this study, we analyze the occurrence and mitigation of gradient explosion both theoretically and empirically, and discover that the correlation between activations plays a key role in preventing the gradient explosion from persisting throughout the training. Finally, based on our observations, we propose an improved adaptive learning rate algorithm to effectively control the training instability.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 23 Apr 2023.
Builds on
37 of 56 references.
References · 37
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2016Layer Normalizationcited by 5,579
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
25 more
- 2018Group Normalizationcited by 1,514
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
- 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
- 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
- 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
- 2017Self-Normalizing Neural Networkscited by 591
- 2017Large Batch Training of Convolutional Networkscited by 529
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
- 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
- 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
- 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
- 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
- 2020The Early Phase of Neural Network Trainingcited by 114
- 2019A Mean Field Theory of Batch Normalizationcited by 58
- 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
- 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 39
- 2016Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descentcited by 26
- 2020Extrapolation for Large-batch Training in Deep Learningcited by 25
- 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
- 2019Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theoremcited by 15
- 2018CosmoFlow: Using Deep Learning to Learn the Universe at Scalecited by 14
- 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
- 2022Pseudo-label Guided Cross-video Pixel Contrast for Robotic Surgical Scene Segmentation with Limited Annotationscited by 4
- 2020Improving Layer-wise Adaptive Rate Methods using Trust Ratio Clippingcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.