mlsanityData to 14 Oct 2026

Characterizing signal propagation to close the performance gap in unnormalized ResNets

arXiv 2101.08692 ↗First posted 21 Jan 2021cs.LG cs.CV stat.MLcited by 88

Not on the map. Why?

Abstract (arXiv)

Batch Normalization is a key component in almost all state-of-the-art image classifiers, but it also introduces practical challenges: it breaks the independence between training examples within a batch, can incur compute and memory overhead, and often results in unexpected bugs. Building on recent theoretical analyses of deep ResNets at initialization, we propose a simple set of analysis tools to characterize signal propagation on the forward pass, and leverage these tools to design highly performant ResNets without activation normalization layers. Crucial to our success is an adapted version of the recently proposed Weight Standardization. Our analysis tools show how this technique preserves the signal in networks with ReLU or Swish activation functions by ensuring that the per-channel activation means do not grow with depth. Across a range of FLOP budgets, our networks attain performance competitive with the state-of-the-art EfficientNets on ImageNet.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 62 of its 81 references that are in the map. First posted on arXiv 21 Jan 2021.

Builds on

62 of 81 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2016Densely Connected Convolutional Networkscited by 6,720
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2016Layer Normalizationcited by 5,669
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
38 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  2. 2017Squeeze-and-Excitation Networkscited by 4,789
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2016Wide Residual Networkscited by 3,070
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  11. 2017Improved Training of Wasserstein GANscited by 2,235
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  13. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  14. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  15. 2018Group Normalizationcited by 1,524
  16. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,512
  17. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  18. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  19. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  20. 2016Deep Networks with Stochastic Depthcited by 979
  21. 2020Designing Network Design Spacescited by 960
  22. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  23. 2017Self-Normalizing Neural Networkscited by 598
  24. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  25. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  26. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 403
  27. 2018How Does Batch Normalization Help Optimization?cited by 357
  28. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  29. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  30. 2015Highway Networkscited by 260
  31. 2019Fixing the train-test resolution discrepancycited by 209
  32. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  33. 2018Understanding Batch Normalizationcited by 159
  34. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  35. 2015All you need is a good initcited by 124
  36. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  37. 2020Normalization Techniques in Training DNNs: Methodology, Analysis and Applicationcited by 90
  38. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 25; 2023: 19; 2024: 10; 2025: 10; 2026: 5.

88 papers.

Sort:

By area:

  1. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  2. 2021Going deeper with Image Transformerscited by 764
  3. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  4. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  5. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  7. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
  8. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  9. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  10. 2025Transformers without Normalizationcited by 70
  11. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 64
  12. 2022Fast Finite Width Neural Tangent Kernelcited by 62
38 more
  1. 2022MogaNet: Multi-order Gated Aggregation Networkcited by 59
  2. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
  3. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 52
  4. 2022Online Training Through Time for Spiking Neural Networkscited by 49
  5. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  6. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 41
  7. 2021A Loss Curvature Perspective on Training Instability in Deep Learningcited by 40
  8. 2022Continual Normalization: Rethinking Batch Normalization for Online Continual Learningcited by 40
  9. 2023Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networkscited by 37
  10. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  11. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 33
  12. 2022Adapting the Linearised Laplace Model Evidence for Modern Deep Learningcited by 29
  13. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  14. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 25
  15. 2022Removing Batch Normalization Boosts Adversarial Trainingcited by 24
  16. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  17. 2022Sampling-free Inference for Ab-Initio Potential Energy Surface Networkscited by 19
  18. 2023Vision-Language Dataset Distillationcited by 19
  19. 2022Scaling ResNets in the Large-depth Regimecited by 18
  20. 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
  21. 2024Federated Unlearning with Gradient Descent and Conflict Mitigationcited by 17
  22. 2024FedMef: Towards Memory-efficient Federated Dynamic Pruningcited by 14
  23. 2023Binarized Neural Machine Translationcited by 13
  24. 2025Stronger Normalization-Free Transformerscited by 13
  25. 2021Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependencecited by 12
  26. 2023Principled Weight Initialisation for Input-Convex Neural Networkscited by 12
  27. 2023FedWon: Triumphing Multi-domain Federated Learning Without Normalizationcited by 11
  28. 2022Regularising for invariance to data augmentation improves supervised learningcited by 9
  29. 2022Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Promptscited by 9
  30. 2023SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networkscited by 8
  31. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  32. 2022Differentially Private Image Classification from Featurescited by 7
  33. 2023How to Use Dropout Correctly on Residual Networks with Batch Normalizationcited by 7
  34. 2024A Method to Facilitate Membership Inference Attacks in Deep Learning Modelscited by 7
  35. 2021Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Trainingcited by 6
  36. 2021A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statisticscited by 6
  37. 2023Improving Normalization with the James-Stein Estimatorcited by 6
  38. 2024Adversarial Training: A Surveycited by 6

50 most cited of 88 All 88 (JSON)