mlsanityData to 14 Oct 2026

Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers

arXiv 2203.08120 ↗First posted 15 Mar 2022cs.LG stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

Training very deep neural networks is still an extremely challenging task. The common solution is to use shortcut connections and normalization layers, which are both crucial ingredients in the popular ResNet architecture. However, there is strong evidence to suggest that ResNets behave more like ensembles of shallower networks than truly deep ones. Recently, it was shown that deep vanilla networks (i.e. networks without normalization layers or shortcut connections) can be trained as fast as ResNets by applying certain transformations to their activation functions. However, this method (called Deep Kernel Shaping) isn't fully compatible with ReLUs, and produces networks that overfit significantly more than ResNets on ImageNet. In this work, we rectify this situation by developing a new type of transformation that is fully compatible with a variant of ReLUs -- Leaky ReLUs. We show in experiments that our method, which introduces negligible extra computational cost, achieves validation accuracies with deep vanilla networks that are competitive with ResNets (of the same width/depth), and significantly higher than those obtained with the Edge of Chaos (EOC) method. And unlike with EOC, the validation accuracies we obtain do not get worse with depth.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 36 of its 61 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

36 of 61 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  8. 2016Wide Residual Networkscited by 3,070
  9. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
24 more
  1. 2017Self-Normalizing Neural Networkscited by 598
  2. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  3. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  4. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  5. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  6. 2015Highway Networkscited by 260
  7. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  8. 2016Deep Information Propagationcited by 195
  9. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  10. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  11. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  12. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  13. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 117
  14. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  15. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
  16. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  17. 2019A Mean Field Theory of Batch Normalizationcited by 58
  18. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
  19. 2020Stable ResNetcited by 49
  20. 2020Deep Isometric Learning for Visual Recognitioncited by 39
  21. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  22. 2014Avoiding pathologies in very deep networkscited by 33
  23. 2021On the validity of kernel approximations for orthogonally-initialized neural networkscited by 4
  24. 2020Bidirectionally Self-Normalizing Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 11; 2024: 6; 2025: 3; 2026: 1.

28 papers.

Sort:

By area:

  1. 2023Understanding plasticity in neural networkscited by 174
  2. 2022Pre-training via Denoising for Molecular Property Predictioncited by 124
  3. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 86
  4. 2024Normalization and effective learning rates in reinforcement learningcited by 58
  5. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  6. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  7. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  8. 2023Simplifying Transformer Blockscited by 37
  9. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  10. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  11. 2024Curvature-Informed SGD via General Purpose Lie-Group Preconditionerscited by 16
  12. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 15
16 more
  1. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  2. 2023Principled Weight Initialisation for Input-Convex Neural Networkscited by 12
  3. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  4. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  5. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  6. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  7. 2025Cutting the Skip: Training Residual-Free Transformerscited by 8
  8. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  9. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  10. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
  11. 2022AutoInit: Automatic Initialization via Jacobian Tuningcited by 3
  12. 2022RepMode: Learning to Re-parameterize Diverse Experts for Subcellular Structure Predictioncited by 3
  13. 2024Generalized Probabilistic Attention Mechanism in Transformerscited by 2
  14. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  15. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  16. 2025ResNets Are Deeper Than You Thinkcited by 0