mlsanityData to 7 Oct 2026

On skip connections and normalisation layers in deep optimisation

arXiv 2210.05371 ↗First posted 10 Oct 2022cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce a general theoretical framework, designed for the study of gradient optimisation of deep neural networks, that encompasses ubiquitous architecture choices including batch normalisation, weight normalisation and skip connections. Our framework determines the curvature and regularity properties of multilayer loss landscapes in terms of their constituent layers, thereby elucidating the roles played by normalisation layers and skip connections in globalising these properties. We then demonstrate the utility of this framework in two respects. First, we give the only proof of which we are aware that a class of deep neural networks can be trained using gradient descent to global optima even when such optima only exist at infinity, as is the case for the cross-entropy cost. Second, we identify a novel causal mechanism by which skip connections accelerate training, which we verify predictively with ResNets on MNIST, CIFAR10, CIFAR100 and ImageNet.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 41 of its 52 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

41 of 52 references.

References · 41

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  10. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  11. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
29 more
  1. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  2. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  3. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  4. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  5. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  6. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  7. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  8. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  9. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  10. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  11. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  12. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  13. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  14. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  15. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  16. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  17. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  18. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  19. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  20. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  21. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  22. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  23. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  24. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  25. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  26. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  27. 2022Re-parameterizing Your Optimizers rather than Architecturescited by 35
  28. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  29. 2021MAE-DET: Revisiting Maximum Entropy Principle in Zero-Shot NAS for Efficient Object Detectioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 3.

7 papers.

  1. 2025Cutting the Skip: Training Residual-Free Transformerscited by 7
  2. 2023On progressive sharpening, flat minima and generalisationcited by 6
  3. 2025Enhancing Transformers Through Conditioned Embedded Tokenscited by 3
  4. 2025SineProject: Machine Unlearning for Stable Vision Language Alignmentcited by 3
  5. 2026Spectral Conditioning of Attention Improves Transformer Performancecited by 3
  6. 2026Preconditioned Attention: Enhancing Efficiency in Transformerscited by 0
  7. 2026Conditioned Initialization for Attentioncited by 0