mlsanityData to 7 Oct 2026

Asymptotic Analysis of Deep Residual Networks

arXiv 2212.08199 ↗First posted 15 Dec 2022cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We investigate the asymptotic properties of deep Residual networks (ResNets) as the number of layers increases. We first show the existence of scaling regimes for trained weights markedly different from those implicitly assumed in the neural ODE literature. We study the convergence of the hidden state dynamics in these scaling regimes, showing that one may obtain an ODE, a stochastic differential equation (SDE) or neither of these. In particular, our findings point to the existence of a diffusive regime in which the deep network limit is described by a class of stochastic differential equations (SDEs). Finally, we derive the corresponding scaling limits for the backpropagation dynamics.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 19 of its 33 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

19 of 33 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2016Wide Residual Networkscited by 3,057
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2019Augmented Neural ODEscited by 396
  7. 2017Stable Architectures for Deep Neural Networkscited by 340
  8. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  9. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  10. 2018The Singular Values of Convolutional Layerscited by 95
  11. 2016Understanding Deep Convolutional Networkscited by 76
  12. 2018A Mean-Field Optimal Control Formulation of Deep Learningcited by 67
7 more
  1. 2020A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depthcited by 56
  2. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  3. 2019Infinitely deep neural networks as diffusion processescited by 21
  4. 2020ResNet After All? Neural ODEs and Their Numerical Solutioncited by 20
  5. 2019Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networkscited by 16
  6. 2021Scaling Properties of Deep Residual Networkscited by 16
  7. 2019Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connectionscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 2.

3 papers.

  1. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  2. 2026Clustering in Deep Stochastic Transformerscited by 5
  3. 2026Correlated initialization of deep residual networkscited by 1