# Asymptotic Analysis of Deep Residual Networks

arXiv: https://arxiv.org/abs/2212.08199 · HTML: https://mlsanity.com/paper/2212.08199/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [Understanding Deep Convolutional Networks](https://mlsanity.com/paper/1601.04920/) (2016)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth](https://mlsanity.com/paper/2003.05508/) (2020)
- [Do Residual Neural Networks discretize Neural Ordinary Differential Equations?](https://mlsanity.com/paper/2205.14612/) (2022)
- [Infinitely deep neural networks as diffusion processes](https://mlsanity.com/paper/1905.11065/) (2019)
- [ResNet After All? Neural ODEs and Their Numerical Solution](https://mlsanity.com/paper/2007.15386/) (2020)
- [Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks](https://mlsanity.com/paper/1910.02934/) (2019)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)
- [Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections](https://mlsanity.com/paper/1904.05263/) (2019)

## Built on by (most cited first, 3 of 3)

- [Neural signature kernels as infinite-width-depth-limits of controlled ResNets](https://mlsanity.com/paper/2303.17671/) (2023)
- [Clustering in Deep Stochastic Transformers](https://mlsanity.com/paper/2601.21942/) (2026)
- [Correlated initialization of deep residual networks](https://mlsanity.com/paper/2609.03589/) (2026)

## Abstract (arXiv)

We investigate the asymptotic properties of deep Residual networks (ResNets) as the number of layers increases. We first show the existence of scaling regimes for trained weights markedly different from those implicitly assumed in the neural ODE literature. We study the convergence of the hidden state dynamics in these scaling regimes, showing that one may obtain an ODE, a stochastic differential equation (SDE) or neither of these. In particular, our findings point to the existence of a diffusive regime in which the deep network limit is described by a class of stochastic differential equations (SDEs). Finally, we derive the corresponding scaling limits for the backpropagation dynamics.