# Neural ODEs as the Deep Limit of ResNets with constant weights

arXiv: https://arxiv.org/abs/1906.12183 · HTML: https://mlsanity.com/paper/1906.12183/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2019.

## Built on by (most cited first, 10 of 10)

- [The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram](https://mlsanity.com/paper/2509.10167/) (2025)
- [Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODE](https://mlsanity.com/paper/2402.02362/) (2024)
- [Learning Subgrid-scale Models with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2212.09967/) (2022)
- [Learning via nonlinear conjugate gradients and depth-varying neural ODEs](https://mlsanity.com/paper/2202.05766/) (2022)
- [Approximation properties of Residual Neural Networks for Kolmogorov PDEs](https://mlsanity.com/paper/2111.00215/) (2021)
- [The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks](https://mlsanity.com/paper/2405.14126/) (2024)
- [On the Convergence and Size Transferability of Continuous-depth Graph Neural Networks](https://mlsanity.com/paper/2510.03923/) (2025)
- [Deep limits and cut-off phenomena for neural networks](https://mlsanity.com/paper/2104.10727/) (2021)
- [Dimensionality Collapse: Optimal Measurement Selection for Low-Error Infinite-Horizon Forecasting](https://mlsanity.com/paper/2303.15407/) (2023)
- [A multilevel approach to accelerate the training of Transformers](https://mlsanity.com/paper/2504.18590/) (2025)

## Abstract (arXiv)

In this paper we prove that, in the deep limit, the stochastic gradient descent on a ResNet type deep neural network, where each layer shares the same weight matrix, converges to the stochastic gradient descent for a Neural ODE and that the corresponding value/loss functions converge. Our result gives, in the context of minimization by stochastic gradient descent, a theoretical foundation for considering Neural ODEs as the deep limit of ResNets. Our proof is based on certain decay estimates for associated Fokker-Planck equations.