# Infinitely deep neural networks as diffusion processes

arXiv: https://arxiv.org/abs/1905.11065 · HTML: https://mlsanity.com/paper/1905.11065/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 21 of 21)

- [Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations](https://mlsanity.com/paper/2102.06559/) (2021)
- [SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equations](https://mlsanity.com/paper/2502.02472/) (2025)
- [On the infinite-depth limit of finite-width neural networks](https://mlsanity.com/paper/2210.00688/) (2022)
- [Width and Depth Limits Commute in Residual Networks](https://mlsanity.com/paper/2302.00453/) (2023)
- [Scaling ResNets in the Large-depth Regime](https://mlsanity.com/paper/2206.06929/) (2022)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)
- [Clustering in pure-attention hardmax transformers and its role in sentiment analysis](https://mlsanity.com/paper/2407.01602/) (2024)
- [Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models](https://mlsanity.com/paper/2310.10767/) (2023)
- [Fractional SDE-Net: Generation of Time Series Data with Long-term Memory](https://mlsanity.com/paper/2201.05974/) (2022)
- [Clustering in Deep Stochastic Transformers](https://mlsanity.com/paper/2601.21942/) (2026)
- [Continuous-Depth Neural Models for Dynamic Graph Prediction](https://mlsanity.com/paper/2106.11581/) (2021)
- [Rademacher Complexity of Neural ODEs via Chen-Fliess Series](https://mlsanity.com/paper/2401.16655/) (2024)
- [Learning Stochastic Optimal Policies via Gradient Descent](https://mlsanity.com/paper/2106.03780/) (2021)
- [Nonlinear controllability and function representation by neural stochastic differential equations](https://mlsanity.com/paper/2212.00896/) (2022)
- [Asymptotic Analysis of Deep Residual Networks](https://mlsanity.com/paper/2212.08199/) (2022)
- [Commutative Width and Depth Scaling in Deep Neural Networks](https://mlsanity.com/paper/2310.01683/) (2023)
- [Neural Options Pricing](https://mlsanity.com/paper/2105.13320/) (2021)
- [Neural Langevin Dynamics: towards interpretable Neural Stochastic Differential Equations](https://mlsanity.com/paper/2211.09537/) (2022)
- [Neural Mean-Field Games: Extending Mean-Field Game Theory with Neural Stochastic Differential Equations](https://mlsanity.com/paper/2504.13228/) (2025)
- [Feature Learning Dynamics in Infinite-Depth Neural Networks](https://mlsanity.com/paper/2512.21075/) (2025)
- [Collective Kernel EFT for Pre-activation ResNets](https://mlsanity.com/paper/2604.15742/) (2026)

## Abstract (arXiv)

When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on restrictive families of functions including constant functions. We consider parameter distributions that shrink as the number of layers increases in order to recover well-behaved stochastic processes in the limit of infinite depth. This leads to set forth a link between infinitely deep residual networks and solutions to stochastic differential equations, i.e. diffusion processes. We show that these limiting processes do not suffer from the aforementioned issues and investigate their properties.