# Stochastic Training of Residual Networks: a Differential Equation Viewpoint

arXiv: https://arxiv.org/abs/1812.00174 · HTML: https://mlsanity.com/paper/1812.00174/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2018.

## Built on by (most cited first, 12 of 12)

- [Sinkformers: Transformers with Doubly Stochastic Attention](https://mlsanity.com/paper/2110.11773/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Momentum Residual Neural Networks](https://mlsanity.com/paper/2102.07870/) (2021)
- [Do Residual Neural Networks discretize Neural Ordinary Differential Equations?](https://mlsanity.com/paper/2205.14612/) (2022)
- [Neural Delay Differential Equations](https://mlsanity.com/paper/2102.10801/) (2021)
- [Understanding Adversarial Robustness of Vision Transformers via Cauchy Problem](https://mlsanity.com/paper/2208.00906/) (2022)
- [PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusion](https://mlsanity.com/paper/2305.15835/) (2023)
- [Neural Network Layer Algebra: A Framework to Measure Capacity and Compression in Deep Learning](https://mlsanity.com/paper/2107.01081/) (2021)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Neural Delay Differential Equations: System Reconstruction and Image Classification](https://mlsanity.com/paper/2304.05310/) (2023)
- [Convergence, design and training of continuous-time dropout as a random batch method](https://mlsanity.com/paper/2510.13134/) (2025)
- [EulerLoRA: Rank-Driven Jump Dynamics for Calibrated Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2608.01142/) (2026)

## Abstract (arXiv)

During the last few years, significant attention has been paid to the stochastic training of artificial neural networks, which is known as an effective regularization approach that helps improve the generalization capability of trained models. In this work, the method of modified equations is applied to show that the residual network and its variants with noise injection can be regarded as weak approximations of stochastic differential equations. Such observations enable us to bridge the stochastic training processes with the optimal control of backward Kolmogorov's equations. This not only offers a novel perspective on the effects of regularization from the loss landscape viewpoint but also sheds light on the design of more reliable and efficient stochastic training strategies. As an example, we propose a new way to utilize Bernoulli dropout within the plain residual network architecture and conduct experiments on a real-world image classification task to substantiate our theoretical findings.