# Generalization bounds for neural ordinary differential equations and deep residual networks

arXiv: https://arxiv.org/abs/2305.06648 · HTML: https://mlsanity.com/paper/2305.06648/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 11 May 2023.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [A Variational Perspective on Diffusion-Based Generative Models and Score Matching](https://mlsanity.com/paper/2106.02808/) (2021)
- [Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit](https://mlsanity.com/paper/1905.09883/) (2019)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Learning Differential Equations that are Easy to Solve](https://mlsanity.com/paper/2007.04504/) (2020)
- [Integrating Expert ODEs into Neural ODEs: Pharmacology and Disease Progression](https://mlsanity.com/paper/2106.02875/) (2021)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Continuous-in-Depth Neural Networks](https://mlsanity.com/paper/2008.02389/) (2020)
- [LEADS: Learning Dynamical Systems that Generalize Across Environments](https://mlsanity.com/paper/2106.04546/) (2021)
- [Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks](https://mlsanity.com/paper/2104.05097/) (2021)
- [Do Residual Neural Networks discretize Neural Ordinary Differential Equations?](https://mlsanity.com/paper/2205.14612/) (2022)
- [Framing RNN as a kernel method: A neural ODE approach](https://mlsanity.com/paper/2106.01202/) (2021)
- [Robust pricing and hedging via neural SDEs](https://mlsanity.com/paper/2007.04154/) (2020)
- [Scaling ResNets in the Large-depth Regime](https://mlsanity.com/paper/2206.06929/) (2022)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)
- [Stateful ODE-Nets using Basis Function Expansions](https://mlsanity.com/paper/2106.10820/) (2021)

## Built on by (most cited first, 21 of 21)

- [Large Language Models as Markov Chains](https://mlsanity.com/paper/2410.02724/) (2024)
- [Implicit regularization of deep residual networks towards neural ODEs](https://mlsanity.com/paper/2309.01213/) (2023)
- [Gaussian Interpolation Flows](https://mlsanity.com/paper/2311.11475/) (2023)
- [Topological Neural Networks go Persistent, Equivariant, and Continuous](https://mlsanity.com/paper/2406.03164/) (2024)
- [On the Residual Scaling of Looped Transformers: Stability and Transferability](https://mlsanity.com/paper/2606.18524/) (2026)
- [Feedback Favors the Generalization of Neural ODEs](https://mlsanity.com/paper/2410.10253/) (2024)
- [Rademacher Complexity of Neural ODEs via Chen-Fliess Series](https://mlsanity.com/paper/2401.16655/) (2024)
- [A Generalization Bound for a Family of Implicit Networks](https://mlsanity.com/paper/2410.07427/) (2024)
- [On the Generalization and Approximation Capacities of Neural Controlled Differential Equations](https://mlsanity.com/paper/2305.16791/) (2023)
- [Bridging Neural ODE and ResNet: A Formal Error Bound for Safety Verification](https://mlsanity.com/paper/2506.03227/) (2025)
- [Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints](https://mlsanity.com/paper/2405.20278/) (2024)
- [Statistical physics analysis of graph neural networks: Approaching optimality in the contextual stochastic block model](https://mlsanity.com/paper/2503.01361/) (2025)
- [Generalization Bound for a General Class of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2508.18920/) (2025)
- [On the Generalization Behavior of Deep Residual Networks From a Dynamical System Perspective](https://mlsanity.com/paper/2602.20921/) (2026)
- [Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers](https://mlsanity.com/paper/2605.07772/) (2026)
- [PAC bounds of continuous Linear Parameter-Varying systems related to neural ODEs](https://mlsanity.com/paper/2307.03630/) (2023)
- [A finite-sample generalization bound for stable LPV systems](https://mlsanity.com/paper/2405.10054/) (2024)
- [Sharper Risk Bound for Multi-Task Learning with Multi-Graph Dependent Data](https://mlsanity.com/paper/2502.18167/) (2025)
- [Dynamical Implicit Neural Representations](https://mlsanity.com/paper/2511.21787/) (2025)
- [Upper Generalization Bounds for Neural Oscillators](https://mlsanity.com/paper/2603.09742/) (2026)
- [Constructive interpolation and generalization rates for neural ODEs: a control perspective](https://mlsanity.com/paper/2606.00469/) (2026)

## Abstract (arXiv)

Neural ordinary differential equations (neural ODEs) are a popular family of continuous-depth deep learning models. In this work, we consider a large family of parameterized ODEs with continuous-in-time parameters, which include time-dependent neural ODEs. We derive a generalization bound for this class by a Lipschitz-based argument. By leveraging the analogy between neural ODEs and deep residual networks, our approach yields in particular a generalization bound for a class of deep residual networks. The bound involves the magnitude of the difference between successive weight matrices. We illustrate numerically how this quantity affects the generalization capability of neural networks.