# Interplay between depth and width for interpolation in neural ODEs

arXiv: https://arxiv.org/abs/2401.09902 · HTML: https://mlsanity.com/paper/2401.09902/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [The Power of Depth for Feedforward Neural Networks](https://mlsanity.com/paper/1512.03965/) (2015)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Identity Matters in Deep Learning](https://mlsanity.com/paper/1611.04231/) (2016)
- [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018)
- [Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity](https://mlsanity.com/paper/1810.07770/) (2018)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [Multi-level Residual Networks from Dynamical Systems View](https://mlsanity.com/paper/1710.10348/) (2017)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [Large-time asymptotics in deep learning](https://mlsanity.com/paper/2008.02491/) (2020)
- [Neural ODE Control for Trajectory Approximation of Continuity Equation](https://mlsanity.com/paper/2205.09241/) (2022)
- [Control of neural transport for normalizing flows](https://mlsanity.com/paper/2307.07817/) (2023)
- [Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems](https://mlsanity.com/paper/2110.12393/) (2021)
- [Quasi-Equivalence of Width and Depth of Neural Networks](https://mlsanity.com/paper/2002.02515/) (2020)
- [Differential Equations for Continuous-Time Deep Learning](https://mlsanity.com/paper/2401.03965/) (2024)
- [Sparsity in long-time control of neural ODEs](https://mlsanity.com/paper/2102.13566/) (2021)
- [Cluster-based classification with neural ODEs via control](https://mlsanity.com/paper/2312.13807/) (2023)
- [Controllability and optimal control of the transport equation with a localized vector field](https://mlsanity.com/paper/1702.07272/) (2017)

## Built on by (most cited first, 9 of 9)

- [Constructive approximate transport maps with normalizing flows](https://mlsanity.com/paper/2412.19366/) (2024)
- [Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networks](https://mlsanity.com/paper/2409.06555/) (2024)
- [Representation and Regression Problems in Neural Networks: Relaxation, Generalization, and Numerics](https://mlsanity.com/paper/2412.01619/) (2024)
- [Exact Sequence Interpolation with Transformers](https://mlsanity.com/paper/2502.02270/) (2025)
- [Control, Optimal Transport and Neural Differential Equations in Supervised Learning](https://mlsanity.com/paper/2503.15105/) (2025)
- [Machine Learning and Control: Foundations, Advances, and Perspectives](https://mlsanity.com/paper/2510.03303/) (2025)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)
- [Constructive interpolation and generalization rates for neural ODEs: a control perspective](https://mlsanity.com/paper/2606.00469/) (2026)
- [Deep learning from a control vantage point](https://mlsanity.com/paper/2610.14385/) (2026)

## Abstract (arXiv)

Neural ordinary differential equations (neural ODEs) have emerged as a natural tool for supervised learning from a control perspective, yet a complete understanding of their optimal architecture remains elusive. In this work, we examine the interplay between their width p and number of layer transitions L (effectively the depth L+1). Specifically, we assess the model expressivity in terms of its capacity to interpolate either a finite dataset D comprising N pairs of points or two probability measures in ℝd within a Wasserstein error margin varepsilon>0. Our findings reveal a balancing trade-off between p and L, with L scaling as O(1+N/p) for dataset interpolation, and L=Oleft(1+(pvarepsilond)-1right) for measure interpolation. In the autonomous case, where L=0, a separate study is required, which we undertake focusing on dataset interpolation. We address the relaxed problem of varepsilon-approximate controllability and establish an error decay of varepsilon~ O(log(p)p-1/d). This decay rate is a consequence of applying a universal approximation theorem to a custom-built Lipschitz vector field that interpolates D. In the high-dimensional setting, we further demonstrate that p=O(N) neurons are likely sufficient to achieve exact control.