# Deep neural networks, generic universal interpolation, and controlled ODEs

arXiv: https://arxiv.org/abs/1908.07838 · HTML: https://mlsanity.com/paper/1908.07838/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2019.

## Built on by (most cited first, 28 of 28)

- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Transformers are Universal In-context Learners](https://mlsanity.com/paper/2408.01367/) (2024)
- [Do Residual Neural Networks discretize Neural Ordinary Differential Equations?](https://mlsanity.com/paper/2205.14612/) (2022)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)
- [Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function](https://mlsanity.com/paper/2107.08649/) (2021)
- [Applications of Signature Methods to Market Anomaly Detection](https://mlsanity.com/paper/2201.02441/) (2022)
- [Near-optimal control of dynamical systems with neural ordinary differential equations](https://mlsanity.com/paper/2206.11120/) (2022)
- [Gradient-free training of neural ODEs for system identification and control using ensemble Kalman inversion](https://mlsanity.com/paper/2307.07882/) (2023)
- [Pricing options on flow forwards by neural networks in Hilbert space](https://mlsanity.com/paper/2202.11606/) (2022)
- [Langevin dynamics based algorithm e-THvarepsilonO POULA for stochastic optimization problems with discontinuous stochastic gradient](https://mlsanity.com/paper/2210.13193/) (2022)
- [Convolutional Neural Networks as 2-D systems](https://mlsanity.com/paper/2303.03042/) (2023)
- [Universal Regular Conditional Distributions](https://mlsanity.com/paper/2105.07743/) (2021)
- [Transformers Can Solve Non-Linear and Non-Markovian Filtering Problems in Continuous Time For Conditionally Gaussian Signals](https://mlsanity.com/paper/2310.19603/) (2023)
- [Cluster-based classification with neural ODEs via control](https://mlsanity.com/paper/2312.13807/) (2023)
- [Randomized Signature Methods in Optimal Portfolio Selection](https://mlsanity.com/paper/2312.16448/) (2023)
- [Constructive approximate transport maps with normalizing flows](https://mlsanity.com/paper/2412.19366/) (2024)
- [On the effectiveness of Randomized Signatures as Reservoir for Learning Rough Dynamics](https://mlsanity.com/paper/2201.00384/) (2022)
- [A Minimal Control Family of Dynamical Systems for Universal Approximation](https://mlsanity.com/paper/2312.12903/) (2023)
- [Control Theoretic Approach to Fine-Tuning and Transfer Learning](https://mlsanity.com/paper/2404.11013/) (2024)
- [Neural networks in non-metric spaces](https://mlsanity.com/paper/2406.09310/) (2024)
- [Geometric Foundations of Tuning without Forgetting in Neural ODEs](https://mlsanity.com/paper/2509.03474/) (2025)
- [The Sherpa.ai Blind Vertical Federated Learning Paradigm to Minimize the Number of Communications](https://mlsanity.com/paper/2510.17901/) (2025)
- [Autonomous-Flow-Based Generation](https://mlsanity.com/paper/2511.09902/) (2025)
- [Convergence analysis of controlled particle systems arising in deep learning: from finite to infinite sample size](https://mlsanity.com/paper/2404.05185/) (2024)
- [Convergence, design and training of continuous-time dropout as a random batch method](https://mlsanity.com/paper/2510.13134/) (2025)
- [Neural Flow Operators can Approximate any Operator: Abstract Frameworks and Universal Approximations](https://mlsanity.com/paper/2605.22557/) (2026)
- [Constructive interpolation and generalization rates for neural ODEs: a control perspective](https://mlsanity.com/paper/2606.00469/) (2026)
- [Universal interpolation for deep residual self-attention networks](https://mlsanity.com/paper/2610.01981/) (2026)

## Abstract (arXiv)

A recent paradigm views deep neural networks as discretizations of certain controlled ordinary differential equations, sometimes called neural ordinary differential equations. We make use of this perspective to link expressiveness of deep networks to the notion of controllability of dynamical systems. Using this connection, we study an expressiveness property that we call universal interpolation, and show that it is generic in a certain sense. The universal interpolation property is slightly weaker than universal approximation, and disentangles supervised learning on finite training sets from generalization properties. We also show that universal interpolation holds for certain deep neural networks even if large numbers of parameters are left untrained, and are instead chosen randomly. This lends theoretical support to the observation that training with random initialization can be successful even when most parameters are largely unchanged through the training. Our results also explore what a minimal amount of trainable parameters in neural ordinary differential equations could be without giving up on expressiveness.