# Approximation Capabilities of Neural ODEs and Invertible Residual Networks

arXiv: https://arxiv.org/abs/1907.12998 · HTML: https://mlsanity.com/paper/1907.12998/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jul 2019.

## Built on by (most cited first, 30 of 56)

- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformers](https://mlsanity.com/paper/2107.13163/) (2021)
- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations](https://mlsanity.com/paper/2206.08311/) (2022)
- [Learning with 3D rotations, a hitchhiker's guide to SO(3)](https://mlsanity.com/paper/2404.11735/) (2024)
- [Topology-Preserving Shape Reconstruction and Registration via Neural Diffeomorphic Flow](https://mlsanity.com/paper/2203.08652/) (2022)
- [Momentum Residual Neural Networks](https://mlsanity.com/paper/2102.07870/) (2021)
- [A Deep Learning approach to Reduced Order Modelling of Parameter Dependent Partial Differential Equations](https://mlsanity.com/paper/2103.06183/) (2021)
- [Implicit Normalizing Flows](https://mlsanity.com/paper/2103.09527/) (2021)
- [On Numerical Integration in Neural Ordinary Differential Equations](https://mlsanity.com/paper/2206.07335/) (2022)
- [Implicit regularization of deep residual networks towards neural ODEs](https://mlsanity.com/paper/2309.01213/) (2023)
- [Policy Analysis using Synthetic Controls in Continuous-Time](https://mlsanity.com/paper/2102.01577/) (2021)
- [On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows](https://mlsanity.com/paper/2402.06578/) (2024)
- [Efficient Image Generation with Variadic Attention Heads](https://mlsanity.com/paper/2211.05770/) (2022)
- [Ab-initio study of interacting fermions at finite temperature with neural canonical transformation](https://mlsanity.com/paper/2105.08644/) (2021)
- [On the expressivity of bi-Lipschitz normalizing flows](https://mlsanity.com/paper/2107.07232/) (2021)
- [Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraints](https://mlsanity.com/paper/2306.09739/) (2023)
- [When are dynamical systems learned from time series data statistically accurate?](https://mlsanity.com/paper/2411.06311/) (2024)
- [Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation](https://mlsanity.com/paper/2305.18460/) (2023)
- [Density estimation on low-dimensional manifolds: an inflation-deflation approach](https://mlsanity.com/paper/2105.12152/) (2021)
- [Equivariant Finite Normalizing Flows](https://mlsanity.com/paper/2110.08649/) (2021)
- [Dynamical systems' based neural networks](https://mlsanity.com/paper/2210.02373/) (2022)
- [Jacobian Determinant of Normalizing Flows](https://mlsanity.com/paper/2102.06539/) (2021)
- [Feedback Favors the Generalization of Neural ODEs](https://mlsanity.com/paper/2410.10253/) (2024)
- [Efficient Regression-Based Training of Normalizing Flows for Boltzmann Generators](https://mlsanity.com/paper/2506.01158/) (2025)
- [Universal Approximation for Log-concave Distributions using Well-conditioned Normalizing Flows](https://mlsanity.com/paper/2107.02951/) (2021)
- [Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE](https://mlsanity.com/paper/2303.05323/) (2023)
- [Whitening Convergence Rate of Coupling-based Normalizing Flows](https://mlsanity.com/paper/2210.14032/) (2022)
- [Partially Stochastic Infinitely Deep Bayesian Neural Networks](https://mlsanity.com/paper/2402.03495/) (2024)
- [Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics](https://mlsanity.com/paper/2104.13962/) (2021)

## Abstract (arXiv)

Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous invertible function remained unresolved. Here, we show that both of these models are limited in their approximation capabilities. We then prove that any homeomorphism on a p-dimensional Euclidean space can be approximated by a Neural ODE operating on a 2p-dimensional Euclidean space, and a similar result for i-ResNets. We conclude by showing that capping a Neural ODE or an i-ResNet with a single linear layer is sufficient to turn the model into a universal approximator for non-invertible continuous functions.