# Universal Approximation Property of Hamiltonian Deep Neural Networks

arXiv: https://arxiv.org/abs/2303.12147 · HTML: https://mlsanity.com/paper/2303.12147/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [Neural Network Approximation: Three Hidden Layers Are Enough](https://mlsanity.com/paper/2010.14075/) (2020)
- [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019)
- [Expressivity of Deep Neural Networks](https://mlsanity.com/paper/2007.04759/) (2020)
- [Learning Functions: When Is Deep Better Than Shallow](https://mlsanity.com/paper/1603.00988/) (2016)
- [Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory](https://mlsanity.com/paper/2007.06007/) (2020)
- [Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design](https://mlsanity.com/paper/2105.13205/) (2021)
- [Robust Classification using Contractive Hamiltonian Neural ODEs](https://mlsanity.com/paper/2203.11805/) (2022)
- [Dynamical systems' based neural networks](https://mlsanity.com/paper/2210.02373/) (2022)
- [Approximate Predictive Control Barrier Functions using Neural Networks: A Computationally Cheap and Permissive Safety Filter](https://mlsanity.com/paper/2211.15104/) (2022)
- [Physically Consistent Neural ODEs for Learning Multi-Physics Systems](https://mlsanity.com/paper/2211.06130/) (2022)
- [Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks](https://mlsanity.com/paper/2204.00187/) (2022)

## Built on by (most cited first, 2 of 2)

- [Symplectic Neural Networks Based on Dynamical Systems](https://mlsanity.com/paper/2408.09821/) (2024)
- [Universal approximation property of ODENet and ResNet with a single activation function](https://mlsanity.com/paper/2410.16709/) (2024)

## Abstract (arXiv)

This paper investigates the universal approximation capabilities of Hamiltonian Deep Neural Networks (HDNNs) that arise from the discretization of Hamiltonian Neural Ordinary Differential Equations. Recently, it has been shown that HDNNs enjoy, by design, non-vanishing gradients, which provide numerical stability during training. However, although HDNNs have demonstrated state-of-the-art performance in several applications, a comprehensive study to quantify their expressivity is missing. In this regard, we provide a universal approximation theorem for HDNNs and prove that a portion of the flow of HDNNs can approximate arbitrary well any continuous function over a compact domain. This result provides a solid theoretical foundation for the practical use of HDNNs.