# Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design

arXiv: https://arxiv.org/abs/2105.13205 · HTML: https://mlsanity.com/paper/2105.13205/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Training robust neural networks using Lipschitz bounds](https://mlsanity.com/paper/2005.02929/) (2020)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Orthogonal Recurrent Neural Networks with Scaled Cayley Transform](https://mlsanity.com/paper/1707.09520/) (2017)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization](https://mlsanity.com/paper/1803.09327/) (2018)
- [Improving training of deep neural networks via Singular Value Bounding](https://mlsanity.com/paper/1611.06013/) (2016)
- [An Ode to an ODE](https://mlsanity.com/paper/2006.11421/) (2020)
- [Distributed Linear-Quadratic Control with Graph Neural Networks](https://mlsanity.com/paper/2103.08417/) (2021)
- [The Microsoft 2016 Conversational Speech Recognition System](https://mlsanity.com/paper/1609.03528/) (2016)
- [Communication Topology Co-Design in Graph Recurrent Neural Network Based Distributed Control](https://mlsanity.com/paper/2104.13868/) (2021)
- [Sparsity Invariance for Convex Design of Distributed Controllers](https://mlsanity.com/paper/1906.06777/) (2019)
- [A unified framework for Hamiltonian deep neural networks](https://mlsanity.com/paper/2104.13166/) (2021)
- [On Separable Quadratic Lyapunov Functions for Convex Design of Distributed Controllers](https://mlsanity.com/paper/1903.04096/) (2019)

## Built on by (most cited first, 18 of 18)

- [Robust Classification using Contractive Hamiltonian Neural ODEs](https://mlsanity.com/paper/2203.11805/) (2022)
- [Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach](https://mlsanity.com/paper/2112.09046/) (2021)
- [Thermodynamics of learning physical phenomena](https://mlsanity.com/paper/2207.12749/) (2022)
- [Dynamical systems' based neural networks](https://mlsanity.com/paper/2210.02373/) (2022)
- [Locally-symplectic neural networks for learning volume-preserving dynamics](https://mlsanity.com/paper/2109.09151/) (2021)
- [Differentiable Simulations for Enhanced Sampling of Rare Events](https://mlsanity.com/paper/2301.03480/) (2023)
- [Unconstrained Parametrization of Dissipative and Contracting Neural Ordinary Differential Equations](https://mlsanity.com/paper/2304.02976/) (2023)
- [Improving Long-Range Interactions in Graph Neural Simulators via Hamiltonian Dynamics](https://mlsanity.com/paper/2511.08185/) (2025)
- [Random orthogonal additive filters: a solution to the vanishing/exploding gradient of deep neural networks](https://mlsanity.com/paper/2210.01245/) (2022)
- [Symplectic Neural Networks Based on Dynamical Systems](https://mlsanity.com/paper/2408.09821/) (2024)
- [Symplectic Neural Flows for Modeling and Discovery](https://mlsanity.com/paper/2412.16787/) (2024)
- [Physically Consistent Neural ODEs for Learning Multi-Physics Systems](https://mlsanity.com/paper/2211.06130/) (2022)
- [Universal Approximation Property of Hamiltonian Deep Neural Networks](https://mlsanity.com/paper/2303.12147/) (2023)
- [Metriplector: From Field Theory to Neural Architecture](https://mlsanity.com/paper/2603.29496/) (2026)
- [On Dissipativity of Cross-Entropy Loss in Training ResNets](https://mlsanity.com/paper/2405.19013/) (2024)
- [Deep Network Trainability via Persistent Subspace Orthogonality](https://mlsanity.com/paper/2508.02882/) (2025)
- [Robust Convolution Neural ODEs via Contractivity-promoting regularization](https://mlsanity.com/paper/2508.11432/) (2025)
- [Stable neural networks and connections to continuous dynamical systems](https://mlsanity.com/paper/2510.22299/) (2025)

## Abstract (arXiv)

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretization of continuous-time Hamiltonian systems and include several existing DNN architectures based on ordinary differential equations. Our main result is that a broad set of H-DNNs ensures non-vanishing gradients by design for an arbitrary network depth. This is obtained by proving that, using a semi-implicit Euler discretization scheme, the backward sensitivity matrices involved in gradient computations are symplectic. We also provide an upper-bound to the magnitude of sensitivity matrices and show that exploding gradients can be controlled through regularization. Finally, we enable distributed implementations of backward and forward propagation algorithms in H-DNNs by characterizing appropriate sparsity constraints on the weight matrices. The good performance of H-DNNs is demonstrated on benchmark classification problems, including image classification with the MNIST dataset.