# Dynamical systems' based neural networks

arXiv: https://arxiv.org/abs/2210.02373 · HTML: https://mlsanity.com/paper/2210.02373/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 77 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 30 of 42)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [The Reversible Residual Network: Backpropagation Without Storing Activations](https://mlsanity.com/paper/1707.04585/) (2017)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks](https://mlsanity.com/paper/1806.05393/) (2018)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [Orthogonal Convolutional Neural Networks](https://mlsanity.com/paper/1911.12207/) (2019)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Deep Learning via Dynamical Systems: An Approximation Perspective](https://mlsanity.com/paper/1912.10382/) (2019)
- [The Modern Mathematics of Deep Learning](https://mlsanity.com/paper/2105.04026/) (2021)
- [Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2002.06262/) (2020)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [Convolutional Proximal Neural Networks and Plug-and-Play Algorithms](https://mlsanity.com/paper/2011.02281/) (2020)

## Built on by (most cited first, 10 of 10)

- [Designing Stable Neural Networks using Convex Analysis and ODEs](https://mlsanity.com/paper/2306.17332/) (2023)
- [Approximation theory for 1-Lipschitz ResNets](https://mlsanity.com/paper/2505.12003/) (2025)
- [Symplectic Neural Networks Based on Dynamical Systems](https://mlsanity.com/paper/2408.09821/) (2024)
- [Symplectic Neural Flows for Modeling and Discovery](https://mlsanity.com/paper/2412.16787/) (2024)
- [Universal Approximation Property of Hamiltonian Deep Neural Networks](https://mlsanity.com/paper/2303.12147/) (2023)
- [Resilient Graph Neural Networks: A Coupled Dynamical Systems Approach](https://mlsanity.com/paper/2311.06942/) (2023)
- [Systematic construction of continuous-time neural networks for linear dynamical systems](https://mlsanity.com/paper/2403.16215/) (2024)
- [Improving regional weather forecasts with neural interpolation](https://mlsanity.com/paper/2505.12040/) (2025)
- [Stable neural networks and connections to continuous dynamical systems](https://mlsanity.com/paper/2510.22299/) (2025)
- [1-Lipschitz Neural Networks on Hadamard Manifolds](https://mlsanity.com/paper/2607.19335/) (2026)

## Abstract (arXiv)

Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a non-autonomous ODE and build neural networks using a suitable, structure-preserving, numerical time-discretisation. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modelling procedure allows a better theoretical understanding of their behaviour. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 and CIFAR-100 datasets.