mlsanityData to 7 Oct 2026

Dynamical systems' based neural networks

arXiv 2210.02373 ↗First posted 5 Oct 2022cs.LG cs.NA math.DScited by 10

Not on the map. Why?

Abstract (arXiv)

Neural networks have gained much interest because of their effectiveness in many applications. However, their mathematical properties are generally not well understood. If there is some underlying geometric structure inherent to the data or to the function to approximate, it is often desirable to take this into account in the design of the neural network. In this work, we start with a non-autonomous ODE and build neural networks using a suitable, structure-preserving, numerical time-discretisation. The structure of the neural network is then inferred from the properties of the ODE vector field. Besides injecting more structure into the network architectures, this modelling procedure allows a better theoretical understanding of their behaviour. We present two universal approximation results and demonstrate how to impose some particular properties on the neural networks. A particular focus is on 1-Lipschitz architectures including layers that are not 1-Lipschitz. These networks are expressive and robust against adversarial attacks, as shown for the CIFAR-10 and CIFAR-100 datasets.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 42 of its 77 references that are in the map. First posted on arXiv 5 Oct 2022.

Builds on

42 of 77 references.

References · 42

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  6. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  7. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2019Deep Equilibrium Modelscited by 603
  11. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  12. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
30 more
  1. 2018Sorting out Lipschitz function approximationcited by 211
  2. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  3. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  4. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  5. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  6. 2020Dissecting Neural ODEscited by 150
  7. 2019Symplectic Recurrent Neural Networkscited by 148
  8. 2019Orthogonal Convolutional Neural Networkscited by 122
  9. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  10. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  11. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  12. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  13. 2021The Modern Mathematics of Deep Learningcited by 63
  14. 2020Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspectivecited by 60
  15. 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
  16. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  17. 2021Neural ODE control for classification, approximation and transportcited by 56
  18. 2020Convolutional Proximal Neural Networks and Plug-and-Play Algorithmscited by 36
  19. 2020Universal Approximation Property of Neural Ordinary Differential Equationscited by 32
  20. 2020Structure preserving deep learningcited by 30
  21. 2020PDE-based Group Equivariant Convolutional Neural Networkscited by 28
  22. 2021Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Designcited by 18
  23. 2016Optimization on Submanifolds of Convolution Kernels in CNNscited by 15
  24. 2022Learning Hamiltonians of constrained mechanical systemscited by 14
  25. 2022Robust Classification using Contractive Hamiltonian Neural ODEscited by 13
  26. 2020Adversarial Feature Desensitizationcited by 12
  27. 2021Locally-symplectic neural networks for learning volume-preserving dynamicscited by 9
  28. 2020Differentiable Implicit Layerscited by 6
  29. 2021Optimization on manifolds: A symplectic approachcited by 5
  30. 2021Better Training using Weight-Constrained Stochastic Dynamicscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 1.

10 papers.

  1. 2023Designing Stable Neural Networks using Convex Analysis and ODEscited by 10
  2. 2025Approximation theory for 1-Lipschitz ResNetscited by 7
  3. 2024Symplectic Neural Networks Based on Dynamical Systemscited by 4
  4. 2024Symplectic Neural Flows for Modeling and Discoverycited by 4
  5. 2023Universal Approximation Property of Hamiltonian Deep Neural Networkscited by 2
  6. 2023Resilient Graph Neural Networks: A Coupled Dynamical Systems Approachcited by 1
  7. 2024Systematic construction of continuous-time neural networks for linear dynamical systemscited by 1
  8. 2025Improving regional weather forecasts with neural interpolationcited by 0
  9. 2025Stable neural networks and connections to continuous dynamical systemscited by 0
  10. 20261-Lipschitz Neural Networks on Hadamard Manifoldscited by 0