A Dynamical Systems Perspective on the Analysis of Neural Networks
Not on the map. Why?
Abstract (arXiv)
In this chapter, we utilize dynamical systems to analyze several aspects of machine learning algorithms. As an expository contribution we demonstrate how to re-formulate a wide variety of challenges from deep neural networks, (stochastic) gradient descent, and related topics into dynamical statements. We also tackle three concrete challenges. First, we consider the process of information propagation through a neural network, i.e., we study the input-output map for different architectures. We explain the universal embedding property for augmented neural ODEs representing arbitrary functions of given regularity, the classification of multilayer perceptrons and neural ODEs in terms of suitable function classes, and the memory-dependence in neural delay equations. Second, we consider the training aspect of neural networks dynamically. We describe a dynamical systems perspective on gradient descent and study stability for overdetermined problems. We then extend this analysis to the overparameterized setting and describe the edge of stability phenomenon, also in the context of possible explanations for implicit bias. For stochastic gradient descent, we present stability results for the overparameterized setting via Lyapunov exponents of interpolation solutions. Third, we explain several results regarding mean-field limits of neural networks. We describe a result that extends existing techniques to heterogeneous neural networks involving graph limits via digraph measures. This shows how large classes of neural networks naturally fall within the framework of Kuramoto-type models on graphs and their large-graph limits. Finally, we point out that similar strategies to use dynamics to study explainable and reliable AI can also be applied to settings such as generative models or fundamental issues in gradient training methods, such as backpropagation or vanishing/exploding gradients.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 36 of its 143 references that are in the map. First posted on arXiv 7 Jul 2025.
Builds on
36 of 143 references.
References · 36
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2016Densely Connected Convolutional Networkscited by 6,720
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 420
- 2019Augmented Neural ODEscited by 397
- 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
24 more
- 2022On Neural Differential Equationscited by 268
- 2022A Survey on Generative Diffusion Modelcited by 240
- 2020The Heavy-Tail Phenomenon in SGDcited by 124
- 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
- 2022Understanding the unstable convergence of gradient descentcited by 86
- 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 79
- 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
- 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
- 2023Generalized Teacher Forcing for Learning Chaotic Dynamicscited by 58
- 2021Neural ODE control for classification, approximation and transportcited by 57
- 2019Approximation Capabilities of Neural ODEs and Invertible Residual Networkscited by 56
- 2024Edge of Stochastic Stability: Revisiting the Edge of Stability for SGDcited by 30
- 2021Neural Delay Differential Equationscited by 26
- 2020Consensus-Based Optimization on the Sphere: Convergence to Global Minimizers and Machine Learningcited by 12
- 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8
- 2014Dynamical Systems on Networks: A Tutorialcited by 7
- 2022Conservative SPDEs as fluctuating mean field limits of stochastic gradient descentcited by 3
- 2023Embedding Capabilities of Neural ODEscited by 3
- 2024On the Convergence of Gradient Descent for Large Learning Ratescited by 3
- 2013Small-world networks of Kuramoto oscillatorscited by 2
- 2021Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theoremcited by 2
- 2024Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functionscited by 2
- 2025Associative Memory and Generative Diffusion in the Zero-noise Limitcited by 2
- 2025The Influence of the Memory Capacity of Neural DDEs on the Universal Approximation Propertycited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.