# A Dynamical Systems Perspective on the Analysis of Neural Networks arXiv: https://arxiv.org/abs/2507.05164 ยท HTML: https://mlsanity.com/paper/2507.05164/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 143 references that are in the map. - First posted on arXiv 7 Jul 2025. ## Builds on (most cited first, 30 of 36) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018) - [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017) - [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018) - [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021) - [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018) - [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018) - [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019) - [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017) - [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022) - [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022) - [The Heavy-Tail Phenomenon in SGD](https://mlsanity.com/paper/2006.04740/) (2020) - [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018) - [Understanding the unstable convergence of gradient descent](https://mlsanity.com/paper/2204.01050/) (2022) - [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018) - [Convergence rates for the stochastic gradient descent method for non-convex objective functions](https://mlsanity.com/paper/1904.01517/) (2019) - [On Linear Stability of SGD and Input-Smoothness of Neural Networks](https://mlsanity.com/paper/2105.13462/) (2021) - [Generalized Teacher Forcing for Learning Chaotic Dynamics](https://mlsanity.com/paper/2306.04406/) (2023) - [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021) - [Approximation Capabilities of Neural ODEs and Invertible Residual Networks](https://mlsanity.com/paper/1907.12998/) (2019) - [Edge of Stochastic Stability: Revisiting the Edge of Stability for SGD](https://mlsanity.com/paper/2412.20553/) (2024) - [Neural Delay Differential Equations](https://mlsanity.com/paper/2102.10801/) (2021) - [Consensus-Based Optimization on the Sphere: Convergence to Global Minimizers and Machine Learning](https://mlsanity.com/paper/2001.11988/) (2020) - [Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learning](https://mlsanity.com/paper/2407.20209/) (2024) - [Dynamical Systems on Networks: A Tutorial](https://mlsanity.com/paper/1403.7663/) (2014) - [Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent](https://mlsanity.com/paper/2207.05705/) (2022) - [Embedding Capabilities of Neural ODEs](https://mlsanity.com/paper/2308.01213/) (2023) ## Built on by (most cited first, 3 of 3) - [Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect](https://mlsanity.com/paper/2603.28591/) (2026) - [Metacognition Should Be the Scientific Framework for Bounded and Effective Self-Governance in Generative AI](https://mlsanity.com/paper/2605.23981/) (2026) - [Barron Optimal Transport I: Generative Modeling](https://mlsanity.com/paper/2610.10875/) (2026) ## Abstract (arXiv) In this chapter, we utilize dynamical systems to analyze several aspects of machine learning algorithms. As an expository contribution we demonstrate how to re-formulate a wide variety of challenges from deep neural networks, (stochastic) gradient descent, and related topics into dynamical statements. We also tackle three concrete challenges. First, we consider the process of information propagation through a neural network, i.e., we study the input-output map for different architectures. We explain the universal embedding property for augmented neural ODEs representing arbitrary functions of given regularity, the classification of multilayer perceptrons and neural ODEs in terms of suitable function classes, and the memory-dependence in neural delay equations. Second, we consider the training aspect of neural networks dynamically. We describe a dynamical systems perspective on gradient descent and study stability for overdetermined problems. We then extend this analysis to the overparameterized setting and describe the edge of stability phenomenon, also in the context of possible explanations for implicit bias. For stochastic gradient descent, we present stability results for the overparameterized setting via Lyapunov exponents of interpolation solutions. Third, we explain several results regarding mean-field limits of neural networks. We describe a result that extends existing techniques to heterogeneous neural networks involving graph limits via digraph measures. This shows how large classes of neural networks naturally fall within the framework of Kuramoto-type models on graphs and their large-graph limits. Finally, we point out that similar strategies to use dynamics to study explainable and reliable AI can also be applied to settings such as generative models or fundamental issues in gradient training methods, such as backpropagation or vanishing/exploding gradients.