# Dynamical stability and chaos in artificial neural network trajectories along training

arXiv: https://arxiv.org/abs/2404.05782 · HTML: https://mlsanity.com/paper/2404.05782/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 59 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel](https://mlsanity.com/paper/2010.15110/) (2020)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [Large Scale Structure of Neural Network Loss Landscapes](https://mlsanity.com/paper/1906.04724/) (2019)
- [Stochastic Training is Not Necessary for Generalization](https://mlsanity.com/paper/2109.14119/) (2021)
- [Shaping the learning landscape in neural networks around wide flat minima](https://mlsanity.com/paper/1905.07833/) (2019)
- [Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function](https://mlsanity.com/paper/2002.06189/) (2020)
- [Do Reservoir Computers Work Best at the Edge of Chaos?](https://mlsanity.com/paper/2012.01409/) (2020)
- [Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions](https://mlsanity.com/paper/1901.02302/) (2019)
- [Acceleration via Fractal Learning Rate Schedules](https://mlsanity.com/paper/2103.01338/) (2021)
- [The Global Optimization Geometry of Shallow Linear Neural Networks](https://mlsanity.com/paper/1805.04938/) (2018)
- [Fusion of complex networks and randomized neural networks for texture analysis](https://mlsanity.com/paper/1806.09170/) (2018)
- [Characterizing Learning Dynamics of Deep Neural Networks via Complex Networks](https://mlsanity.com/paper/2110.02628/) (2021)
- [Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent](https://mlsanity.com/paper/2303.13093/) (2023)
- [Improving Deep Neural Network Random Initialization Through Neuronal Rewiring](https://mlsanity.com/paper/2207.08148/) (2022)
- [An effective theory of collective deep learning](https://mlsanity.com/paper/2310.12802/) (2023)
- [Loss Surface Modality of Feed-Forward Neural Network Architectures](https://mlsanity.com/paper/1905.10268/) (2019)
- [Correlations of network trajectories](https://mlsanity.com/paper/2206.03334/) (2022)
- [Quasi-universal scaling in mouse-brain neuronal activity stems from edge-of-instability critical dynamics](https://mlsanity.com/paper/2111.12067/) (2021)
- [Griffiths phases and the stretching of criticality in brain networks](https://mlsanity.com/paper/1308.6661/) (2013)
- [Optimal input representation in neural systems at the edge of chaos](https://mlsanity.com/paper/2107.05709/) (2021)
- [Deep Neural Networks as Complex Networks](https://mlsanity.com/paper/2209.05488/) (2022)

## Built on by (most cited first, 6 of 6)

- [Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes](https://mlsanity.com/paper/2406.05033/) (2024)
- [Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region](https://mlsanity.com/paper/2509.25351/) (2025)
- [Towards certification: A complete statistical validation pipeline for supervised learning in industry](https://mlsanity.com/paper/2411.02075/) (2024)
- [Leveraging chaotic transients in the training of artificial neural networks](https://mlsanity.com/paper/2506.08523/) (2025)
- [Temporal Complexity and Self-Organization in an Exponential Dense Associative Memory Model](https://mlsanity.com/paper/2601.11478/) (2026)
- [Scalar Representations of Neural Network Training Dynamics](https://mlsanity.com/paper/2606.30384/) (2026)

## Abstract (arXiv)

The process of training an artificial neural network involves iteratively adapting its parameters so as to minimize the error of the network's prediction, when confronted with a learning task. This iterative change can be naturally interpreted as a trajectory in network space -- a time series of networks -- and thus the training algorithm (e.g. gradient descent optimization of a suitable loss function) can be interpreted as a dynamical system in graph space. In order to illustrate this interpretation, here we study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network, and its evolution through learning a simple classification task. We systematically consider different ranges of the learning rate and explore both the dynamical and orbital stability of the resulting network trajectories, finding hints of regular and chaotic behavior depending on the learning rate regime. Our findings are put in contrast to common wisdom on convergence properties of neural networks and dynamical systems theory. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning