mlsanityData to 14 Oct 2026

Dynamical stability and chaos in artificial neural network trajectories along training

arXiv 2404.05782 ↗First posted 8 Apr 2024cs.LG cond-mat.dis-nn nlin.CDcited by 6

Not on the map. Why?

Abstract (arXiv)

The process of training an artificial neural network involves iteratively adapting its parameters so as to minimize the error of the network's prediction, when confronted with a learning task. This iterative change can be naturally interpreted as a trajectory in network space -- a time series of networks -- and thus the training algorithm (e.g. gradient descent optimization of a suitable loss function) can be interpreted as a dynamical system in graph space. In order to illustrate this interpretation, here we study the dynamical properties of this process by analyzing through this lens the network trajectories of a shallow neural network, and its evolution through learning a simple classification task. We systematically consider different ranges of the learning rate and explore both the dynamical and orbital stability of the resulting network trajectories, finding hints of regular and chaotic behavior depending on the learning rate regime. Our findings are put in contrast to common wisdom on convergence properties of neural networks and dynamical systems theory. This work also contributes to the cross-fertilization of ideas between dynamical systems theory, network theory and machine learning

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 28 of its 59 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

28 of 59 references.

References · 28

  1. 2016An overview of gradient descent optimization algorithmscited by 1,033
  2. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  4. 2014The Loss Surfaces of Multilayer Networkscited by 280
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  6. 2016Deep Learning without Poor Local Minimacited by 234
  7. 2018Deep Learning: A Critical Appraisalcited by 217
  8. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  9. 2020Training independent subnetworks for robust predictioncited by 177
  10. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  11. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  12. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
16 more
  1. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26
  2. 2020Do Reservoir Computers Work Best at the Edge of Chaos?cited by 17
  3. 2019Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functionscited by 13
  4. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  5. 2018The Global Optimization Geometry of Shallow Linear Neural Networkscited by 11
  6. 2018Fusion of complex networks and randomized neural networks for texture analysiscited by 8
  7. 2021Characterizing Learning Dynamics of Deep Neural Networks via Complex Networkscited by 8
  8. 2023Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descentcited by 8
  9. 2022Improving Deep Neural Network Random Initialization Through Neuronal Rewiringcited by 7
  10. 2023An effective theory of collective deep learningcited by 7
  11. 2019Loss Surface Modality of Feed-Forward Neural Network Architecturescited by 5
  12. 2022Correlations of network trajectoriescited by 4
  13. 2021Quasi-universal scaling in mouse-brain neuronal activity stems from edge-of-instability critical dynamicscited by 3
  14. 2013Griffiths phases and the stretching of criticality in brain networkscited by 2
  15. 2021Optimal input representation in neural systems at the edge of chaoscited by 2
  16. 2022Deep Neural Networks as Complex Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizescited by 14
  2. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 7
  3. 2024Towards certification: A complete statistical validation pipeline for supervised learning in industrycited by 4
  4. 2025Leveraging chaotic transients in the training of artificial neural networkscited by 3
  5. 2026Temporal Complexity and Self-Organization in an Exponential Dense Associative Memory Modelcited by 0
  6. 2026Scalar Representations of Neural Network Training Dynamicscited by 0