# Leveraging chaotic transients in the training of artificial neural networks

arXiv: https://arxiv.org/abs/2506.08523 · HTML: https://mlsanity.com/paper/2506.08523/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 50 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [How Two-Layer Neural Networks Learn, One (Giant) Step at a Time](https://mlsanity.com/paper/2305.18270/) (2023)
- [Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function](https://mlsanity.com/paper/2002.06189/) (2020)
- [Finite-time Lyapunov exponents of deep neural networks](https://mlsanity.com/paper/2306.12548/) (2023)
- [Transition to chaos in random neuronal networks](https://mlsanity.com/paper/1508.06486/) (2015)
- [An effective theory of collective deep learning](https://mlsanity.com/paper/2310.12802/) (2023)
- [Dynamical stability and chaos in artificial neural network trajectories along training](https://mlsanity.com/paper/2404.05782/) (2024)
- [Forgetting leads to chaos in attractor networks](https://mlsanity.com/paper/2112.00119/) (2021)
- [Correlations of network trajectories](https://mlsanity.com/paper/2206.03334/) (2022)
- [Scalar embedding of temporal network trajectories](https://mlsanity.com/paper/2412.02715/) (2024)
- [Initialisation and Network Effects in Decentralised Federated Learning](https://mlsanity.com/paper/2403.15855/) (2024)
- [Deep Neural Networks via Complex Network Theory: a Perspective](https://mlsanity.com/paper/2404.11172/) (2024)
- [Discontinuous transition to chaos in a canonical random neural network](https://mlsanity.com/paper/2405.14607/) (2024)
- [On Learnable Parameters of Optimal and Suboptimal Deep Learning Models](https://mlsanity.com/paper/2408.11720/) (2024)

## Built on by (most cited first, 3 of 3)

- [Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region](https://mlsanity.com/paper/2509.25351/) (2025)
- [Intuition emerges in Maximum Caliber models at criticality](https://mlsanity.com/paper/2508.06477/) (2025)
- [Scalar Representations of Neural Network Training Dynamics](https://mlsanity.com/paper/2606.30384/) (2026)

## Abstract (arXiv)

Traditional algorithms to optimize artificial neural networks when confronted with a supervised learning task are usually exploitation-type relaxational dynamics such as gradient descent (GD). Here, we explore the dynamics of the neural network trajectory along training for unconventionally large learning rates. We show that for a region of values of the learning rate, the GD optimization shifts away from purely exploitation-like algorithm into a regime of exploration-exploitation balance, as the neural network is still capable of learning but the trajectory shows sensitive dependence on initial conditions --as characterized by positive network maximum Lyapunov exponent--. Interestingly, the characteristic training time required to reach an acceptable accuracy in the test set reaches a minimum precisely in such learning rate region, further suggesting that one can accelerate the training of artificial neural networks by locating at the onset of chaos. Our results --initially illustrated for the MNIST classification task-- qualitatively hold for a range of supervised learning tasks, {learning architectures (including both shallow and deep multilayer perceptrons and convolutional neural networks) and other hyperparameters (different activation functions and weight regularisation),} and showcase the emergent, constructive role of transient chaotic dynamics in the training of artificial neural networks.