# Anomalous diffusion dynamics of learning in deep neural networks

arXiv: https://arxiv.org/abs/2009.10588 · HTML: https://mlsanity.com/paper/2009.10588/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2020.

## Built on by (most cited first, 9 of 9)

- [The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion](https://mlsanity.com/paper/2107.09133/) (2021)
- [Characterization of Generalizability of Spike Timing Dependent Plasticity trained Spiking Neural Networks](https://mlsanity.com/paper/2105.14677/) (2021)
- [High-dimensional manifold of solutions in neural networks: insights from statistical physics](https://mlsanity.com/paper/2309.09240/) (2023)
- [Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent](https://mlsanity.com/paper/2503.22478/) (2025)
- [Differentially Private Stochastic Gradient Descent with Low-Noise](https://mlsanity.com/paper/2209.04188/) (2022)
- [Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation](https://mlsanity.com/paper/2311.14120/) (2023)
- [Extended critical regimes of deep neural networks](https://mlsanity.com/paper/2203.12967/) (2022)
- [Rolling Ball Optimizer: Learning by ironing out loss landscape wrinkles](https://mlsanity.com/paper/2505.19527/) (2025)
- [Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetry](https://mlsanity.com/paper/2609.02373/) (2026)

## Abstract (arXiv)

Learning in deep neural networks (DNNs) is implemented through minimizing a highly non-convex loss function, typically by a stochastic gradient descent (SGD) method. This learning process can effectively find good wide minima without being trapped in poor local ones. We present a novel account of how such effective deep learning emerges through the interactions of the SGD and the geometrical structure of the loss landscape. Rather than being a normal diffusion process (i.e. Brownian motion) as often assumed, we find that the SGD exhibits rich, complex dynamics when navigating through the loss landscape; initially, the SGD exhibits anomalous superdiffusion, which attenuates gradually and changes to subdiffusion at long times when the solution is reached. Such learning dynamics happen ubiquitously in different DNNs such as ResNet and VGG-like networks and are insensitive to batch size and learning rate. The anomalous superdiffusion process during the initial learning phase indicates that the motion of SGD along the loss landscape possesses intermittent, big jumps; this non-equilibrium property enables the SGD to escape from sharp local minima. By adapting the methods developed for studying energy landscapes in complex physical systems, we find that such superdiffusive learning dynamics are due to the interactions of the SGD and the fractal-like structure of the loss landscape. We further develop a simple model to demonstrate the mechanistic role of the fractal loss landscape in enabling the SGD to effectively find global minima. Our results thus reveal the effectiveness of deep learning from a novel perspective and have implications for designing efficient deep neural networks.