# The boundary of neural network trainability is fractal

arXiv: https://arxiv.org/abs/2402.06184 · HTML: https://mlsanity.com/paper/2402.06184/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 13 references that are in the map.
- First posted on arXiv 9 Feb 2024.

## Builds on (most cited first, 5 of 5)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Understanding and correcting pathologies in the training of learned optimizers](https://mlsanity.com/paper/1810.10180/) (2018)
- [Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function](https://mlsanity.com/paper/2002.06189/) (2020)
- [From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regression](https://mlsanity.com/paper/2310.01687/) (2023)

## Built on by (most cited first, 13 of 13)

- [Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration](https://mlsanity.com/paper/2512.22382/) (2025)
- [The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions](https://mlsanity.com/paper/2506.13234/) (2025)
- [FOCUS: First Order Concentrated Updating Scheme](https://mlsanity.com/paper/2501.12243/) (2025)
- [Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Region](https://mlsanity.com/paper/2509.25351/) (2025)
- [Wasserstein Distributionally Robust Shallow Convex Neural Networks](https://mlsanity.com/paper/2407.16800/) (2024)
- [Spring-block theory of feature learning in deep neural networks](https://mlsanity.com/paper/2407.19353/) (2024)
- [Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects](https://mlsanity.com/paper/2508.03222/) (2025)
- [Hyperparameter Loss Surfaces Are Simple Near their Optima](https://mlsanity.com/paper/2510.02721/) (2025)
- [Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models](https://mlsanity.com/paper/2501.04286/) (2025)
- [Revisiting Learning Rate Control](https://mlsanity.com/paper/2507.01724/) (2025)
- [Fractal basin geometry and the limits to predictability and reproducibility in deep learning](https://mlsanity.com/paper/2510.05606/) (2025)
- [Kohn-Sham Spectral Embedding on Sparse Graphs at the Nishimori Temperature for Image Classification](https://mlsanity.com/paper/2607.28428/) (2026)
- [Fractional Optimizers Meet Fractal Activation Functions: An Empirical Study of Multi-Scale Optimization in Neural Network](https://mlsanity.com/paper/2608.14636/) (2026)

## Abstract (arXiv)

Some fractals -- for instance those associated with the Mandelbrot and quadratic Julia sets -- are computed by iterating a function, and identifying the boundary between hyperparameters for which the resulting series diverges or remains bounded. Neural network training similarly involves iterating an update function (e.g. repeated steps of gradient descent), can result in convergent or divergent behavior, and can be extremely sensitive to small changes in hyperparameters. Motivated by these similarities, we experimentally examine the boundary between neural network hyperparameters that lead to stable and divergent training. We find that this boundary is fractal over more than ten decades of scale in all tested configurations.