# Fractal basin geometry and the limits to predictability and reproducibility in deep learning

arXiv: https://arxiv.org/abs/2510.05606 · HTML: https://mlsanity.com/paper/2510.05606/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 71 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Critical Learning Periods in Deep Neural Networks](https://mlsanity.com/paper/1711.08856/) (2017)
- [Randomness In Neural Network Training: Characterizing The Impact of Tooling](https://mlsanity.com/paper/2106.11872/) (2021)
- [Stability and Convergence Trade-off of Iterative Optimization Algorithms](https://mlsanity.com/paper/1804.01619/) (2018)
- [Nondeterminism and Instability in Neural Network Optimization](https://mlsanity.com/paper/2103.04514/) (2021)
- [Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks](https://mlsanity.com/paper/2306.04251/) (2023)
- [Sources of Irreproducibility in Machine Learning: A Review](https://mlsanity.com/paper/2204.07610/) (2022)
- [On the Reproducibility of Neural Network Predictions](https://mlsanity.com/paper/2102.03349/) (2021)
- [Unveiling the structure of wide flat minima in neural networks](https://mlsanity.com/paper/2107.01163/) (2021)
- [Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function](https://mlsanity.com/paper/2002.06189/) (2020)
- [Reproducibility in Optimization: Theoretical Framework and Limits](https://mlsanity.com/paper/2202.04598/) (2022)
- [Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent](https://mlsanity.com/paper/2402.07193/) (2024)
- [Churn Reduction via Distillation](https://mlsanity.com/paper/2106.02654/) (2021)
- [The boundary of neural network trainability is fractal](https://mlsanity.com/paper/2402.06184/) (2024)
- [Parameter Symmetry Potentially Unifies Deep Learning Theory](https://mlsanity.com/paper/2502.05300/) (2025)
- [Understanding Mode Connectivity via Parameter Space Symmetry](https://mlsanity.com/paper/2505.23681/) (2025)
- [Remove Symmetries to Control Model Expressivity and Improve Optimization](https://mlsanity.com/paper/2408.15495/) (2024)

## Abstract (arXiv)

Fundamental limits to predictability are central to our understanding of many physical and computational systems. In deep learning, training outcomes are notoriously difficult to predict and reproduce, yet the mechanism underlying these difficulties has remained unclear. Here we propose fractal, riddled basin geometry as a unifying dynamical-systems mechanism: every initialization leading to one solution lies arbitrarily close to initializations leading to different solutions, forming a fat fractal with structure at arbitrarily fine scales. Using a minimal neural network model, we analytically establish the classical route to riddling and derive a near-zero uncertainty exponent: even sharp increases in initialization precision yield only marginal gains in outcome predictability. In deep networks, intertwined training destinations and near-zero parameter-space exponents provide robust signatures consistent with riddling across tested architectures, optimizers and training protocols, offering a common dynamical account of otherwise disparate empirical observations. We further introduce functional uncertainty exponents, which extend these diagnostics of scale-insensitive irreproducibility from parameter space to the learned functions themselves. Together, our results support riddled basin geometry as a candidate organizing principle for irreproducibility in deep learning, with practical implications for optimization and the reliable deployment of artificial intelligence.